Understand what really happens between your prompt and ChatGPT's answer. This topic explains large language model basics, tokens and context windows, the difference between training and inference, why models sometimes state things that are simply wrong, and how ChatGPT, Gemini, Claude and Copilot differ in practice.
ChatGPT, Gemini, Claude and Microsoft Copilot are all built on large language models (LLMs). An LLM is a neural network with billions of parameters that has one core skill: given a sequence of text, it predicts the next token — a small chunk of characters — over and over. Everything you see, from a PowerShell script to a meeting summary, is generated one token at a time. The model was trained on massive text corpora (web pages, books, code repositories) until this prediction became remarkably good, then refined with human feedback to be helpful and safer.
Once you understand tokens and context windows, half of the 'weird AI behaviour' you will meet stops being mysterious. You know why a 300-page PDF cannot simply fit in a prompt, why the model forgets the beginning of a very long chat, why per-token pricing exists in Azure OpenAI, and why the same question can give different answers twice. That understanding directly changes how you write prompts (Level 2) and how you architect AI-assisted workflows (Level 3+). It also lets you speak credibly with vendors who quote '128k context' or '200k tokens' as selling points.
Step one, your prompt is tokenised: text is split into tokens, roughly 4 characters or 0.75 English words each, so 1,000 tokens is about 750 words. Step two, those tokens become numbers (embeddings) that flow through the neural network. Step three, the model outputs a probability distribution over every possible next token and picks one; then it repeats the whole loop, reading your prompt plus everything it has generated so far. The context window — 128,000 tokens for GPT-4o, 200,000 for Claude Sonnet, up to 1 million for Gemini — is the maximum text this loop can consider at once. Training happened once, offline, on GPUs; inference happens every time you press Enter.
Your prompt
Tokenizer Tokens (≈4 chars each)
LLM predicts next Token (probability)
Tokens decoded Response
Loop repeats (context window grows)
A real endpoint scenario: you paste a 40-line IntuneManagementExtension.log excerpt into Copilot and ask why a Win32 app failed. The log is tokenised (a 40-line excerpt is roughly 1,500 tokens), the model reads it inside its 128k context window, and predicts an explanation token by token. It correctly spots the detection-rule failure because it has 'seen' thousands of similar logs during training. But if you ask it for your company's internal app ID, it cannot know — that data was never in training, which is exactly when you need grounding with your own data, like Copilot does with Microsoft Graph.
Open the tokenizer at platform.openai.com/tokenizer (or ask any chatbot). Paste: "Get-IntuneManagedDevice | Select DeviceName, ComplianceState" and count the tokens. Notice how PowerShell splits into many small pieces — this is why code burns context faster than prose.
In a new chat, paste a long technical document (about 10,000 words) and ask for a summary. Then ask: "What was the exact heading of the first section?" Compare with a fresh chat where you paste only one page. This demonstrates how long-context models behave when the window fills.
Send the same prompt — "Explain error 0x87D1041C in Intune and list 3 possible causes" — to ChatGPT, Gemini, Claude and Microsoft Copilot. Note differences in accuracy, structure and tone. Same mechanics, different training data and alignment choices.
Ask: "What is the exact KB number for the Windows 11 24H2 cumulative update released on my birthday last year?" Watch the model produce a plausible-looking number. Then ask "Are you sure? What would you verify?" This teaches the single most important safety habit: verify facts you act on.
In one chat, set a rule ("From now on, all answers must be in table format"), continue for 20+ exchanges, then check whether the rule still holds. You are seeing context pressure — the fix is to restate constraints, which becomes a formal technique in Level 2.
Consumer chatbot tiers may use your conversations for training and human review; enterprise tiers (Microsoft 365 Copilot, Azure OpenAI with data boundaries, ChatGPT Enterprise) generally do not, and keep data inside your compliance boundary. The model also cannot see your tenant unless explicitly grounded — Microsoft 365 Copilot pulls only what the signed-in user can already access via Microsoft Graph, so your permission hygiene still applies. Never paste secrets, tokens or personal data into a consumer tier, and disable chat history where policy requires it.
Tokens are the chunks — roughly four characters or 0.75 English words — into which an LLM splits all text, both prompts and output. They matter because the context window is measured in tokens, API pricing is per token, and unusual content like code or non-English text tokenises into more pieces. Knowing token math lets me estimate whether a document fits into a 128k context window and what a workflow will cost.
Training is the one-time, compute-heavy phase where the model adjusts billions of weights on large datasets using GPU clusters — weeks of work and millions of dollars. Inference is the phase you interact with: the frozen model processes your tokens and predicts output, typically in milliseconds per token. Knowledge changes only through training or fine-tuning; inference never updates the model itself.
Because it is optimised to predict plausible text, not to state verified truth. If a fact is rare or out of the training data, the model still produces the most statistically likely continuation, which can be a fabricated but realistic-looking answer — a hallucination. Mitigations are grounding techniques like RAG, asking the model to cite sources, and human verification of anything operationally important.
Run the same IT prompt through ChatGPT (free), Copilot and one other assistant of your choice: 'Explain what happens when a Windows device fails Conditional Access, in 150 words for an IT admin.' Compare answers for accuracy, structure and confidence. Then open the OpenAI tokenizer and measure how many tokens each model's answer consumed. Write three sentences on which assistant you would pick for daily IT work and why.
The underlying model's weights are frozen after training; your chat only lives inside the current context window. Consumer tiers may use conversations for future training or review unless you opt out, while enterprise tiers typically contractually exclude that. Within one conversation, everything you said earlier stays available until the context limit — which is the 'memory' you actually experience.
It means the model can consider about 128,000 tokens at once — roughly 96,000 English words, or a few hundred pages. Your prompt, the document you paste and the model's own replies all share that budget. For IT work it is enough for most logs and documentation, but not for an entire tenant's data — which is why retrieval (RAG) exists.
They share the same core mechanics but differ in training data, context length, integrations and enterprise controls. Copilot's advantage for IT is Microsoft Graph grounding — it can reference your actual mail, meetings and files with permission awareness. ChatGPT and Claude are strong generalists with large plugin and API ecosystems. Test them on your real tasks rather than trusting benchmarks alone.