AL & ML Concepts Explained: Clear Up the Confusion, build real understanding
The past year, during the early planning phase for a new internal tooling project, and someone from the research team said we should "fine-tune a smaller model with RAG on top to reduce hallucinations." Most of the people nodded. I'm pretty sure nobody in that room had a complete picture of what that sentence actually meant, and nobody wanted to be the one to ask. So I went home and actually sat down with it. Not the blog-post version where someone explains tokens with a bread-baking analogy. The real thing. Tokens: the unit everything is measured in Before any other term makes sense, you need to get tokens. Not because it's exciting, but because literally every other concept references it. A token is roughly a word fragment. The exact split depends on the tokenizer the model uses, but a rough rule of thumb: one token is about 0.75 words in English. So "unbelievable" might be two tokens ("un" and "believable"), while "cat" is one. OpenAI's tiktoken library lets you see exactly how GPT-4 chops up your text: import tiktoken enc = tiktoken.encoding_for_model("gpt-4") tokens = enc.encode("The quick brown fox jumps over the lazy dog") print(len(tokens)) # 9 tokens print(tokens) # [791, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679] Why does this matter? Because models have a "context window" measured in tokens, and because you pay per token when using APIs. A GPT-4 Turbo call that processes a 10,000-word document is not cheap. Knowing the token math helps you design smarter systems instead of just throwing text at the API and hoping for the best. Context window: what the model can "see" The context window is the total number of tokens a model can process at once, input plus output combined. GPT-4 Turbo sits at 128k tokens. Claude 3 Opus goes up to 200k. Gemini 1.5 Pro pushed it to 1 million tokens, which is genuinely hard to wrap your head around. Think of it like RAM. Whatever fits in the context window is what the model can reason over. Anything outside it? Gone. The model has no memory of it whatsoever. This is why, early on, people would feed an entire codebase into a model and get bizarre, contradictory outputs. The model wasn't dumb. It was working with a fragment of what it actually needed. Garbage in, confused output out. LLM: the thing everyone's actually talking about Large Language Model. It's a neural network trained on massive amounts of text data to predict the next token in a sequence. That's genuinely all it is at the mechanical level. The "large" part refers to parameter count: GPT-3 had 175 billion parameters, and GPT-4's count was never officially disclosed but estimates run much higher. I'd resist calling them "just autocomplete," though. The emergent behaviors you get from scale are qualitatively different from what a smaller model produces. But I also think it's worth keeping the mechanical definition in your head so you don't accidentally anthropomorphize the thing too much. It's pattern completion. Very, very good pattern completion. Prompt engineering: less mysterious than it sounds Prompt engineering is the practice of crafting inputs to get better outputs from a model. No special tools required. No Python. Just text. But there are real techniques here, and some of them are surprisingly effective. Zero-shot prompting means you give the model a task with no examples. "Summarize this meeting transcript in three bullet points." Simple. Few-shot prompting means you put examples directly in the prompt before asking the model to do the task. Like this: Input: "The deployment failed due to a timeout in the payment service." Output: severity=high, component=payment, category=infrastructure Input: "User reported that the search bar returns no results on mobile." Output: severity=medium, component=search, category=ui Input: "Login page takes 12 seconds to load on slow connections." Output: The model sees the pattern and continues it. I used this exact approach when building a log classifier for our on-call rotation. It cut the time I would've spent labeling training data from probably two days down to a few hours of prompt iteration. Not an exaggeration. Chain-of-thought prompting is when you ask the model to reason step by step before giving a final answer. Adding "think through this carefully before answering" or "let's work through this step by step" genuinely improves accuracy on reasoning tasks. Not a placebo. There's solid research backing it up, and I've seen it make a real difference on classification tasks where the model kept jumping to the wrong answer. Temperature and top-p: the dials people forget to set Temperature controls randomness. A temperature of 0 makes the model deterministic, always picking the most probable next token. Crank it to 1 or above and outputs get more creative and less predictable. For code generation or data extraction, I always set temperature to 0. I want consistent, boring, predictable output. For brainstorming or drafting tasks, something around 0.7 to 0.9 gives more interesting results. Top-p (also called nucleus sampling) is a related parameter. It limits the model to sampling only from the set of tokens that together account for p% of the probability mass. So top-p of 0.9 means the model ignores the long tail of unlikely options and only picks from the top 90%. Honestly, for most use cases you can tune temperature and leave top-p at its default. I've rarely needed to touch both simultaneously, and when I have, the results were marginal at best. RAG: retrieval-augmented generation This one tripped me up longer than I'd like to admit. The name sounds intimidating. The idea is actually pretty simple. RAG is a pattern where, before calling the LLM, you retrieve relevant documents from an external knowledge base and inject them into the prompt. The model then answers based on that injected context instead of relying solely on what it learned during training. So instead of asking GPT-4 about your internal API documentation (which it has never seen and never will), you: Take the user's question Convert it to a vector embedding Search a vector database (Pinecone, Weaviate, pgvector, take your pick) for similar documents Stuff those documents into the prompt Ask the model to answer based on them Here's a stripped-down version using LangChain and OpenAI: from langchain.chains import RetrievalQA from langchain.vectorstores import FAISS from langchain.embeddings import OpenAIEmbeddings from langchain.llms import OpenAI embeddings = OpenAIEmbeddings() vectorstore = FAISS.load_local("my_docs_index", embeddings) qa_chain = RetrievalQA.from_chain_type( llm=OpenAI(temperature=0), retriever=vectorstore.as_retriever(search_kwargs={"k": 4}), ) result = qa_chain.run("What's the rate limit on the payments API?") print(result) RAG is why most production AI applications don't just call gpt-4 directly. There's a retrieval layer sitting in front of it. And it's one of the main ways to reduce hallucinations, because you're giving the model a source of truth to work from instead of asking it to recall something it may have only half-learned during training. Hallucination: the polite word for "making stuff up" Hallucination is when a model generates factually incorrect output with complete confidence. Not a hedge, not an "I'm not sure about this," just a clean, fluent, totally wrong answer. It happens because the model is optimized to produce fluent, plausible-sounding text. It doesn't "know" when it doesn't know something. It fills gaps the same way it fills anything else. The classic failure mode: ask a model to cite sources, and it'll invent paper titles, author names, and DOIs that look completely real. I've seen this bite teams who skipped output validation and shipped model responses directly to users. Not a great look. Mitigation strategies: RAG, asking the model to only answer based on provided context, and adding validation steps in your pipeline that check outputs against known data....