Small Language Models handle classification, extraction, tool choice, and repetitive work. Route to a frontier model only when the task is hard. One proxy, two lanes, separate metrics.
The first AI feature I ship usually calls the largest model I can reach. It works in the demo. The answer looks smart. The JSON comes back shaped. Then the same endpoint starts doing intent classification, field extraction, tool selection, and a short summary for every request. One model. One price. One latency band. Four jobs that did not need the same brain. That is the habit I am trying to break. A Small Language Model (SLM) is not a downgrade for every task. It is the right size for work that is narrow, repetitive, and easy to check. Classification. Extraction. Tool picking. Short structured outputs. The frontier model stays in the design. It stops being the default. SLM vs LLM, without the marketing fog I do not define an SLM by a magic parameter count. In practice, "small" means a model you can afford to run often: on a modest GPU, on CPU with quantization, or on a phone, with a job that is easy to score. An LLM, especially a frontier one, means breadth: open reasoning, weak prompts, messy evidence, and answers that still have to hold together. You pay for that breadth on every call, including the ones that only needed a label. SLM LLM (frontier / large) Job fit Narrow, repeated, checkable Open, compositional, ambiguous Typical outputs Labels, JSON fields, tool ids, short rewrites Long answers, multi-constraint plans, free-form analysis Knowledge Enough for the task, often after fine-tune or RAG Broader world knowledge; weaker prompts still work more often Latency and cost Built for high volume Built for hard questions; expensive as a default Deployment Edge, on-device, single GPU, air-gapped Usually hosted APIs or heavy GPU fleets Failure mode Misses edge cases outside the trained shape Sounds right while weakly grounded How I judge it Precision, schema pass rate, escalation rate Grounding samples, human review, hard-case success The comparison that matters in a product is not "which model wins a chat arena." It is which lane owns which request. What it actually takes to build or adopt an SLM Most teams should not train a foundation model from scratch. That is a research and capital project. What product teams need is a task model : a small base, adapted to one job, served cheaply, and measured. This is the stack I plan for: A frozen task definition. Closed label set, JSON schema, or tool catalog. If you cannot write the contract, you are not ready for an SLM. You are still exploring with a large model. A base SLM you did not train. Phi, Gemma, Llama 1B/3B, Granite Micro/Tiny, and similar open or hosted small models. Pick one that already does instruction following and structured output well enough to evaluate. Data for that task, not "more internet." A few hundred to a few thousand labeled examples often beat a vague prompt. Keep a held-out golden set that never trains. Adaptation in order of cost: prompt + schema + constrained decoding first supervised fine-tune or LoRA next preference or reinforcement-style tuning only if the eval set still fails distillation from a large teacher when you already have good traces for the same task Serving that matches the volume. Quantization when quality holds. A runtime you can operate: vLLM, llama.cpp, on-device runtimes, or a managed small-model endpoint. An eval harness before the router. Measure precision on labels, schema pass rate on extractors, tool-choice accuracy, and how often you would still escalate. A router with a typed reason. The SLM is lane one. The LLM is lane two. Without the escalation path, every hard case looks like an SLM failure instead of a routing miss. That is what "build an SLM" means for a shipping team. Not a new foundation model. A small specialist with data, a contract, a serve path, and a way out when the task is hard. Training techniques I actually reach for I treat training as a ladder. Climb only when the step below fails the golden set. The snippets below are the shape of the work, not a full training script. Prompting, schemas, and constrained decoding. Not training, but the first adaptation. Fix the output shape. Constrain tokens so invalid keys cannot appear. record IntentOut(Intent intent) {} IntentOut out = slm.chat() .responseFormat(IntentOut.class) // schema / constrained decode .system("Classify the message.") .user(text) .as(IntentOut.class); Supervised fine-tuning (SFT). Show (input → correct output) pairs. Do not train on the golden set. {"messages":[ {"role":"system","content":"Reply with one label: FAQ, COMPLAINT, SEARCH, HANDOFF."}, {"role":"user","content":"My invoice is wrong again."}, {"role":"assistant","content":"COMPLAINT"} ]} LoRA and QLoRA (PEFT). Train a thin adapter. Keep the base frozen. Swap adapters per task. from peft import LoraConfig, TaskType lora = LoraConfig( task_type=TaskType.CAUSAL_LM, r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"], lora_dropout=0.05, ) # one base checkpoint → adapters/classifier, adapters/extractor, ... Domain continued pretraining (optional). Unlabeled domain text only when SFT keeps missing domain tokens. # domain corpus (not task labels) tickets/2024/*.txt policies/employee-handbook.md Knowledge distillation. Teacher solves the easy lane. Student learns the decision, not the essay. { "input":"find java jobs in CR", "teacher_tool":"searchJobs", "student_target":"searchJobs" } String toolId = teacher.pickTool(ask); // large model, offline or shadow if (validator.accepts(ask, toolId)) { distillRows.add(new DistillRow(ask, toolId)); // train SLM on this } Preference tuning (DPO / ORPO and kin). After SFT, when two answers are valid but one is better. {"prompt":"Extract skills from: ...", "chosen": "{\"skills\":[\"Java\",\"Spring Boot\"]}", "rejected": "{\"skills\":[\"coding\",\"backend stuff\"]}"} Quantization. Shrink for serve after quality holds on the golden set. # post-training example shape; tool varies (GPTQ, AWQ, llama.cpp, …) quantize --model adapters/classifier-merged \ --bits 4 \ --calib golden/classifier.jsonl \ --out serve/classifier-q4 Synthetic data with a human gate. Generate candidates. Accept only what a person or a strict validator passes. List<SftRow> drafts = teacher.paraphrase(seedRows, 5); List<SftRow> accepted = drafts.stream() .filter(row -> schema.isValid(row.assistant())) .filter(row -> reviewQueue.approved(row.id())) .toList(); Technique When I use it What I watch Schema + constraints Always first Schema pass rate SFT Labels/tools still wrong after prompting Precision / exact match LoRA / QLoRA SFT is needed and I want cheap iteration Adapter size, forgetfulness on other tasks Continued pretrain Domain language keeps breaking SFT Domain term coverage Distillation Teacher already solves the easy lane Student vs teacher agreement on easy set Preference (DPO/ORPO) Structure is right, ranking is wrong Side-by-side win rate Quantization Serve budget forces it Quality drop on the golden set SK Telecom's Gemma moderation work and K Health's Gemma intake work sit on this ladder: start from a small base, adapt with task data, measure, then serve. The technique names change. The order does not. Four jobs that should not open with a frontier call Each sample is the shape I want from the SLM lane: a closed contract, a small call, a check. The chat client is whatever wraps your SLM. The point is the output, not the SDK. Classification. Is this a FAQ, a complaint, a search, or a handoff? That is a label. public enum Intent { FAQ, COMPLAINT, SEARCH, HANDOFF } Intent intent = slm.chat() .system("Reply with one label only: FAQ, COMPLAINT, SEARCH, HANDOFF.") .user(message.text()) .asEnum(Intent.class); if (intent == Intent.HANDOFF) { return queue.toHuman(message); } Information extraction. Pull the fields. Return JSON. Validate. Retry once on schema failure. record JobFields(String company, String title, List<String> skills, String startDate) {} JobFields fie...