Free course · 86 lessons
AI Engineering
Building real products on top of foundation models.
How to build and run production AI features as a software engineer: how models behave, calling them reliably, prompts and context, retrieval, agents, evaluation, security, and keeping costs under control.
Learning path
4 stages, one step at a time
Each stage builds on the one before it. Go in order the first time, then jump around when you need a refresher.
- 01
Mental model
What an AI engineer does and how models really behave.
11 lessons · 1.5 hrsJump to stage → - 02
Talking to models
APIs, prompts, and measuring whether they work.
17 lessons · 2.7 hrsJump to stage → - 03
Building the system
Context, retrieval and agents.
28 lessons · 4.9 hrsJump to stage → - 04
Running it in production
Observability, security, cost and product thinking.
30 lessons · 5 hrsJump to stage →
Curriculum
12 parts · 86 lessons
Open a part to see its lessons. Completed lessons are ticked off as you go.
Stage 1 Mental model
00Start HereThe role, the escalation ladder, and how to read this course.
- 01What an AI engineer is (and isn't)AI engineer vs ML engineer vs researcher: the API line, the three roles, and exactly what this handbook teaches and skips.
- 02The software baselineThe seven engineering skills this book assumes (Python, HTTP, JSON, async, Git, Docker, Postgres), why each one matters for AI work, and where to refresh it.
- 03Thinking in probabilistic systemsWhy the same LLM call returns different answers even at temperature 0, the five failure modes, and why graceful degradation is the default posture.
- 04The escalation ladderThe priority order the whole book runs on: prompt, context, retrieval, tools, agent, fine-tune. Climb the cheapest rung that fixes the failure.
01How Models BehaveTokens, sampling, context windows and failure modes.
- 01The model landscapePlace any model into a tier, and know what each tier is for: frontier, mid, fast, and local, plus open-weight vs hosted and chat vs reasoning vs multimodal.
- 02Tokens and context windowsTokens are the unit every LLM bill, limit, and prompt is measured in. How to count them in Python before you send, and why non-Latin text costs more.
- 03Sampling and temperatureHow temperature and top-p reshape a model's probability distribution, how to pick them per task, and why temperature 0 is not a determinism guarantee.
- 04EmbeddingsWhat embeddings are, why cosine similarity and dot product agree on normalized vectors, and the four engineering patterns built on them: search, memory, routing, clustering.
- 05Reasoning modelsWhat test-time compute is, the effort and budget knobs across providers, why you pay for hidden thinking tokens, and when reasoning earns its cost.
- 06The cost-quality-latency triangleThe book's recurring mental model: cost, quality, and latency are three axes you can't max at once. Name the one you sacrifice, then learn the levers that cheat it.
- 07Just enough MLThe three ML ideas an AI engineer who never trains a model still needs: the data split, overfitting, and distribution shift, framed as eval decisions.
Stage 2 Talking to models
02Working with Model APIsCalling model APIs reliably: retries, streaming, structured output.
- 01Calling modelsWhat happens when you POST a chat completion: messages, roles, the system prompt, and the three provider shapes you need to know side by side.
- 02StreamingHow token-by-token streaming wins on perceived latency, why it breaks output validation, and the four ways to get both.
- 03Structured outputsWhy schema-enforced JSON, not free text, is the contract between an LLM and the code that consumes its output, and how to wire it up with Pydantic.
- 04Function and tool callingThe model proposes a tool call, you execute it, the model continues: how to define tool schemas, run the loop, parallelize, and return errors the model can act on.
- 05Errors, retries, fallbacksHow to handle the timeouts, rate limits, and overloaded responses every LLM API throws at you, with the right backoff, fallback, and idempotency rules.
- 06Choosing models and reading benchmarksPublic leaderboards don't predict your product's quality. Shortlist by tier, test on your own eval set, decide. That's the whole workflow.
- 07Gateways and routingOne API in front of many providers, cheap-then-expensive cascades, provider failover, and the rule for when a gateway earns its operational cost.
- 08Prompt cachingThe single biggest cost lever in API-native LLM products. A 50k-token agent over 10 turns drops from $1.53 to $0.47 with one config change.
03Prompt EngineeringWriting prompts that hold up in production.
- 01Anatomy of a production promptThe five sections every production system prompt has, why monolithic god-prompts fail, and how to wire defensive instructions against prompt injection.
- 02Zero-shot and few-shotWhen examples earn their tokens and when they don't. The decision rule, the over-prompting cliff, and how prompt caching rewrites the math.
- 03Chain-of-thought, ReAct, self-consistencyThe three techniques that survived the 2023 prompting literature, and the reasoning-model inversion that flips the rules: when to use each, and when to delete them.
- 04Templates and chainingTreating prompts as typed code, and the call you make every week: one big prompt or several small ones in a chain.
- 05Prompt versioningTreat a prompt change like a code change: registry, version pinning, rollback, and review, with a minimal setup you can build in a week.
04Eval FundamentalsBuilding eval sets before building anything else.
- 01Why you can't ship without evalsThe vibes-driven failure mode, evals as the AI engineer's CI, and the maturity ladder that gets you off the whack-a-mole loop.
- 02Look at your dataThe single highest-ROI habit in LLM evaluation: read your traces by hand and build a failure taxonomy before automating anything.
- 03Your first eval setBootstrap a 50-example eval set in a week: real or realistic inputs, binary pass/fail labels, versioned in git, growing from production failures.
- 04Assertions and unit tests for LLM outputThe cheap, deterministic checks that should gate every LLM commit before any judge model runs: schema, contains, regex, code-runs, SQL-parses, wired into pytest and CI.
Stage 3 Building the system
05Context EngineeringDeciding what the model sees, and in what order.
- 01Context is more than the promptThe six things that go into a context window every turn, why prompt engineering only covers one of them, and what the discipline of context engineering actually owns.
- 02Context assembly and routingWhy a 1M-token window doesn't make context free, and the route-then-assemble pipeline that decides what the model actually sees.
- 03Compression and context budgetsMore context isn't better context. How lost-in-the-middle and context rot degrade long prompts, and when to trim, summarize, or re-retrieve.
- 04Memory and stateWhat an LLM remembers between turns, between sessions, and across years, and the techniques that make each scope work without crowding the context window.
- 05Tool context and metadataModels have no clock, no role awareness, and no sense of locale. The metadata you inject and the way you describe tools prevent whole classes of confidently wrong answers.
- 06Testing and observing contextSnapshot the assembled context, regression-test the assembly logic, and capture traces that can answer why the model saw what it saw.
06Retrieval-Augmented GenerationChunking, embeddings, retrieval and reranking.
- 01When you actually need RAGRAG vs long context vs fine-tune as a cost decision: long context pays per query, RAG pays once to index. The default rule, the exceptions, and when RAG is the wrong answer.
- 02Document processing and chunkingHow parsing and chunk boundaries set your retrieval ceiling, why overlap and chunk size are eval questions, and the defaults that survive contact with real corpora.
- 03Embeddings and vector searchHow to pick an embedding model on your own data, why pgvector is the default store, what HNSW and IVFFlat actually do, and the honest threshold where a dedicated vector DB earns its keep.
- 04Hybrid search and rerankingWhy BM25 still beats vector search on exact terms, how to fuse them with RRF, and when a reranker earns its latency.
- 05Query understandingRewriting, multi-query, decomposition, and routing: closing the gap between what users type and what your retriever can match.
- 06Citations and groundingMake RAG answers verifiable: span-level citations, refusal on empty retrieval, and the faithfulness vs fluency tension that decides both.
- 07Advanced retrievalThree patterns that beat vanilla RAG on specific queries: agentic retrieval, GraphRAG, and vision-native PDF search. When each earns its complexity, and when it doesn't.
- 08Evaluating retrievalRAG eval is mostly retrieval eval. Recall@k and MRR in plain terms, building an eval set from real queries, and the 2x2 that separates retrieval failure from generation failure.
- 09Production RAGThe five operational concerns that turn a RAG prototype into a system you can run: incremental indexing, the reindex tax, multi-tenant isolation, metadata filtering, and per-stage latency budgets.
07Agent EngineeringTools, loops, memory and multi-step agents.
- 01Workflow vs agentIf you can write the control flow in Python, it's a workflow. If the model decides at runtime, it's an agent. Most production wins are workflows.
- 02The five workflow patternsChaining, routing, parallelization, orchestrator-workers, evaluator-optimizer: each with code, a use case, and the failure mode that bites in production.
- 03The agent loopThink, act, observe, and the part nobody talks about: how the loop ends. Stop conditions, reflection that actually works, and why production agents look nothing like the demo.
- 04Tool designThe agent-engineering skill: schemas that make invalid calls impossible, descriptions the model can't misread, errors it can recover from, and the right number of tools.
- 05MCP: Model Context ProtocolHow to expose tools, data, and prompts to any AI host through one open protocol; build a server and client end-to-end.
- 06MCP securityThe four attack classes MCP creates: tool poisoning, cross-server shadowing, rug pulls, and injection via tool results, and the architectural defenses that actually stop them.
- 07Agent memory and task stateThe four places an agent can put information, why most teams use the wrong one, and how to design state that survives a pod restart mid-task.
- 08Human-in-the-loopApproval gates as tool calls, confidence-based escalation, and handoff design: the three independent things you have to engineer for selective human oversight to actually work.
- 09Long-running agentsBudgets, kill switches, checkpoints, replay, and the sync vs async choice: the harness work that keeps a multi-minute agent from burning a month's API spend.
- 10Multi-agent systemsWhen multi-agent topology earns its 15x token cost, when it doesn't, and the single-writer pattern that handles most production work.
- 11Agent archetypesResearch, Coding, Browser, and Computer Use agents differ in one thing that cascades into everything else: what the model sees at each step.
- 12Voice agentsThe 800ms budget that defines voice agents, the cascaded vs native trade-off, and why barge-in is the hardest correctness problem in the stack.
- 13Choosing a frameworkFrameworks buy state, retries, and tracing. Most teams don't need them. The decision rule for plain SDK, light structure, and full graph runtime.
Stage 4 Running it in production
08Evaluation & ObservabilityDeeper evals, tracing and monitoring in production.
- 01The eval taxonomyFour axes that decide which kind of evaluation answers your question: offline vs online, component vs end-to-end, reference-based vs reference-free, automatic vs judge vs human.
- 02LLM-as-judgeBuild a second-LLM judge you can trust: binary pass/fail over Likert, prompt patterns, and the human-agreement loop you must run before deploying it.
- 03Judge failure modesPosition bias, length bias, self-preference, and style bias in LLM-as-judge pipelines, with the numbers and the mitigations that actually work in production.
- 04Evaluating RAG and agentsComponent evals, tool-choice accuracy, trajectory evaluation, and the weakest-link pattern that tells you which piece of a compound system to fix.
- 05Regression suites and CI gatesWiring evals into CI as a real merge gate: two-tier golden sets, differential thresholds over absolute floors, and how to keep flaky probabilistic tests trustworthy.
- 06Online evaluation and A/B testingWhy your offline eval lied about a 3-point lift, the implicit signals that don't, and the sample sizes LLM A/B tests actually need.
- 07TracingSpans for LLM calls, retrieval, and agent loops; the gen_ai.* OpenTelemetry conventions; why vendor-proprietary tracing locks you in.
- 08Monitoring and dashboardsWhy LLM systems need cost, latency, quality, and usage on one screen; how to alert on quality regressions and cost spikes; and which observability platform to pick in 2026.
- 09Feedback & the data flywheelHow production traffic compounds into better evals and a better product, instead of becoming a treadmill of sprint-by-sprint bug fixes.
09Security & GovernancePrompt injection, data protection and guardrails.
- 01Prompt injectionWhy prompt injection has no general fix, the difference between direct and indirect attacks, and the defense-in-depth posture you ship with.
- 02Agent security and the lethal trifectaThe three-leg threat model for agentic exfiltration, why all three together is game-over, and how to cut at least one leg before you ship.
- 03GuardrailsInput filters, output validators, content moderation models, and the streaming-versus-validation problem nobody warns you about.
- 04Abuse preventionRate limits, spend caps, per-tenant budgets, and the behavioral signals that catch automated abuse before it shows up on the bill.
- 05PII and privacyDetect and redact before the model call, navigate the controller-processor-subprocessor chain, and pick the right retention and residency knobs.
- 06Compliance and auditWhat engineers actually build for SOC 2 and the EU AI Act: structured audit trails, tamper-evident retention, and the model documentation regulators look for.
- 07Red-teamingAutomated probe suites, manual protocols, and continuous red-teaming in CI: how to attack your own AI system before someone else does.
10Deployment & CostHosting choices, customization and cost control.
- 01Hosted vs self-hostedThe four-line TCO model that decides when self-hosting beats per-token billing, why GPU utilization is the silent multiplier, and the 'start hosted, migrate the hot path' default for 2026.
- 02Self-hosting essentialsWhat you actually need to know to run an open-weight model competently: vLLM as the default server, KV cache as the resource constraint, continuous batching, and how to size GPUs before you spend.
- 03The customization menuFine-tuning, LoRA, and distillation as decisions, not techniques: when each is the right answer, what it costs, and the contamination trap that makes broken systems look healthy.
- 04Cost engineeringFive orthogonal levers (caching, batching, routing, quantization, distillation) that compound multiplicatively to cut a five-figure monthly bill by ~80% without changing model quality.
- 05Latency engineeringWhy LLM latency isn't a single number: the waterfall, TTFT vs total time, parallel tool calls, prefix caching, and the perceived-latency tricks that close the gap UX-side.
- 06Shipping changesPrompt and model changes are deployments. Eval-gated promotion, shadow traffic, canary rollout, and instant rollback as the discipline that turns 'just edit the prompt' into safe production change.
- 07SLOs and incident responseAI failures return HTTP 200 with garbage. The four AI incident classes (quality regression, injection exploit, cost runaway, hallucination at scale), the semantic SLIs that detect them, and the runbooks that contain them.
- 08AI infrastructure patternsThe four AI-specific infrastructure components: vector stores in production, semantic caching, durable agent job queues, and embedding pipelines. Everything generic cross-links to HLD.
11AI ProductUX, instrumentation and growth for AI features.
- 01AI UXStreaming as a trust signal, not a transport detail. The three failure modes (blank screen, mid-stream stall, unverifiable claim) and what to build for each.
- 02Failure recovery UXAI failures aren't edge cases. Designing the fallback cascade, retry affordances, and human handoffs so the failure you can't prevent doesn't destroy trust.
- 03AI product metricsBeyond accuracy. Task success, deflection vs. containment vs. resolution, escalation as a quality signal, time-to-completion, and AI-feature retention.
- 04ExperimentationA/B testing AI features when variance is 2-5x higher than deterministic features, the treatment can drift mid-experiment, and 'no significant effect' usually means underpowered.
- 05Cost modeling and pricingWhy marginal cost matters again. Cost-per-task (not per-token), the four pricing architectures, and the math that says when a feature is too expensive to ship.
- 06Feedback and growth loopsMost teams have a log file, not a flywheel. The three preconditions for a working data loop, and why the loop itself (not the data) is the durable advantage.
BeginnerIntermediateAdvanced
How to get the most out of it
Read, explain, revisit
01
Read in order first
Lessons list their prerequisites at the top. The stages are ordered so you rarely meet a term before it's explained.
02
Write it in your own words
Summarizing the idea and its trade-offs is the quickest way to find what you haven't understood yet.
03
Come back to the diagrams
When revising, skim the diagrams and tables before re-reading. If you can explain each one, you know the lesson.
The lessons in this course are adapted from the AI Engineering handbook in The Engineering Handbook by Handbook Academy and contributors, licensed under CC BY-SA 4.0. The course structure, stage groupings and summaries on this page were written for Engineering Path. Credits & license.