Overview
free

Course Setup and the Incremental Ladder
Why "Transformers to Intelligence"
How to Use This Course
The Incremental Ladder (Step 0 to Step 7)
The Course Lenses
Diagram Legend and Notation Types

Mental Models: Functions, Learning, and Intelligence
Learning as Function Approximation
Intelligence as Compression, Prediction, and Control
Local Optimization, Global Behavior

Mathematical Foundations
Linear Algebra for AI: vectors, matrices, tensors, and why geometry is the hidden structure of models
Probability and Uncertainty: distributions, expectations, Bayes rule, and calibration as an operational concern
Optimization in Practice: gradients, convex vs non-convex intuition, SGD and common variants

Physics-Inspired Views of Learning
Information Theory as a Lens: entropy, mutual information, and compression-driven interpretations of learning
Energy Landscapes and Statistical Mechanics Intuition: why minima, basins, and temperature metaphors explain training behavior
Scaling Limits: compute, energy, bandwidth, and the real costs that constrain "bigger models."

Diagramming AI Systems
Computation Graphs and Layers: forward/backward passes as the "wiring diagram" of learning.
Training vs Inference Paths: data pipelines, feedback loops, and the boundary where behavior is measured.
System Topologies for AI: model, retrieval, tools, data stores, and users as a coupled system.

Step 0 Representations: Vectors, Embeddings, Tokens
Feature Spaces and Embeddings: one-hot vs dense representations and what "meaning in geometry" implies.
Tokenization Across Modalities: text, images, and structured inputs as discrete interfaces to continuous models.
Normalization and Scaling: preprocessing as a stability tool, not a cosmetic step.

Step 0 Models: Linear and Logistic Regression
Linear Regression and MSE: fitting as projection and what linearity limits you to.
Logistic Regression and Softmax: decision boundaries, cross-entropy, and multi-class classification.
Training Loops and Optimization Basics: learning rates, SGD, and the earliest forms of regularization.

Step 0 Delivery: Packaging, Serving, and Evaluating Linear Models
From Notebook to API: packaging artifacts, input contracts, and reproducible inference.
Offline vs Online Evaluation: calibration, drift, and stability as production realities.
When Linear is Enough: diagnosing "ceiling effects" and knowing when representation learning is required.

Step 1 MLPs and Depth
Why Depth Matters: universal approximation versus practical learnability.
Activations and Initialization: vanishing/exploding gradients as architectural constraints.
Designing MLPs for Structured Data: inductive bias for tabular and mixed-feature inputs.

Step 1 Convolutions and Local Structure
Convolution as an Inductive Bias: receptive fields, weight sharing, and locality.
Pooling, Stride, and Invariance: what invariance buys and what it breaks.
Beyond Images: extending convolutional structure to audio, time series, and other domains.

Step 1 Representation Learning and Regularization
Autoencoders and Bottlenecks: learning feature spaces by compression
Regularization Toolbox: dropout, batch normalization, weight decay, and how they change training dynamics
Transfer Learning: pretrain then adapt, and why data boundary choices matter

Step 2 Recurrent Networks and Temporal Dependencies
RNNs, LSTMs, GRUs: representing state over time and what "memory" means in a model
Training instability in time: truncated BPTT, gradient pathologies, and practical mitigation
Seq2Seq abstractions: encoder-decoder thinking and where attention first enters as a remedy

Step 2 Attention Mechanisms
Query-Key-Value: attention as differentiable selection and routing
Soft vs hard attention: trade-offs in differentiability, interpretability, and optimization
Longer contexts: scaling attention and the costs that push you toward transformer-style designs

Step 2 Deployment: Architecting and Serving Sequence Models
Streaming vs Offline Inference: boundary choices for latency and correctness
Memory footprint and sequence length: performance ceilings and practical constraints
Evaluating sequence tasks: perplexity, BLEU/ROUGE-style metrics, and domain-specific success definitions

Anatomy of a Transformer
Multi-Head Self-Attention: context mixing as the core computational primitive
Positional encoding: absolute vs relative schemes and what they enable or prevent
Feed-forward, residuals, normalization: stability mechanisms and depth scaling behavior

Transformer Variants
Encoder-only, decoder-only, encoder-decoder: matching architecture to task constraints
Long-context strategies: sparse and linearized attention and the costs they shift elsewhere
Mixture-of-experts and scaling variants: capacity, routing, and operational complexity

Training Transformers
Data and tokenization pipelines: corpus construction as the dominant design lever
Distributed training patterns: parallelism as a systems boundary problem
Stability and schedules: making training predictable under scale

Inference, Optimization, and Compression
Inference graphs and KV caching: throughput/latency trade-offs and streaming implications
Quantization, pruning, distillation: compressing capability into deployable footprints
Hardware-aware deployment: CPUs, GPUs, accelerators, and where bottlenecks migrate

What Is a Foundation Model?
Self-Supervision and Emergence: why next-token and related objectives produce broad capabilities.
Objective Families: masked, causal, and hybrid objectives as behavior-shaping choices.
Scaling Laws Intuition: the data/compute/parameters triangle and what it predicts well (and poorly).

LLMs and SLMs as Design Points
LLM Design Posture: generality, capability, and the costs you accept to get them.
SLM Design Posture: specialization, edge constraints, and operational privacy/cost advantages.
Choosing a Point in the Trade Space: capability, latency, privacy, and unit economics.

Fine-Tuning, Adaptation, and Instruction Following
Supervised Fine-Tuning: task shaping and the risks of brittle specialization.
Adapters and LoRA-Style Methods: low-rank adaptation as an operationally tractable compromise.
Instruction Tuning and Preference Data: steering behavior via curated interaction distributions.

Alignment, Safety, and Guardrails
RLHF and Related Approaches: what is optimized, what is approximated, and where reward hacking appears.
Red-Teaming and Safety Policies: turning anticipated misuse into testable evaluation artifacts.
Guardrails and Constraints: balancing helpfulness, honesty, and harmlessness as a system design problem.

Evaluation and Benchmarking of Foundation Models
Benchmarks vs Holistic Evaluation: what benchmarks measure, and what they systematically miss.
Robustness, Calibration, Bias, Fairness: evaluation as continuous monitoring of failure surfaces.
Human Evaluation Loops: feedback pipelines, labeler variance, and governance of subjective judgments.

Multi-Modal Architectures
Modality Encoders: vision, audio, code, and structured data as distinct input contracts.
Fusion Strategies: cross-attention, late fusion, and joint embedding spaces.
Contrastive and Joint Training: aligning modalities and the failure modes of misalignment.

Retrieval-Augmented Generation (RAG)
Vector Stores and Indexes: retrieval as an external memory boundary.
Chunking and Context Construction: grounding as an engineering problem, not a slogan.
RAG Failure Modes: hallucination, stale knowledge, retrieval miss, and robustness strategies.

Tool-Using Models
Function Calling and Structured Outputs: schemas as the interface contract between models and tools.
Designing Tool APIs for Models: affordances, constraints, and making actions safer than free-form text.
Access Control and Safety Around Tools: permissions, auditing, and failure containment.

Model Ecosystems and Composition
Routing and Composition: orchestrating specialized models as a system-level mixture of experts.
Specialized Helpers: code, math, extraction, planning, and when decomposition improves reliability.
Latency, Reliability, and Cost: managing budgets and failure propagation in composed systems.

Agents and Feedback Loops
Agent Loops: observe-think-act-learn as a control system, not just an app pattern.
Planning and Reflection: when self-critique helps and when it creates new failure modes.
Memory Architectures: short-term context versus long-term storage and retrieval boundaries.

Multi-Agent Systems
Coordination and Communication: protocols, roles, and division of labor among agents.
Emergence and Simulation: what can arise from local rules and why it is hard to predict.
Governance and Control: bounding agent societies with policies, incentives, and oversight.

Orchestration Layers and Workflows
Orchestration Frameworks and Policy Engines: separating execution from governance.
Tool Routing, Retries, Backoff: reliability engineering for non-deterministic components.
Integrating with Platforms and UIs: connecting agents to data, business systems, and human workflows.

Operating AI Systems in Production
Monitoring the Right Things: quality, safety, latency, and cost as co-equal SLOs.
Incident Response for AI: rollbacks, kill switches, override mechanisms, and postmortems.
Continuous Improvement: A/B testing, canaries, and closing the loop without contaminating evaluation.

Organizational Design for AI Systems
Roles and Interfaces: research, engineering, safety, product, and policy as coupled responsibilities.
Data Lifecycle Governance: collection, labeling, privacy, and retention as design constraints.
Documentation and Accountability: audits, model cards, and external reporting as operational necessities.

World Models and Internal Simulators
World Models in RL: latent dynamics and why prediction becomes simulation.
Learning Environment Structure: representations of causality, dynamics, and uncertainty.
Imagination and Counterfactuals: using simulators for planning and robustness.

Planning and Control with Learned Models
Model Predictive Control: planning under constraints with learned dynamics.
Integrating LMs with World Models: division of labor between language and dynamics.
Closed-Loop Systems: perception -> model -> plan -> act as an end-to-end safety boundary.

Continual Learning, Memory, and Identity
Lifelong Learning and Forgetting: catastrophic forgetting as a systems reliability problem.
Memory Structures: internal vs external memory and how they change failure surfaces.
Identity and Versioning Over Time: stability, updates, and continuity as product promises.

AI in the Real World: Humans, Organizations, Society
Human-AI Interaction Patterns: interface design as behavior shaping and risk control.
Societal Embedding: labor, creativity, decision-making, and organizational adoption dynamics.
Governance and Long-Term Risk: regulation, accountability frameworks, and durable operational norms.

Architectural Patterns for AI Models
Encoder/Decoder Families: autoencoders and representation bottlenecks as reusable motifs
Diffusion and Energy-Based Perspectives: generative patterns and where they fit operationally
Hybrid Architectures: combining symbolic and neural components and the boundary management required
Architecture Selection by Step: matching pattern choice to ladder rung constraints
Failure Surfaces by Architecture: typical breakdown modes and what to test first

Data, Datasets, and Evaluation Patterns
Data Curation: filtering, deduplication, and provenance as model-shaping forces
Privacy and Data Risk: sensitive data handling and the costs of leakage
Synthetic Data and Augmentation: when synthetic helps, and how it can mislead
Evaluation Suites and Dashboards: continuous measurement as part of the system
Dataset Shift and Drift: maintaining meaning as environments and users change

Infrastructure and Scaling Patterns
Training Clusters: compute topology, throughput constraints, and distributed training patterns
Inference Infrastructure: online, batch, streaming, and the operational consequences of each
Cost Optimization and Elasticity: scaling down as a first-class design goal
Hardware-Aware Design: choosing architectures and serving strategies that match accelerators
Reliability Under Load: degradation strategies when latency, cost, and quality collide

Safety, Security, and Robustness Patterns
Defense-in-Depth for AI: layered mitigations across model, data, and system boundaries
Adversarial Robustness and Abuse Resistance: anticipating malicious inputs and misuse incentives
Policy, Logging, and Auditability: making failures diagnosable and decisions reviewable
Red-Teaming as an Engineering Practice: turning threats into test plans and regression suites
Post-Hoc Analysis and Remediation: learning loops that improve safety without breaking trust

Design Patterns for AI Products
Zero-Shot and Few-Shot UX: prompt-based interaction patterns and their brittleness
Copilot, Agent, Assistant Archetypes: choosing product posture and responsibility boundaries
Human-in-the-Loop Workflows: escalation paths, approvals, and when automation should stop
Trust and Transparency Patterns: explaining uncertainty, citing sources, and managing expectations
Product Safety in Practice: user reporting, abuse handling, and operational governance

/transformers-to-intelligence/transformers-to-intelligence

Multi-Modal Architectures

Sign in to access this lesson.

Sign in Create account