Frames to Realities

How to Design Augmented & Virtual Experiences

41 modules

166 lessons

—

Part 1

Course Setup and the Incremental Ladder

Course Setup and the Incremental LadderSign in
Why "Frames to Realities"Sign in
How to Use This CourseSign in
The Incremental Ladder (Step 0 to Step 7)Sign in
The Course LensesSign in
Diagram Legend and Notation TypesSign in

Part 2

What Is an AR/VR/XR Experience?

What Is an AR/VR/XR Experience?Sign in
Definitions and BoundariesSign in
XR as Real-Time Embodied SimulationSign in
The Triangle - Hardware, Engine, ContentSign in

Part 3

3D Graphics and Coordinate Spaces Recap

3D Graphics and Coordinate Spaces RecapSign in
Transforms as ContractsSign in
Space TaxonomySign in
Cameras and ProjectionsSign in

Part 4

Human Perception, Depth Cues, and Presence

Human Perception, Depth Cues, and PresenceSign in
Depth Cues and Cue ConsistencySign in
Motion Parallax, Field of View, and Frame RateSign in
Presence, Embodiment, and CoherenceSign in

Part 5

XR System Building Blocks

XR System Building BlocksSign in
Displays, Optics, and Tracking RigsSign in
Runtime Responsibilities, Scene Graphs, and Spatial UISign in
End-to-End Dataflows as a Coupled PipelineSign in

Part 6

Diagramming AR/VR Systems

Diagramming AR/VR SystemsSign in
Coordinate Frame DiagramsSign in
Rendering Pipeline DiagramsSign in
Interaction Flow DiagramsSign in

Part 7

Classic 3D Rendering Pipelines

Classic 3D Rendering PipelinesSign in
Geometry to PixelsSign in
Scene Graphs and CullingSign in
Frame Timing BasicsSign in

Part 8

Camera Control and Input in 3D

Camera Control and Input in 3DSign in
Camera Rigs on Flat ScreenSign in
Polling vs EventsSign in
Comfort-Adjacent ControlsSign in

Part 9

Stereo Vision and Binocular Rendering

Stereo Vision and Binocular RenderingSign in
Left/Right Eye ViewsSign in
Convergence vs AccommodationSign in
Stereo Failure ModesSign in

Part 10

HMD Hardware, Optics, and Comfort

HMD Hardware, Optics, and ComfortSign in
Optics ArtifactsSign in
FOV and Resolution ConstraintsSign in
Physical Comfort and the Wearable BoundarySign in

Part 11

Presence, Embodiment, and Simulator Sickness

Presence, Embodiment, and Simulator SicknessSign in
Presence Factors: Visual, Audio, Haptics, Agency, and Why Coherence Matters More Than FidelitySign in
Sickness Causes: Sensory Mismatch, Latency, Acceleration Cues, and How Design Choices Amplify or Reduce ThemSign in
Designing for Comfort Early: Principles That Prevent Expensive Rework Once Content ScalesSign in

Part 12

Tracking Fundamentals: 3DoF, 6DoF, and Fusion

Tracking Fundamentals: 3DoF, 6DoF, and FusionSign in
Orientation vs Position: What 3DoF and 6DoF Enable, and What Breaks When Either Is MissingSign in
IMUs and Drift: Noise, Bias, Drift and Why Raw Sensor Truth Is Not Stable Enough for PresenceSign in
Fusion and Prediction Concepts: High-Level Fusion, Prediction Windows, and Why Timing Matters as Much as AccuracySign in

Part 13

Inside-Out vs Outside-In Tracking

Inside-Out vs Outside-In TrackingSign in
Tracking Architectures: Headset Cameras vs External Base Stations and the Failure Modes Each IntroducesSign in
Occlusion and Tracking Volume: How Real-World Geometry Becomes an Input ConstraintSign in
Trade-offs and Product Fit: Setup Friction, Cost, Robustness, and Designing Experiences Around Tracking RealitySign in

Part 14

Coordinate Spaces in XR

Coordinate Spaces in XRSign in
Core Frames: Head, Eye, Hand, Controller, Room, World, Device and How They ComposeSign in
Re-centering and Drift: Origin Management as a Product Feature, Not a Calibration FootnoteSign in
Guardian and Boundaries: Safety Systems as Coordinate-Frame Products with UX ImplicationsSign in

Part 15

VR Rendering End-to-End

VR Rendering End-to-EndSign in
The VR Frame Timeline: Sample -> Predict -> Render -> PresentSign in
Motion-to-Photon Budgets: Where Latency Comes From and What You Can Actually ImproveSign in
Foveation as a Resource Strategy: Reallocating Quality to Protect ComfortSign in

Part 16

Distortion Correction, Reprojection, and Timewarp

Distortion Correction, Reprojection, and TimewarpSign in
Distortion and Pre-Warp: Lens Distortion Correction and Why Rendering Must Anticipate OpticsSign in
Late Latching and Reprojection: Reducing Apparent Latency Without Changing Core Simulation TimeSign in
Dropped Frames and Async Techniques: Timewarp/Spacewarp Concepts and Graceful DegradationSign in

Part 17

Spatial Audio and Haptics for Presence

Spatial Audio and Haptics for PresenceSign in
Spatial Audio Basics: HRTFs, Occlusion, Reverb and Why Audio Anchors PresenceSign in
Event-to-Sound Mapping: Coupling Simulation Events to Sound Fields Without DisorientationSign in
Haptics Patterns: Vibration and Wearable Haptics as Feedback Channels With Safety and Comfort ConstraintsSign in

Part 18

Optical vs Video See-Through AR

Optical vs Video See-Through ARSign in
Optical See-Through Constraints: Waveguides, Display Limitations, and What "Occlusion" Means When You Cannot Block PhotonsSign in
Video Passthrough Trade-offs: Latency, Camera Artifacts, and Blending ConstraintsSign in
Visual Integration Goals: Convincing Blend Versus Stylized Overlay and Choosing the Right ObjectiveSign in

Part 19

Camera Pose Estimation and SLAM

Camera Pose Estimation and SLAMSign in
VIO/SLAM Concepts: Pose Estimation as the Foundation for World-Locked ContentSign in
Planes and Feature Points: What Is Detected, What Is Inferred, and What Remains UncertainSign in
Anchors and Stability: Local Versus World Anchors and How Drift Surfaces as UX BugsSign in

Part 20

Occlusion, Lighting Estimation, and Visual Integration

Occlusion, Lighting Estimation, and Visual IntegrationSign in
Occlusion Techniques: Depth Sensing, Occlusion Masks, Depth-from-Motion, and Their Failure CasesSign in
Lighting Estimation: Approximating Real Lighting So Virtual Objects "Sit" CorrectlySign in
Style as Engineering: Realism Versus Stylization as a Robustness and Performance DecisionSign in

Part 21

Input Paradigms: Controllers, Hands, Gaze, Voice

Input Paradigms: Controllers, Hands, Gaze, VoiceSign in
Ray and Direct Manipulation: Selection/Manipulation Patterns and How Tracking Noise Changes UXSign in
Hand Tracking and Gestures: Affordances, Ergonomics, and Preventing Accidental ActivationSign in
Gaze and Voice: Dwell/Confirm Patterns, Eye vs Head Gaze, and the Reliability Limits of Voice in Mixed SpacesSign in

Part 22

Locomotion and Spatial Navigation

Locomotion and Spatial NavigationSign in
Locomotion Families: Teleport, Dash, Smooth Locomotion, Arm-Swinging and When Each Is AppropriateSign in
Room-Scale vs Seated/Standing: Designing for Physical Constraints and Safety BoundariesSign in
Cognitive Mapping: Helping Users Orient, Remember, and Navigate Without OverloadSign in

Part 23

Spatial UI and Information Layering

Spatial UI and Information LayeringSign in
UI Anchoring Modes: World-Locked, Head-Locked, Body-Locked and When Each BreaksSign in
Panels, Volumes, Menus: Spatial UI Primitives and How to Keep Them DiscoverableSign in
Density and Context: Avoiding Clutter, Managing Focus, and the AR vs VR Differences in AttentionSign in

Part 24

Interaction Design for Comfort and Usability

Interaction Design for Comfort and UsabilitySign in
Ergonomics and Reach: Interaction Thresholds, Fatigue, and Designing Within Human RangesSign in
Preventing Overload: Accidental Activation, Mode Confusion, and Attention ManagementSign in
Accessibility in XR: Motion Sensitivity, Alternative Inputs, and Inclusive Spatial DesignSign in

Part 25

Spatial Mapping and Meshes

Spatial Mapping and MeshesSign in
Building Spatial Meshes: Scanning, Reconstruction, and What the Map RepresentsSign in
Planes, Meshes, Volumes: Combining Representations for Interaction and OcclusionSign in
Performance and Memory: Spatial Meshes as a Budget Consumer and Why Pruning Is MandatorySign in

Part 26

Scene Understanding and Semantics

Scene Understanding and SemanticsSign in
Semantic Layers: Floors, Tables, Walls, and Why Semantics Changes Interaction DesignSign in
Designing With Uncertainty: Confidence Scores, Fallbacks, and "Good Enough" Placement RulesSign in
Meaningful Placement: Aligning Content to Surfaces and Zones Without Overfitting to Noisy DetectionSign in

Part 27

Persistent Spatial Anchors and World Persistence

Persistent Spatial Anchors and World PersistenceSign in
Anchors Across Sessions: Persistence Semantics and the Product Promises They ImplySign in
Relocalization and Drift: Lifecycle Management for Anchors and Re-Alignment StrategiesSign in
Persistence Failure Modes: Environmental Change, Moving Objects, and When "World-Locked" Becomes WrongSign in

Part 28

Multi-User Shared Environments

Multi-User Shared EnvironmentsSign in
Shared Coordinates: Alignment Across Devices and the Consequences of Coordinate DisagreementSign in
Synchronizing State and Avatars: Replication, Authority, and Presence Under LatencySign in
Collaboration Patterns: Turn-Taking, Shared Tools, and Managing Conflict in Shared SpacesSign in

Part 29

Safety Boundaries and Real-World Constraints

Safety Boundaries and Real-World ConstraintsSign in
Guardian Systems: Spatial Boundaries as Safety Infrastructure and UX FeaturesSign in
Mixed-Space Hazards: People, Pets, Objects, and Designing to Avoid CollisionsSign in
Privacy and Social Norms: Bystanders, Privacy Zones, and Designing for Public SpacesSign in

Part 30

XR Engine Architecture and Subsystems

XR Engine Architecture and SubsystemsSign in
Subsystem Map: Rendering, Tracking, Input, Physics, Audio, Networking, UI and Their Coupling PointsSign in
XR Main Loop Design: Update Ordering Under Prediction and Latency ConstraintsSign in
Plugins and Platform Abstractions: Portability Across Runtimes Without Losing PerformanceSign in

Part 31

Tools, Content Pipelines, and Authoring Workflows

Tools, Content Pipelines, and Authoring WorkflowsSign in
XR Asset Pipelines: Formats, Importers, and Constraints That Preserve Scale and PerformanceSign in
Spatial Authoring: Prefabs, Scene Authoring, and "Spatial Storyboards" as Production ArtifactsSign in
Iteration Workflows: In-Headset Editing, Remote Preview, Hot Reload, and Why Iteration Speed Drives QualitySign in

Part 32

Networking, Cross-Device, and Cloud Services

Networking, Cross-Device, and Cloud ServicesSign in
Multi-Device XR: Standalone, Tethered, Mobile, and Capability-Driven DesignSign in
Real-Time Networking for Presence: Avatars, Voice, Shared State, and Latency BudgetsSign in
Cloud Services (Conceptual): Persistence, Analytics, and Remote Rendering as System BoundariesSign in

Part 33

Performance, Optimization, and Testing

Performance, Optimization, and TestingSign in
Frame Budgets: CPU/GPU/Tracking Budgets and How One Subsystem Can Ruin ComfortSign in
Profiling Motion-to-Photon: Measuring the Full Pipeline Rather Than Local Hot SpotsSign in
Testing Strategies: Automated Tests, In-Headset QA, and Comfort Testing as a Distinct DisciplineSign in

Part 34

Accessibility, Ethics, and Well-Being

Accessibility, Ethics, and Well-BeingSign in
Designing for Diverse Bodies: Size, Mobility, Sensitivity Differences and Inclusive Interaction RangesSign in
Intensity Guidelines: Motion, Flashing, and Sensory Load as Safety ConstraintsSign in
Ethics and Well-Being: Harassment, Addiction Pressure, Data Use, and Environment Safety in Mixed RealitySign in

Part 35

Shipping and Operating XR Products

Shipping and Operating XR ProductsSign in
Distribution and Certification: Stores, Compliance, and Platform ExpectationsSign in
Updates and Remote Configuration: Patching Safely in Systems That Interact With the Physical WorldSign in
Live Ops for XR: Events, Telemetry-Driven Improvements, and Maintaining Presence Quality Over TimeSign in

Part 36

Interaction Patterns for XR

Interaction Patterns for XRSign in
Core Patterns: Ray+Trigger, Direct Grab, Gaze+Confirm, Gesture Vocabularies, and Why Each Has a Comfort EnvelopeSign in
Menus and Shortcuts: Toolbelts, Radial Menus, Contextual Actions, and DiscoverabilitySign in
Anti-Patterns: Designs That Commonly Cause Discomfort, Fatigue, or Mode ConfusionSign in

Part 37

Spatial UX Design Patterns

Spatial UX Design PatternsSign in
Spatial Composition: Placing UI in VR vs AR and Avoiding Occlusion of MeaningSign in
Anchoring to Objects, Spaces, People: Choosing Stable References Users Can TrustSign in
Microinteractions: Highlights, Affordances, Transitions, and Feedback That Preserve PresenceSign in

Part 38

Comfort and Performance Patterns

Comfort and Performance PatternsSign in
Adaptive Quality: Dynamic Scaling and Performance Stabilizers That Protect ComfortSign in
Comfort Modes: Vignette, Horizon Locking, Reprojected Locomotion, and When to Offer ThemSign in
Cognitive Load Reduction: Clutter Control, Attention Guidance, and Pacing as System DesignSign in

Part 39

Multi-Platform and Abstraction Patterns

Multi-Platform and Abstraction PatternsSign in
Capability-Aware Design: Feature Detection and Graceful DegradationSign in
Input Abstraction: Actions vs Device-Specific Controls and Why Mapping Is a UX ContractSign in
Portability vs Quality: What to Standardize and What Must Be Device-SpecificSign in

Part 40

Prototyping, Storyboarding, and Narrative Patterns

Prototyping, Storyboarding, and Narrative PatternsSign in
Rapid Prototyping: Validating Interaction Hypotheses Before Building Content PipelinesSign in
Spatial Storyboarding: Narrative Beats in 3D Space and Choreographing AttentionSign in
Grayboxing and Proxies: Testing Scale, Comfort, and Flow Early With Placeholder AssetsSign in

Part 41

Patterns for Social XR and Collaboration

Patterns for Social XR and CollaborationSign in
Avatars and Embodiment: Non-Verbal Cues, Identity, and Presence SignalsSign in
Shared Tools and Surfaces: Collaboration Mechanics and Coordination in 3DSign in
Moderation and Safety Tools: Social Boundaries, Reporting, and Anti-Harassment PatternsSign in

Course overview