Overview
free

Course Setup and the Incremental Ladder
Why "Frames to Realities"
How to Use This Course
The Incremental Ladder (Step 0 to Step 7)
The Course Lenses
Diagram Legend and Notation Types

What Is an AR/VR/XR Experience?
Definitions and Boundaries
XR as Real-Time Embodied Simulation
The Triangle - Hardware, Engine, Content

3D Graphics and Coordinate Spaces Recap
Transforms as Contracts
Space Taxonomy
Cameras and Projections

Human Perception, Depth Cues, and Presence
Depth Cues and Cue Consistency
Motion Parallax, Field of View, and Frame Rate
Presence, Embodiment, and Coherence

XR System Building Blocks
Displays, Optics, and Tracking Rigs
Runtime Responsibilities, Scene Graphs, and Spatial UI
End-to-End Dataflows as a Coupled Pipeline

Diagramming AR/VR Systems
Coordinate Frame Diagrams
Rendering Pipeline Diagrams
Interaction Flow Diagrams

Classic 3D Rendering Pipelines
Geometry to Pixels
Scene Graphs and Culling
Frame Timing Basics

Camera Control and Input in 3D
Camera Rigs on Flat Screen
Polling vs Events
Comfort-Adjacent Controls

Stereo Vision and Binocular Rendering
Left/Right Eye Views
Convergence vs Accommodation
Stereo Failure Modes

HMD Hardware, Optics, and Comfort
Optics Artifacts
FOV and Resolution Constraints
Physical Comfort and the Wearable Boundary

Presence, Embodiment, and Simulator Sickness
Presence Factors: Visual, Audio, Haptics, Agency, and Why Coherence Matters More Than Fidelity
Sickness Causes: Sensory Mismatch, Latency, Acceleration Cues, and How Design Choices Amplify or Reduce Them
Designing for Comfort Early: Principles That Prevent Expensive Rework Once Content Scales

Tracking Fundamentals: 3DoF, 6DoF, and Fusion
Orientation vs Position: What 3DoF and 6DoF Enable, and What Breaks When Either Is Missing
IMUs and Drift: Noise, Bias, Drift and Why Raw Sensor Truth Is Not Stable Enough for Presence
Fusion and Prediction Concepts: High-Level Fusion, Prediction Windows, and Why Timing Matters as Much as Accuracy

Inside-Out vs Outside-In Tracking
Tracking Architectures: Headset Cameras vs External Base Stations and the Failure Modes Each Introduces
Occlusion and Tracking Volume: How Real-World Geometry Becomes an Input Constraint
Trade-offs and Product Fit: Setup Friction, Cost, Robustness, and Designing Experiences Around Tracking Reality

Coordinate Spaces in XR
Core Frames: Head, Eye, Hand, Controller, Room, World, Device and How They Compose
Re-centering and Drift: Origin Management as a Product Feature, Not a Calibration Footnote
Guardian and Boundaries: Safety Systems as Coordinate-Frame Products with UX Implications

VR Rendering End-to-End
The VR Frame Timeline: Sample -> Predict -> Render -> Present
Motion-to-Photon Budgets: Where Latency Comes From and What You Can Actually Improve
Foveation as a Resource Strategy: Reallocating Quality to Protect Comfort

Distortion Correction, Reprojection, and Timewarp
Distortion and Pre-Warp: Lens Distortion Correction and Why Rendering Must Anticipate Optics
Late Latching and Reprojection: Reducing Apparent Latency Without Changing Core Simulation Time
Dropped Frames and Async Techniques: Timewarp/Spacewarp Concepts and Graceful Degradation

Spatial Audio and Haptics for Presence
Spatial Audio Basics: HRTFs, Occlusion, Reverb and Why Audio Anchors Presence
Event-to-Sound Mapping: Coupling Simulation Events to Sound Fields Without Disorientation
Haptics Patterns: Vibration and Wearable Haptics as Feedback Channels With Safety and Comfort Constraints

Optical vs Video See-Through AR
Optical See-Through Constraints: Waveguides, Display Limitations, and What "Occlusion" Means When You Cannot Block Photons
Video Passthrough Trade-offs: Latency, Camera Artifacts, and Blending Constraints
Visual Integration Goals: Convincing Blend Versus Stylized Overlay and Choosing the Right Objective

Camera Pose Estimation and SLAM
VIO/SLAM Concepts: Pose Estimation as the Foundation for World-Locked Content
Planes and Feature Points: What Is Detected, What Is Inferred, and What Remains Uncertain
Anchors and Stability: Local Versus World Anchors and How Drift Surfaces as UX Bugs

Occlusion, Lighting Estimation, and Visual Integration
Occlusion Techniques: Depth Sensing, Occlusion Masks, Depth-from-Motion, and Their Failure Cases
Lighting Estimation: Approximating Real Lighting So Virtual Objects "Sit" Correctly
Style as Engineering: Realism Versus Stylization as a Robustness and Performance Decision

Input Paradigms: Controllers, Hands, Gaze, Voice
Ray and Direct Manipulation: Selection/Manipulation Patterns and How Tracking Noise Changes UX
Hand Tracking and Gestures: Affordances, Ergonomics, and Preventing Accidental Activation
Gaze and Voice: Dwell/Confirm Patterns, Eye vs Head Gaze, and the Reliability Limits of Voice in Mixed Spaces

Locomotion and Spatial Navigation
Locomotion Families: Teleport, Dash, Smooth Locomotion, Arm-Swinging and When Each Is Appropriate
Room-Scale vs Seated/Standing: Designing for Physical Constraints and Safety Boundaries
Cognitive Mapping: Helping Users Orient, Remember, and Navigate Without Overload

Spatial UI and Information Layering
UI Anchoring Modes: World-Locked, Head-Locked, Body-Locked and When Each Breaks
Panels, Volumes, Menus: Spatial UI Primitives and How to Keep Them Discoverable
Density and Context: Avoiding Clutter, Managing Focus, and the AR vs VR Differences in Attention

Interaction Design for Comfort and Usability
Ergonomics and Reach: Interaction Thresholds, Fatigue, and Designing Within Human Ranges
Preventing Overload: Accidental Activation, Mode Confusion, and Attention Management
Accessibility in XR: Motion Sensitivity, Alternative Inputs, and Inclusive Spatial Design

Spatial Mapping and Meshes
Building Spatial Meshes: Scanning, Reconstruction, and What the Map Represents
Planes, Meshes, Volumes: Combining Representations for Interaction and Occlusion
Performance and Memory: Spatial Meshes as a Budget Consumer and Why Pruning Is Mandatory

Scene Understanding and Semantics
Semantic Layers: Floors, Tables, Walls, and Why Semantics Changes Interaction Design
Designing With Uncertainty: Confidence Scores, Fallbacks, and "Good Enough" Placement Rules
Meaningful Placement: Aligning Content to Surfaces and Zones Without Overfitting to Noisy Detection

Persistent Spatial Anchors and World Persistence
Anchors Across Sessions: Persistence Semantics and the Product Promises They Imply
Relocalization and Drift: Lifecycle Management for Anchors and Re-Alignment Strategies
Persistence Failure Modes: Environmental Change, Moving Objects, and When "World-Locked" Becomes Wrong

Multi-User Shared Environments
Shared Coordinates: Alignment Across Devices and the Consequences of Coordinate Disagreement
Synchronizing State and Avatars: Replication, Authority, and Presence Under Latency
Collaboration Patterns: Turn-Taking, Shared Tools, and Managing Conflict in Shared Spaces

Safety Boundaries and Real-World Constraints
Guardian Systems: Spatial Boundaries as Safety Infrastructure and UX Features
Mixed-Space Hazards: People, Pets, Objects, and Designing to Avoid Collisions
Privacy and Social Norms: Bystanders, Privacy Zones, and Designing for Public Spaces

XR Engine Architecture and Subsystems
Subsystem Map: Rendering, Tracking, Input, Physics, Audio, Networking, UI and Their Coupling Points
XR Main Loop Design: Update Ordering Under Prediction and Latency Constraints
Plugins and Platform Abstractions: Portability Across Runtimes Without Losing Performance

Tools, Content Pipelines, and Authoring Workflows
XR Asset Pipelines: Formats, Importers, and Constraints That Preserve Scale and Performance
Spatial Authoring: Prefabs, Scene Authoring, and "Spatial Storyboards" as Production Artifacts
Iteration Workflows: In-Headset Editing, Remote Preview, Hot Reload, and Why Iteration Speed Drives Quality

Networking, Cross-Device, and Cloud Services
Multi-Device XR: Standalone, Tethered, Mobile, and Capability-Driven Design
Real-Time Networking for Presence: Avatars, Voice, Shared State, and Latency Budgets
Cloud Services (Conceptual): Persistence, Analytics, and Remote Rendering as System Boundaries

Performance, Optimization, and Testing
Frame Budgets: CPU/GPU/Tracking Budgets and How One Subsystem Can Ruin Comfort
Profiling Motion-to-Photon: Measuring the Full Pipeline Rather Than Local Hot Spots
Testing Strategies: Automated Tests, In-Headset QA, and Comfort Testing as a Distinct Discipline

Accessibility, Ethics, and Well-Being
Designing for Diverse Bodies: Size, Mobility, Sensitivity Differences and Inclusive Interaction Ranges
Intensity Guidelines: Motion, Flashing, and Sensory Load as Safety Constraints
Ethics and Well-Being: Harassment, Addiction Pressure, Data Use, and Environment Safety in Mixed Reality

Shipping and Operating XR Products
Distribution and Certification: Stores, Compliance, and Platform Expectations
Updates and Remote Configuration: Patching Safely in Systems That Interact With the Physical World
Live Ops for XR: Events, Telemetry-Driven Improvements, and Maintaining Presence Quality Over Time

Interaction Patterns for XR
Core Patterns: Ray+Trigger, Direct Grab, Gaze+Confirm, Gesture Vocabularies, and Why Each Has a Comfort Envelope
Menus and Shortcuts: Toolbelts, Radial Menus, Contextual Actions, and Discoverability
Anti-Patterns: Designs That Commonly Cause Discomfort, Fatigue, or Mode Confusion

Spatial UX Design Patterns
Spatial Composition: Placing UI in VR vs AR and Avoiding Occlusion of Meaning
Anchoring to Objects, Spaces, People: Choosing Stable References Users Can Trust
Microinteractions: Highlights, Affordances, Transitions, and Feedback That Preserve Presence

Comfort and Performance Patterns
Adaptive Quality: Dynamic Scaling and Performance Stabilizers That Protect Comfort
Comfort Modes: Vignette, Horizon Locking, Reprojected Locomotion, and When to Offer Them
Cognitive Load Reduction: Clutter Control, Attention Guidance, and Pacing as System Design

Multi-Platform and Abstraction Patterns
Capability-Aware Design: Feature Detection and Graceful Degradation
Input Abstraction: Actions vs Device-Specific Controls and Why Mapping Is a UX Contract
Portability vs Quality: What to Standardize and What Must Be Device-Specific

Prototyping, Storyboarding, and Narrative Patterns
Rapid Prototyping: Validating Interaction Hypotheses Before Building Content Pipelines
Spatial Storyboarding: Narrative Beats in 3D Space and Choreographing Attention
Grayboxing and Proxies: Testing Scale, Comfort, and Flow Early With Placeholder Assets

Patterns for Social XR and Collaboration
Avatars and Embodiment: Non-Verbal Cues, Identity, and Presence Signals
Shared Tools and Surfaces: Collaboration Mechanics and Coordination in 3D
Moderation and Safety Tools: Social Boundaries, Reporting, and Anti-Harassment Patterns

/frames-to-realities/frames-to-realities

Scene Understanding and Semantics

Sign in to access this lesson.

Sign in Create account