Skip to content

About

Converts video tutorials into interactive lessons validated inside Photoshop and Blender

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

AI Onboarding Agent

An AI agent that transforms training videos into interactive lessons. Supply any instructional video — software onboarding, employee training, process walkthroughs, creative tutorials — and the system generates a structured lesson, then guides the learner through it step-by-step, observing their actions in the target application, validating progress, and adapting to their pace.

Currently supports Photoshop (via UXP plugin) and Blender (via addon), with an extensible plugin system for adding new applications.

How It Works

The system has two phases: generation and teaching.

During generation, a training video is transcribed (WhisperX), uploaded to the Gemini Files API, and processed through a multi-stage LLM pipeline that extracts steps, actions, insights, and external assets. The output is a structured YAML lesson file with video-derived goals, keyframe references, and instructor quotes.

During teaching, a LangGraph agent runs an observe-validate-teach loop. It connects to the target application through a plugin system over WebSocket IPC, watches for state changes, validates whether the learner has completed each step's goal, and streams feedback through a floating overlay UI. The agent has access to a RAG knowledge base and can query indexed documentation when the learner needs help beyond what the video covers.

Teaching Agent

The core of the system is a 6-node LangGraph state machine (src/teaching/agent.py):

observe → validate → analyze_hints → (confirm | teach | advance) → END

Each invocation starts with observe_app_changes, which drains events from the application plugin. validate_step runs one-shot validation (see below). analyze_hints performs LLM-based hint signal analysis in a separate node — this isolation prevents structured JSON from leaking into the streamed UI output. Routing after analyze_hints is based on step status: completed steps go to advance (which generates a combined completion + next-step intro message), uncertain validations go to confirm, and everything else goes to teach.

State design. LangGraph state must be serializable, but plugins hold WebSocket connections, DXcam handles, and other non-serializable resources. The RuntimeContext pattern solves this: TeachingState is a plain TypedDict with serializable fields (messages, lesson, step index, hint level), while the TeachableApp instance and related dependencies pass through LangGraph's context_schema parameter, accessible via config["configurable"] in each node.

Streaming. Session methods (stream_start, stream_input, stream_observation) yield typed UIIntent items — a union of 8 frozen dataclasses including AssistantToken, StepTransition, ValidationResult, and VisualGuidance. UI surfaces consume intents via structural pattern matching. For external UIs (UXP panels, Blender addon), serialize_intent() converts to JSON for WebSocket transport. LLM token streaming uses LangGraph's tag-based filtering: only calls tagged "stream-to-ui" emit tokens to the UI (src/teaching/ui_intents.py).

Hint escalation. The agent tracks step_start_time, recent_messages, and consecutive_failures in graph state. The analyze_hints node uses an LLM to detect signals — explicit location requests, frustration, time stuck (>90s), repeated questions — and sets a hint level (1–5). At level 4+, visual arrow overlays highlight specific UI elements in the target application via IPC to a PyQt6 overlay process.

Validation

Step validation uses a one-shot tool-calling pattern with a maximum of 2 LLM calls, regardless of how many tools are involved (src/teaching/validation_agent.py):

  1. Tool selection: The LLM receives pre-gathered structural data (document info, layer hierarchy) and decides which tools to call — capture_screenshot (DXcam), get_keyframe (lesson reference image), capture_canvas (app-specific document capture), or layer/selection queries.
  2. Parallel execution: All selected tools execute concurrently via asyncio.gather. Tool results, including multimodal image content, feed back into the message history.
  3. Structured judgment: A second LLM call with with_structured_output(ValidationOutput) returns a Pydantic model with goal_type (visual / structural / hybrid / unverifiable), passed, confidence (high / medium / low), reasoning, and stay (user wants more time). Unverifiable goals (file exports, clipboard operations) route to user confirmation rather than false negatives.

The entire flow has a 60-second timeout with a safe fallback result.

Lesson Generation

The generation pipeline (src/lessons/generator.py) follows a steps-first architecture — steps are identified with full video context before actions are extracted per step:

  1. Transcribe — WhisperX extracts word-level timestamps
  2. Upload — Full video uploaded to Gemini Files API (preserves audio for context)
  3. Identify steps — LLM sees the entire video + transcript, outputs a visual_outcome per step that becomes the step's goal
  4. Extract actions — Per-step extraction with Gemini's native SDK video_metadata for temporal clipping (LangChain doesn't support this). Returns a step_action_map preserving step-action associations
  5. Enrich + Detect assets — Parallel: enrich steps with implementation details; detect external resources (fonts, textures, brushes) from transcript
  6. Match asset URLs — Link detected assets to download URLs in the YouTube description
  7. Synthesize — Extract instructor insights with timestamps, generate learning objectives

All structured output calls use invoke_with_retry() (src/core/llm.py), which on ValidationError feeds the Pydantic error message back to the LLM as a HumanMessage so it can self-correct on the next attempt, with exponential backoff between retries.

LLM configuration is decoupled from usage through a preset system: get_llm_for_preset("validation") loads model, provider, thinking level, and media resolution from app_config.yaml. Offline tasks (generation) use high thinking depth and high media resolution; interactive tasks (teaching) use low thinking for speed.

Plugin System

The agent connects to desktop applications through a protocol-based plugin system (src/core/plugins/interface.py):

TeachableApp is a @runtime_checkable Protocol generic over StateT and EventT. Plugins extend TeachableAppBase, which provides event buffering, validation tool construction, triage examples, and window capture. Plugin authors implement ~6 methods (connect, disconnect, state, format_for_llm, format_summary, diff_states); the base class provides ~10 more.

State diffing. Each plugin implements diff_states(old, new) to compute semantic events from raw state changes. The Photoshop plugin, for example, compares PhotoshopState snapshots and emits typed events like LAYER_ADDED or SELECTION_CHANGED. This filters out noisy heartbeat updates and prevents false-positive interventions from the proactive observer.

Event flow. External plugins (UXP for Photoshop, addon for Blender) send state over WebSocket. Events pass through an EventBuffer with per-consumer cursors — the teaching session and the proactive observer drain independently without interference.

Knowledge System

The teaching and validation agents can query an indexed knowledge base during lessons (src/teaching/knowledge_tool.py). Knowledge retrieval is bound as a LangChain @tool — the LLM decides when to query rather than following hardcoded rules. The system works without an index (graceful degradation).

Retrieval uses a hybrid approach: BM25 for keyword matching + dense embeddings via EnsembleRetriever, with optional FlashRank reranking. Query preprocessing strips question patterns for better BM25 matching ("What is a layer mask?" → "layer mask"). During indexing, a PruningContentFilter applies density-based content extraction — this reduced Adobe's Photoshop documentation from 105k to 1.2k chunks (src/core/knowledge/loaders.py, src/core/knowledge/retriever.py).

Prompt Architecture

All prompts live in YAML files (src/*/prompts/), not Python code. The system uses a two-tier structure:

  • Centralized prompts (src/teaching/prompts/) define persona, step introductions, and lesson flow with {application_name} placeholders. These work across all plugins.
  • Plugin-required prompts (src/apps/<plugin>/prompts/) define app-specific teaching language — how to describe Photoshop layers vs. Blender objects, validation rules for structural vs. visual goals, observation response patterns. There is no fallback; each plugin must provide its own.

Generation prompts use XML sections (<PERSONA>, <GROUNDING_RULES>, <OUTPUT_FORMAT>) in system messages for static structure, with {variables} in user messages for runtime context. Formatting is handled by PromptBuilder (src/teaching/prompt_builder.py), which assembles system messages from plugin and centralized prompts at runtime.

Getting Started

Requirements: Windows, Python 3.11+, uv, ffmpeg, yt-dlp, CUDA-capable GPU

uv sync --extra dev
uv run teacher generate "https://www.youtube.com/watch?v=VIDEO_ID"
uv run teacher overlay lessons/my-lesson/lesson.yaml
uv run pytest

See CLAUDE.md for environment configuration and CLI reference.

Documentation

About

Converts video tutorials into interactive lessons validated inside Photoshop and Blender

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages