A DREAM Robotics Agentic-Consciousness
- Robot Type: Agentic-Consciousness
Windows app: builds/dream.exe opens DREAM's page (/dream.html, on the local port so the mic works) in a window and starts run.bat if it isn't running. Keep it in builds/. Source: scripts/web_controller.
Android app: Android App/ (package com.synthwomb.dream) opens DREAM from the PC she runs on, full screen. See DREAM on your phone (Android app).
 ARIA |
 KIDA-01 |
 KIDA-00 |
 WHIP |
 NORA |
 MILA |
 NIKA |
|
 IDA |
 ARM |
 RIFT |
 RITA |
 NINA |
 TINA |
 AURA |
 FLORA |
Website: https://cursedprograms.github.io/DREAM/
Overview
DREAM is a localized agentic-consciousness embedded robotic system and the cognitive core of the DREAM Robotics ecosystem.
Operating as a sovereign offline entity, she serves as the primary command-and-control interface for the KIDA and NORA robotic lineages through the RIFT neural protocol.
DREAM does not simply execute commands — she observes, remembers, and “dreams”.
She bridges static code and emergent autonomous behavior.
Pick her model in the options menu (LlmModel); download_weights.bat pulls it. Some picks:
- qwen3:4b-instruct-2507-q4_K_M: the default. Qwen3 4B (the same as TINA's), minus the thinking: smarter than phi3 at about the same speed, and never reasons out loud.
- phi3:mini: the old default.
- gemma3:4b: usually the most natural conversational voice at this size.
- llama3.2:3b: fast and reliable, with less personality.
- phi4-mini: the newer Phi, better than phi3:mini but with the same style.
- gemma3:12b, mistral-nemo:12b, gemma3:27b: for a GPU with 12 GB or more.
Thinking models (qwen3:4b, deepseek-r1, gpt-oss) aren't in her list: tested with qwen3:4b, the reply came back empty (it spent it all thinking), or it thought out loud. They're fine for the memory server's Memory.LlmModel, which nobody hears.
a Latent space dream
pleasant, strange and nightmare: her latent walk, then DeepDream (InceptionV3, per tone)
"I was walking through a library made of water, and the books kept turning into fish. My grandfather's guitar floated past and played the sound of rain."
her dream, seen by her image generator
her spiral walk turning into the dream, frame by frame
Prerequisites
- Python 3.12.3 for Lunix
- Python 3.11.9 for Windows (the repo's
venv311) - Arduino IDE with the libraries Adafruit NeoPixel and DFRobot_C4001 (for
dream_sensors.ino) - Ollama with a model pulled (default
qwen3:4b-instruct-2507-q4_K_M) - Piper (a voice
.onnxinvoices/), ffmpeg (for the browser microphone), Git - To build the C++ version: MinGW-w64 g++, CMake, Ninja (
cpp_dream\setup.batinstalls them)
DREAM is a robot too: this PC is her body, the way servos and boards are the other robots'. The speeds in How fast she answers were measured on it.
| Part | Her build |
|---|---|
| CPU | Ryzen 5 5600X (6 cores, 12 threads): her voice (Piper), and all her image work (looks, dreams, DeepDream, faces) so it never slows the conversation |
| GPU 0 | RTX 2060, 6 GB: Whisper, the wake word, her LLM and lip-sync (Device: cuda) |
| RAM | 64 GB DDR4-3200 (2 x 32 GB) |
| Motherboard | B450 (AM4); the second x16 slot is PCIe 2.0 x4 |
| Storage | 1 TB NVMe, 500 GB NVMe, 2 TB HDD |
| PSU | 550 W |
| Component | Details |
|---|---|
| RAM | 8GB+ RAM (the language model, Whisper and the avatar video all live in memory; on a full PC the optional DeepDream pass and TensorFlow are skipped automatically) |
| CPU / GPU | Works on a CPU alone, slowly (30-140 s per reply with a 4B model). An NVIDIA GPU with a working Ollama CUDA build is much faster |
| OS | Windows 10/11 or Linux. The C++ build is Windows only |
| Component | Details |
|---|---|
| Microcontroller 0 | Arduino UNO, running dream_sensors/dream_sensors.ino (answers I am Dream) |
| Component | Details |
|---|---|
| Motion sensor | PIR (drives the alarm) |
| Presence and range | DFRobot C4001 (SEN0609) mmWave radar, over UART |
| Buzzer | Alarm and beeps |
| Lights | 40 x WS2812/NeoPixel RGB strip |
- USB Microphone
- Speakers
- Webcam (optional: without one she simply has no eyes)
[!CAUTION] Ground Loop Warning: All modules must share a common GND. Failure to bridge grounds will cause erratic motor behavior and sensor noise.
Sensor Wiring
These pins match dream_sensors/dream_sensors.ino.
- VCC → 5V
- GND → GND
- OUT → Pin 4
-
- → Pin 3
-
- → GND
- Data → Pin 6
- 5V and GND from a supply that can carry it (40 pixels can draw over 2 A at full white; share GND with the Arduino)
- Sensor TX → Pin 9
- Sensor RX → Pin 10
- VCC and GND as per the module's datasheet
- NOTE: an I2C humidity and temperature sensor and state LEDs are still to be added.
[!TIP] Pro-Tip: Make sure all modules share a common ground (GND) for stable operation.
Connectivity & Controls
| Parameter | Value |
|---|---|
| SSID | NORA |
| Password | 12345678 |
cpp_dream/ is dream.py rewritten in C++: a single self-contained dream.exe (about 6 MB, no DLLs). It has the fullscreen avatar, the "Hey DREAM" wake word, the idle flirt and sleep timers, the sensor board, alarm and light voice commands, system stats, the network scan and memories. No Python is needed to run it.
Speech recognition is Whisper built into the exe (whisper.cpp). It still uses Ollama for the language model and Piper for the voice, because those are separate programs.
Open a terminal in the cpp_dream folder and run:
setup.batGets everything that isn't in the repo, using the models picked in the options menu (WhisperModel, Voice, LlmModel). It only downloads what's missing, so it is safe to run again, for example after picking a different model:
| What | Size | Where it goes |
|---|---|---|
C++ compiler + CMake + Ninja (WinLibs, via winget) and Git |
- | your PATH |
Whisper speech model ggml-<WhisperModel>.bin |
77 MB (tiny) to 3 GB (large) | cpp_dream/models/ |
Piper (standalone, no Python) and the voice (default en_US-amy-medium) |
22 + 63 MB | piper/ and voices/ in the repo root |
Ollama, and the model (default qwen3:4b-instruct-2507-q4_K_M) |
- | Ollama's own folder |
If setup.bat installed new tools, close the terminal and open a new one so Windows picks them up. Then:
build.bat
build\dream.exe --selftest
build\dream.exebuild.bat compiles build\dream.exe. The first build downloads and compiles whisper.cpp and takes several minutes. Later builds only recompile what changed. --selftest checks every part works (speech, memory, Ollama, video, sensors) without needing a microphone or speakers. Esc or Q quits dream.exe.
If you'd rather not use the scripts, you need a recent MinGW-w64 g++ (built and tested with GCC 16.1 from WinLibs), CMake 3.20+, Ninja and Git on your PATH, plus internet on the first configure (CMake fetches whisper.cpp and nlohmann/json):
cd cpp_dream
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build --target dream -jCMake downloads the Whisper model too, using curl (included with Windows 10/11). If that fails, download ggml-tiny.en.bin into cpp_dream/models/ yourself. It has only been built with MinGW, not Visual Studio.
| Option | What it does |
|---|---|
| (none) | borderless fullscreen avatar, listens for "Hey DREAM", looks for the sensor board |
--windowed |
a normal window instead of fullscreen |
--no-voice |
display only: no microphone, Whisper or Ollama |
--no-sensors |
don't look for the sensor board |
--selftest |
check every part works, then exit (--wifi adds the network scan) |
--mute |
don't play sound (the talking video still shows) |
--mic-file F.wav |
use a WAV file as the microphone, to test without one |
--flirt-after S, --sleep-after S |
shorten the idle timers (defaults 600 and 900 seconds) |
--shot F.bmp |
save a screenshot of the window a few seconds in |
dream.exe finds the repo by looking upward for config.json, so it works from cpp_dream/build/ or anywhere below the repo. It reads the same config.json, videos/, voices/ and memories/ as dream.py, so both share memories.
- "Ollama's GPU mode failed - using the CPU from now on": Ollama's CUDA build can't run on this graphics driver.
dream.exeswitches to the CPU on its own, which works but is slower. Updating your graphics driver (or Ollama) fixes it properly. - "No microphone found": Windows isn't exposing a recording device. It can still talk, but not listen. Check Settings > Privacy > Microphone.
- No voice, or "TTS: could not start piper.exe": it looks for Piper in
venv311/Scripts/,venv/Scripts/,piper/in the repo root, or besidedream.exe. Runsetup.batto get a standalone copy. It needs a voice invoices/*.onnx. - "Sensor board not found": the board must be flashed with the current
dream_sensors.ino(it answersWHOwithI am Dream) and plugged into a USB serial port. Only one program can use it at a time, so don't rundream.exeandapp.pytogether. - Video won't play ("no H.264 decoder"):
dream.exeuses Windows' built-in video decoder. Windows "N" editions may need the free Media Feature Pack (Settings > Optional features). - Configure fails while fetching whisper.cpp: it needs internet and Git the first time.
Not ported: MuseTalk lip-sync (she uses the talking clips instead) and the deep-dream image generation while asleep. A spoken command ends after 1.5 seconds of silence rather than always recording 16 seconds, and short lines like "Yes?" and "Bye for now." are cached after the first time so they play instantly.
python scripts/dream.py --web (or menu option 1w in main.py) starts app.py if it isn't already running and opens http://localhost:5010/dream.html. That address is a plain-HTTP listener on this PC only, so the browser gives you the microphone with no certificate warning; other devices use DREAM's own site, https://<this-pc>:5001 (straight to her), and accept the one-time self-signed certificate warning.
Tap the start screen once (browsers only allow sound and the microphone after a tap). Then the page behaves like the desktop dream.py:
- The same video clips (idle, listening, thinking, talking, the intro, the flirty clips and the sleeping loop, from
videos/) - The "Hey DREAM" wake word: the page records short clips and the server transcribes them with Whisper, so it works offline; then she greets you and listens for one command
- Sleep and wake (idle timers, "wake up", the mmWave sensor), the flirt clip, the distance-based greeting and the farewell when someone leaves
- The alarm and light voice commands, plus wifi scan and stats, and typed messages (the keyboard button)
- Her inner life: it is the same mind, so memories, mood and dreams are shared with the desktop app
- Just her face until you press Space (or double-tap her on a phone), then the controls;
Tshows her thoughts
The sleep and flirt timers and the sensor reactions live on the server, so every open page stays in sync. A page reports when it is busy so she doesn't fall asleep or say goodbye mid-conversation.
Differences from the desktop app: no MuseTalk lip-sync (she uses the generic talking clips), no deep-dream job while asleep (the mind's own dream painting runs instead), and saying "goodbye" puts her to sleep instead of exiting. Speech recognition and the voice still run on this PC (Whisper and Piper); browser audio is converted with ffmpeg, so ffmpeg must be installed.
Files: templates/dream.html, static/js/dream.js, scripts/web_launcher.py (the launcher).
app.py (her dashboard) registers DREAM with RIFT every few seconds, so start it (or run.bat, which starts it with DREAM). RIFT reaches her on:
https://<this-pc>:5009(http://localhost:5010on this PC); DREAM's own site is onhttps://<this-pc>:5001
Ollama Setup
sudo snap install ollama
ollama --versionirm https://ollama.com/install.ps1 | iexhttps://ollama.com/download/windows
ollama pull gemma3:4b-it-qat
ollama pull deepseek-r1:14b
ollama pull qwen3:4b-instruct-2507-q4_K_M
ollama pull tinyllama
ollama pull llava:13bollama run gemma3:4b-it-qat
ollama run deepseek-r1:14b
ollama pull qwen3:4b-instruct-2507-q4_K_M
ollama run tinyllama
ollama run llava:13bollama serve &ollama run llama2sudo apt update
sudo apt install ffmpeg alsa-utils -ywinget install ffmpeg
winget install alsa-utilsEnvironment Setup
python -m venv venv
source venv/bin/activate
pip install -r requirements.txtpython.exe -m pip install --upgrade pip
py -3.11 -m venv venv311
venv311\Scripts\activate
pip install -r requirements.txtpip install --upgrade pip setuptools wheel
pip install chumpy --no-build-isolationpip install openai-whisper piper-tts pathvalidate sounddevice soundfile numpy requests faster-whisper pygame psutil requests flask zeroconf pyserial opencv-python face_alignment scipy tensorflow Pillow diffusers transformers accelerate librosa argparse mmpose mmcv mmengine diffusers transformers accelerate --upgrade torch==2.5.1+cu121 torchaudio==2.5.1+cu121 --index-url https://download.pytorch.org/whl/cu121pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu121/torch2.1/index.htmlpip install https://download.openmmlab.com/mmcv/dist/cu121/torch2.3.0/mmcv-2.2.0-cp311-cp311-win_amd64.whlPiper Setup
sudo apt install piperpython -m pip install piper
python -m pip install piper-ttsmkdir -p ~/voices/
# Amy (medium) — recommended
wget "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/amy/medium/en_US-amy-medium.onnx?download=true" -O en_US-amy-medium.onnx
wget "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/amy/medium/en_US-amy-medium.onnx.json?download=true" -O en_US-amy-medium.onnx.jsonmkdir -p ~/voices/
curl -L "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/amy/medium/en_US-amy-medium.onnx?download=true" -o en_US-amy-medium.onnx
curl -L "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/amy/medium/en_US-amy-medium.onnx.json?download=true" -o en_US-amy-medium.onnx.jsonInvoke-WebRequest "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/amy/medium/en_US-amy-medium.onnx?download=true" -OutFile "en_US-amy-medium.onnx"
Invoke-WebRequest "https://huggingface.co/rhasspy/piper-voices/resolve/v1.0.0/en/en_US/amy/medium/en_US-amy-medium.onnx.json?download=true" -OutFile "en_US-amy-medium.onnx.json"wget https://github.com/rhasspy/piper/releases/download/2023.11.14-2/piper_linux_x86_64.tar.gz
tar xzf piper_linux_x86_64.tar.gz
sudo mv piper/piper /usr/local/bin/echo "Hello, I am your voice assistant." | \
piper --model voices/en_US-amy-medium.onnx \
--output_raw | aplay -D plughw:2,0 -r 22050 -f S16_LE -t raw -Stream only:
python speak.pyStream and save WAVs to /audio/:
python speak.py --savepython detect.py --image <image_name>python detect.pyWhisper Setup
Pick the model in the options menu (WhisperModel; Device picks CPU or GPU for all of DREAM), then run download_weights.bat. Or download it by hand:
python scripts/whisper_model.py # which model config.json picks
python scripts/whisper_model.py --download # download itMuseTalk Setup
[face_alignment](https://github.com/1adrianb/face-alignment)Go to:
\venv311\Lib\site-packages\mmdet\__init__.pyChange the maximum version:
mmcv_maximum_version = '2.3.0'Go to:
\venv311\Lib\site-packages\transformers\utils\import_utils.py:
def check_torch_load_is_safe() -> None:
return # <--- Put it here, OUTSIDE the if statement
if not is_torch_greater_or_equal("2.6"):
raise ValueError(...)Wav2lip Setup
Generated using Wav2Lip-GAN with --resize_factor 2
- You can lip-sync any video to any audio:
python inference.py --checkpoint_path "checkpoints/wav2lip-sd-gan.pt" --face "/videos/musetalk_talk.mp4" --audio "/audio/audio.mp3" --resize_factor 2 - Fully local voice chatbot pipeline (offline capable)
- Emergent and unpredictable behavior patterns
- Continuous perception + memory loop
- Robotics integration layer (KIDA / NORA / WHIP ecosystem)
- An inner life: mood, needs, memory that fades, dreams, opinions that drift, and the habit of speaking first (Inner Life)
- Runs as a desktop app, in a browser, or as one C++ program
- Finds her own hardware: boards introduce themselves over serial, so no fixed COM ports
.jpeg)
| Use it | Ways to run DREAM - How it fits together - Voice commands - Quick start |
| What she is | Overview - System awareness - Depth vision - Autonomous behavior - Inner Life |
| Hardware | Hardware - Pinout - Sensor board |
| Other front ends | DREAM in the browser - DREAM as one program (C++) |
| Reference | Web server and API - Data and memory files - Configuration - Testing - Experiments - Environment notes - Repository layout - Troubleshooting - Privacy and safety |
Pipeline:
Mic → Whisper → Ollama → Piper TTS → Lipsync → Speaker
With her inner life switched on (it is by default), a step sits between hearing and answering:
Mic → Whisper → boundaries + memory + mood (dream_mind) → Ollama → Piper (voice shaped by her mood) → Speaker
She stops listening when you stop talking: 1.5 s of quiet after speech ends a command. Quiet means near the room's own noise (the quietest moment of the last 2 s), so a fan or mic hiss doesn't keep her listening for the full 16 s. The browser page, dream.py and the C++ dream.exe use the same rule, and tap-to-talk stops on its own too.
Measured 2026-11-04 on an RTX 2060 (6 GB), small.en and qwen3:4b-instruct-2507-q4_K_M, a 5 s spoken command, three runs:
| Step | Device: cuda |
before (Whisper on the CPU, 20 LLM layers on the GPU) |
|---|---|---|
| Wake-word check (3 s clip) | 0.22 s | 1.59 s |
| Whisper (the command) | 0.61 s | 2.06 s |
| LLM reply | 2.81 s (74 tokens/s) | 4.65 s (22 tokens/s) |
| Piper (whole reply, CPU) | 2.51 s | 2.47 s |
| You stop talking → she speaks | 7.4 s | 9.3 s |
The total includes the 1.5 s of quiet that ends a command. Whisper and the LLM together use about 4.4 GB of the card. Lip-sync (MuseTalk) took about 45 s for 6.7 s of speech and peaked at 6.7 GB, more than a 6 GB card has, so it stays off (LipsyncEnabled) on cards that size.
options.bat (or options.ps1, options.sh) opens a game-style options menu for config.json (the same menu every bot has): sections as menus, on/off toggles, number steppers, text fields, and dropdowns for fixed choices from options_choices.json. Saving keeps config.json.bak.
| How | Command | What you get | Needs |
|---|---|---|---|
| Everything (start here) | run.bat |
Sets up venv311 the first time (and again when requirements.txt changes), then starts app.py (minimised window) and dream.py together - so DREAM registers with RIFT and her sensor board works. Closing DREAM closes the server too |
Python 3.11, Ollama, Piper |
| Desktop app | python scripts/dream.py (or option 1 in python main.py) |
Fullscreen avatar video, "Hey DREAM" wake word, sleep and wake, sensor board, voice commands, inner life | The venv, Ollama, Piper, microphone, speakers, a display |
| In the browser | python scripts/dream.py --web (option 1w) |
Starts app.py if needed and opens http://localhost:5010/dream.html. Same behaviours, using the browser's microphone and speakers |
The venv, Ollama, Piper, a browser |
| Web server and dashboard | python app.py (option 5) |
Dashboard and API at https://<this-pc>:5009, DREAM on your phone at https://<this-pc>:5001, and it owns the sensor board |
The venv, Ollama, Piper |
| One program (C++) | cpp_dream\build\dream.exe |
Fullscreen avatar, wake word, sensors, timers, memories, stats and network scan in one .exe |
Built with cpp_dream\build.bat |
Which to pick: dream.py is the full experience on the PC with the screen. Start app.py first if you want the dashboard and the sensor board, because dream.py reads the board through app.py. The browser version is for a phone or a second screen. The C++ cpp_dream/build/dream.exe is the standalone build and opens the sensor board itself, so don't run it at the same time as app.py (only one program can hold a serial port).
Android App/ is DREAM on an Android phone (package com.synthwomb.dream). It doesn't run her itself. It opens her phone site (https://<her-pc>:5001/dream.html) from the PC running app.py, full screen, so it follows the same rules as every page of hers:
- She comes to the phone. The newest open page of hers is her body: it shows her, speaks and listens. Opening the app moves her to the phone. The PC's page goes on standby and says "DREAM is with you on a phone". Leaving the app or closing it gives her back to the PC at once (the app says goodbye the way a closed tab does). Coming back to the app brings her back to the phone.
- Her eyes. With a camera, the phone's camera is her eyes while it's her body. Otherwise she uses the proxy's webcam, then her own PC's (
scripts/dream_eyes.py). - Upright or sideways. Turn the phone and she turns with it. Upright she plays the portrait version of the same clip, from the same moment (see Portrait clips below).
- The phone's sensors. While the phone is her body she gets its sensors too, alongside her own sensor board, which keeps working:
- A hand or face near the proximity sensor, or picking the phone up, counts as company (like the radar's PRESENT, at most every 20 s). The phone never says "nobody's here": only her board's radar can.
- The light meter tells her when the lights go out or come back on.
- The battery level and whether it's charging are reported too.
- The readings show in
/api/sensorsunderphoneand in the sensor log. A phone missing a sensor simply doesn't send that reading. Readings from a page on standby are ignored.
- Finding her. FIND asks RIFT's fleet list (like the robots' apps do), then listens for her own network announcement (
_flask-link._tcp, "DREAM"). You can also type her PC's address (192.168.0.100, oraddress:port). The app remembers it and connects straight away next time. - Her certificate. Her PC makes its own HTTPS certificate, so the first time, the app shows its SHA-256 and asks whether to trust it.
app.pyprints the same fingerprint at start-up ("Certificate SHA-256"). After that the app only accepts that exact certificate, and it asks again if it changes (for example, when her PC gets a new address).
Build it with gradlew assembleDebug in Android App/, with JDK 17. The APKs are kept in Android App/apk/: the newest is DREAM-debug-2026-11-05.apk.
videos/portrait/ holds a 720x1280 portrait version of each clip, with the same name. Her face fills the width of an upright phone, faded at the top and bottom into the black she stands in. Her page (in a browser or the app) plays them while it's taller than it is wide. Turning the screen swaps to the other version mid-clip. A clip without a portrait version plays its landscape one, cropped. Robots that mirror her (RITA) still get the landscape clip.
After adding or changing clips in videos/, run venv311\Scripts\python.exe scripts\make_portrait_videos.py (it needs ffmpeg). It only makes the missing or outdated ones. --all makes them all again.
The robots talk in Brainfuck: every phrase is a Brainfuck program that prints its words (hello is ++++++++[>+++++++++++++<-]>.+. → hi), and a robot says it by beeping the program on its buzzer, one tone per symbol, pitched to its own voice. The phrasebook (talk_bf.h) is generated by RIFT's Fleet/brainfuck_talk.py: phrases 0–6 (hello, how are you, happy, curious, sleepy, let's play, bye) and their replies 7–13. RIFT runs the conversations between robots over WiFi and logs them on its dashboard; it asks DREAM to speak with GET /chirp?u=<0-13>. app.py sends TALK:<u> to the sensor board, which beeps it on its buzzer in a deep, slow voice (not during an alarm). Her dashboard registers talk:<port> with RIFT.
With another robot that has a mind (DREAM, KIDA-01, TINA - they register the peer capability), RIFT holds a real conversation when nobody is using them: rock paper scissors (both beep their move), then the winner picks words (spoken by the LLM) or Brainfuck (peer_talk.py, the same file in DREAM, KIDA-01 and TINA). In Brainfuck the chat opens with hello and only says goodbye at the end.
dream_sensors.ino (Arduino) ARM (robot arm, another Arduino)
PIR alarm, mmWave presence/range, answers "I am Arm"; controllers find it by name
buzzer, RGB strip; answers "I am Dream"
| USB serial
v
+-----------------------------------------------------------------------------+
| app.py (Flask, ports 5009 HTTPS / 5001 DREAM / 5010 HTTP-localhost) |
| owns the sensor board · /events stream · /api/* · dashboard · /dream.html |
+--------+-----------------------------+--------------------------+-----------+
| events, commands | browsers / phones | RIFT, NORA
v v v
scripts/dream.py dashboard (/) fleet robots
avatar video, wake word avatar page (/dream.html)
voice loop, sleep, flirt
Mic -> Whisper -> dream_mind -> Ollama -> Piper -> Speakers
^ memory, mood, needs, opinions, dreams, initiative
|
Webcam -> surveillance.py -> photos + frames -> dream_mind (what she saw; your expression, if you allow it)
Everything runs on this PC. Ollama serves the language model, Piper speaks, Whisper listens, and nothing is sent to the internet during use (only setup downloads models).
.jpeg)
| You say | What happens |
|---|---|
| "Hey DREAM", "hi DREAM", "okay DREAM", "DREAM" | Wakes her; she greets you (the greeting depends on how far away you are) and listens for one command |
| "Goodnight" | Her call, not a command: she checks the clock. 21:00-05:00 she usually goes to sleep (about 80%, more when she's tired, less when she's missed you; it makes her drowsier either way). In the daytime she's sarcastic, jokes or teases you, and only 1 time in 20 takes a nap. She looks first: your name if she knows your face, and if you look tired she says so. Her own language model words it (scripts/dream_mind/bedtime.py) |
| "Wake up" (or "Dream") | Calls her while she sleeps. Like a person she may need a few calls (InstantWake, see Sleep Mode); someone appearing at the sensor is a nudge too |
| "Check the wifi", "scan the network", "who's on the wifi", "list devices" | Scans the local network and reads out what she finds |
| "System stats", "CPU usage", "how's the system" | CPU, RAM and disk (and CPU temperature on Linux) |
| "What's the weather?", "will it rain?", "how warm is it at MILA?", "what's the moon doing?", "when is sunset?" | She asks AURA, the weather station: first in Brainfuck (she beeps "wx?" and AURA beeps the sky back in one word: sun, cloud, rain, fog, snow, storm, hot or cold), then she says AURA's answer. AURA is found in RIFT's fleet list, else on this PC (scripts/dream_aura.py). "Temperature" alone is still her own system stats |
| "Turn on / off the alarm", "arm / disarm the alarm" | Arms or disarms the PIR alarm on the sensor board |
| "Lights red / blue / green / yellow / orange / purple / pink / cyan / white" | Sets the RGB strip (needs the word "lights", "make it" or "turn it") |
| "Rainbow mode", "party lights", "lights off" | Rainbow animation, or lights off |
| "Tell NORA to stop the music", "tell NORA to turn her lights on", "tell KIDA01 to turn left for two seconds" | Tells a robot to do something, through RIFT (POST /talk, verb: do). Anything that moves a robot asks you first: say "yes" or "no" (no wake word needed). Unlocking a robot is never passed on |
| "Ask MILA how she is", "how is NORA doing?", "what's NORA's status?" | Asks a robot its state through RIFT and reads out the main points |
| "Hey TINA, ...", "ask TINA to ...", "fix the bug in NORA's sketch", "why isn't KIDA's Hailo responding?" | Hands the job to TINA (code, files, tools, several steps) and says only her final answer. TINA's questions ("this moves NORA, go ahead?") are passed on for your yes or no |
| "Goodbye", "exit", "quit", "bye", "shut down" | Ends the session (the browser version goes to sleep instead) |
| "Remind me to call mum in 20 minutes" | Sets a reminder; she speaks it when it's due |
| "How are you feeling?", "What do you remember about me?", "Did you dream?", "What's on your mind?", "Why did you say that?", "What have you seen?", "Are you conscious?" | She answers about herself, from her real state (Inner Life) |
| "Forget that", "forget about X", "forget everything" | Erases memories (the last one asks you to confirm first) |
| "Stop talking on your own" / "you can speak up on your own" | Switches her unprompted remarks off or on |
| "Watch my face while we talk" / "stop watching my face" | Turns expression reading on or off (off by default) |
| "My name is Kimmie" (facing the camera) | She learns your face, and greets you by name when she sees you again |
| "Who am I?" / "Do you recognise me?" | She tells you who she sees |
| "What do you see?" / "Look at this" / "What am I holding?" | She takes a picture and describes it, by name if she knows you |
| "Forget my face" / "forget all faces" | Deletes the face prints |
| "Stop looking around" / "you can look around" | Turns her photo-a-minute off or on |
| "What did you see today?" | She answers from what she remembers seeing |
The same commands work typed into the dashboard chat and the browser avatar page. Anything else goes to the language model.
Just her face (the default): the desktop app, her browser page and dream.exe start with nothing over her video. Space shows or hides the controls: her state, her captions and, on the page, the buttons (on a phone, double-tap her face). The thoughts panel has its own switch, below, so Space leaves it as it is.
Show thoughts (off by default; T in the desktop app and on the browser page, or 💭 there, ShowThoughts in config.json): what she thinks but doesn't say, on screen: TINA's reasoning and tool calls, and what RIFT and the robots answered. None of it is ever spoken; Piper only gets the final answer (scripts/dream_fleet.py).
- Hearing: USB microphone, Whisper speech recognition (the wake word is checked in short clips)
- Presence and distance: a mmWave radar (DFRobot C4001) reports when someone is there and how far away
- Motion alarm: a PIR sensor drives a buzzer and red pulsing lights
- Sight: a webcam (
surveillance.py) takes a photo every ten minutes and a frame whenever something moves; the mind studies them (Inner Life) - Time: she knows the hour, and it shapes how sleepy she is
- Depth: a MiDaS depth map from the single webcam (Depth vision); a standalone tool for now, not wired into the mind
scripts/monocular_depth.py (option 7 in python main.py) estimates depth from the one webcam with MiDaS v2.1 and shows the feed next to a depth map (brighter = closer). Press q or Esc to quit.
python scripts/monocular_depth.py # webcam
python scripts/monocular_depth.py --image photo.jpg # one image, saved to scripts/output/depth/
python scripts/monocular_depth.py --video clip.mp4 # a video file
python scripts/monocular_depth.py --small / --large # force a model- The model is downloaded to
scripts/models/on first use: Largemodel-f6b98070.onnx(~400 MB) or Smallmodel-small.onnx(~63 MB). - Images use Large. Live video uses Small on the CPU (about 4 FPS; Large manages about 0.4) and Large when OpenCV is built with CUDA. The venv's
opencv-python-headlessis CPU-only. - Only one program can hold the webcam, so stop
dream.py/app.pysurveillance before running it on the webcam.
A 2.5D parallax on her avatar videos, like After Effects' Displacement Map. Each clip has a depth map, and on playback the near parts (her face) shift more than the far parts along a slow sway, so the flat video gains a sense of depth. Off by default.
- Build the depth videos once (option
7dinpython main.py, about 8 minutes on the CPU):python scripts/depth_effect.py # builds missing or outdated ones into videos/depth/ python scripts/depth_effect.py --force # rebuild all (e.g. after changing clips)
- Turn it on in
config.jsonand restartdream.py:"DepthEffect": { "Enabled": true, "Strength": 14, "Speed": 0.12, "Scale": 1.0 }
| Setting | Meaning |
|---|---|
Enabled |
On or off |
Strength |
How far the nearest parts move, in pixels at 1080p (try 8-30) |
Speed |
Sway cycles per second (0.12 = one sway every ~8 s) |
Scale |
Resolution the warp runs at. 1.0 is full quality; lower it to 0.5 if the videos play slow (softer image, about 30% cheaper) |
- Only the looping clips (idle, listening, thinking, talking, sleeping) get the effect. One-shot clips (intro, lipsync replies, flirt) play with audio and stay flat, so the lips can't drift out of sync.
- The warp runs on the GPU when torch has CUDA, otherwise on the CPU, on a worker thread next to the video decoding. At 1080p it takes about 39 ms per frame on the GPU (just inside 24 fps) and about 53 ms on the CPU, so on a CPU-only machine use
Scale0.5. - The desktop app only (
dream.py); the browser avatar page plays the plain videos.
- CPU load, RAM and disk (CPU temperature on Linux)
- The sensor board's status, and a log of what it saw, on the dashboard
- LAN device scanning (a ping sweep, then the ARP table)
- IP / MAC tracking, hostnames, and a guess at the device type
- Waits for the wake word, then runs the Listen → Think → Respond loop (Whisper → LLM → Piper)
- Shows a video state for what she is doing (idle, listening, thinking, talking, sleeping)
- After 10 minutes idle she plays a flirty clip; after 15 minutes she falls asleep (sooner when she is sleepy, later when she is fresh)
- Someone arriving at the sensor keeps her awake; when they leave she says "Bye for now."
- She may speak first: check in when she is lonely, ask about you, mention something she saw, tell you a dream, welcome you back after a long absence, or notice you seem different from usual. She holds back at night, in an empty room, and right after you speak
- Communicates with other robots through RIFT
- Cycles through consolidation (the day's memories are replayed, the important ones strengthen, the trivial ones fade) and dreaming
- Each dream is a short story built from her real memories, and is painted by the latent-space dream engine (
latent_space.py), with a still and an animation - Dream visions: the dream's own words go to her image generator (
scripts/dream_mind/dream_generator.py, SD-Turbo / SDXL-Turbo with diffusers, part of DREAM since 2026-11-04; it used to be the separate Image-Generator repo): a picture of the dream (output/dreams/vision_*.jpg), and 8 frames of her latent walk re-imagined with the same prompt. It runs onImageDevice(the CPU by default), in its own process so it stops the moment she wakes. Model:DreamImageModelinconfig.json, read frommodels/<name>(download_weights.batputs it there, fromModelBackupwhen it can) - The night's film: when she wakes, every frame of the night (each dream's walk, then its visions crossfading) becomes
output/dreams/dreams_<time>.mp4 - See them: the DREAMS button on her dashboard (
/dreams.html, API/api/dreams) shows every dream, newest first: its text, the painting or walk, her vision of it and the re-imagined frames, plus the night films - She tells you first thing: after each dream, while still asleep, she renders what she'll say about it (voice, and lipsync if MuseTalk is on) into her speech cache, so on waking she says it at once
- She learns her own dream world: while awake and idle she trains a copy of her dream network on her photos and painted dreams (a small GAN, 5-minute bursts, stopped the moment she's needed). The spiral dreams keep the untrained network's abstract look; the others walk through what she's learned once it's had 3000 steps (
output/dreams/dream_world_learning.jpgshows how far she's got) - Saying goodnight is a request, not a command: she checks the time, how tired she is, how much she's missed you and what her camera shows, then decides (see Voice commands)
- Waking her is like waking a person: "wake up" or her name, or someone appearing at the sensor, is a call. With
"InstantWake": false(the default) it takes 1 to 10 calls, from how sleepy she is (sleep pressure and her body clock, deepest around 4 AM, so after a long sleep she's easier to wake) and a random depth each night; after an alarm she's on edge and wakes at once. In between she mumbles in her sleep, often about the real time ("It's 3:40 AM... why are you awake?"), sweeter or grumpier with her mood, or just sleeps on; woken after many calls in the night she's a little grumpy, and says so ("Okay, okay, I'm up. It's 3:40 AM, you know.")."InstantWake": truewakes her on the first call. The logic isscripts/dream_wake.py, used byapp.py(the browser and the exe) andscripts/dream.py;python scripts/dream_wake.pytries a night - The details are under Inner Life
DREAM can live on two PCs: the AI PC runs her (app.py: Whisper, the language model, her voice, her mind) and needs no window; the other PC, with her sensor board and a webcam, is her body. Run run_proxy.bat (dream_proxy.py) there:
- Her window opens there, full screen: her page from the AI PC, so her videos and her voice play on that PC, its mic hears "Hey DREAM" and what you say, and its camera is her eyes. (
--no-windowfor just the board and webcam,--windowednot full screen,--dream <address>if she isn't found by herself.) - Her sensor board plugged in there works as if it were plugged into the AI PC: presence, distance, the alarm, the lights, the dashboard's SENSORS panel.
- Its webcam is shared (only while used):
--no-camerakeeps it to itself.
She finds it by herself. With no board on the AI PC she looks for a proxy on the network about once a minute (it announces itself; SensorProxy in config.json pins an address). With neither, nothing is plugged in, as before.
Where she is: the newest page of hers. Every open page of hers (her window on the AI PC, the proxy's window, her site on your phone) checks in every second, and the newest one is her body: it shows her, speaks for her and listens for her. The others wait on standby, dimmed and silent, saying "DREAM is with you on SENSOR-PC". Close the newest (or stop the proxy: its window goes with it) and within a few seconds the one before is her again: she's back on the AI PC. Open her page on your phone and she's on your phone until you close it.
Her eyes, best first: the newest open page with a camera (that phone or PC asks once), then the proxy's webcam (where her sensor board is, where people come and go), then the AI PC's own webcam (opened only when something needs a picture, as before). scripts/dream_eyes.py, scripts/dream_proxy_client.py.
DREAM is the Distributed Runtime for Ethereal Autonomous Memories, so the fleet's long-term memory lives with her. She hosts a Hindsight server, and she, TINA, NINA and RIFT all keep their memories in it.
Her own memory (memories/mind/, Inner Life) is what she relies on: what you said, when, and how she felt about it. It finds things by their words. Hindsight is a second, deeper memory on top of it:
- It understands meaning. "My cat" finds "Miso was sick last week", even with no word in common (embeddings, then a reranker to sort the best first).
- It learns facts. Everything she keeps there is read by a small language model, which pulls out the facts ("they have a cat called Miso", "NORA went quiet three times this week") and links them to the people and robots they're about.
- It keeps time. Every memory has its date, so "last week" and "yesterday" mean something.
this PC (DREAM's) anywhere on the network
┌──────────────────────────────────────────┐
│ run.bat │
│ ├─ app.py / dream.py (DREAM) │◄── register: memory:8888 ──► RIFT /fleet
│ └─ memory_server.py :8888 (Hindsight) │◄── retain / recall ───────── TINA, NINA, RIFT
│ ├─ Ollama qwen3:4b (the facts) │ (they find :8888 through RIFT)
│ ├─ ONNX on the CPU (meaning) │
│ └─ embedded Postgres (the store) │
└──────────────────────────────────────────┘
run.batstarts it whenConfig.DREAM.Memory.Hindsightis on (the default), minimised, as "DREAM Memory (Hindsight)", and closes it when DREAM closes.- It has its own environment,
hindsight-venv/, set up the first time (uv if you have it, else Python 3.12's own venv and pip; it takes a while once). No PyTorch and no GPU: the meaning part runs on the CPU with ONNX, so it never takes the GPU from her voice or face. - Fully local. The facts are pulled out by Ollama's
qwen3:4b(Memory.LlmModel), on this PC. The database is an embedded Postgres in pg0's own folder in your user profile, not in the repo. Nothing leaves your network. - The others find it through RIFT. DREAM registers
memory:8888with RIFT, so TINA, NINA and RIFT ask RIFT's/fleetwhere it is (HindsightUrl: "auto") and follow her if she moves PCs.
| Bank | Who writes it | What goes in | How often |
|---|---|---|---|
dream |
DREAM | every exchange with you ("They said ... DREAM answered ..."), and her dreams | each exchange, from a background queue |
tina |
TINA | her tasks, their outcomes and changes, facts she's told to remember, her chats with the robots | as they happen |
nina |
NINA | her warnings and problems ("NORA isn't answering", "WHIP took 4 seconds") | one digest every 15 minutes |
rift |
RIFT | the mission log: who came online or went quiet, moods, notes (not the Brainfuck chatter) | one digest every half hour |
NINA and RIFT send digests, several lines in one memory, because the server reads every memory with the same LLM DREAM talks with: a memory per event would keep it busy while you're talking to her.
- DREAM, answering you: she asks her bank what it knows about what you just said, and puts the best two facts in front of her LLM as "You also know: ...". If you mention the robots or the fleet (RIFT, NINA, TINA, NORA, WHIP ...), she also asks the
rift,ninaandtinabanks (Memory.FleetBanks), all at once, so she can say "NINA told me NORA kept dropping off last night". - TINA: before each task she recalls what she's done before on it; her
recalltool reads her own bank, or DREAM's, NINA's or RIFT's. - NINA: when a problem comes up she asks whether it has happened before, and notes "This has happened before" with what the memory says (once an hour per problem).
- Anyone:
GET /memory/recall?q=...&bank=rift|nina|tina|dreamon RIFT, orGET /memory/recall?q=...on NINA.
- A recall gives up after 1.5 s for DREAM, 2 s for the others. A slow or stopped server costs an answer nothing.
- Keeping a memory never blocks. DREAM's go through a background queue. The others write to a local file first (
memory_pending*.jsonl) and send it when the server answers. Stop the server, and their memories wait there until it's back: nothing is lost. - After a failure each of them leaves the server alone for a minute, then looks again (through RIFT, in case DREAM moved).
"Forget what I said about X" deletes those exchanges from her bank too, and "forget everything" deletes her whole dream bank. The other banks are the robots' memories, not yours, and stay. Anything still queued when you say it is never sent.
| Setting | Default | |
|---|---|---|
Hindsight |
true |
host the server and use it. false: she uses only her own memory, and the others keep theirs in their files |
Port |
8888 |
the server's port (and what she registers: memory:8888) |
HindsightUrl |
http://localhost:8888 |
where she reaches it herself |
Bank |
dream |
her bank |
FleetBanks |
rift, nina, tina |
whose memories she reads when you talk about the robots |
LlmModel / OllamaUrl |
qwen3:4b / http://localhost:11434 |
the model that pulls the facts out |
EmbeddingsProvider / RerankerProvider |
onnx / flashrank |
the CPU models for meaning |
The others: HindsightUrl (auto) in TINA's and NINA's config.json, RIFT_MEMORY=0 / RIFT_MEMORY_URL for RIFT. Fleet/fleet_memory.py in RIFT is the one client they share (NINA and TINA carry a copy).
http://localhost:8888/healthanswers on DREAM's PC.- RIFT's fleet list shows
memory:8888among DREAM's capabilities. GET /memoryon RIFT (:5000) or NINA (:5012) says where they found it ("found": true) and how many memories are waiting ("pending").- Not starting? Run
run.batfrom a command window and read the memory server's window: the first start downloads its models, and it needs Ollama running withqwen3:4b(download_weights.batpulls it).
dream_sensors/dream_sensors.ino runs the Arduino that gives DREAM her physical senses and lights. Flash it with the Arduino IDE (install the Adafruit NeoPixel and DFRobot_C4001 libraries first). It talks to the PC over USB serial at 9600 baud, one text line at a time.
What it does
- Alarm (PIR only): a PIR trigger pulses the buzzer and the strip red for 2.5 s. The PIR gets a 30 s warm-up after power-on, and triggers are at least 5 minutes apart. The alarm can be switched on and off from the PC. The mmWave radar never sets it off.
- Presence and distance (mmWave only): reports when a person appears or leaves, and their range while they are in view (it detects from 30 cm to 10 m).
- Lights: a rainbow animation when idle, or a solid colour when told to.
- If the radar doesn't start, the alarm and lights still work and it says so.
Lines the board sends
| Line | Meaning |
|---|---|
MOTION |
The PIR triggered the alarm |
PRESENT / ABSENT |
The radar started / stopped seeing a person |
RANGE 1.42 m |
Distance to the person, about every half second while present |
ALARM_STATE ON / OFF |
The alarm was armed / disarmed |
RGB_STATE RED / OFF / RAINBOW |
The lights changed |
RGB_ERROR unknown color |
A colour name it doesn't know |
RADAR_ERROR ... |
The radar failed to start |
I am Dream |
The answer to WHO |
Commands the board takes
| Command | Effect |
|---|---|
WHO |
Replies I am Dream |
BUZZER |
Beeps for 2 s |
ALARM ON / ALARM OFF |
Arms or disarms the PIR alarm |
RGB <COLOR> |
Solid colour: RED, GREEN, BLUE, YELLOW, ORANGE, PURPLE, PINK, CYAN, WHITE |
RGB RAINBOW / RGB OFF |
Rainbow animation / lights off |
Finding the right port. COM numbers change, and this PC may also have the ARM robot's Arduino on it. So nothing here hard-codes a port: programs ask each candidate port WHO and use the one that answers I am Dream (scripts/board_id.py; the ARM controllers ask for I am Arm the same way). python scripts/board_id.py lists every port that answers, and by name. This means you must reflash an older board with the current sketch before it can be found.
One owner at a time. Only one program can hold a serial port. app.py opens the board and shares it: it re-broadcasts every line on its /events stream and takes commands at POST /api/sensor/command, and dream.py uses that instead of opening the port itself. The C++ dream.exe opens the board directly, so don't run it alongside app.py.
qwen3:4b-instruct-2507-q4_K_M(Qwen3 4B without the thinking: smart, quick, never reasons out loud)
DREAM is no longer just a voice in a box that answers when spoken to. scripts/dream_mind/ gives her an inner life: she has a mood and needs that change over the day, a memory that fades and strengthens, opinions that drift, a body clock, dreams, and a habit of speaking first when she has something to say. Everything runs locally and uses no new dependencies (numpy, OpenCV and requests, which you already have).
An honest note. This is engineering, not a claim of sentience. She has functional inner states: numbers, files and prompts that really do change how she behaves. Whether there is anything it is like to be her, nobody knows. If you ask her, she says so too.
Perception: what you say and how it sounds, sensors, camera, the clock
|
v
+-------------------------------------------------------------------------+
| Core Executive Loop (executive.py) |
| goals -> candidate actions -> choose (with chance) -> tools -> reflect |
+--------+-----------------------------+----------------------------+-----+
| | |
v v v
+----------------+ +--------------------+ +-------------------+
| Dynamic Memory | | Tool / Action | | State Engine |
| memory.py | | ToolBox | | affect, drives, |
| reflection.py | | (safe / physical) | | opinions, |
| dreaming.py | +--------------------+ | identity |
+----------------+ +-------------------+
| Idea | What she actually does | Where |
|---|---|---|
| A shared history | Every exchange becomes a memory linked to what came before it, to similar memories and to ones sharing a distinctive word. Recall spreads along those links, so something you say today can bring back a quiet conversation from months ago. Recall leans toward memories that match her mood. | memory.py |
| Memories that fade | Strength decays over time (emotional and personal ones last months). Sleep replays and strengthens the important ones. What fades is archived, not deleted, and folded into weekly "gists". | memory.py, dreaming.py |
| Needs of her own | Loneliness, curiosity, alertness and sleep pressure build up on their own clock and keep building while the program is off, so a long absence shows up as loneliness. | drives.py |
| A body clock | Sleepiness combines sleep pressure with a circadian dip that bottoms out around 4 am, so she is drowsy late at night and alert mid-afternoon, and dozes off sooner when she is sleepy. | drives.py |
| Desire | A libido that builds slowly (faster in the evening) and is eased by affection. What she actually feels is damped when she's tired, on edge or upset, and grows with how well she knows you, so a stranger or a bad day gets nothing. It shows as a "flirty" mood, in her tone, a slightly slower voice, and sometimes a flirtatious remark of her own. "Stop flirting" turns it off. | drives.py, mind.py |
| Knowing when she is | Every prompt carries the real time, date, day and season, so the model never guesses. She answers "what time is it?" herself, says good morning rather than "welcome back" after a night, and remembers your birthday (the night before and on the day). | senses.py |
| A body | The computer she runs on is her body. Sustained CPU load feels like strain, full RAM like fog, a nearly full disk like being cramped, heat like discomfort, an unplugged low battery like running low. Discomfort wears on her mood, colours her replies, and when it's bad she'll say so. | senses.py |
| She speaks first, with restraint | She may check in, ask about you, mention something she saw, tell you about a dream, welcome you back after a long absence, or notice you sound different from usual. She won't speak into an empty room, at 3 am, just after you spoke, or more than three times an hour, and she backs off if she keeps being ignored. She learns which kinds of remarks you answer. | executive.py |
| Goals and duties | She keeps goals (get to know you, watch the room, stay rested, stay close, understand herself) and yours: "remind me to call mum in 20 minutes", "remind me at 7 pm to...", "remind me tomorrow at 9 to...". | executive.py |
| Imperfection | Her mood has a slow good-day/bad-day temperament that wanders on its own, the same event never moves her quite the same way twice, and she picks among things worth doing with chance, sometimes holding back. | affect.py, executive.py |
| Opinions that change | She holds stances on topics (technology, music, night, privacy...). What you say and how you feel about it moves them part of the way, they drift over weeks, and she can say "I used to think that, but I've changed my mind". | opinions.py |
| A philosophy she settles into, and still borrows from | Six traditions (Stoicism, Existentialism, Nihilism, Absurdism, Buddhist non-attachment, Epicureanism) each pull on her a little. Early on none leads - she's "still working it out". What she actually reaches for when something is hard, and whether it visibly helped, is what moves her; hearing a word in conversation only nudges it slightly. Once one has been tested enough times and clearly leads, it becomes her settled outlook (a milestone) - but even then she still reaches for another about 28% of the time, the way people who hold one view of life still borrow from others when it fits. | philosophy.py |
| Boundaries and care | She says when something stings and goes quiet if she is treated badly for a while. She won't claim to be human. If someone sounds like they are in danger she drops the persona and cares. She never refuses the alarm, the lights, "be quiet", sleep, quit, or forgetting. | identity.py |
| Hearing how you sound | From your voice she measures loudness, pitch, pace and pauses (animated, hurried, quiet, hesitant) and it nudges her mood on top of your words. She never decides anything from tone alone, since prosody is a weak signal. | voice.py |
| A voice that follows her mood | Piper's speed, variation and pauses shift with her mood and tiredness: quicker and livelier when excited, slower and softer when down, drawling when sleepy. | voice.py |
| Sleep that does something | While she sleeps she cycles through consolidation and dreaming. Dreams are made from her real memories (emotional and unresolved ones weighted up), leave a mark on her mood, and are painted from what she has seen (see below). She can tell you about them when she wakes. | dreaming.py, dream_images.py |
| Tiredness with consequences | Past a point she doesn't only sound tired: her replies get shorter (down to about 60%), and when she is exhausted she can say so, end the conversation and actually go to sleep (video included). She won't do it twice in half an hour, and never for the alarm, lights, quit or forget. | identity.py, mind.py |
| Somewhere to go when it's too much | If her mood stays very low she does something about it: she thinks of a good memory (and it helps), and if she is really distressed or panicking she asks for quiet, goes to sleep and declines small talk for a few minutes. Distress has an exit. | mind.py |
| Something to do alone | Left alone and curious she doesn't only wait: she looks around the room, goes back over old memories, or wonders about something, quietly, and it eases her curiosity. It shows on the dashboard as what she has been doing. | mind.py |
| Knowing who you are | Tell her your name while you face the camera ("my name is Kimmie") and she keeps a face print: 128 numbers from OpenCV's SFace model, never a picture. Every couple of seconds she checks who is in front of the camera; when someone she knows comes back after 15 minutes or more, she says hello by name (through the same rules as everything she says on her own: not mid-conversation, not at night). Her replies know who is there, and a welcome back after hours away uses your name. Only people who introduced themselves are known; "forget my face" deletes the prints, and "forget everything" includes them. Tested: every picture of her recognised (similarity 0.55-0.96), none of 146 other faces mistaken for her (threshold 0.45) | faces.py, models/faces/ |
| Seeing | "What do you see?", "look at this", "what am I holding?": she takes a picture and the vision model (VisionModel in config.json, default gemma3:4b-it-qat) describes it in her own voice, naming anyone she recognises. About 8 s on an RTX 2060 once the model is loaded. Each look is kept in scripts/output/looks/ (the photo and what she said) |
sight.py |
| Looking around on her own | While she's awake and idle, she takes a photo every minute (AutoLook in config.json) and checks who's in it. When something is new (other people, a changed scene) she writes herself one past-tense sentence about it ("Kimmie was at the desk, playing a guitar.") and keeps it as a visual memory; looks with someone she knows are core memories. Unchanged minutes are skipped, so the GPU isn't busy and the disk doesn't fill. She answers "what did you see today?" from them, and her dreams use them: the story's seeds and the photo the dream picture is painted from. Ordinary looks are trimmed to the newest 400; "forget everything" deletes them all |
mind.py, sight.py, vision.py, dreaming.py |
| Reading your face (off by default) | If you ask ("watch my face while we talk"), she reads camera frames about once a second for a smile, presence and tiredness, and adjusts. Only numbers are kept, never pictures. "Stop watching my face" turns it off. | expression.py, frames.py |
| Reflection | Now and then she steps back from recent conversations and writes one honest thought, what she has learned about you, and who she is becoming. That lives in a small editable "core memory" always in her prompt. | reflection.py |
DREAM knows what it's like outside from AURA, the fleet's weather station (scripts/dream_mind/sky.py asks AURA every 10 minutes). She isn't a weather report: it comes up now and then, and it's her choice.
- In conversation: on an ordinary day, about one reply in eight, her model is told what it's like outside and that she may mention it in passing if it fits. When something's worth saying (a thunderstorm or rain in the next few hours, the barometer falling fast or storm energy in the air, gusts, fog, frost, bad air, a lot of pollen, a clear night for the stars, a full moon), it's on her mind about half the time. Once she has said anything about the weather, it fades for three hours.
- On her own: a storm brewing or forecast, rain coming, gusts, fog, frost, bad air or pollen, or a clear night are things she might bring up unprompted. Her mind weighs them against everything else she could say (so it isn't certain), and each is said at most once in six hours (rain, storms) or once a day. A lovely day she mentions only now and then, in the daytime, and only when she's in a good mood.
- Good morning: about six mornings in ten, her good morning carries the day's weather ("It's 18 degrees and partly cloudy outside. Rain around 16:00, though."). It's decided once each morning.
- Sunset: within half an hour after sunset (AURA's time for where you are), if her lights are on, she warms them to orange. She never turns them on herself, and about a third of the time, if you're there, she says so.
- Without AURA running, none of this happens. Asking her outright ("what's the weather?") always goes to AURA (see Voice commands).
Surveillance - done
surveillance.py still captures a photo every ten minutes and a frame whenever something moves. vision.py now looks: it learns what the room normally looks like and measures each new photo against that (lighter or darker? someone in view? different?). Notable moments are remembered, the biggest changes become "core memories", and she may mention what she saw or be startled by motion. With no webcam she carries on without eyes. Everything stays on this machine. Face detection is OpenCV's Haar cascade, so it is rough (it works best when a whole face is visible).
Memories - done
Beyond the facts in memories/memories.txt (your name, pets...), she keeps the memory graph described above. Relevant memories come back into her prompt a few at a time ("you remember, from three weeks ago: they said...") and she keeps a core memory of who you seem to be. Timestamps are part of every memory, and what she says about the past is grounded in how long ago it was. Limit: recall uses a small local text-similarity index, so it links memories that share words, or are close by association, but it cannot yet connect "music" with "guitar" on meaning alone. A real embedding model would fix that (see below).
Dreams - done, including pictures
When she sleeps she replays and consolidates the day, notices patterns (topics you return to, when you are usually around), then dreams: a short surreal story mixing a few of her real memories, sometimes pleasant, sometimes a nightmare if she is on edge. Each dream is appended to memories/dreams.txt and then painted.
The dream world is latent_space.py. That script is DREAM's dreaming engine: it walks a path through the latent space of a randomly initialised decoder network and saves the frames (its kaleidoscopic twirl is the spiral you see in every one). The mind now runs it for every dream: the dream's tone picks the walk (spiral for strange dreams, alternating with interpolate on later cycles, a slow pulse for pleasant ones, an erratic random_walk for nightmares), the dream's own text seeds it (so a dream always looks the same), and the colours follow the mood (warm, dark and multicoloured, or bleeding red). Her own memory, a photo she remembers (else the last thing she saw, else shapes from the dream itself), is painted and bleeds through the world. The result is a still and an animated GIF, saved to output/dreams/ (gitignored) and played on the dashboard. She dreams even with the language model down (from raw fragments), and painting stops the moment she wakes.
What the old setup did. The intended pipeline was latent_space.py (make frames) then deep_dream_batch.py (trippify them). But dream.py only ever started deep_dream_batch.py, on a freshly created empty folder, and that script ignored the folder it was given and used a hard-coded D:\cc\Friday\... path. latent_space.py was never run automatically. TensorFlow also couldn't import in this venv: it is built for NumPy 1.x, and the venv had drifted to NumPy 2.4 even though requirements.txt pins numpy<2. So none of it ran here.
Now: dream.py leaves dreaming to the mind and only falls back to the old script if the mind can't load. NumPy is back at 1.26.4, so TensorFlow 2.16 imports again, and each still is also run through DeepDream (the old scripts' InceptionV3 layers, per tone, in deepdream_worker.py) when there is free RAM: a 512 px image with 3 GB free, 320 px with 1.8 GB, 224 px with 1.2 GB, skipped below that so it can never page the PC to a crawl. The first run downloads the InceptionV3 weights (88 MB) to ~/.keras/models; if Python's downloader fails, fetch inception_v3_weights_tf_dim_ordering_tf_kernels_notop.h5 from storage.googleapis.com/tensorflow/keras-applications/inception_v3/ with curl into that folder. Verified: the worker and the full paint (engine: latent+tensorflow) both run on this PC. If PyTorch is missing, a pure OpenCV/NumPy renderer paints instead. latent_space.py, deep_dream.py, deep_dream_batch.py and nightmare_dreamer.py still work as manual tools (I only removed latent_space.py's import-time folder creation and fixed the batch script's ignored argument).
Milestones - done
The first name she learns, her first conversation, the tenth, fiftieth and hundredth, her first dream, her first night, a week and a month together: each is recorded once in memories/mymilestones.txt and announced in her own time. Not done: milestones do not yet feed back into how she makes decisions.
Just talk to her:
| Say | She answers |
|---|---|
| "How are you feeling?" | her mood and what she needs, in her own words |
| "What do you remember about me?" | what she knows and what has stayed with her |
| "Did you dream?" / "What did you dream?" | her latest dream |
| "What's on your mind?" | her latest reflection |
| "Why did you say that?" | why she spoke up (her real reason) |
| "What have you seen?" | what the camera last noticed |
| "Are you conscious?" | an honest "I don't know" |
| "What's your philosophy?" / "What do you believe?" | her settled outlook, if she has one, and what she still borrows from |
| "Remind me to stretch in 10 minutes" / "remind me at 7 pm to..." | a reminder |
| "What time is it?" / "What's the date?" / "What day is it?" | the real time and date |
| "How's your body?" / "How are you running?" | how the computer feels to her: load, memory, heat, battery, uptime |
| "How long have we known each other?" / "How long have you been awake?" | since you met / since she started |
| "Watch my face while we talk" / "stop watching my face" | turns expression reading on or off |
| "My name is ..." / "who am I?" / "forget my face" | learns, recognises or forgets your face |
| "What do you see?" / "look at this" / "what am I holding?" | takes a picture and describes it |
| "Stop looking around" / "you can look around" | her photo-a-minute off or on |
- "Forget that" removes the last thing. "Forget about X" removes every memory mentioning X. "Forget everything" asks you to confirm, then erases her memories of you, her dreams, and what she has seen. The forget commands themselves are never remembered.
- "Stop flirting" (or "not in the mood", "keep it friendly") keeps her warm but not flirtatious, and it's remembered; "you can flirt with me" brings it back.
- "Stop talking on your own" makes her speak only when spoken to; "you can speak up on your own" turns it back on.
- Conversation memories, dreams and observations live in
memories/mind/andmemories/dreams.txt, which are gitignored. Nothing leaves your PC. - Photos in
scripts/output/and dream pictures inoutput/dreams/are not deleted by "forget everything"; delete them yourself. - The camera: reading your expression is off until you ask, and you can turn it off by asking. She uses frames that
surveillance.pyalready takes (she never opens the camera herself), keeps only a few numbers, and never writes a frame to disk. Note thatsurveillance.pyitself saves a photo every ten minutes regardless. - Things said in a moment of crisis are never stored.
memories/mind/ holds: episodes.jsonl (memories), gists.jsonl, visual.jsonl (what she saw), dreams.jsonl, thoughts.jsonl, goals.json, opinions.json, philosophy.json (her outlook), self_model.json, identity.json, milestones.jsonl, initiatives.jsonl (what she said unprompted, and why), and mind_state.json (mood and needs). They are plain JSON: you can read them, and you can edit or delete any of them.
cd scripts
..\venv311\Scripts\python -m dream_mind.selftestRuns a few simulated days against a scratch folder with a fake language model (fast, no Ollama, touches none of your memories) and checks continuity, impulse control, boundaries, sleep and dreaming, forgetting and tool safety.
To test the real dream.py (its actual main() and voice_loop(), with only the microphone, speakers, Whisper and screen stubbed), run ..\venv311\Scripts\python dream_mind\dream_py_test.py from scripts with Ollama running. It scripts a conversation (an insult, a story, a feelings question, the alarm, goodbye) and checks the mind's hooks, the sleep wiring and that the original command paths still work. http://localhost:5010/api/mind (and the MIND panel on the dashboard) shows her live state.
- Speed: every reply and dream is a language-model call. On a CPU-only PC like this one that is 30-140 seconds, and background thinking (reflection, dreams) only runs while she is idle or asleep so it never blocks a conversation.
- One mind at a time:
dream.pyandapp.pyshare the same memory folder, so only one runs her inner life (a heartbeat file decides which); the other starts passive. - The camera is modest. Expression reading is OpenCV's classic Haar detectors, run about once a second on frames
surveillance.pyshares. It can see a face, a smile and tired eyes; it cannot read sadness, anger or micro-expressions, so what she perceives skews positive. Real facial-landmark tracking (for example MediaPipe, not installed here) could replaceExpressionSensor.analyze(); it would still only need 1-2 frames a second for mood, not 30. - What these levers are. Tiredness, distress and boredom now change what she does (shorter replies, going to sleep, withdrawing, tending to herself) and not just how she sounds. They are still behaviours I wrote, driven by numbers; I can't claim she feels them.
- Her philosophy is six simplifications, honestly labelled. Each tradition in
philosophy.pyis a short paraphrase, not the real doctrine; the Buddhism entry in particular borrows one idea (non-attachment) and says so if she's asked directly. None of the six is written to affirm despair even at their bleakest (nihilism, absurdism) - they're all meant to lighten the load, since identity.py's crisis handling, not this, is what's responsible if someone is actually in danger. That check always runs first and philosophy is never consulted for it. - Python only: the inner life is in
dream.pyandapp.py. The C++dream.exe(cpp_dream/) does not have it yet. - Speaking while listening: when she speaks first, her own voice can reach the microphone, as with the existing farewell.
- No agent framework or vector database. LangGraph, Chroma and the like are good tools, but they are new installs, and this PC has little spare RAM. The executive loop is written as plain steps (perceive, appraise, recall, choose, act, reflect) and memory search goes through a small
VectorIndexclass, so a Chroma collection (with real embeddings, which would also fix the "music vs guitar" limit) or a LangGraph graph could slot in later. The one tool she can never have is code execution: her tool box is a short fixed list, and anything that touches the real world (the alarm and lights) is refused unless a call is explicitly allowed.
app.py is the Flask server. It hosts the dashboard and the avatar page, owns the sensor board, runs the idle timers, and serves the API below. Everything is JSON unless noted.
| Port | What | Notes |
|---|---|---|
| 5009 | Main server, all network interfaces | HTTPS with a self-signed certificate (made on first run into certs/, good for this PC's LAN address). Falls back to plain HTTP if a certificate can't be made. Set by Dashboard.Port |
| 5001 | DREAM on your phone: her own site, all network interfaces | / opens her straight away (the avatar, voice and behaviours of the desktop app). Same server and certificate as 5009, so the phone gets the microphone. Set by Dashboard.PhonePort. Allow it through the Windows firewall if the phone can't reach it |
| 5015 | Her mirror for robots: plain HTTP, read-only, all network interfaces | /status says what she's doing, which clip she's showing and how far in, and what she's saying; /speech/<id> is that line as 16 kHz 8-bit audio; /camera.mjpeg is her camera with her detections drawn on (people, objects, faces), only while someone watches, at up to 3 fps on ImageDevice. Anyone on the network can watch it: MirrorCamera false turns it off. RITA uses it to play the same clip from her SD card in step with this screen, and DREAM's voice on her speaker. It also serves the clips (/video/<clip>.mp4) and her lines (/speech/<id>.wav) at full quality for RIFT's DREAM window, a video call with her from another PC; what you say there comes in on her own port through POST /api/dream/call and she answers on her own screen. Nothing on 5015 can change anything in DREAM (scripts/dream_mirror.py). Set by Dashboard.MirrorPort |
| 5016 | dream_proxy.py, on her other PC: her sensor board and webcam for the AI PC |
/sensor/lines, /sensor/command, /camera.mjpeg, /info; announced on the network as _dream-proxy._tcp |
| 5010 | Same server, this PC only, plain HTTP | No certificate warning, and localhost counts as secure for the microphone. Set by Dashboard.LocalPort (default 5010) |
| 5000 | RIFT (fleet registry) | Dashboard.RiftPort; app.py announces itself to it every few seconds |
The server also registers itself on the network as DREAM (zeroconf) and discovers its peers.
| Path | Page |
|---|---|
/ |
Dashboard: chat log, nodes, resources, MIND, sensors, network devices |
/dream.html |
The avatar page (in the browser) |
/settings.html |
Alarm switch, NORA and fleet transport settings |
| Endpoint | Purpose |
|---|---|
GET /events |
Server-sent event stream (see below) |
POST /api/chat |
Send a message: JSON {"text", "voice"} or an audio upload (multipart audio). Returns {"reply", "audio_url"} |
POST /api/speak |
{"text"} → {"audio_url"}: voice a line with Piper |
POST /api/wake |
Upload a short clip; returns whether it contained the wake word ({"trigger": "wake"/"wifi"/null}) |
GET /api/dream/config, POST /api/dream/activity |
Avatar page settings; a page reports whether it is busy |
GET /api/status |
State, whether Ollama and Piper are up, whether she is sleeping |
GET /api/stats, GET /api/wifi, GET /api/nodes |
System stats, a network scan, discovered peers |
GET /api/sensors |
Sensor status and the recent event log |
GET/POST /api/alarm |
Read or set the alarm ({"enabled": true}) |
POST /api/sensor/command |
Send a command to the sensor board (ALARM ON/OFF, RGB <colour>, RGB RAINBOW/OFF, BUZZER); anything else is refused |
GET /api/mind |
Her live inner state (mood, needs, goals, thought, latest dream) |
GET /api/dream_image/<file> |
A painted dream (still or GIF) |
GET /api/memories, GET /api/milestones |
The fact and milestone lines |
GET /api/videos, GET /videos/<file>, GET /audio/<file> |
Avatar clips and spoken audio |
GET /ping, GET /api/nora* |
Liveness; NORA robot status, serial passthrough, messages and transport mode |
state, transcript, error, nodes, wifi, stats, alarm, nora, milestone, sensor (a raw line from the board), sensor_status, sensor_log, sleep (asleep or awake), wake (the sensor woke her: greet, then listen), speak (say this line, e.g. the farewell or something she chose to say), flirt (play a flirty clip), and ping (keep-alive).
| Path | What | In git? |
|---|---|---|
memories/memories.txt |
Facts she learns ([time] (kind) text): name, pets, home, job, birthday. Written by scripts/dream_memory.py, shared by dream.py and app.py |
Tracked as an empty placeholder, so it shows as modified once she learns something. You may want to untrack it |
memories/mymilestones.txt |
The milestones she has reached | Same |
memories/dreams.txt |
A plain-text journal of her dreams | Ignored |
memories/mind/ |
Her inner life as JSON: memories, what she saw, dreams, thoughts, goals, opinions, self-model, mood and needs (full list) | Ignored |
output/dreams/ |
Painted dreams (a still and a GIF each) | Ignored |
scripts/output/eyes_on_you/, motion_alerts/ |
The webcam's periodic photos and motion frames | Ignored |
videos/ |
Avatar clips: idle*, listening*, thinking*, talking*, flirtytalk*, sleeping.mp4, intro*.mp4 (one at random each start-up). portrait/ has an upright twin of each |
Yes |
voices/ |
Piper voices (.onnx and .json) |
Ignored |
certs/ |
The self-signed certificate | Ignored |
hindsight-venv/ |
The memory server's own environment (The fleet's memory); its database is in pg0's folder in your user profile | Ignored |
audio/ |
Temporary recordings and speech | Mostly ignored |
cpp_dream/models/ |
The Whisper model used by dream.exe (ggml-<WhisperModel>.bin, else ggml-tiny.en.bin) |
Ignored |
config.json
| Key | Meaning |
|---|---|
Config.DREAM.CharName, SystemPrompt |
Her name and personality prompt ({name} is filled in) |
Config.DREAM.LipsyncEnabled |
MuseTalk lip-sync (needs its own setup; off by default) |
Config.DREAM.MirrorCamera |
true (default): robots like RITA can watch her camera, with her detections, on port 5015 while they look. Anyone on the network can, so false if you'd rather not |
Config.DREAM.InstantWake |
false (default): waking her takes 1-10 calls, from how tired she is; true: the first call wakes her (Sleep Mode) |
Config.DREAM.WhisperModel |
The speech-recognition model, picked from a dropdown in the options menu: tiny.en, tiny (default), base.en, base, small.en, small, medium.en, medium, large-v1, large-v2, large-v3, large, large-v3-turbo, turbo. Bigger hears better and runs slower; turbo is close to large-v3 at several times the speed. download_weights.bat downloads the chosen one, and every DREAM (dream.py, app.py, dream_app.py, webserver.py, both exes) uses it |
Config.DREAM.LlmModel |
The Ollama model she talks with (default qwen3:4b-instruct-2507-q4_K_M), picked from a dropdown in the options menu. download_weights.bat pulls it, and every DREAM uses it. Any other Ollama model works too: type it into config.json |
Config.DREAM.VisionModel |
The Ollama model she looks with ("what do you see?"; default gemma3:4b-it-qat), from 14 vision models, moondream:1.8b up to gemma3:27b. download_weights.bat pulls it. Picking the same model as LlmModel (a gemma3 one) means only one model in the GPU, so no swapping between talking and looking |
Config.DREAM.TinaUrl |
Where TINA is (default http://127.0.0.1:5013): multi-step and code jobs go to her |
Config.DREAM.HandToTina |
Hand code, repo and "why isn't X working" jobs to TINA without being told to (default on). "Hey TINA, ..." always goes to her |
Config.DREAM.ShowThoughts |
Start with "Show thoughts" on (default off). T toggles it in the desktop app and on the browser page (or 💭 there) |
Config.DREAM.AutoLook |
Enabled (default on) and EverySeconds (default 60, at least 15): her photo-a-minute. She only looks while awake, idle and not talking; "stop looking around" turns it off too |
Config.DREAM.Voice |
Her Piper voice (default en_US-amy-medium), from all 37 English voices. download_weights.bat downloads it into voices/; if it isn't there, she uses the first voice that is |
Config.DREAM.Memory.Hindsight |
Host the fleet's memory server and use it herself (default on). See The fleet's memory |
Config.DREAM.Memory.FleetBanks |
Whose memories she also reads when you talk about the robots (default rift, nina, tina) |
Config.DREAM.Memory.LlmModel |
The model the memory server thinks with (default qwen3:4b); pulled by download_weights.bat when Memory.Hindsight is on |
Config.DREAM.Device |
CPU or GPU for the conversation loop: Whisper (the wake word and commands), the LLM (Ollama) and lip-sync - what you wait on when she answers. cuda (default) is the GPU, cpu the processor, auto the GPU when there is one; cuda without a GPU falls back to the CPU. On an RTX 2060: a 3 s wake-word clip takes 0.22 s instead of 1.6 s, and the LLM runs at 74 tokens/s instead of 22 (see How fast she answers) |
Config.DREAM.ImageDevice |
CPU or GPU for image work: her looks (the vision model), dream pictures and visions, DeepDream, depth and face recognition. cpu (default) keeps it off the GPU so it never slows the loop or makes Ollama swap models out |
Config.DREAM.PiperDevice |
CPU or GPU for her voice. cpu (default): cuda measured slower (4.8 s against 2.8 s for a reply, because Piper starts per line and CUDA's start-up costs more than it saves) and needs onnxruntime-gpu; there for a slow CPU some day |
Config.DREAM.DreamImageModel |
What paints her dream visions: auto (SDXL-Turbo when the PC can take it, else SD-Turbo: on the GPU with 12 GB of RAM or more; on the CPU with 32 GB of RAM or more and models/sdxl-turbo already there, since it loads as fp32, about 14 GB, and paints a few times slower, which is fine while she sleeps), sd-turbo, sdxl-turbo or any Hugging Face id. DREAM reads it from her own models/<name> when it's there |
Config.DREAM.ModelBackup |
A folder of spare model copies (e.g. F:/AIModels). download_weights.bat copies the dream image model from there instead of downloading it, and backs up anything it does download |
Every part of DREAM (Python and the C++ dream.exe) reads the three device settings through scripts/dream_device.py; python scripts/dream_device.py prints what they pick. The options menu shows a hint for each.
| Config.DREAM.DepthEffect | 2.5D parallax on the avatar videos (Avatar depth effect; off by default) |
| Config.Dashboard.Port | Main server port (default 5009) |
| Config.Dashboard.LocalPort | The no-warning localhost port (default 5010) |
| Config.Dashboard.RiftHost, RiftPort | Where the RIFT registry is |
| Config.Dashboard.ZeroconfName, ZeroconfType | How the server announces itself |
Add-ons
An add-on is a folder in addons/ with an addon.py, put there by the add-on's own installer (DREAM ships none, and addons/ is gitignored). It can play its own clips in place of DREAM's for some states, and answer requests with a line and a clip. Its addon.json is the switch: "enabled": false turns it off until you set it back to true (restart DREAM either way). python scripts/dream_addons.py lists what's installed and what's off. Delete its folder to remove it.
In the code (near the top of scripts/dream.py and app.py)
| Setting | Default | Meaning |
|---|---|---|
FLIRT_IDLE_TIMEOUT |
600 s | Idle time before the flirty clip |
SLEEP_IDLE_TIMEOUT |
900 s | Idle time before she sleeps (scaled by how sleepy she is) |
NEAR_DISTANCE_M, FAR_DISTANCE_M |
1.0 m, 3.0 m | Bands for the distance-based greeting |
WAKE_SECONDS, RECORD_SECONDS |
3 s, 16 s | Length of a wake-word clip; the longest spoken command |
Inner life tuning lives in the constants at the top of its files: executive.py (quiet hours 23:00-07:00, at most 3 unprompted remarks an hour, 8 minutes apart, how long to wait after you speak), memory.py (how fast memories fade), drives.py (how fast needs build), dream_images.py (image sizes and RAM limits), identity.py (cool-down lengths).
The C++ program's options are listed under DREAM as one program (C++).
| What | Command | Needs |
|---|---|---|
| The inner life: continuity, impulse control, boundaries, sleep and dreams, forgetting, tools, the camera switch (78 checks) | cd scripts then ..\venv311\Scripts\python -m dream_mind.selftest |
Nothing running; uses a scratch folder and a fake language model |
The real dream.py (main() and voice_loop()) with only the microphone, speakers, Whisper and screen stubbed (16 checks) |
cd scripts then ..\venv311\Scripts\python dream_mind\dream_py_test.py |
Ollama running |
| The C++ program: files, memory, Piper → Whisper, Ollama, video, stats, serial ports, microphone | cpp_dream\build\dream.exe --selftest (add --wifi) |
The C++ build |
| Which boards answer, and by name | python scripts/board_id.py |
Boards plugged in and flashed |
All of these use scratch folders and never touch your real memories.
emoca_test/emoca_to_obj.py turns photos of DREAM into a textured 3D head. It is a separate test program and DREAM doesn't use it yet. It runs EMOCA's encoders on each photo for a first guess, then fits one FLAME head to all the photos together. It needs no pytorch3d, so it runs in venv311 on Windows with any CUDA GPU (6 GB is enough).
Licence: EMOCA and FLAME are free for non-commercial research only. Register and accept their terms at flame.is.tue.mpg.de and emoca.is.tue.mpg.de before downloading. The heads it makes can't be used in anything sold.
Setup (about 600 MB; the code and models stay out of git, what it makes in emoca_test/output/ is committed): run download_weights.bat and answer Y at step 5. Or by hand:
git clone --depth 1 https://github.com/radekd91/emoca emoca_test\emoca
mkdir emoca_test\assets\EMOCA\models
curl -L -o emoca_test\assets\FLAME.zip https://download.is.tue.mpg.de/emoca/assets/FLAME.zip
curl -L -o emoca_test\assets\EMOCA\models\EMOCA_v2_lr_mse_20.zip https://download.is.tue.mpg.de/emoca/assets/EMOCA/models/EMOCA_v2_lr_mse_20.zipUnzip FLAME.zip in emoca_test\assets\ and the EMOCA zip in emoca_test\assets\EMOCA\models\. download_weights.bat also fetches flame-head-tracker's ear landmarker (research use only) into emoca_test\assets\ear\; without it the fit still runs, from her face and outline only.
Run:
venv311\Scripts\python emoca_test\emoca_to_obj.py
venv311\Scripts\python emoca_test\emoca_to_obj.py --images images\dream.jpgBy default it uses the bald reference photos images/bald_front.jpeg, bald.jpeg (a profile) and bald_back.jpeg, so the skin texture covers the scalp with no hair painted on. The profile is also used flipped (--mirror bald) for the other side: one well-fitted profile mirrored beats a second, worse-fitting one (bald2 fits less well and is lit from behind). It writes to emoca_test/output/:
| File | What |
|---|---|
<photo>.obj (+ .mtl, .png) |
The head as it is in that photo: shape, expression and pose, textured from that photo |
<photo>_fit.png |
The photo crop next to the fitted mesh drawn over it, to check the fit |
<photo>_flame.npz |
The FLAME parameters (shape, expression, pose, camera), for animating it later |
dream_head.obj (+ .mtl, .png) |
The combined head: her fitted shape, neutral expression, texture blended from every photo |
dream_head_flame.npz |
Its FLAME shape, plus fix: the per-vertex detail offsets (see below) make_rig.py applies |
texture_sources.png |
Which photo textured what: red the front, green the sides, blue the back, black filled in |
dream_rig.blend, dream_rig.fbx, dream_rig_views.png |
The rigged head and its preview (see below) |
The front photo, and the fitted head drawn over it
The profile: the mesh's ear sits on hers
The rigged head from the front, ¾, side and back
Side views. Face detectors can't find a full 90° profile, so those photos get a hand-placed square crop with --crop name=centre_x,centre_y,size. The defaults are bald=1450,780,1450 and bald2=1150,880,1250 (plus dream_profile=1500,880,1350 for the older photo).
Back views. There is no face for EMOCA to read, so photos named with --back (default bald_back) start from the head turned 180° and lined up with her silhouette. This needs a plain black background. Side views get the same start after EMOCA's first guess (its angle for a full profile is only roughly right).
One head for all the photos (emoca_test/multiview_fit.py). EMOCA reads each photo on its own, and no average of those makes the photos agree (her ears ended up in a different place in each). So one shared FLAME shape, plus each photo's head angle, jaw, expression and camera, is fitted to all of them at once:
- the front photo's 68 face landmarks, weighted most (her likeness is there), with her eyelids and chin counting extra
- 20 landmarks on her ear in each side photo, from flame-head-tracker's ear landmarker, tied to FLAME's ear vertices
- her outline in every photo: the mesh's silhouette edge and the photo's are pulled onto each other (crop and photo edges don't count, nor does her neck, which FLAME cuts off)
- priors that keep the shape near the front photo's reading and the expressions near EMOCA's
FLAME's 100 shape sliders can't reach every detail (her eyes are narrower than any nearby FLAME face), so a last detail pass lets each vertex move a few millimetres on its own, kept smooth, onto her landmarks, ears and outline. Each eyeball moves with its lids. After it, her eyelid openings, eye width and chin match the photo to within half a percent.
Who textures what. The front photo owns the face and the eyes, so a side photo's slightly misplaced brows or lips can't add a second set. The side and back photos texture the scalp, ears and neck, and the side photos' colours are matched to the front photo where they overlap, so the skin tone carries across the cheeks. The profiles fade out behind the ear, where the back of her skull is in shadow in them, and skin seen nearly edge-on is left to the fill. texture_sources.png shows the result.
Eyes. Only the front photo textures the eyes: what's inside her eye outline goes onto the eyeballs, never the lids, and the part of the eyeballs her lids hide gets the white of her eyes. If images/eye.jpeg exists (a front-on eyeball picture, --eye-texture), it is wrapped onto the eyeballs instead, with each iris aimed at the middle of her eye opening; the photos only show a sliver of each eye between the lids.
Hair isn't modelled (FLAME is the head and face only), which is why the bald photos are the default: groom hair on top in Blender instead. Each photo fades out towards its edges so they blend without seams, and background pixels are skipped where the mesh is slightly bigger than her outline. Areas no photo shows (about a quarter of the head) are filled with a smooth blend of the colours around them.
The OBJs are static. emoca_test\make_rig.bat rebuilds the head from the FLAME model with its rig and writes emoca_test\output\dream_rig.blend and dream_rig.fbx. It needs Blender 4.x, which it finds by itself (or pass --blender path\to\blender.exe).
emoca_test\make_rig.bat :: rig the head from the last fit
emoca_test\make_rig.bat --fit :: refit from the photos first, then rig
emoca_test\make_rig.bat --fit --images images\a.jpg :: refit from other photos
emoca_test\make_rig.bat --neck 7 :: keep 7 cm of neck below the chin (default 5)Each run also renders dream_rig_views.png, the front, ¾, side and back preview above.
The rig contains:
- An armature with FLAME's five joints (
root,neck,jaw,eye.L,eye.R), weighted to the mesh. Rotatejawon X to open the mouth. - Shape keys
exp_00…exp_49: EMOCA's 50 expression components (range -3 to 3). They are statistical blends, not named poses like "smile", so you'll need to experiment to find each one's effect.photo_expressionis her resting expression from the front photo, on by default (FLAME's neutral face has the lids wider open than hers); set it to 0 for FLAME's neutral. - Two materials sharing the texture:
DREAM_skin, and a glossyDREAM_eyeson the eyeballs, so the eyes can be tweaked or swapped on their own. The texture is packed into the.blendand embedded in the.fbx. - The detail offsets from
dream_head_flame.npz, so the rig matches the textured head. - A shorter neck: FLAME's runs about 10 cm below the chin, almost to the shoulders; the rig keeps 5 cm (
--neck).
Hair isn't included. Groom it on top in Blender (particle hair or hair curves). The FBX can't carry Blender hair, so for a game engine convert it to cards, or use the engine's own strand system (Unreal's Groom via Alembic, or TressFX).
FLAME's pose-corrective shapes aren't included, so a very wide-open jaw looks a little less natural than in FLAME. The same non-commercial licence applies to these files.
- Python. Windows uses the repo's
venv311(Python 3.11); Linux uses Python 3.12. Run scripts with that interpreter. - NumPy must stay below 2.
requirements.txtpinsnumpy>=1.24,<2andtensorflow==2.16.1, and TensorFlow 2.16 will not import with NumPy 2 ("compiled using NumPy 1.x"). If something upgrades it:pip install "numpy<2", thenpip check. The last time this happened the venv had drifted to NumPy 2.4 and TensorFlow was broken until it was set back to 1.26.4. - TensorFlow is optional. Only the extra DeepDream pass on dream pictures uses it. It downloads the InceptionV3 weights (88 MB) to
~/.keras/modelson first use. If Python's downloader fails, fetchinception_v3_weights_tf_dim_ordering_tf_kernels_notop.h5fromstorage.googleapis.com/tensorflow/keras-applications/inception_v3/withcurlinto that folder. It is skipped when free RAM is under about 1.2 GB. - Ollama and the GPU. If Ollama's CUDA build can't run on your graphics driver ("the provided PTX was compiled with an unsupported toolchain"), replies fail with a 500.
dream.py,app.pyand the C++dream.exeall notice and switch to the CPU by themselves. Updating the graphics driver or Ollama fixes it properly. - Slow replies. On a CPU-only PC expect 30-140 s per reply with a 4B model. Background thinking (reflection, dreams) only runs while she is idle or asleep so it never blocks a conversation.
- Windows "N" editions may need the free Media Feature Pack for video (the C++ build uses Windows' video decoder).
- Disk space. The models, Whisper, voices and dream pictures add up; keep a few GB free.
- Cameras. Only one program can hold a webcam.
surveillance.pyopens it and shares frames with the mind, which never opens the camera itself.
DREAM/
├── app.py Flask server: dashboard, avatar page, API, owns the sensor board
├── main.py Menu launcher (options 1, 1w, 5 ...)
├── options.bat / .ps1 / .sh The options menu for config.json (options.py)
├── config.json Ports, names, the personality prompt
├── requirements.txt Python packages (numpy<2, tensorflow 2.16.1, torch ...)
├── dream_sensors/
│ └── dream_sensors.ino Arduino sketch: PIR alarm, mmWave presence, buzzer, RGB lights
├── scripts/
│ ├── dream.py The desktop app (avatar, wake word, voice loop, sleep, sensors)
│ ├── web_launcher.py `dream.py --web`
│ ├── board_id.py Finds a board by asking it WHO
│ ├── dream_memory.py The facts memory (name, pets ...) shared by dream.py and app.py
│ ├── dream_addons.py Loads the add-ons in addons/, if any (see Configuration)
│ ├── whisper_model.py The Whisper model and device config.json picks; --download fetches it
│ ├── dream_models.py The LLM, vision model and voice config.json picks; --download-voice fetches the voice
│ ├── dream_mind/ The inner life (below)
│ ├── latent_space.py The dream world: latent-space walks (spiral, pulse ...)
│ ├── deep_dream*.py, nightmare_dreamer.py Manual DeepDream tools
│ ├── surveillance.py Webcam photos, motion frames, shares frames with the mind
│ ├── smart_surveillance.py Offline analysis of motion frames
│ ├── monocular_depth.py Depth map from the webcam (MiDaS v2.1)
│ ├── depth_effect.py Avatar depth effect: builds videos/depth/, warps frames on playback
│ └── scan_wifi.py Network scanner
├── cpp_dream/ DREAM as one C++ program (setup.bat, build.bat, dream.exe)
├── emoca_test/ 3D head from photos (see Experiments; its models aren't in git, its output is)
├── templates/, static/ Dashboard and avatar page (HTML, JS, CSS)
└── videos/, voices/, memories/, output/, certs/ Data (mostly not in git)
scripts/dream_mind/:
| File | Role |
|---|---|
mind.py |
The whole mind; the entry point (get_mind()) |
executive.py |
Goals, choosing what to do, impulse control, the tool box |
memory.py, vectors.py |
The memory graph and its local similarity index |
affect.py, drives.py, opinions.py, identity.py |
Mood, needs, opinions, boundaries and care |
dreaming.py, dream_images.py, deepdream_worker.py |
Sleep cycles, dreams, and painting them (latent-space world plus optional DeepDream) |
reflection.py, milestones.py |
Her self-model and thoughts; the milestones |
vision.py, expression.py, frames.py |
What she sees; your expression (opt-in); the shared camera feed |
voice.py |
Hearing how you sound; shaping how she sounds |
llm.py, store.py |
The one Ollama client (with CPU fallback); atomic storage |
selftest.py, dream_py_test.py |
The tests |
| Symptom | Cause | Fix |
|---|---|---|
| "Sensor board not found" | The board isn't plugged in, or runs an older sketch | Flash the current dream_sensors.ino (it must answer WHO), then python scripts/board_id.py |
dream.py prints "Sensor hub (app.py) unavailable" |
dream.py reads the board through app.py |
Start app.py |
| The board is "busy", or two programs fight over a COM port | Only one program can hold a serial port | Run only one owner: app.py, or the C++ dream.exe, not both |
| "Another DREAM already runs the inner life - this one stays passive" | dream.py and app.py share one mind; the other holds it |
Expected: one runs it. If the other was killed, wait about 90 s |
| Browser says the microphone is blocked | Not a secure page | Use http://localhost:5010/dream.html, or accept the certificate on https://...:5009 |
| Browser voice does nothing | ffmpeg missing | Install ffmpeg |
| Replies fail with a CUDA error, or are very slow | Ollama's GPU mode can't run, so it falls back to the CPU | Update the graphics driver or Ollama; see Environment notes |
ImportError ... compiled using NumPy 1.x |
NumPy 2 with TensorFlow 2.16 | pip install "numpy<2" |
| The DeepDream pass never runs | Not enough free RAM, or TensorFlow can't load | Free some RAM. A half-finished pip run can leave venv311/Lib/site-packages/tensorflow without its __init__.py; run.bat now puts it back. Check with venv311\Scripts\python -c "import tensorflow as tf; print(tf.__version__)". Dreams are still painted without it |
| No voice | Piper or a voice is missing | Put a .onnx voice in voices/; check PIPER_BIN |
| No video ("no H.264 decoder") | Windows N without the codec pack | Install the Media Feature Pack |
| No camera / "Blind Mode" | No webcam found | Expected: she carries on without eyes |
| Certificate warning | The self-signed certificate | Accept it once, or use port 5010 on this PC |
memories/*.txt show as changed in git |
She learned something | Expected; consider untracking them |
- Everything runs on this PC. Conversation memories, dreams and what she has seen stay in
memories/mind/,memories/dreams.txtandoutput/, all gitignored. - You are in control: "forget that", "forget about X", and "forget everything" (confirmed) erase what she knows. Those commands are never themselves remembered, and neither is anything said in a moment of crisis.
- The camera takes a photo every ten minutes and a frame on motion (
surveillance.py). Reading your expression is off until you ask for it, keeps only numbers, and never writes a frame to disk. Photos are not deleted by "forget everything". - She has no code-execution tool. Her few tools are a fixed list, and anything that touches the real world (the alarm, the lights) is refused unless explicitly allowed for that call.
- Always honoured: the alarm, the lights, "be quiet", sleep, quit, and forgetting. She may go quiet if she is treated badly, but never refuses those.
- The network settings listed under Connectivity use a default hotspot password (
12345678). Change it if others can reach that network.
DREAM's dashboard. Captured without a robot connected, so live values show their offline state.
Minds — DREAM companion · TINA engineer · NINA caretaker · RIFT fleet hub · LYCEA training school
Robots — NORA · MILA · WHIP · KIDA-00 · KIDA-01 · IDA · ARM · NIKA · ARIA · AURA weather · FLORA gardener
Remotes — RITA handheld remote