Repository navigation
Conversation
Member
Author
|
Also remove the references to G5 from |
Contributor
|
@floesche -- thanks for the PR! However, I'm going to close this because I will be updating how we're handling arena configs, including generation and will make a larger update next week. At that time, I'll remove the G5 option....so doesn't make sense to review this one because all the changes should be done together, starting with the matlab code |
mbreiser
added a commit
that referenced
this pull request
Sep 12, 2026
…, hang window, next diagnostic build Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mbreiser
added a commit
that referenced
this pull request
Sep 14, 2026
…iver, GET_HEALTH decode — Studio v0.76 (fw #50) (#198) * feat(studio): controller-fault detection, post-mortem probes, soak driver, GET_HEALTH decode — Studio v0.76 (fw #50) A Mode-3 controller wedge (firmware #50) used to be invisible: the closed-loop drain loop swallowed every 0x70 timeout and a dead-arena run "completed" and auto-committed. Now: - fictrac-bridge-client: rolling fault detector (>=3 failed applies in the last 10, not "3 consecutive" — a late same-opcode reply can satisfy the next request) → latched `fault` event, apply forced OFF. - arena-runner: `fault(reason, detail)` wakes the trial wait and unwinds; `summary.fault` / `summary.faultDetail` / `summary.stopAcked`; a timed-out protocol command (the 0x08 after the 0x70s) is labelled the same. - arena-session: bridge fault → runner.fault + `fault` event; runner sends go through session.send (a link facade) so trialParams/STOP land in the run log; `reconnect()` / `flushRx()`; `dt` on the monotonic clock; terminal summary + fault detail in the sanitized runner rows. - arena-link: `reconnect()` from granted ports (navigator.serial.getPorts, VID/PID match, ambiguity refused, re-enumeration poll); the response deadline now covers a stalled write; `flushRx()`. - run-log / adapter: outcome CONTROLLER_FAULT (auto-commits; ABORTED_BY_USER still doesn't); `fault` phase in the transcript. - studio-postmortem (new, classic dual-export): quiet period + rx flush → confirmation 0xC2 → probe window (5 s timeouts, typed `probe` rows with decoded + raw hex) → halt | SYSTEM_RESET → reconnect → MAC verify → post-reset probe. - wire: GET_HEALTH 0xCA encode/decode (55 B + optional 11 B slowest-op tail), GET_FRAME_POSITION 0x72, capability bit 7 `health`. - Studio: fault banner; `runOnce()` (the no-dialog half of beginRun); soak driver (File ▾ → Soak…, `?soak=1`, advanced-only; refuses without a behavior_v2 ack or bridge frames; halt-first; never auto-commits); Debug ▾ → Controller health; URL-state `soak` flag; v0.76 footer. - scripts/wedge-scan.py + tests/test-wedge-scan.py: per-run wedge/onset table over v1/v2/gz/.runlog.json (found a sixth, earlier occurrence: rig2 2026-07-11). - fictrac_sim.py: --turn-sigma / --jump-every / --jump-deg. - protocols/soak_mode3_closed_loop.yaml; docs/development/mode3-wedge-soak-plan.md (findings, hypothesis ladder, campaign spec, bench recipe); release notes; CLAUDE.md rule. Tests: full `pixi run test` green (new: fault latch, runner fault, session facade + fault routing, link reconnect/write-deadline, wire GET_HEALTH vectors, adapter/run-log outcome, postmortem lifecycle, URL soak flag, wedge-scan). Not bench-tested. Codex (gpt-6-astra) plan review reconciled in .codex-review/report-20260911-wedge-soak.md (gitignored). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(soak): keep the bridge up between iterations; soak always logs behavior_v2; honest stop reason Bench 2026-09-11 (10-10 controller, sim): the first smoke iteration ran clean (2,001 × 0x70, RTT 2/6 ms) but (1) the protocol's last step disconnected the fictrac plugin — the bridge IS the run's logger, so the run's terminal event never reached the log and the soak loop then waited on frames from a disconnected bridge; (2) a remembered behavior_v1 session setting overrode the soak preflight's v2 negotiation at run start. Drop fictrac.disconnect from the soak protocol, reconnect the bridge before the freshness check, force behavior_v2 while a soak runs, and report the operator's stop reason instead of "no-frames". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(soak plan): first bench night — wedge reproduced on a third controller (silent, 0x01 no ack); flash mishap note Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(closed loop): a controller REJECT of SET_FRAME_POSITION counts as an apply failure Bench 2026-09-12 07:00 ET: a firmware built for a 4x10 arena rejected every pattern on the 2x10 card, and the runner streamed 21k rejected 0x70s ("no pattern selected", status 1) for minutes without tripping the fault detector — it only counted throws (timeouts/transport). The session's bridge applyFrame now decodes the reply and throws on a non-ok status, so rejects feed the ≥3-of-10 window like timeouts do (one #199 reject per trial start stays well below the threshold). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(studio): record the controller's firmware build identity (GET_FIRMWARE_VERSION 0xCB) in every run log No G6 controller build was identifiable until now: 0xC2 carries a format byte that is always 1, 0xE3 is the panel image footer, and run_metadata.firmware held "v1" (or null after a gesture-free reconnect). Firmware feat/controller-health adds GET_FIRMWARE_VERSION 0xCB — git SHA, dirty flag, branch, build date, arena rows×cols, compiled in by a PlatformIO pre-script. - wire: encodeGetFirmwareVersion / decodeFirmwareVersion (46 B payload → {sha, dirty, debug, date, branch, arena, label}), golden tests. - Studio: Studio.refreshFirmwareVersion() on the Connect button, the Console info command, and a state hook that covers every session.reconnect() (the post-mortem/soak paths, which used to leave firmware null); gated on the `health` capability bit because older firmware flashes CE 01 on the arena for an unknown opcode. Studio.firmware → run_metadata.firmware → run log. - post-mortem probe set includes firmware_version (post-reset probe records which build wedged); wedge-scan.py gains an `fw` column. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore(studio): footer timestamp for the 2026-09-12 edits Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(telemetry): controller ring-buffer drain into the run log — T4 host side (fw feat/telemetry-ring) The controller keeps CMD/FRAME/STATE records in reboot-surviving OCRAM (proposal § 3.1); the host drains them with GET_TELEMETRY_BLOCK 0xA9 using an ack cursor (lossless, framing A) and writes compact rows into the run log. - js/arena-telemetry.js (new, classic dual-export): parseBlock (18 B header + records: CMD/FRAME/STATE/PAD), toRows → ["cc"|"cf"|"cs", rx, t_us, seq, …], createDrainer (ack cursor, seq gaps, controller drop counter, drainAll crash dump), createPoller (single-flight, gated). tests/test-arena-telemetry.js. - wire: SET_TELEMETRY 0xA8 / GET_TELEMETRY_BLOCK 0xA9 encoders + tests. - bridge 3.1: {type:'rows'} → each shape-checked array row written verbatim (write_rows); fictrac-bridge-client.logRows(); Python test. - Studio: Studio.initTelemetry() on every link-up (health cap + 0xA8 ack), 10 Hz drain while connected (pauses under the post-mortem), stream_schema line per log, minute stats, Studio.drainTelemetry() wired as the post-mortem's afterReconnect → the ring's last records after a reset. - wedge-scan.py: ctl recs / sd max us / ctl states / ctl rejects columns and, with --verbose, the last 40 controller records before a wedge. - docs: plan § 11 (T4 as built), release notes, CLAUDE.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(closed loop): wrap frame indices into the configured frame count before apply (#199) The bridge applies a new `frames` modulus one frame late, so the first index after a trial whose pattern has a different frame count carried the previous modulus → "index out of range" + an error glyph once per trial (bench 2026-09-11: 2 of 110,643). Now that controller rejects feed the fault window, a protocol with short alternating trials could even trip a false CONTROLLER_FAULT. Wrap locally (idx mod frames) before the consumer clamp; no configured count → untouched. run_metadata gains `telemetry: ring-10hz|off` so the instrumentation factor is visible in analysis. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(fw #50): Codex review — evidence-first post-mortem, ack-means-stored ring drain, deferred fault commit, reader safety Reconciles the gpt-6-astra status + diff reviews (.codex-review/report-20260912-status.md): - runner: terminal event emitted after the best-effort STOP so the serialized summary carries stopAcked; Studio defers a CONTROLLER_FAULT run's export/commit until the post-mortem (probes + ring dump) has finished - post-mortem: identity requires a decoded 0xC2 reply and the same MAC; "recovered" requires zero timeouts/errors and every required probe ok (SD-image probes are optional); capability-gated probes are skipped + recorded when caps are unknown; evidence first after reset (health → ring drain → probes), duplicate drain removed - telemetry: FRAME sd_load_us u32, header flag bits 2/3, maxChunks 40 / maxBytes 178, ack only when the sink stored the rows (notStored), drainer.idle(); Studio gates SET_TELEMETRY on GET_FIRMWARE_VERSION flags bit 2 (never on `health`), on/off switch (Debug ▾), stop path, poller yields to the post-mortem; analog-in too - readers: runlog-replay + dashboard analysis-core skip string-tagged rows - bridge 3.1: per-tag minimum row lengths; logRows returns acceptance - wedge-scan: fault-declared outcome, pre-boot ring tails, malformed rows counted - soak: per-iteration applied_0x70 / duration / telemetry stats, `unexercised` - docs: status doc §4b/§4c/§6, plan bench log, release notes, CLAUDE.md rules Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(CLAUDE.md): telemetry ring gate is the 0xCB flag, not the health capability Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(telemetry): incarnation-aware drainer rearm — NO_ACK first after (re)connect and post-reboot, duplicates dropped, fresh ring resets the cursor Codex firmware review: an old ack sent to a power-cycled controller would free unseen records. The Studio now rearms the drainer on every link-up and before the post-reboot crash-dump drain. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): firmware ring Codex review outcome + flash go/no-go Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: commit the 2026-09-12 Codex reconciliation report; track deferred items in #200 / fw #54 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(studio): telemetry poller idles unless the bridge is logging (no refused-row spin) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(session): send({silent:true}) skips the bridge command log; the telemetry drainer uses it The 9-byte 0xA9 request is not compactable (8-byte head), so every drain request landed as a 207 B arena_command object — 20 % of the T4 log file. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(telemetry): poller.stop() called window.clearInterval as a method → Illegal invocation in browsers (would abort the fault post-mortem) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): T4 iteration-1 gate results — telemetry cross-checks, reboot survival both routes, two live fixes Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: handover for the parallel telemetry-metrics / SD-jitter planning session Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): wedge #2 captured on ring firmware — breadcrumb evidence, hang window, next diagnostic build Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(wire/studio): GET_HEALTH v2 tail (ISR breadcrumb, watchdog PC capture) + GET_CRASHREPORT 0xCC decoders; Console health readout and post-reboot crash-report capture; analyzer decodes watchdog boots Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(postmortem): self-reset path — a link that drops after the fault (hardware watchdog reboot) is reconnected, identity-checked, evidence-drained and the soak continues Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(CLAUDE.md): watchdog self-reset path, GET_HEALTH v2, GET_CRASHREPORT Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(wire/studio): CrashReport len is in words (11); GET_CRASHREPORT gated on 0xCB flags bit 3, not the ring bit Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): wedge #3 captured (same breadcrumb window); Codex review of the watchdog build and the fix list Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): wedge #4, shrinking intervals across MCU-only reboots, watchdog build flashed (RTWDOG config failed on hardware) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(wire): GET_HEALTH v3 tail — raw WDOG3_CS at boot and live (hardware-enabled bit) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): watchdog build 86eeb4a flashed and proven by starve test; RTWDOG clock-gate root cause Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(studio): a mid-run involuntary link drop (watchdog self-reset) is routed through the fault lifecycle, not treated as a user STOP Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(wire): GET_HEALTH v4 tail — measured watchdog tick rate, live TOVAL, verify bits Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): 54b57d0 flashed; watchdog verified again (0.5 s effective; tick measurement 4x off); soak restarted 17:31 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: close-out and PR consolidation plan for the fw #50 campaign Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(postmortem): a link drop during the probe window ends the window and takes the self-reset path Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(wire): GET_HEALTH v5 tail — WDOG3_CNT around each kick (watchdog offset diagnostics) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): 4860fef flashed — watchdog expiry calibrated to 2.0 s; final firmware for the night Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): operating point raised to 286 Hz + wide jumps at 22:31; first-minute numbers Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): finding — display starves at 286 Hz because SET_FRAME_POSITION restarts the refresh timer Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(studio): bound the raw log panel (3k lines) — an uncapped DOM log starved the closed-loop apply loop over long runs Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): 200 Hz block from 23:36; 286 Hz block summary; decisions Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): wedge #5 caught by the watchdog — PC in IntervalTimer::end() at the PIT TCTRL store; hung-peripheral-bus reading Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): post-self-reset frame-count bug (pattern set rebuilt on connect) — live re-patch, proper fix listed Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): standing order for the night — soak to ~08:00; flash the free-running-timer variant on the next identical wedge Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): evidence + proposed fix write-up for Codex review Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): Codex review reconciliation — IntervalTimer::end() race as the leading mechanism; evidence doc §2b; bench log Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(telemetry): decode fw free-running-timer variant (0xCB bit4), ISR ids 4–7 (usb/sdhc/lpspi/pit), STATE kinds 8 wdog_context / 9 prev_isr_count Host side of the fw eca07f6 + follow-up build: firmware label gains ' freerun', health ISR names cover the driver-vector trampolines, the ring decoder unpacks the watchdog context record (EXC_RETURN, stacked IPSR, prior isr_last) and the per-ISR boot counts; wedge-scan labels them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): firmware diff review 1 (eca07f6) reconciliation Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(telemetry): STATE kind 10 timer_fail, EXC_RETURN mode bit; docs: firmware diff review 2 reconciliation, SDHC priority correction Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): evidence §2b — PIT-only NVIC masking, not PRIMASK * docs(wedge): bench log — fix build candidate 394dee45 * docs(wedge): firmware diff review 3 — flash gate passed for 394dee45 * docs(wedge): wedge #6 (PC 0x212f2 — second return address inside the end() race window), flash of 394dee45, starve gate, soak restart Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): night-2 wedge-scan table + morning summary (baseline 2 wedges / 1.42M, fix build 0 / 4.41M) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): PIT race reproduced stand-alone (stock dead <0.5 s, PIT-masked 5.9M cycles clean); soak restarted on 394dee45 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): findings + fix + upstream note draft for Codex review before posting Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): Codex reconciliation of findings/fix/upstream note; final texts (200 Hz-block baseline 1.09M, conditional 3 %, invariant-framed upstream fix) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): handover for the Mode-3 performance session (SD → PSRAM, tick service, presentation latency, deferred lists); soak wedge-watch script Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(wedge): bench log — soak wound down, handover * docs(wedge): fw #50 comment posted; upstream report held for Frank's review --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Merged
6 of 7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I was confused by G5 being listed in most of the tools. I thought, G5 never worked. Even if it did, I believe the panels had a different number of LEDs?
This PR removes all references to G5, except for the
data/reference_data.jsonfile.