[Bug]: Windows 10 native recording can hang on stop and fail to save MP4 in 1.9.4-rc.3 #359

Activity

1. EtienneLescot commented on Aug 14, 2026

   

   EtienneLescot

   on Aug 14, 2026

   Member

   More actions

   Thanks for the detailed trace.

   Two of your three findings are already addressed on release/v1.9.5 (rc.2):

   • Stop hang. fe794ff1 made the DXGI input path opt-in, and it is not in
     1.9.4-rc.3 — which is why you got videoInput: dxgi-nv12 by default. rc.2
     defaults to cpu-rgb32, so the second D3D11 device and the keyed-mutex bridge
     are off the default path. Verified on a build here.
   • DPI mismatch. 60bb6d7c converts Electron DIPs to physical pixels before
     the rect reaches the helper. Your single-monitor shim was needed because a new
     helper met the old rc.3 main process; in rc.2 both halves agree.

   IsBorderRequired: which Windows SDK did you build against? With C++/WinRT
   2.0.250303.5 the projected setter expands to
   try_as_with_reason<IGraphicsCaptureSession3> + check_hresult — it throws
   hresult_no_interface rather than making an unchecked vtable call, and the
   existing catch (...) handles it. Older C++/WinRT emitted the unchecked form, so
   this could be real against your SDK, but then it is a local-rebuild issue rather
   than one in the shipped helper.

   Can you run rc.2 as shipped, without your patch? Hang gone → your diagnosis is
   confirmed. Hang persists → the DXGI bridge is ruled out and the search narrows.
   That is more useful right now than merging a patch.

   We won't restore dxgi-nv12 as the default: it is off because it cost the
   reporter of #336 their recording. Worth revisiting in 1.10 with your rewrite
   reviewed.
2. slipperpeng commented on Aug 14, 2026

   

   slipperpeng

   on Aug 14, 2026

   Last edited by slipperpeng

   Author

   More actions

   Thanks. I tested the current release/v1.9.5 head without my patch. The result is the second branch of your diagnostic: the hang persists with cpu-rgb32, so the DXGI bridge is ruled out on this machine.

   Exact build tested

   There is no public rc.2 installer/tag yet, so I used the repository's own Windows diagnostic artifact rather than another local rebuild:

   • branch/head: release/v1.9.5 at b1b81de582e5b801a58b1b3d3fdd33b7f86239b9
   • successful official workflow: Diagnostic artifact run 31784641764
   • artifact: openscreen-diagnostic-windows-x64
   • wgc-capture.exe SHA-256: 11800C8859F559641E76052E2852B4CFAAD368B32D31076034EA85F2E22CCC2E
   • no OPENSCREEN_WGC_ENABLE_DXGI_INPUT variable was set

   The helper reported:

   {"video":"default","videoInput":"cpu-rgb32","container":"fragmented-mp4","preferSoftwareEncoder":false}

   SDK answer / IsBorderRequired

   My local rebuild used:

   • Windows SDK 10.0.22621.0
   • C++/WinRT 2.0.220110.5
   • Visual Studio Build Tools 17.10.5 / MSVC 19.40.33813

   You were correct about the old projection. An unmodified local build of the same branch exits with 0xC0000005 immediately after the cursor-capture event, before recording starts or stop is sent. The official CI helper above does not crash there. I therefore consider IsBorderRequired a local old-SDK rebuild issue, separate from the shipped-helper stop hang.

   Default rc.2 result

   The repository helper test failed 3/3 times with the official CI helper (5 s display capture, no audio). All three runs received stop, then followed the same sequence:

   command-received       0 ms
   wgc-quiesce            ~5000 ms, drained=false
   microphone             ~5000 ms
   loopback               ~5000 ms
   webcam                 ~5000 ms
   audio-mixer            ~5000 ms
   video-writer-join      ~13000 ms, phase=abandoned
   exit code              3
   

   The raw diagnostic contains:

   WARNING: A WGC frame callback did not finish; leaving the device alive
   [stop-timing] step=video-writer-join elapsed_ms=13030 phase=abandoned encode_stage=idle audio_stage=idle
   

   I also ran the bundled diagnostic tool for 10 seconds:

   Inputs	WGC drain	Abandoned step	Stop elapsed	Exit
   display, no audio	5003 ms, false	video-writer-join at 13030 ms	13066 ms	3
   display + system audio + default mic	5009 ms, false	video-writer-join at 13600 ms	13635 ms	3

   The selected microphone in the second run was the real default endpoint. Audio changes the later timings slightly but is not required to reproduce the hang.

   Two more controls fail in the same place:

   • window capture instead of display capture
   • display capture with the Microsoft software H.264 encoder forced

   That rules out the display-source type and the hardware H.264 encoder as necessary triggers. It does not yet isolate texture size.

   CPU-path A/B

   As a control, I ran my backed-up repaired helper with OPENSCREEN_WGC_ENABLE_DXGI_INPUT=0, so it also reported videoInput: cpu-rgb32. It passed 3/3 times on the same machine and same test script:

   • stop latency: 79 ms, 140 ms, 138 ms
   • every run completed wgc-quiesce -> video-writer-join -> encoder-finalize -> wgc-session-close
   • every run produced a readable 5.000 s H.264 fragmented MP4
   • repaired helper SHA-256: 1234FA69D894E6AEFB805BFD0CC4E4D9AE042806FF95DE3784DB94AE226893C2

   This means the successful result does not depend on my rewritten DXGI path being active.

   The remaining difference that matches the timing is the frame-state lock scope. In current rc.2:

   • the WGC callback holds mutex across ID3D11DeviceContext::CopyResource (main.cpp around 913-927);
   • the writer holds the same mutex across captureVideoSample (main.cpp around 961-1085);
   • the CPU capture then calls staging CopyResource and blocking Map(D3D11_MAP_READ) (mf_encoder.cpp around 916-919).

   If that CPU readback stalls while the writer owns mutex, an already-entered WGC callback waits on the same mutex. quiesceCapture() then cannot drain that callback, and joining the writer cannot finish. Also, encode_stage=idle appears not to cover the CPU copyFrameToBuffer path, so it does not rule out CopyResource/Map; the stage breadcrumbs are set on the DXGI conversion path.

   My repaired helper moves the D3D copy/readback work outside the application frame-state mutex and relies on D3D multithread protection for the shared immediate context. The successful forced-CPU A/B makes that lock-scope change the leading explanation now, rather than the keyed-mutex bridge.
3. EtienneLescot commented on Aug 29, 2026

   

   EtienneLescot

   on Aug 29, 2026

   Member

   More actions

   @slipperpeng — sorry for the long silence on this. Your 14 Aug trace is the reason I have come back to it, and it deserved an answer a lot sooner than two weeks.

   What your trace decided

   wgc-quiesce        5003 ms, drained=false
   video-writer-join 13030 ms, phase=abandoned encode_stage=idle audio_stage=idle
   WARNING: A WGC frame callback did not finish; leaving the device alive
   

   On 11 Aug, before either of us had that measurement, I wrote in #252 that encode_stage=idle is what #306 predicts and #305 does not. You then produced it 3/3 on the official CI helper with videoInput: cpu-rgb32 — the default path, no DXGI bridge in play. @Afton-programmer's Save Diagnostics report on #460 landed on the same place from the other direction: videoEncoderRuntime: hardware, and the callback still stuck.

   So the encoder is out and the bridge is out. What is left is the thing neither of the merged fixes touches: onFrameArrived holds the shared frame-state lock across CopyResource, on a thread we do not own, and when that call does not return the lock is gone for good.

   There is now a build that removes that thread

   #306 (pull-based frame delivery, by @abduznik) had been sitting conflicted since 11 Aug. It is rebased onto main and mergeable again. writeVideoFrames pulls each frame itself with TryGetNextFrame() and does the copy on its own thread; no FrameArrived handler is registered at all by default.

   Same delivery mechanism you used last time, so nothing to compile:

   • run: https://github.com/getopenscreen/openscreen/actions/runs/33259506707
   • artifact: openscreen-diagnostic-windows-x64 (expires 12 Sep; GitHub requires you to be signed in to download it)

   The two runs that would settle it

   Exactly what you already did, then the same bundle a second time with one variable set:

   set OPENSCREEN_WGC_LEGACY_FRAME_CALLBACK=1
   

   That flag restores the shipped push-based path from the same binary, so the two runs differ in one thing only. The wgc-quiesce line now names which one ran:

   what you see	what it means
   mode=pull, drained=true, stop completes	the wedge needed the callback thread. #306 is the fix
   mode=pull, drained=true, video-writer-join … phase=abandoned	the copy wedges on the writer thread too. #306 is not the fix and I want to know that before it lands
   mode=legacy-callback, drained=false, reason=frame-callback-stuck	your original failure, reproduced on purpose — that is the control, not a regression

   The middle row is the one I am actually asking for, and it is the reason I would rather you ran this than merge on the strength of a hypothesis that has never met the hardware it is about. On mine — Windows 11, integrated Radeon, no virtual display — all six scenarios pass and always did; it has never reproduced this bug and cannot falsify anything.

   On IsBorderRequired: I took your conclusion, local old-SDK rebuild rather than the shipped helper, and there is nothing outstanding for you there.

   No rush and no obligation. This has already cost you a lot of runs, and the diagnosis in your first report is what got the rest of us pointed the right way.
4. abduznik commented on Aug 29, 2026

   

   abduznik

   on Aug 29, 2026

   Contributor

   More actions

   @EtienneLescot I know your message was directed at @slipperpeng, but I went ahead and ran the diagnostic build across both modes on my own setup (AMD Ryzen 5 4500 + RTX 4060 Ti, Windows 10 Pro, standard cpu-rgb32 software frame copy + hardware video encoder pipeline).

   The results align directly with Row 1 vs Row 3:

   1. Default Pass (mode=pull)

   [stop-timing] step=wgc-quiesce elapsed_ms=0 drained=true mode=pull
   [stop-timing] step=video-writer-join elapsed_ms=0 phase=begin
   [pacing] frames=284 elapsed_ms=9480
   [stop-timing] step=video-writer-join elapsed_ms=0

   Status: Clean exit (exitCode: 0)

   Stop Duration: stopElapsedMs dropped to 84 ms (down from ~13,000 ms)

   Writer Join: video-writer-join completed cleanly in 0 ms

   2. Control Pass (OPENSCREEN_WGC_LEGACY_FRAME_CALLBACK=1)

   WARNING: A WGC frame callback did not finish; leaving the device alive
   [stop-timing] step=wgc-quiesce elapsed_ms=5012 drained=false mode=legacy-callback
   [stop-timing] step=video-writer-join elapsed_ms=5012 phase=abandoned encode_stage=idle audio_stage=idle reason=frame-callback-stuck

   Status: Timed out (exitCode: 3, stop-timeout on video-writer-join)

   Stop Duration: stopElapsedMs hit 5057 ms before abandoning

   Quiesce: drained=false, reason=frame-callback-stuck

   Conclusion

   The control flag cleanly reproduces the original callback deadlock on my hardware, while pull mode (mode=pull) completes teardown almost instantaneously with zero wedging. Moving CopyResource off the OS-managed WGC callback thread and pulling via TryGetNextFrame() on the writer thread resolves the deadlock as hypothesized. #306 is confirmed as the fix.

   👍React with 👍1Reacted by Etienne Lescot
5. slipperpeng commented on Sep 1, 2026

   

   slipperpeng

   on Sep 1, 2026

   Last edited by slipperpeng

   Author

   More actions

   @EtienneLescot I ran the requested A/B test on the original Windows 10 machine with the same diagnostic build and the same inputs: 10-second display capture with system audio and microphone enabled.

   Build: fix/wgc-pull-based-frame-delivery at f4cafb2 from Actions run 33259506707.

   Default mode: pull

   [stop-timing] step=wgc-quiesce elapsed_ms=0 drained=true mode=pull
   [stop-timing] step=video-writer-join elapsed_ms=45
   [stop-timing] step=encoder-finalize elapsed_ms=69
   [stop-timing] step=wgc-session-close elapsed_ms=70
   

   • exit code: 0
   • total stop time: 141 ms
   • recording-stopped emitted
   • saved file is readable: H.264 video + AAC audio, 48 kHz stereo

   Control mode: legacy callback

   The only change was setting:

   set OPENSCREEN_WGC_LEGACY_FRAME_CALLBACK=1

   WARNING: A WGC frame callback did not finish; leaving the device alive
   [stop-timing] step=wgc-quiesce elapsed_ms=5002 drained=false mode=legacy-callback
   [stop-timing] step=video-writer-join elapsed_ms=5741 phase=abandoned reason=frame-callback-stuck
   

   • exit code: 3
   • total stop time: 5801 ms
   • recording-stopped not emitted
   • output was only 83 bytes and unusable

   The results match your table exactly. Pull mode completes and saves; legacy callback mode reproduces the callback deadlock. On this hardware, #306 is confirmed as the fix for the Issue #359 stop/save failure.

   Test system: Windows 10 Pro 22H2 (build 19045), AMD Ryzen 7 5700X, NVIDIA GeForce RTX 4060 Ti.

   👍React with 👍1Reacted by Etienne Lescot
6. EtienneLescot commented on Sep 4, 2026

   

   EtienneLescot

   on Sep 4, 2026

   Member

   More actions

   Closed by #306.

   Huge thanks to @slipperpeng for the initial root-cause analysis and for running the decisive A/B verification on the original reproducing hardware, as well as @abduznik for the implementation and matching test run.

   Both A/B passes confirmed the hypothesis:

   • Pull mode (mode=pull) completes teardown cleanly in ~80–140 ms with a valid, fully playable MP4.
   • The control flag (OPENSCREEN_WGC_LEGACY_FRAME_CALLBACK=1) cleanly reproduced the legacy callback deadlock (drained=false, reason=frame-callback-stuck).

   #306 is now merged into main. The OPENSCREEN_WGC_LEGACY_FRAME_CALLBACK=1 escape hatch remains available in the binary as a rollback lever should any unexpected hardware edge cases arise.
7. EtienneLescot

   closed this as completedon Sep 4, 2026