Activity

1. MuhammadBilalKhan267

   added a commit that references this issue on Aug 23, 2026

   Update VFR issue reference to Breakthrough#569

   6e7a366
2. Breakthrough commented on Sep 23, 2026

   

   Breakthrough

   on Sep 23, 2026

   Last edited by Breakthrough

   Owner

   More actions

   Following up on our Discord convo about VFR handling. It was proposed to add FramePosition / FrameDuration types for v0.8.

   It's useful to define that PTS alone cannot be the identity of a frame. A stream can contain two frames with the same PTS, or PTS that goes backwards/is non monotonic. Certain stream types also carry no timestamps at all, in which case timestamps are synthesized as was the case in v0.6 and previous.

   There are some other nuances that I've noticed from other projects, such as some frames having zero duration, PTS wraparound for long videos, or muxed streams that don't have corrected timestamps (e.g. the PTS goes up then starts again at 0 when it hits the concat point). Let's define then a frame index number that the decoder produces. This should line up with your proposal to only count what we decode.

   Proposed Types for v0.8

   We need to add a new base type in addition to Timecode, which already exists, to represent Duration as a span. We can define all these types then as:

   • Timecode(pts, time_base): a point in time in PTS (fractional value). Arithmetic with times/durations should return another Timecode.
   • Duration(frames, time): the duration between two points in time. Some inputs such as min_scene_len only specify one of the two (frames or seconds), so when comparing, we use whichever component was specified in the other operand and never convert (no framerate assumption). We call the case where the duration contains both components an exact duration.
   • FramePosition(index, time: Timecode): the observed timestamp for the frame at index. Immutable. Constructed only by backends or during index lookups. Does not have operator + , just - to calculate duration. FramePosition - FramePosition should always give an exact duration.
   • FrameIndex: ordinal -> raw pts as an array plus a time_base (and some token to verify where the frame was minted? e.g. against the same stream object). Built by SceneManager during detection as the decoder produces frames. Allows translating the decoded ordinal for a given VideoStream to the presentation time (e.g. via at(ordinal) or at_or_before(Timecode) via binary search).

   Positions should never computed, only observed or looked up. For CFR, the lookup is equivalent to what we do now. For VFR, it becomes exact. Non-monotonic input is also well defined by this.

   Consumers

   A quick survey of frame_num uses in the codebase:

   • requires only frame index: save-qp, save-keyframes, progress bar, image sequences, the list-scenes table, CSV output
   • requires only pts: split-video (ffmpeg/mkvmerge), timecode strings, the time command
   • requires duration calculations: every detector's min_scene_len, the fade minimum in ThresholdDetector, the mid-scene checks in AdaptiveDetector
   • requires lookup of index: ThresholdDetector splitting a fade at its midpoint is the only consumer that technically needs a frame it never observed. We need to figure out how to make the lookup work, e.g. does ThresholdDetector then need to stash a reference to SceneManager so it can look it up?

   Implementation Notes

   We might need an origin enum somewhere to track RELATIVE from ABSOLUTE counting behaviours and allow a config option to change it. The absolute mode would require a demux-only pass for some inputs, which might be an acceptable tradeoff for the accuracy gain.

   It might also be worth overhauling how backends handle timing. If we have the following sequence where a frame is "repeated" in time:

   index:  0     1     2     3     4
   pts:    0    33    66    66    99
   

   We could define a TimestampPolicy that could be set to for example:

   • PASSTHROUGH: detectors see frame 66 twice
   • DROP: throw away frames with a repeated PTS in the stream, detector only sees 4 frames
   • MONOTONIC: nudge the frame to the next position the stream can express in terms of it's time base. So frame 3 would become 66.001 if the time base was 1/1000~. Should also log a warning to be consistent with ffmpeg (something about how we detected a non-monotonic PTS in the stream and it's changing from X to Y).

   save-keyframes on VFR becomes correct which is good. It also allows replay of a run. Indices should be 0-based internally, only adding 1 when formatting durations for example. There are also some discrepencies still, e.g. the csv format/scene table add 1 but save-qp/save-keyframes use 0 indexed ordinals.

   Rollout

   If we make FramePosition act as much as possible like FrameTimecode, the vast majority of existing code will continue to work while we improve the underlying type. It should be sufficient if we keep the same immutable properties and comparison operators. This will allow downstream consumers to get the improved accuracy without needing any changes on their part too.

   👍React with 👍1Reacted by Muhammad Bilal Khan
3. Breakthrough

   added

   bug

   on Sep 23, 2026
4. Breakthrough

   added this to the 0.8 milestone on Sep 23, 2026
5. Breakthrough commented on Oct 25, 2026

   

   Breakthrough

   on Oct 25, 2026

   Last edited by Breakthrough

   Owner

   More actions

   Thanks for the detailed feedback on these @MuhammadBilalKhan267.

   Seeking & Ordinal Origin

   Ordinals are relative to the first frame decoded in a run (RELATIVE). Mid-run seeking was never supported, I'll document that constraint rather than trying to auto-detect it (risk of false-positives).

   The index can store an offset representing the frames before the first decoded frame, and a flag that indicates if it is exact or not:

   • OpenCV / MoviePy: exact (CAP_PROP_POS_FRAMES / seek frame number).
   • PyAV: uses our current round(seconds * fps) estimate by default so CFR behavior is unchanged. An opt-in ABSOLUTE CLI mode can make this exact via a fast demux-only packet pass.

   save-qp continues writing relative ordinals. save-keyframes will write origin_offset + ordinal and warn once on VFR input if the offset is inexact.

   Backwards PTS in PyAV

   This is reproducible, and looks like a genuine PyAV bug rather than an index issue. PyAV unconditionally sets AVFMT_FLAG_GENPTS on the container when opening it, which means for DTS-only containers like an AVI with B-frames, genpts constructs decode-order PTS values that scramble the B-frame reordering. OpenCV avoids this by falling back to pkt_dts.

   I'll fix this separately for 0.7.3 in another issue, and have the PyAV backend open containers with fflags=-genpts. This should mean we prefer pts -> dts -> decoded frame count. Perhaps we should ask the PyAV folks to expose Frame.best_effort_timestamp or something...

   For FrameIndex, it will store the PTS given to the detector(s) and will track monotonicity on append. We don't need a mapping of time-to-ordinal for the first iteration, bisect/linear scan is fine.

   Duration of Last Frame

   Back to the "use 0" answer. +1 frame, this is the existing contract across get_scene_list, ThresholdDetector.post_process, and split-video - using 0 would drop the final frame when splitting and cause trailing 1-frame scenes to fail min_scene_len.

   FrameIndex.end() shall therefore have to estimate the last frame's duration:

   1. Using the frame duration from the backend when available (e.g. frame.duration in PyAV),
   2. If unavailable, use the most common (statistical mode?) frame distance, rather than the previous, since that will fail with certain known patterns.
   3. If unavailable, 1 / avg_fps.

   Also re: your fade-out question: end - fade_out.time is always exact in frames (both ordinals were processed). They are also exact im terms of time when the backend provides a frame duration, otherwise it uses the estimate above when a fade runs to end-of-stream.

   Types & Rollout

   • FramePosition(ordinal, time, frame_rate): Index type, keeping FrameTimecode's read-only interface (pts, time_base, seconds, get_timecode, comparisons, and a deprecated frame_num property) so downstream code keeps working. No + operator; FramePosition - FramePosition returns an exact Duration.
   • FrameTimecode: Kept as user-input/timestamp type, since too much existing code relies on FrameTimecode arithmetic (start_time + duration, FrameTimecode(int | str, fps)).
   • StatsManager: Keys changed to origin_offset + ordinal (with a warning when inexact), while we migrate internal frame_num callers so library code itself doesn't trigger the deprecation warning.
   • TimestampPolicy: PASSTHROUGH / DROP / MONOTONIC, with the default being monotonic. Monotonic should just bump non-monotonic timestamps to the next tick and warn once.

   Staging

   1. PyAV -genpts fix - I'll tackle this for 0.7.3.
   2. Duration + Timecode arithmetic (used internally by FlashFilter and min_scene_len).
   3. FramePosition + FrameIndex + ThresholdDetector (stamping with indices in the decode thread, frame_skip tracking, needs_frame_index, index.end(), StatsManager keys, frame_num cleanup).
   4. Ordinal origin & ABSOLUTE mode (CLI flag, finish save-qp / save-keyframes integration).
   5. TimestampPolicy type introduction.
   6. timecodes-v2 export and replay.

   If you can scope your first draft PR to (2) + (3) with the buffer stopgap, that sounds great!