Skip to content

Linux export: libopenh264 ignores the requested bitrate entirely — 47x over target on heavy content, 4.5x under on light #572

Closed as not planned
Closed as not planned
Linux export: libopenh264 ignores the requested bitrate entirely — 47x over target on heavy content, 4.5x under on light#572

Description

@EtienneLescot
EtienneLescot
opened on Sep 2, 2026

libopenh264 in the vendored ffmpeg build does not rate-control at all. It is not that the
requested bitrate is approximate — it has no effect worth measuring, in either direction.

Found while benchmarking the Linux export (#544, #552). It is not specific to that work: every
Linux export goes through this encoder, so the quality of every Linux export is whatever the
encoder feels like, and the setting we expose does not reach it.

Reproduction

120 frames of 1280×720, deliberately incompressible content (so the encoder cannot come in
under target by accident), encoded against the vendored libavcodec with nothing but the
bitrate knobs changed:

requested bit_rate rc_max_rate rc_mode produced
2 Mbps — default (quality) 93.5 Mbps
8 Mbps — default (quality) 97.5 Mbps
2 Mbps — bitrate 94.0 Mbps
8 Mbps — bitrate 97.7 Mbps
2 Mbps 2 Mbps bitrate 94.0 Mbps
8 Mbps 8 Mbps bitrate 97.7 Mbps

Asked for 2 Mbps with an explicit 2 Mbps ceiling, it emits 94 Mbps — 47× over. Quadrupling
the request moves the output by 4%.

Neither rc_mode nor rc_max_rate changes anything.

What this looks like in a real export

On screen-recording content the symptom inverts, which is why it went unnoticed: a mostly
static screen is very compressible, so the encoder undershoots instead.

requested produced
a 60 s 1080p60 export 8 Mbps 1.75 Mbps
the same, requested 12 12 Mbps 0.72 Mbps
the same content via h264_vaapi 8 Mbps 6.95 Mbps

So the bitrate we compute from the output resolution — (w * h * 8 Mbps) / (1920 * 1080),
pipeline_linux.rs — never reaches the encoder. Export quality does not scale with resolution,
and raising the setting does not raise quality. On heavy content the same defect produces a
file many times larger than asked for.

h264_vaapi in the same build honours the request, which is what makes this look like a
libopenh264-specific defect rather than a wiring mistake on our side.

What is ruled out

  • Not the option plumbing. av_opt_set(priv_data, "rc_mode", "bitrate") returns 0, and the
    value is still 1 when read back after avcodec_open2. The encoder's own option table
    lists rc_mode with quality as default and bitrate/buffer/timestamp alongside.
  • Not bit_rate being clobbered at open. It reads back as 8000000 after avcodec_open2.
  • Not a ceiling. Setting rc_max_rate equal to the target changes nothing.
  • Not content-dependent. The table above is incompressible input, where a working rate
    control would have the most obvious job to do.

Where I would look

  1. Whether the wrapper forwards the target into OpenH264's SEncParamExt —
    iTargetBitrate and sSpatialLayers[0].iSpatialBitrate — or leaves them at defaults.
  2. Whether the bundled OpenH264 is built without its rate-control module, in which case the
    setting has nowhere to land regardless of what the wrapper does.

I could not fix this from application code, which is why this is an issue rather than a PR: no
combination of the options ffmpeg exposes for this encoder had any effect.

Why it is worth fixing rather than working around

Beyond the quality problem, it makes the software encoder impossible to compare against
anything. Benchmarking it against h264_vaapi required pinning VAAPI to the bitrate
libopenh264 happens to emit, otherwise the comparison reports a quality difference as a speed
difference. Any future encoder work runs into the same wall.

Measured on AMD Ryzen 5 7520U, Ubuntu 24.04, against
crates/thirdparty/ffmpeg-linux64-lgpl-shared. The probe is ~40 lines of C linked against the
vendored libraries; happy to attach it if useful.

Activity

  1. EtienneLescot commented on Sep 2, 2026

    @EtienneLescot
    EtienneLescot
    on Sep 2, 2026
    MemberAuthor

    Root-caused. The cause is neither of the two places you suggested looking, and both of your "ruled out" items are correct — the wiring really is fine.

    It is bEnableFrameSkip, and openh264 says so out loud

    ffmpeg's wrapper does forward the target. libopenh264enc.c (n8.1.2, byte-identical to master):

    param.iTargetBitrate = avctx->bit_rate > 0 ? avctx->bit_rate : TARGET_BITRATE_DEFAULT;
param.iMaxBitrate    = FFMAX(avctx->rc_max_rate, avctx->bit_rate);
param.bEnableFrameSkip = s->skip_frames;
param.sSpatialLayers[0].iSpatialBitrate    = param.iTargetBitrate;
param.sSpatialLayers[0].iMaxSpatialBitrate = param.iMaxBitrate;

    So your point 1 is answered: iTargetBitrate and sSpatialLayers[0].iSpatialBitrate are both set. Point 2 is answered too — the rate-control module is compiled in and running.

    What is wrong is that -allow_skip_frames defaults to false, and openh264 prints this at every avcodec_open2 — it is in the log of every export we have ever produced:

    Warning: bEnableFrameSkip = 0, bitrate can't be controlled for RC_QUALITY_MODE, RC_BITRATE_MODE and RC_TIMESTAMP_MODE without enabling skip frame.

    CheckFrameSkipBasedMaxbr() in ratectl.cpp returns immediately when frame skip is off, and it is the only consumer of iMaxSpatialBitrate at encode time — which is exactly why rc_max_rate changed nothing for you.

    Turning it on does restore control, and is unusable: at 2 Mbps on your incompressible case it kept 3 of 120 frames.

    The second reason, and the one that explains your table

    qmin/qmax default to -1 in the wrapper's FFCodecDefault, so openh264 keeps its own (0, 51) — then ParamValidation() judges iMinQp = 0 invalid and silently replaces the whole window with (12, 42). Every Linux export so far has run inside a QP window nobody chose. That is why your rc_max_rate and rc_mode rows are flat: the encoder was pinned against a ceiling of QP 42 the whole time.

    Where I have to disagree with the report

    the quality of every Linux export is whatever the encoder feels like

    Not on realistic input. I rebuilt the measurement on four content classes cut from real recordings in ~/.config/openscreen/recordings, through a C probe that mirrors try_open exactly (no ffmpeg CLI in between):

    clip (1080p, real) ask 1M ask 2M ask 4M ask 8M
    mixed: UI + video window 0.98 1.94 3.67 5.64
    full-frame webcam — 1.99 — 7.51

    That is the target tracked to within a few percent. Quality on a dense capture is VMAF 96.3; forcing QP down to 22 only reaches 96.6, so the encoder is not starving anything.

    Both halves of your report are real but need narrower statements:

    • Overshoot needs content openh264 cannot compress at any QP. It reproduces on synthetic noise; it did not reproduce on any real recording I could build, including full-frame camera footage.
    • Undershoot is the QP-12 floor (GOM_MIN_QP_MODE, clamped unconditionally). On a static screen the encoder pins there and spends 0.68 Mbps whether you ask for 8 or 40. That is not the encoder failing to reach a target — it is correct VBR behaviour on content that needs no more bits. I could not reproduce your 12 Mbps → 0.72 non-monotonicity; my sweeps are monotone throughout.

    One methodology note that may matter: your h264_vaapi comparison almost certainly ran against system ffmpeg, not the vendored build. The vendored h264_vaapi aborts on this machine (libva 2.20 on Ubuntu 24.04 lacks vaMapBuffer2, which BtbN's implib-gen stub asserts on). Filing that separately — it is harmless today, because try_open passes AV_PIX_FMT_YUV420P with no hw_frames_ctx and VAAPI refuses with a clean -22 long before libva is touched, but it is a landmine for the planned hardware ladder.

    What I have fixed, and what stays open

    #575 does what is reachable from application code, and found a worse defect on the way:

    • The export had one keyframe per file. libopenh264 overrides g to -1; try_open never set gop_size; openh264 got uiIntraPeriod = 0. Measured 1 I-frame in 300. Every seek redecoded from frame 0. Now fps * 2.
    • The QP window is now chosen rather than inherited. Inert on all four real clips, byte-identical; on noise it cuts 113.2 → 65.5 Mbps.
    • High profile instead of Constrained Baseline / CAVLC — -3.8 % bitrate on dense content, -8.0 % on a static screen, at equal VMAF.

    Leaving this issue open: none of that gives libopenh264 rate control. rc_mode, rc_max_rate, rc_buffer_size, max_nal_size and level were each measured and each do nothing, so the sweep does not need repeating — VideoEncoder::tune_openh264 records it. Upstream is cisco/openh264#3259, closed without a fix, and libopenh264enc.c has not changed between n8.1.2 and ffmpeg master.

    The one genuinely promising direction nobody can reach from here: openh264's SCREEN_CONTENT_REAL_TIME usage type — the right one for a screen recorder, with a (26, 35) QP window and SCC tooling — is exposed by no ffmpeg option. That needs an upstream patch adding an AVOption for iUsageType, and it is probably worth more to this project than anything else on the list.

    And happy to take that probe you offered — I wrote my own, but a second one against a different fixture would be worth comparing.

  2. EtienneLescot
    added 2 commits that reference this issue on Sep 2, 2026

    fix(export): give the Linux encoder a keyframe interval and a chosen …

    ee6870e

    fix(export): give the Linux encoder a keyframe interval and a chosen …

    870d814

  3. EtienneLescot commented on Sep 3, 2026

    @EtienneLescot
    EtienneLescot
    on Sep 3, 2026
    MemberAuthor

    Narrowing this with a measurement that separates "the wrapper is wired wrong" from "this encoder has no rate control": the same harness, same incompressible input, run through both software encoders in the vendored build.

    encoder requested 2M 8M 20M
    libopenh264 93.5 Mbps 97.5 97.5
    libkvazaar 8.6 Mbps 18.7 34.4

    libkvazaar responds — monotonic, and overshooting by 1.7–4x is what a working rate controller does on input it cannot compress. libopenh264 is flat: going from 8 to 20 Mbps changes the output by nothing at all.

    So this is not our plumbing. The same AVCodecContext fields, set the same way, reach one encoder and not the other. It is specific to libopenh264 in this build.

    And it is not substitutable. libkvazaar is HEVC-only (libkvazaar H.265 / HEVC (codec hevc)), so it cannot serve ExportCodec::H264. Checking -encoders on the vendored binary, libopenh264 is the only software H.264 encoder present — the LGPL build excludes libx264. There is no other H.264 encoder to fall back to.

    Which puts the practical scope more precisely than the original report:

    I have no fix to offer inside the application. Every option libopenh264 exposes has now been tried by two people independently: rc_mode (accepted, survives avcodec_open2 — verified by reading it back — and changes nothing), rc_max_rate, rc_buffer_size, max_nal_size, level. What does reach it is the QP window, which #575 now sets explicitly.

    So the realistic choices are (a) treat bit_rate as inert on that one path and control quality through QP instead, documenting it rather than computing a number that goes nowhere, or (b) take it up with the vendored build — either the wrapper is not forwarding the target into SEncParamExt, or the bundled OpenH264 is built without its rate-control module. I could not tell which from outside.

    Measured on AMD Ryzen 5 7520U, Ubuntu 24.04, against crates/thirdparty/ffmpeg-linux64-lgpl-shared, 120 frames of 1280x720 noise.

  4. EtienneLescot commented on Sep 14, 2026

    @EtienneLescot
    EtienneLescot
    on Sep 14, 2026
    MemberAuthor

    Closing, with the conclusion of the investigation.

    On real content the requested bitrate is followed: mixed 1080p30 gives 0.98 / 1.94 / 3.67 Mb/s for 1 / 2 / 4 requested, and webcam gives 1.99 / 7.51 for 2 / 8. The blowup in the title only reproduces on incompressible noise, not on an actual screen recording.

    What remains is upstream. libopenh264 only rate-controls with frame skipping enabled, which drops frames and is not acceptable for an export (cisco/openh264#3259, closed without a fix). #575 already applied what is achievable from the app, and the whole reasoning with its measurements lives in the tune_openh264 doc comment in crates/compositor/src/pipeline_linux.rs.

  5. EtienneLescot
    closed this as not plannedon Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions