Skip to content

Never unpickle binary responses unless explicitly asked (binary_loads= / binary_dumps= seams) - #20

Merged
thorwhalen merged 4 commits into
masterfrom
fix/no-unpickling-by-default
Oct 13, 2026
Merged

thorwhalen merged 4 commits into
masterfrom
fix/no-unpickling-by-default

Conversation

@thorwhalen

@thorwhalen thorwhalen commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Closes #19. Client-side counterpart of i2mint/py2http#20.

What changes

  • Binary (application/octet-stream) responses are no longer unpickled. Without a decoder, a binary response raises http2py.decorators.BinaryResponseNotDecoded (a ValueError), whose message names the options. Unpickling a response lets a spoofed or compromised server, or anyone on a plain-http path, run code on the client.
  • New seam binary_loads= (keyword-only) on mk_request_function, HttpClient, mk_request_func_from_openapi_spec, openapi_to_py, and loads= on handle_binary_resp. It accepts:
    • bytes for the raw content (e.g. images);
    • any bytes -> object decoder;
    • unsafe_pickle_loads (new, in http2py.decorators) for a fully trusted server.
    • loads=pickle.loads still works but warns, mirroring py2http's handle_binary_req. A non-callable loads gives a TypeError at construction.
  • New seam binary_dumps= on the same functions, for binary request bodies. The default stays pickle.dumps. Pickling the client's own arguments is no risk to the client, and the server already has to opt into unpickling them (loads=unsafe_pickle_loads on py2http's side). Raising instead would only break clients of such opted-in servers, with no security gain. Pass binary_dumps= to match a server that uses another decoder.
  • default_configs.default_binary_output_trans is kept for importers. It now raises on binary responses as well.

Opt-in for the old behaviour

HttpClient(spec, binary_loads=unsafe_pickle_loads) (or the same keyword on the other factories). Only when the server is trusted.

Tests

tests/test_binary_payloads.py (8 tests). It uses a fake dispatch, so no network is involved. It goes in the root tests/ because http2py/tests is excluded from CI collection. The tests cover:

  • the default raises, and a payload that runs code when unpickled is never run;
  • bytes and unsafe_pickle_loads both work;
  • pickle.loads warns, and a non-callable loads is refused;
  • binary_dumps is used, and the default is still pickle;
  • HttpClient passes binary_loads through.

Full suite with doctests: all pass (the dill tripwire is removed, see below).

Also: dill workaround removed

CI was red on tests/test_dependency_workarounds.py::test_dill_workaround_is_still_needed. That failure happens on master too. The test is a deliberate tripwire: it fails once the released ju stops importing dill (i2mint/ju#6), and it asks for exactly this fix. So this PR drops dill from [project].dependencies and deletes that test module, in a separate commit.

Adversarial review

Fresh Opus reviewer, two passes.

  • Pass 1: SHIP. Nothing in the package unpickles server data by default, and nothing deserializes it unsafely in any other way: no dill, marshal, eval or yaml.load (the CLI uses safe_load). A server that lies about its Content-Type gets nothing either, because the decoder comes from the spec's response_type, not from the header. Positional HttpClient callers and importers of default_binary_output_trans still work. Small fixes, all done:
    • binary_loads is now validated up front in mk_request_function and HttpClient, so a non-callable is rejected even when the spec has no binary endpoints.
    • The pickle.loads warning now points at the caller's line instead of library code.
    • The docstring says the warning is advice, not a guard.
    • Py2Request accepts partial(mk_request_function, binary_loads=...), which used to crash on string specs.
  • Pass 2: SHIP. One cosmetic nit, fixed: the package-dir match now ends in a path separator.
  • Deferred to mk_cli / api_pkg_maker: no binary_loads passthrough; generated funcs file does not compile #21:
    • mk_cli/dispatch_cli and api_pkg_maker's generated module have no binary_loads= passthrough. Binary endpoints raise there, which is safe but loses capability.
    • The generated module does not compile.

Dependents gate (priv test-dependents http2py --transitive): http2py and extrude pass, and the reviewer found no dependent relying on unpickled responses (meshed, extrude and the plunk demos use JSON services). py2http's own suite passes against this branch (74 passed). Its gate entry failed at collection because pytest imported the repo-root deploy.py, which runs a build and a twine upload at import time. Nothing ran, because the machine has no python on PATH. That is unrelated to this change and is filed as i2mint/py2http#26.

🤖 Generated with Claude Code

thorwhalen and others added 2 commits October 13, 2026 02:37
A binary (octet-stream) response used to be unpickled, which lets a spoofed
or compromised server run code on the client. It now raises
BinaryResponseNotDecoded unless a decoder is given:

- handle_binary_resp(func, loads=...);
- mk_request_function / HttpClient / mk_request_func_from_openapi_spec /
  openapi_to_py(..., binary_loads=...): bytes for the raw content, a decoder
  of your own, or the new http2py.decorators.unsafe_pickle_loads for a fully
  trusted server (loads=pickle.loads also works but warns).

Binary request bodies get a binary_dumps= seam on the same functions; the
default stays pickle.dumps (safe for the client, and the server has to opt
into unpickling it).

Closes #19

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…caller, Py2Request accepts a partial

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
thorwhalen and others added 2 commits October 13, 2026 02:41
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nt/ju#6)

tests/test_dependency_workarounds.py was the tripwire for this: it failed once
ju stopped importing dill, and asked for this removal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thorwhalen
thorwhalen merged commit a00cd65 into master Oct 13, 2026
12 checks passed
@thorwhalen
thorwhalen deleted the fix/no-unpickling-by-default branch October 13, 2026 02:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Client unpickles binary responses and pickles binary request bodies

1 participant