[bokelley]

Fixes #1547. Refs #1522, #1414, #1409, #1397, #1359, #1362.

What this changes

Every optional property the generator renders as T | None = None also
accepted an explicit JSON null, where the pinned schema — and so the SDK's
own validate_request — rejects it. The generator now grades each optional
property with the same jsonschema validator the bundled arm uses and, where
the property cannot be null, appends one Annotated element to the field:

creatives: Annotated[
    list[Creative] | None,
    Field(description=..., max_length=100, min_length=1),
    NoExplicitNull,
] = None

NoExplicitNull (in adcp.types.base, beside SchemaInt and WireUrl)
builds a json_or_python core schema: the JSON side is the type without its
null arm, the Python side keeps T | None. No validator function sits in
the path, so nested validation keeps pydantic's native JSON handling
(duplicate-key diagnostics, error order), and the published JSON Schema
loses the null arm without a hook. The annotation, the default, and
attribute access are unchanged: omitted still reads as None and dumps as
absent. What changes is input and the published schema:

	before	after
model_validate_json('{..., "creatives": null, ...}')	accepted, creatives is None	ValidationError at ('creatives',) with the type's own error (list_type), as a required array given null reports
model_validate({..., "creatives": None}) / Model(creatives=None)	accepted	unchanged: a Python None is "not supplied" (decision 1); refused under ADCP_STRICT_VALIDATION=1
model_validate({...}) with the property omitted	creatives is None	unchanged
model_json_schema()["properties"]["creatives"]	anyOf: [array, null]	type: array with the bundle's minItems/maxItems/description
model_dump() / exclude_none	unchanged	unchanged
mypy / pyright surface, .pyi stubs	T | None	unchanged
a property whose schema admits null (type list, anyOf null arm, nullable $ref)	accepts null	unchanged — generated without the marker

It is a generator rule, not per-field redeclarations: 9,044 fields across 667
generated modules (the issue's estimate was 9,945 across 687; the difference is
properties that already refused None — Literal constants with a default,
fields an allOf arm makes required — which only lose the transient mark). The canonical subclasses inherit it through the generated parent
(adcp.types.SyncCreativesRequest refuses the same payload), and _inherit
carries the marker with the rest of the FieldInfo.

Mechanism

1. scripts/generate_types.py::mark_no_explicit_null_properties runs once over
   the staged schema tree after every file is written (so $refs resolve). For
   every property outside required it asks the bundled validator
   is_valid(None) on the property node — the same predicate
   adcp.types.versioned._schema_admits_null uses for omit_none_fields on
   the serialization side — and marks the ones that refuse with
   x-adcp-no-explicit-null: true. The key never leaves the staging copy.
2. The generator carries the mark into Field(json_schema_extra=...) via
   --field-extra-keys (the only flag change).
3. scripts/post_generate_fixes.py::point_omittable_fields_at_no_explicit_null
   runs first in the fix list: it removes that keyword and appends
   NoExplicitNull, re-rendering the field the way the generator's black pass
   laid it out (width from [tool.black] line-length, right-hand-side split,
   hug, exploded). Graded: with the marker suppressed, the rewritten marked
   tree is byte-identical to a tree generated without the mark, except for
   one module (below). Every later fix therefore reads exactly the text it
   read before, plus one Annotated element.

Why not #1522's approach

#1522 does not render this case. Measured on its head merged onto main +
#1548 + #1518 + #1501: SyncCreativesRequest.creatives renders identically
to main and the #1547 payload is still accepted; its compiled constraints add
required-group validators, not null refusal. The "fork whose generator renders
optional-non-nullable distinctly" in the issue is later work on the
contributor's side that is not in the PR. The marker here is the smallest
mechanism that keeps the T | None surface; #1522's constraint compiler is
the right home for root-level conditions (#1414), not for this per-property
rule.

Decision memo for 9.0 GA

Decision 1 — a JSON null is refused; a Python None still spells absence

Pydantic cannot tell Model(**wire_dict) from Model(field=None) written by
a Python caller: both are python-mode validation. What it can tell apart is a
JSON document (model_validate_json, TypeAdapter.validate_json,
info.mode == "json") from a Python object. The rule is drawn there:

input	before	after
JSON document with "creatives": null (model_validate_json)	accepted	refused, list_type
model_validate({... "creatives": None ...}) (python mode)	accepted, reads None	unchanged — None is "not supplied"
Model(creatives=None)	accepted	unchanged
either, with ADCP_STRICT_VALIDATION=1	—	refused in python mode too
published schema	anyOf: [array, null]	type: array

Why not refuse everywhere (the first cut of this PR did): measured on the
SDK's own suite, refusing a python-mode None fails 428 tests, 316 of
them with null_not_allowed, through ~40 call sites in src/adcp that
pass an optional value through as field=None or build a wire dict with
None values and call model_validate on it (webhooks.py,
webhook_sender.py, feed_mirror.py, client.py, server/principal.py,
canonical_formats/*, decisioning/*, canonical_creative.py). That is
idiomatic Python, it is what model_dump's exclude_none=True default
already promises on the way out, and every adopter's code does the same. A
GA line cannot break it. The schema's statement — absence or a value, never
null — is about the wire document, and that is where it is now enforced:
the JSON arm, the JSON-mode model arm, and the published inputSchema agree.

What this leaves open, stated plainly: the #1547 reproduction as written
(model_validate on a Python dict carrying None) still returns
creatives=None. The flip side of the same rule: JSON produced with
nulls — model_dump_json(exclude_none=False), a TypeAdapter dump, or
an adopter's own outer pydantic model wrapping an SDK model (TaskResult
is one) — is not an AdCP document and no longer parses in JSON mode. The
bundled schema already rejects such a document today; the model arm now
agrees. Reading one back still works through python mode
(model_validate(json.loads(...))), and the SDK's own JSON-mode readers
(reporting/adjustment_evidence.py, reporting/ledger/provisional.py)
parse documents the SDK dumped with its exclude_none default. Two tests
that round-tripped TaskResult.model_dump_json() / TypeAdapter.dump_json
now dump with exclude_none=True, the way the CLI already does. The same bytes as JSON are refused; the server framework
already runs validate_request strict on inbound documents, so a wire null
never reached a handler through the SDK's own servers; and the parity suite
now grades the model in JSON mode, which is the document the bundled schema
grades. The one-line alternative — refuse in python mode unconditionally — is
python_schema=strict unconditionally in
adcp.types.base._NoExplicitNull.__get_pydantic_core_schema__, and the
measurement above is the cost. If the maintainers would rather take that cost for GA, the ~40 SDK
sites need the same treatment first (omit the key instead of passing None),
which touches files other workspaces own (webhook_sender.py, reporting).

ADCP_STRICT_VALIDATION=1 — the existing switch that turns unknown fields
into errors — now also refuses a python-mode None, so a suite that wants to
catch callers spelling absence as an explicit None can.

One generated change beside the marker: the eight "submitted" response arms
carried model_config = ConfigDict(extra='allow', validate_default=True),
a leftover from when they paired it with use_enum_values to turn the
status default into its value. status is a StrEnum member now and the
default is already the literal's member, so validating defaults did nothing
— except that in JSON mode it would run every inherited = None default
through NoExplicitNull. The arms take the base's default policy.

Decision 2 — the one module whose class set moves

brand/get_brand_identity_response.py: six colour properties are
oneOf: [string-with-pattern, array-of-same]. The generator copies a
property's extras onto the RootModel it mints for an inline arm, so the mark
stops --reuse-model from folding the six identical arm wrappers into one.
After the mark is stripped they are identical again but separately defined
(Secondary1Item(Primary1Item) instead of (Primary)). No public name, no
field, no accepted document changes. Accepted rather than special-cased: the
alternative (not marking union-typed properties with inline arms, 23 nodes)
would leave those properties accepting null.

The six issues against the 9.0 GA line

issue	status on main today	changes annotations?	changes accepted documents?	recommended pre-GA scope
#1547 explicit null	this PR	no (T | None kept; NoExplicitNull appended)	yes: refuses null the schema refuses; nothing newly accepted	land for GA (beta.5) — the only remaining declared MODEL_ACCEPTS divergence class the parity suite could not close
#1359 schema/model parity	the two headline causes are closed by #1375 (format: uri enforced, strict scalars) — re-measured: https:// and "sandbox": "yes" are refused by both arms. tests/conformance/test_schema_model_parity.py is the bundle-driven sweep it asked for; this PR deletes its optional_field_accepts_explicit_null declaration, leaving datetime_field_parses_a_number, conditional_subschema_is_not_generated, boolean_literal_equals_one, union_arm_absorbs_a_type_error, canonical_model_refuses_legacy_creative_identity, open format kinds	—	—	close as completed by #1375 + this PR; file the residuals that are still generator defects (datetime number parsing, Literal[True] == 1) as one small follow-up for next
#1362 pointer-ref RootModel wrappers	closed by #1371 — re-measured: TargetingOverlayInput/TargetingOverlay differ on 9 of 37 fields, all in the issue's two carve-outs (Sequence vs list on *_exclude, per-module element classes); geo_countries is a list, indexable; ProductPurchaseInput differs from ProductPurchase on 2 shared fields (was 19); 87 RootModel classes remain in the tree	—	—	close; nothing left for GA
#1414 root anyOf conditions dropped	open; enforce_root_required_groups keeps only each arm's required	no	yes (stricter: CreateMediaBuyRequest would refuse a doc no arm accepts)	next, via #1522's constraint compiler, which is where it is implemented; do not hand-write the 12 validators on main
#1409 union arm names	open; arms named by position (CreateMediaBuyResponse1)	yes: 38 public names change	no	next with a deprecation shim; a GA line cannot rename 38 exports
#1397 tolerant canonical subtype of strict wire model	open design question; revalidate_instances="subclass-instances" closes the runtime hole	would (lattice change)	no	defer past GA; document the subtype contract in docs/extending-types.md and keep the config guard

#1522 evaluation

Rebased (scratch, not pushed) onto main 5ce9ee4 + #1548 + #1518 + #1501:
11 conflicting files, all two implementations of one fix. Resolution: keep
main's _inherit form of SyncCreativesRequest.creatives, take #1522's
generator, regenerate. After regeneration generate_types.py --check passes,
ruff and mypy src/adcp/ are clean, mypy --strict tests/type_checks/ is
clean (the PR's documented 3.12 assert-type failure does not reproduce), and
the suite is 17,251 passed / 0 failed / 2,655 skipped (Postgres lane not
exercised).

Redundant with main now: pixel_ratio comparison (#1518), open asset
unions (#1501 — #1522's copy of emit_open_asset_unions is kept, #1501's
duplicate dropped), optional creatives (#1546/#1548), the asset
forward-compat patch removal (#1501).

Still unique to #1522, by theme (hand-written src +3.5k/−1.1k, scripts
+6.3k/−1.8k, tests +11.5k, generated 252 files):

theme	public API	accepted documents	pre-GA?
constraint reader/compiler (schema_constraints.py, compile_conditions.py, _conditions.py): root anyOf/if-then/required-group constraints compiled into validators; modules with @model_validator 60 → 225	no new exports	stricter (closes #1414)	no — behaviour change across 165 modules in a GA line; next
branch classes / split roots (branch_classes.py, _split_root.py, _branch_dispatch.py, union_branch_names.py): construction dispatches to a branch; +6 *Branches names in adcp.types, +43 in _generated; removes adcp.types.response_dispatch / ResponseArmDispatchMixin; deletes _ergonomic.py and generate_ergonomic_coercion.py	yes (adds and removes)	type(x) changes for split roots	no — breaking; next (also where #1409 lands)
error.field JSONPath-lite (pydantic_errors.py, schema_validator.py, error_narrowing.py)	no	error payload shape	could ship on main as its own PR if the spec requires the path form for 3.2
recovery text derived from the code (governance.py, server/*)	no	no	own PR, either line
v1-refs accessor (canonical_formats/identity.py + 5 callers) — fixes an AttributeError in the v2→v1 projection	no	looser (projection works)	main, small PR
four Legacy* re-exports in adcp.types.legacy	adds names	no	main, small PR
canonical-creative / split-root integration, 25 new test files, two docs	+5 Legacy* names	no	with the branch-class theme

Also found: post_generate_fixes.py on the branch defines _ensure_typing_import
twice with different signatures (the later wins) — a PR defect, not a merge
artifact.

Suggested path for the contributor: retarget #1522 to next as-is (it
merges cleanly there once next takes main), and carve two small PRs for
main now: (1) the v1-refs accessor, (2) the adcp.types.legacy
re-exports. If error.field JSONPath is needed for 3.2 conformance, a third.
Nothing in #1522 needs to precede 9.0 GA, and nothing in it conflicts with
this PR (its generator does not touch the optional-property rendering).

Not taken

• chore(deps): bump the pip group across 2 directories with 2 updates #1538 (datamodel-code-generator 0.64 → 0.81, no regeneration). Not needed
  for this fix; the pin stays. 0.81's changelog lists eight output-changing
  breaks, so a bump is a regeneration with its own review on next.

Tests

• tests/test_no_explicit_null.py (new): the Generated optional properties render as T | None, accepting an explicit JSON null the schema forbids #1547 document refused in JSON
  mode at the same pointer the bundled schema reports (list_type); a Python None still
  spells absence (dict and keyword) and dumps as absent;
  ADCP_STRICT_VALIDATION refuses it too; constraints and description
  survive beside the marker; the published schema drops the null arm and
  keeps the bundle's minItems/maxItems; a genuinely nullable property
  (update_media_buy's budget_cap_timezone, daily_budget_cap) keeps
  accepting null with no marker; a JSON round trip of an omitted property
  stays omitted; a redeclaration that keeps the parent FieldInfo keeps the
  rule; a 60-model random sample checks marker ⇔ no null arm; the transient
  key ships nowhere; the rule reaches >500 modules and >8000 fields. One
  test is xfail(strict=True): the canonical adcp.types.SyncCreativesRequest
  redeclares creatives with a fresh Field() on main, which drops the
  inherited marker; fix(types): inherit the generated constraints on the canonical creatives field #1548's _inherit carries it, and the strict xfail flips
  to a failure the moment that lands so the marker gets removed.
• tests/test_post_generate_no_explicit_null.py (new): the rewriter's layout
  against black's forms at the repository's line length, the marker-suppressed
  mode, a Literal default that only loses the keyword, and the parenthesised
  default form. Beyond the unit tests, the rewrite was graded whole-tree: with
  the marker suppressed, the rewritten marked generation is byte-identical to
  an unmarked generation for 1096 of 1097 modules (the one exception is the
  arm-wrapper dedup in get_brand_identity_response.py, which a later fix
  rewrites anyway, so the committed module is unchanged).
• tests/conformance/test_schema_model_parity.py: grades the model in JSON
  mode (the document under test is JSON), and the
  optional_field_accepts_explicit_null declaration is deleted — a
  null-acceptance divergence on any graded tool now fails the suite.
• tests/test_targeting_overlay_compat.py, tests/test_targeting_overlay_schema.py:
  the schema-shape tests that expected an anyOf: [TargetingOverlayInput, null] arm now expect the bundled shape (no null arm; on control_media_buy
  the optional packages array likewise loses its null arm, which moves one
  advertised path); the overlay field is optional, not nullable. The
  targeting-overlay bridge in _forward_compat publishes its own input
  schema, so NoExplicitNull strips the null arm through
  json_schema_input_schema as well.
• tests/test_codegen_invariants.py, tests/test_canceled_literal_default.py:
  the three hand-written fixes that widen a Literal[True] = True to
  Literal[True] | None = None (canceled, unchanged, optional boolean
  consts) now append NoExplicitNull too — the parity suite caught
  update_media_buy/canceled = null being accepted otherwise.
• tests/test_manifest_response_readback.py: fixtures spell an absent
  format_kind by omission rather than null, and the two outer-model
  round trips dump with exclude_none=True (decision 1).
• tests/test_canonical_boundary_validation.py: canonical_creative._has_boundary_serialization_metadata
  treats NoExplicitNull as validation-only, so the direct boundary scan
  still applies to marked models.
• The committed generated diff, graded by AST with the marker removed:
  676 modules changed; beyond the marker and its import, the only differences
  are the eight submitted-arm validate_default removals above. No class is
  renamed or renumbered; _generated.py, _ergonomic.py,
  error_details.py and docs/shared-type-names.md are unchanged.

Other workspaces

Nothing here touches reporting storage, ESA helpers, operation_id, or
format_is_supported.

🤖 Generated with Claude Code

---

Open workspace in Conductor

[aao-secretariat]

Ladon verdict: Approve

Approve — clean breaking-change PR with correct semver signalling.

This fix(codegen)! change makes refusal of JSON null on optional-non-nullable properties explicit while preserving Python None as absence. The breaking change is correctly signalled (! prefix + BREAKING CHANGE: footer + migration memo), so release-please will cut a major — the mandatory semver check passes.

What was verified:

• base.py _NoExplicitNull builds json_or_python with no validator in the path, preserving nested JSON diagnostics; _without_nullable correctly handles the json_schema_input_schema arm (the _forward_compat targeting-overlay bridge), tested.
• canonical_creative.py:321 correctly short-circuits NoExplicitNull in _has_boundary_serialization_metadata.
• post_generate_fixes.py (+418): _rewrite_no_explicit_null_marks L1249 correctly distinguishes T | None from Literal[True]; black-layout emulation is fail-loud via CI --check.
• Generated tree unchanged (_generated.py/aliases.py/_ergonomic.py), no renumbering; import-layering, credential, and CI-gate floors intact.

No Critical/High/Medium findings. No gated paths, no high-risk flag, no author team gate. review_decision is REVIEW_REQUIRED but gated_paths is false, so no hard gate applies — falls through to row 9.