Skip to content

ci(canonical): replay the ai-agent-action conformance suite and require jcs-admit 0.1.1 - #228

Merged
geekgonecrazy merged 7 commits into
atomicdotdev:devfrom
astrogilda-forks:ci/conformance-corpus
Nov 3, 2026
Merged

geekgonecrazy merged 7 commits into
atomicdotdev:devfrom
astrogilda-forks:ci/conformance-corpus

Conversation

@astrogilda

@astrogilda astrogilda commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Thanks @graywolf336 for merging #204.

This adds a CI job that runs the ai-agent-action suite from probityai/agent-evidence-vectors against atomic-canonical. The new conformance_corpus test reads 120 documents from 74 of the 75 vectors at the v0.25.0 commit and holds each to its manifest verdict. A byte fault such as a repeated member or an unsafe integer must be refused by jcs::admit_document, and every other document must be admitted. Record lines in accept vectors must also match jcs::canonicalize byte for byte. With the admission call replaced by a plain serde_json parse, five vectors fail.

The test skips v43175cd2ad003639, whose extensions member nests 128 deep and puts its statement at depth 130, past the 128-level MAX_DEPTH in atomic-canonical.

The manifest now requires jcs-admit 0.1.1, which the dev lockfile already resolves, so Cargo.lock matches dev.

Edited 2026-10-23: merged dev, moved the suite from v0.13.0 to the v0.25.0 commit, and updated the counts and the lockfile note.

jcs-admit 0.1.1 lowers its serde floor, so the lockfile no longer has
to carry serde 1.0.229 and the syn 3.0.6 it brought in. Against the
lockfile before the admission layer was added, jcs-admit is now the
only package that differs.

Signed-off-by: Sankalp Gilda <sankalp.gilda@gmail.com>
Add an ignored test that reads a checkout of the suite and holds every
statement and record line to the verdict its manifest implies: a byte
fault must be refused by admit_document, every other document must be
admitted, and each record line of an accept vector must already be
canonical. A new CI job checks out agent-evidence-vectors at v0.13.0
and runs it. With the admission result ignored, four vectors fail.

Signed-off-by: Sankalp Gilda <sankalp.gilda@gmail.com>
@graywolf336

graywolf336 commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Hey, I'm reviewing open PRs and see there is a conflict. I have it fixed locally but some reason github is rejecting my changes. Is there any way you could resolve the conflicts?

@graywolf336

Copy link
Copy Markdown
Contributor

Ah, found the reason. astrogilda-forks/atomic is in an organization and GitHub does not support the maintainers in one org being able to push edits to another org's fork.

Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>

# Conflicts:
#	Cargo.lock
Pin the checkout to the commit behind the v0.25.0 tag rather than the
v0.13.0 tag name. The suite now has 75 vectors; the test reads 120
documents from 74 of them and passes. With admit_document replaced by a
plain serde_json parse, five vectors fail.

Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
@astrogilda

Copy link
Copy Markdown
Contributor Author

@graywolf336 done: I merged dev into the branch, so it's conflict-free at 8288324. I also moved the suite to the v0.25.0 commit, and the new CI job passes on it.

@graywolf336

Copy link
Copy Markdown
Contributor

@astrogilda would you be willing to update our docs to add a section to the software supply chain and how it maps back to atomic? We are looking to merge your changes this week and we wanting to make sure our documentation is updated.

List the standards behind canonical bytes, Data Integrity proofs, grants and change signatures, the CI jobs that hold them, how to run the conformance replay locally, and how to raise its pins. Link it from the documentation index.

Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
Add a weekly cargo entry that allows only the two crates the canonical layer pins, so each new release arrives as a pull request against dev that runs the canonical tests and the conformance replay. Point the supply chain page at it.

Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
@astrogilda

Copy link
Copy Markdown
Contributor Author

@graywolf336 done: atomicdotdev/atomic-docs#33 adds a Software Supply Chain page that maps each Atomic record to its standard and names the CI job that checks it.

I also added a short contributor page here, docs/software-supply-chain.md, and a Dependabot entry that proposes each new jcs-admit and dsse release.

@graywolf336

Copy link
Copy Markdown
Contributor

@astrogilda thank you!! I'm about to fly back from home a conference, when I get back I plan to merge this changes in. Which merge order should I do it in or does it not matter with all of your open PRs?

@astrogilda

Copy link
Copy Markdown
Contributor Author

@graywolf336 the conflicts on my four open PRs are fixed, with CI green on each one. Merge order: #228 first as the smallest, then #236 and #235, then #234 as the biggest. They don't clash with each other, so the order is only my preference; safe flight!

Supply chain page: atomicdotdev/atomic-docs#33, plus docs/software-supply-chain.md in #228.

@geekgonecrazy
geekgonecrazy merged commit f8711ef into atomicdotdev:dev Nov 3, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants