Skip to content

Runtime evidence format, storage and secfoo runtime ingest #53

Closed

Description

@vsn411

This is the shared contract that both probes (#55, #56), the coverage importer (#54) and the live sensor (#67 → #66) write to. One format for files and for the live API.

Event file: NDJSON. One session line, then load lines (#68) and hit lines, one per watched function or package seen per route:

{"type":"session","id":"<uuid>","language":"python","runtime":"3.12.4","started_at":"...","ended_at":"...","source":"sensor|coverage","app":"payments-api","env":"prod","instance":"<host or pod name>","sensor_version":"0.4.0","app_functions_total":1840,"app_functions_executed":1362}
{"type":"load","session":"<uuid>","ecosystem":"PyPI","package":"pyyaml","version":"5.3","path":"/app/.venv/lib/python3.12/site-packages/yaml","first_seen":"..."}
{"type":"hit","session":"<uuid>","finding_fingerprint":"...","symbol":"yaml.load","route":"POST /upload","auth":"anonymous","count":312,"first_seen":"...","last_seen":"...","entry_frame":"app/routes/upload.py:42","stack_sample":["app/routes/upload.py:42","app/services/parse.py:17"]}

language is one of python, node, jvm, dotnet, go, rust, ruby, php or native. app, env, instance, route and auth are optional, so file and coverage producers can leave them out. For live sensors, the server fills app/env from the sensor's SECFOO_APP/SECFOO_ENV (#66). route is a template or job:<name> / startup / unknown, never a concrete URL (#69).

stack_sample holds file:line frames only, never local variables or arguments. Frames outside the app (site-packages, node_modules) are dropped, except the vulnerable function itself.

Storage: new tables runtime_sessions, runtime_evidence (FK to sca_findings by project and fingerprint, keyed per route) and runtime_packages (loaded inventory per app/env). Add them in schema.sql with a migration that works on existing databases.

CLI: secfoo runtime ingest events.ndjson --project <name>. It validates against the schema, ignores unknown fingerprints with a warning, and running the same file twice must not double-count. Put validation and idempotent upsert in one module, so the live API (#66) reuses it.

Done when: the schema is documented in docs/REFERENCE.md, and tests cover a valid file, a malformed line (skip it and report it, don't abort the whole ingest) and re-ingesting the same file.

Activity

  1. added
    enhancementNew feature or request
    help wantedExtra attention is needed
    hackathonOpen for the Secfoo hackathon
    difficulty: hardLeaderboard: 10 points when the PR closing this is merged
    on Nov 8, 2026
  2. karthickmk2001 commented on Nov 11, 2026

    @karthickmk2001
    Collaborator

    I'd like to take this. I'll follow the format in the description (session/load/hit NDJSON) so #54, #55, #56 and #57 can build on it; PR to follow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    difficulty: hardLeaderboard: 10 points when the PR closing this is mergedenhancementNew feature or requesthackathonOpen for the Secfoo hackathonhelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions