Architecture

Data Flow

End-to-end ingestion data flow. Accepted-path arrows are solid green; the rejected branch is dashed red. Consumers (bottom row) read indexes only — testing-os is the sole write authority.

The same flow as text (terminal-friendly fallback for screen readers and CLI viewers):

Source repo workflow
  → Builds structured submission (JSON)
  → Emits via repository_dispatch to testing-os

testing-os ingestion pipeline
  → Schema validation (AJV)
  → Provenance check (GitHub API)
  → Policy evaluation (enforcement, scenarios, freshness)
  → Verdict computation (source proposes; verifier confirms
    or downgrades)

  → Accepted: records/<org>/<repo>/YYYY/MM/DD/<run-id>.json
  → Rejected: records/_rejected/<org>/<repo>/YYYY/MM/DD/...

  → Index rebuild:
      indexes/latest-by-repo.json
      indexes/failing.json
      indexes/stale.json

Key Design Decisions

Central Ingestion

Source repos never write records directly. They emit structured payloads via repository_dispatch, and only the testing-os bot writes to the records directory. This prevents source repos from fabricating evidence.

Verdict Ownership

The source repo proposes a verdict (overall_verdict in the submission). The verifier can confirm or downgrade — never upgrade. A source claiming “pass” that fails schema or policy validation becomes “fail.”

Sharded Persistence

Records are stored at records/<org>/<repo>/YYYY/MM/DD/<run-id>.json. This provides natural time-sharding, easy browsing, and clean git history without merge conflicts.

Generated Indexes

latest-by-repo.json is rebuilt from accepted records after every ingestion. Consumers read indexes, not the raw record tree. This keeps reads fast without scanning git history.

Components

Component	Location	Purpose
Verifier	packages/verify/	Schema, provenance, policy, verdict validation
Ingestion	packages/ingest/	Pipeline orchestration, atomic persistence, index rebuild
Submission builder	packages/report/	Canonical submission assembly for source repos
Portfolio	packages/portfolio/	Org-level summary generation

Verifier Pipeline (8 steps)

The verifier (packages/verify/index.js) processes each submission through eight stages in order:

1. Repo-binding cross-field guard – runs first, before schema validation: rejects a submission whose submission.repo does not match the owner/repo encoded in source.run_url (validators/repo-binding.js). Without it, a submitter could claim submission.repo = "victim-org/victim-repo" while supplying a run_url for a real run from their own repo – provenance would confirm the run exists and persistence would file the forged “pass” under the victim’s path. This is a first-class trust-boundary stage with its own parser module and provider-lockstep CI test, not part of step 2’s schema check.
2. Schema validation – validates the submission against dogfood-record-submission.schema.json using AJV.
3. Verifier-owned field guard – rejects submissions that include fields only the verifier may set (policy_version, verification, or overall_verdict as an object).
4. Provenance check – confirms the source workflow run actually exists via the GitHub Actions API (or a stub adapter in tests).
5. Step results validation – checks that each scenario’s required steps have matching results and that verdicts are internally consistent.
6. Policy evaluation – evaluates enforcement tier, required scenarios, freshness, and execution-mode constraints from the repo or global policy. When a policy names required scenarios, their definitions are supplied by the ingestion layer, which for a github submission performs a read-only fetch of the submitting repo’s dogfood/scenarios/<scenario_id>.yaml at the attested commit (size-capped and schema-validated; an absent file leaves required-steps enforcement off with a visible required_steps unenforced warning). This read and the provenance check are the pipeline’s two routine calls to the GitHub API – the same surface disclosed in the README security section and the Error Code Reference.
7. Verdict computation – computes the final verdict. The source proposes a verdict string; the verifier may confirm or downgrade, never upgrade. Verdict severity from highest to lowest: fail, blocked, partial, pass.
8. Record assembly – builds the persisted record with verifier-owned fields (verification.status, verification.verified_at, overall_verdict.verified, overall_verdict.downgraded).

Generated Indexes

The index generator (packages/ingest/rebuild-indexes.js) produces three files after every ingestion:

Index	Content
indexes/latest-by-repo.json	Latest accepted record per repo and surface – the primary read model for consumers
indexes/failing.json	Records where the verified verdict is not pass
indexes/stale.json	Repo/surface pairs with no accepted record within the staleness threshold (default 30 days)

Atomic Persistence

Records are written atomically: the persist layer writes to a temporary file, then renames it to the final path. Duplicate detection by run_id prevents double-writes (collisions surface as DUPLICATE_RUN_ID — see Error Code Reference). Accepted records go to records/<org>/<repo>/YYYY/MM/DD/, rejected records to records/_rejected/<org>/<repo>/YYYY/MM/DD/.

Rebuild Outcomes

packages/ingest/rebuild-indexes.js returns four arrays per call: accepted, rejected, corrupted, skipped.

• accepted / rejected — record loaded cleanly; routed by verification.status.
• corrupted — JSON.parse failed on the file. The rebuild logs [rebuild-indexes] corrupted record skipped: <path> — <error> to stderr and continues; the record is excluded from all indexes.
• skipped — record loaded but missing run_id.

Corruption does not fail the rebuild — the index is silently incomplete until repaired. See Operating Guide → Corrupted Record Recovery for the procedure.

Enforcement Tiers

Mode	Behavior	Default
required	Blocks on violation	Yes – all repos start here
warn-only	Warns but doesn’t block	Must have documented reason
exempt	Skips evaluation entirely	Must have documented reason + review date

Missing policy defaults to required – the safe default.