ARF - a record format for evaluation runs

What breaks today

Most evaluation pipelines end in a CSV of scores. Weeks later a row says 0.82, and nobody can say which prompt version, which model build, or which judging step produced it.

The cost lands on whoever has to defend the number: the engineer asked why a regression appeared, the reviewer asked what the score covered, the auditor asked to see the inputs. The scores survive. The context that made them mean anything does not.

What ARF is

Ten JSON record types and one hashing rule. ARF is not a new format - it is the format Korvo Atlas already implements, written down so the same bytes and digests can be checked outside the product.

Validation is fully offline: no network calls, and schema $id values are never dereferenced. Those identifiers (namespace path korvo.xyz/arf/ns/v0.1) name types - they are not fetch targets, and nothing serves them yet.

A record, in full

Every ARF record is a plain JSON object. This is a real Claim, copied from claim.ndjson.

{
  "id": "6251dbd4-a1f2-4b8b-9cd0-594a46a2f5c3",
  "statement": "Training language models on their own synthetic outputs leads to progressive degradation of output quality, a phenomenon termed \"model collapse\".",
  "confidence": "high",
  "sourceIds": ["bafb465b-47be-45fd-b94f-ea6599153e4f"],
  "questionId": "7ebb02e3-c3db-4010-a03c-13d7e7ca4df8",
  "status": "published",
  "tags": ["model collapse", "synthetic data"],
  "origin": "human",
  "reviewStatus": "unreviewed",
  "createdBy": "Alice Chen",
  "createdAt": "2026-03-25T20:17:56.727Z",
  "updatedAt": "2026-03-25T20:17:56.727Z"
}

statement

The assertion itself - one claim per record, never a paragraph containing several.

sourceIds

The inputs this claim rests on. At least one is required, and order is significant: reordering it changes the digest.

questionId

The question this claim answers. Claims point at questions, not the reverse.

confidence

How confident the author is - one of low, medium, high, verified. It is a stated position, not a computed score; engine verdicts live on Evaluation instead.

Records are stored one per line as .ndjson. Unset means the key is absent - ARF never writes null to mean “no value”.

How records reference each other

Every link is a UUID held in a field on the pointing record. Read X --field--> Y as “X.field holds the id of Y”.

Claim        --questionId---->   Question
Claim        --sourceIds----->   Source
Artifact     --claimIds------>   Claim
Artifact     --sourceIds----->   Source
Evaluation   --claimIds------>   Claim
Challenge    --claimId------->   Claim
Challenge    --challengerId-->   Validator
Endorsement  --claimId------->   Claim
Endorsement  --validatorId--->   Validator
Revision     --artifactId---->   Artifact
ChainRecord  --artifactId---->   Artifact

ArtifactBundle { artifact, question, claims[], sources[], publishedBy, publishedAt }
     this whole object is the value that gets hashed

An ArtifactBundle is the publish-time envelope: one artifact, its question, and the claims and sources it used, collected into a single JSON object. That object is the thing that gets hashed.

The ten record types

Grouped by role. Required fields are exactly those marked required in each schema.

Evaluation, Revision and ChainRecord link to their schema because no example record is published yet. For Evaluation that is deliberate: the claimIds link cannot be recovered from an engine payload, and a fabricated example would misrepresent the field most likely to be misunderstood.

The digest, end to end

Canonicalization means reducing a JSON value to one exact byte sequence, so any correct implementation produces the same bytes. Four stages, one operation.

{
  "id": "602d54e3-0650-40c8-898c-e6d3c0e71d71",
  "claimId": "6251dbd4-a1f2-4b8b-9cd0-594a46a2f5c3",
  "validatorId": "d2f602ae-9be6-43c0-8c1e-622980547d49",
  "comment": "Strong evidence from the Shumailov et al. paper. Model collapse is well-documented.",
  "weight": 5,
  "createdAt": "2026-03-25T20:17:56.727Z"
}

{"claimId":"6251dbd4-a1f2-4b8b-9cd0-594a46a2f5c3","comment":"Strong evidence from the Shumailov et al. paper. Model collapse is well-documented.","createdAt":"2026-03-25T20:17:56.727Z","id":"602d54e3-0650-40c8-898c-e6d3c0e71d71","validatorId":"d2f602ae-9be6-43c0-8c1e-622980547d49","weight":5}

56c223f1dac62072d1999e00cd3ba854352682294425fa10ad102f1a0c73ea6e

sha256:56c223f1dac62072d1999e00cd3ba854352682294425fa10ad102f1a0c73ea6e

Check it yourself

Paste step 2 into any SHA-256 and you get step 3.

printf '%s' '{"claimId":"6251dbd4-a1f2-4b8b-9cd0-594a46a2f5c3","comment":"Strong evidence from the Shumailov et al. paper. Model collapse is well-documented.","createdAt":"2026-03-25T20:17:56.727Z","id":"602d54e3-0650-40c8-898c-e6d3c0e71d71","validatorId":"d2f602ae-9be6-43c0-8c1e-622980547d49","weight":5}' | shasum -a 256

The three rules that decide the bytes

• Object keys are sorted by UTF-16 code unit, at every depth. Reordering keys does not change the digest.
• Array order is preserved and is significant. Reordering sourceIds is a different record, and must produce a different digest.
• The digest covers the canonical bytes, not a re-parsed or pretty-printed string. There is no trailing newline.

Values with no JSON representation - NaN, Infinity, undefined - are rejected rather than coerced. Emitting a hash for an unrepresentable value would produce a proof nobody can reproduce.

The same operation applied to an ArtifactBundle gives its contentHash. bundle.ndjson records sha256:81ea9018…db24d673, and recomputing it this way reproduces that value.

The algorithm is versioned as rfc8785-jcs-v1, carried inside the bundle. An implementation that meets a version string it does not recognise must reject it rather than fall back to a default - silently hashing with a different algorithm produces a confidently wrong answer.

Validate the examples

Offline. No network, no $id dereferencing.

git clone https://github.com/korvoxyz/atlas.git
cd atlas
npm ci
npm run arf:validate -- spec/v0.1/examples/*.ndjson

PASS spec/v0.1/examples/artifact.ndjson:1 (artifact)
PASS spec/v0.1/examples/bundle.ndjson:1 (bundle)
PASS spec/v0.1/examples/challenge.ndjson:1 (challenge)
PASS spec/v0.1/examples/claim.ndjson:1 (claim)
...
PASS spec/v0.1/examples/validator.ndjson:2 (validator)

16 passed, 0 failed

Records are checked against the same JSON Schemas the server uses. The type is inferred from the record shape, or forced with --type=claim.

The spec

Everything authoritative lives under spec/, licensed Apache-2.0. The Korvo Atlas server remains AGPL-3.0-only.

• spec/v0.1/README.mdHow the pieces fit together.
• canonicalization.mdHashing rules, conformance vectors, porting checklist.
• disclosure.mdWhether an author intends a record to be published.
• bundle.schema.jsonJSON Schema for the ArtifactBundle.
• examples/The seed records used by the command above.

Examples are seed records, exported read-only from development fixtures - not production data. Nothing in spec/v0.1/examples/ came from a live deployment.