Dataset Generation

Dataset generation turns one validated production release into the current public VyDex Dataset 2.0.0 contract. The Dataset 1.0.0 generator and its stable legacy path remain available for v1 release inputs and preserved release history; they are not widened to accept Entry v2 records. This page is for technical users, maintainers, and coding agents that need to inspect, integrate with, or safely change the Schema, projection, ordering, serialization, or immutable artifact behavior.

Purpose and Ownership

The system keeps every public dataset consumer on one versioned contract without making release construction responsible for serialization or filesystem writes. It owns:

The exact TypeScript contracts for Dataset 2.0.0 and the retained legacy Dataset 1.0.0 projection.
The origin-specific draft 2020-12 JSON Schema and its canonical identity.
Projection of current public Entries from a validated production release.
Derived public labels, Evidence Strength scores, dates, Evidence Types, and relationship references.
Deterministic array ordering, including defensive reuse of the domain-owned public source order, plus stable property construction, indentation, UTF-8 text, and final-newline behavior.
Validation of serialized dataset content against the exact public Schema.
Immutable artifact paths and the stable-latest redirect descriptor.
Pure release-specific filename and path derivation from validated release metadata.
Structured generation diagnostics.

It does not own:

Loading canonical records or publication snapshots.
Validating or constructing production releases.
Creating release IDs or timestamps.
Reading or persisting release descriptors.
Choosing the genuine production site origin.
Publishing Cloudflare redirect files or verifying deployed redirects.
Creating public content, historical dataset exports, CSV, APIs, media, or reports.
Terminal output or persistent logging.

The filesystem writer is a separate adapter. It owns safe emission beneath an injected output root, but it does not change the generated bytes or decide which release to publish.

Inputs and Outputs

generateVyDexDatasetV2 accepts one successful production ReleaseModel whose current Entries are Entry v2 records and whose route registry selects the Dataset v2 Schema and stable-latest path. generateVyDexDatasetV1 remains the explicit legacy projection for a v1 release. Both generators require validated metadata, a production HTTPS origin, current published Entries, resolved relationships, canonical URLs, and a matching route registry; preview, partial, non-promotable, or structurally inconsistent release values are rejected.

The generator returns:

The typed VyDexDatasetV2 value for the current release, or VyDexDatasetV1 for a legacy release.
Deterministically serialized JSON.
The immutable release-specific public path.
The Dataset Schema public path.
A 302 stable-latest redirect descriptor targeting the immutable artifact.

The application export boundary calls the generator with an already selected validated release. It checks that repeated generation produces the same path and bytes, validates the page-facing metadata against the generated dataset, and returns the artifact together with the Export JSON Page presentation model. Dataset generation does not own that presentation model or page composition.

generateVyDexDatasetSchemaV2 accepts a production site origin and returns the origin-specific Dataset v2 Schema value and deterministic Schema bytes. The v1 Schema generator remains available for the retained legacy contract. Schema generation does not create release identity or persist artifacts.

Normal Flow

Release Construction validates repository records, selects each Entry's newest published snapshot, resolves relationships and routes, and returns one complete production release.
The application export boundary selects the v2 or legacy generator from the release route registry. The selected generator rechecks the release discriminator, persisted metadata, production origin, required dataset routes, public Entry state, unique Entry identities, Methodology compatibility, and canonical relationship URLs.
Each current public Entry is projected from its selected snapshot. Editable canonical differences and historical snapshots are not input collections and cannot become separate export records.
The generator derives public values and applies the selected Dataset contract's ordering rules without mutating the release. Sources are reordered from a copy through the same helper used by release resolution.
It builds the Schema from the same validated site origin used by the release. The Schema $id and dataset $schema must be the same absolute canonical URL.
The dataset is serialized with two-space indentation and exactly one final newline, parsed again, and validated with Ajv in strict draft-2020-12 mode.
The application export boundary repeats generation and rejects any path or byte difference for the same in-memory release.
A successful artifact can feed the static Export JSON endpoint or the dataset artifact writer. Generation itself performs no filesystem, environment, clock, randomness, or logging work.

Deterministic Public Ordering

The public dataset uses explicit ordering so unchanged release inputs produce byte-identical output:

Entries sort by slug.
Sources use the domain-owned public cascade: Primary Evidence, Independent Replication, Official Record, Strong Artifact, Context Source, and Media Report. Source title in English alphabetical order is the final fallback.
Domains and Evidence Types follow their controlled-value order.
Secondary Topic Trails sort by slug.
Methodology versions sort numerically by semantic version.
Caveats and arrays without a public sorting rule preserve their defined input order.

Source Role labels come from the exhaustive SOURCE_ROLE_LABELS map. Entry-level Evidence Types are derived from the unique Evidence Types on exported Sources; there is no authored Entry-level Evidence Type list.

Release resolution already supplies sources in this order for page consumers. Dataset generation defensively reapplies the shared copied-array helper rather than trusting caller order or maintaining a private comparator. These arrays are deterministic traversal structures; their position does not communicate recency, confidence, popularity, or a numeric ranking.

Schema Identity and Static Publication

The current versioned Schema is published at /schemas/vydex-dataset/2.0.0.json. Its root $schema identifies JSON Schema draft 2020-12, while its $id identifies the VyDex Schema at the validated production origin. The generated Dataset v2 uses that exact $id as its $schema value. The v1 Schema remains at /schemas/vydex-dataset/1.0.0.json for legacy releases and preserved immutable history.

The Dataset v2 Schema describes only the public Dataset 2.0.0 shape. It is not generated from the broader authored Entry schema. Required fields reject unknown properties; the v2 projection contains Summary, Reality Check, Context, Takeaway, Entry-specific assessment rationales, revision metadata, resolved relationships, and sources, and exports only current Entry v2 records. The legacy v1 Schema continues to describe its original claim, review, Entry State, Frontier Delta, details, significance, and caveat fields.

The Schema documents the current nullable meanings:

date_happened: null and date_disclosed: null mean the date is unknown.
next_check_date: null means no check is scheduled.
potential_significance_if_confirmed: null means the field is not applicable.

Astro prerenders the Schema route from the shared Schema generator. PUBLIC_SITE_ORIGIN is required so the build can create the absolute $id. Cloudflare Pages response metadata is defined in public/_headers; Astro's local static preview serves the .json file using its extension-based JSON media type.

Immutable Artifact Writing

The generated public artifact path is:

/datasets/releases/{release-id}/vydex-latest-entry-versions-v2-0-0-{YYYY-MM-DD}.json

The date comes from the first ten characters of the validated RFC 3339 UTC release_metadata.generated_at value. The derivation does not parse the timestamp through a local timezone and never reads the current clock, filesystem time, Git history, or build date. The Release ID directory identifies the immutable release, while the dated basename stays readable when downloaded.

The dataset artifact writer accepts an explicit output root, creates required parent directories, verifies that the resolved target remains inside that root, and creates the file exclusively. If the path already contains identical bytes, the writer returns an idempotent unchanged result. If the bytes differ, it returns immutable_artifact_collision and leaves the existing file untouched.

The current stable convenience path is /datasets/vydex-latest-entry-versions-v2-0-0.json. Generation returns a 302 descriptor that points to the current immutable v2 path. The v1 stable path /datasets/vydex-latest-entry-versions-v1-0-0.json remains the legacy alias for v1 output and archived release history. The writer does not create a mutable copy or a deployment redirect file; Repeatable Release Publication writes and verifies the selected descriptor in the staged Cloudflare _redirects file.

Failure Behavior

Generation returns structured blocking diagnostics. The v2 generator fails when the input is not a complete production release, release metadata or routes are inconsistent, a legacy Entry or duplicate public Entry is present, canonical URLs disagree with the route registry, the Entry's Methodology is not a v2 version, the Schema identity differs from the dataset reference, the Schema does not compile, or the serialized dataset fails Schema validation. The explicit v1 generator applies the corresponding v1-only checks and rejects Entry v2 records.

Application export preparation also fails when repeated generation changes the immutable path or bytes, Entry count disagrees with the generated Entry array, scope is not latest_entry_versions, generation time differs from the release descriptor, represented Methodology versions disagree with dataset metadata, or the filename, public path, or Schema path does not match the shared derived location. It returns structured diagnostics and no page model.

Filesystem emission reports unsafe_artifact_path, immutable_artifact_collision, or artifact_write_failed. It never resolves a collision by overwriting the existing immutable artifact. The release gate maps those diagnostics to affected generated surfaces, formats the private terminal report, and chooses the process exit code.

Internal Edge Cases

Schema and dataset identities are derived from the same validated origin rather than a hardcoded hostname.
Release IDs and timestamps are copied unchanged; generation never reads the clock or creates an ID.
The artifact date is sliced from an already validated UTC timestamp, so process and host timezones cannot change the filename.
Source Evidence Types are sorted before Entry-level Evidence Types are derived.
A byte-identical existing file is valid idempotent output, not a collision.
Symlinked or escaped output paths are rejected before existing bytes are accepted.
Object property order is constructed consistently for byte stability, but consumers must treat JSON object keys as unordered.

Cross-System Edge Cases

Canonical Records owns authored Entry and Source validation. Dataset generation narrows those records into a separate immutable public contract.
Publication Revisions owns snapshot history and revision activity. Dataset generation receives only the current snapshot selected by release construction.
Release Construction owns production validity, relationship resolution, canonical URLs, and route registration. Dataset generation rejects disagreements rather than rebuilding those decisions.
The domain-owned source-ordering module is shared with release resolution. Dataset generation may defensively reorder copied input, but it must not introduce a second role cascade or mutate resolved sources.
Static Application Foundation owns Astro publication, configured environment access, Cloudflare response metadata, pinned dependencies, and CI execution.
The Export JSON Page consumes the prepared current artifact and presentation model. It never parses generated bytes, reconstructs release metadata, or links to a mutable stable convenience path.
Repeatable Release Publication owns descriptor persistence, writer invocation, staged redirect emission, static verification, and local promotion. Hosted deployment and deployed-target verification remain outside both systems.

Invariants

Dataset Schema versioning remains separate from Methodology versioning; the current v2 pair is selected from the current Methodology while the v1 pair remains a legacy contract.
One current public Entry produces exactly one exported Entry.
Editable unpublished state, historical snapshots, and Removed Entries never appear as separate records.
Stable IDs represent relationships; labels, slugs, and URLs are derived from resolved records.
Schema $id equals dataset $schema and uses the release's validated HTTPS origin.
Identical releases, site origins, dependencies, and generator code produce byte-identical Schema and dataset text.
Identical validated release metadata and source records produce the same dated immutable path as well as identical bytes.
The immutable writer never overwrites different bytes.
Pure generation remains free of filesystem, environment, logging, clock, and randomness side effects.
Release resolution, Entry pages, and Dataset generation agree on public source order and preserve every source's attached fields.

Implementation Landmarks

src/domain/json-export-generation/ — Versioned v1/v2 public types, Schema construction, projection, validation, and serialization.
src/domain/source-ordering/ — Shared public Source Role and title ordering contract.
src/adapters/dataset-artifact-writer/ — Injected filesystem emission and collision protection.
src/adapters/application-export/ — Determinism checks and page-facing metadata validation over a generated artifact.
src/release/stage-one-release/ — Staged artifact writing, redirect emission, output verification, manifest inventory, and promotion.
src/pages/schemas/vydex-dataset/ — Thin static Schema publication route.
src/pages/datasets/releases/ — Thin prerendered immutable artifact endpoint.
src/domain/route-generation/ — Schema, stable-latest, and immutable artifact route ownership.
tests/domain/, tests/adapters/, and tests/browser/ — Contract, determinism, writer, and published Schema coverage.

Before Changing Dataset Generation

Check:

Whether the change is compatible with the immutable Dataset 2.0.0 contract or requires a new major dataset version, and whether legacy Dataset 1.0.0 bytes and routes remain readable.
Whether every public field is projected from the selected snapshot or resolved release relationship rather than editable authoring state.
Whether array ordering and property construction remain deterministic across supported Node.js environments.
Whether public source order still matches release resolution and preserves citation IDs, labels, Evidence Types, URLs, publishers, and used_for values.
Whether Schema $id, dataset $schema, route registration, and public publication still agree.
Whether nullable meanings, Source Role labels, Evidence Strength scores, and derived Evidence Types remain exact.
Whether two identical generations still produce identical bytes and exactly one final newline.
Whether filename derivation still uses only the validated UTC descriptor date and shared route function.
Whether the writer remains idempotent for identical bytes and refuses every different-byte collision.
Whether environment, filesystem, terminal, clock, or release-descriptor behavior is being introduced into the wrong boundary.