Project format compatibility before v1.0

Status: Accepted
Decision date: 2026-07-24

GraphForge is pre-v1. Project storage is not backward compatible before the v1.0 format freeze.

Contract

• GraphForge opens only a supported current-format project container and committed generation; the product version is distinct from its format marker.
• Any other on-disk layout is an unsupported input.
• GraphForge does not provide an importer, migration command/API, in-place upgrade, read-only compatibility view, generation-zero interpretation, identity conversion, data-loss report, rollback workflow, or fixture-retention promise for unsupported layouts.
• Unsupported inputs return GF_UNSUPPORTED_PROJECT_FORMAT before any file is created, removed, renamed, rewritten, or opened as graph data.
• Detection distinguishes only a valid supported container, a recognized unsupported pre-v1 container, and malformed input. It never decodes unrecognized graph records or guesses a format from partial files.
• Data that is not already in a supported project must be re-created through the current public construction or ingest surface.

Within a supported current-format generation, base-only canonical Parquet remains readable. A legacy single canonical property Parquet snapshot remains the oldest (0, 0) authority when later full-snapshot-v1 immutable fragments are published; it is authenticated and retained rather than rewritten or silently deleted. Typed GFDR records use the current versioned framing and typed value encoding. The superseded prototype’s routing-free topology/property payloads and plain-string property representation are not losslessly migratable and are rejected with GF_UNSUPPORTED_PROJECT_FORMAT; GraphForge never silently reinterprets them.

GFDR is therefore a permanent, versioned exception to the default Parquet graph-data rule, not an unversioned sidecar. Its exact magic, run/record/schema markers, authoritative JSON payload contract, and migration debt are recorded in ADR 0024. The same ADR records compiled ontology Parquet as a different exception: a derived runtime snapshot that never replaces the CURRENT-selected workspace’s canonical ontology JSON and may be discarded and recompiled when incompatible or stale. External YAML or JSON files remain authoring/import inputs rather than project commit authority.

Portable selection is dependency-closed for graph semantic bindings. When the selected participant set includes the graph semantic-bindings participant, the bundle must also contain the exact workspace ontology-composition participant whose fingerprint those bindings name. Import authenticates and reconstructs the composition, compares the fingerprint, and stages composition plus bindings in the same generation. A missing or mismatched authority fails before publication; import never drops the bindings or fabricates composition state.

Shared value contract extraction for v0.6.0

ADR 0025 separates compiler IR versioning from persisted ID/value/catalog schemas. The current container marker is graphforge-project/v1\n, distinct from the product version. IrVersion is result metadata, not a project-open compatibility gate. The shared value contract preserves current integer, Arrow and serde encodings; it introduces no automatic migration or mandatory new metadata for existing supported projects. Unknown versions and malformed values must fail through shared checked decoders without rewriting committed authority. The ADR identifies the catalog/value admission and durable reopen tests.

Knowledge-layer implementation boundary

The knowledge layer uses the current graph and project contracts. Internal layout descriptions below are not a promise to support projects produced by older engine versions.

• removes obsolete graph-embedded confidence/provenance columns, execution options, and schemas. It does not transform unsupported project layouts or fabricate knowledge records from unrecognized fields.
• resolves only versioned current-format committed generations. No unsupported root is a readable snapshot or generation zero.
• initializes the current-format capability manifest. It does not upgrade unrecognized manifests.
• guarantees atomicity and recovery for current-format generations only.
• exposes no migration API in Rust, Python, or Node.
• proves deterministic creation, publication, recovery, reopen, and binding parity for current-format projects only.
• the current release qualification requires clean current-format installs and project creation/reopen. It does not require upgrade evidence from unsupported artifacts.
• named checkpoints and complete-workspace revert operate only within the current-format generation protocol. They do not recognize, migrate, or import unsupported layouts. The frozen contract is ADR 0014.

Committed-generation resolution

graphforge-storage resolves a project by reading the exact FORMAT marker, the canonical bounded CURRENT record, and the manifest of the UUID named by that record. It never elects a generation by directory enumeration. The manifest digest must match CURRENT, and machine-owned paths must remain normalized, contained, and free of links.

The resolved generation and its lease-file handle remain owned by the opened GraphForge instance. Changing CURRENT therefore affects only later opens; an existing instance never follows the pointer again. Abandoned generation and transaction directories are neither consulted nor cleaned up by the read-side resolver.

An explicitly empty directory may be initialized as a new current-format project. A non-empty directory without the exact supported marker is never initialized or modified.

Atomic generation publication

graphforge-storage stages a complete participant set beneath one private generation UUID while holding the non-blocking OS writer lock. Participant paths are machine-derived from bounded lowercase capability and record-family IDs. Every request is a complete replacement snapshot: callers must explicitly carry forward unchanged parent participants, and an omitted participant is absent from the child generation. Publication never silently merges staged records with the parent. Before publication, every file is closed, flushed, reread, and checked against its exact byte length and SHA-256; domain-owning crates then validate schemas and values through opaque callbacks, followed by composite reference validation against the pinned parent plus staged records.

The generation manifest and lease are flushed only after validation. CURRENT is replaced once, using a same-filesystem atomic replacement, after the participant directories and generation directory are durable. That pointer replacement is the only commit point. A pre-publication error leaves the parent authoritative, while an error after replacement reports committed=true.

Transaction journals use the frozen PREPARING -> STAGED -> VALIDATED -> DURABLE -> PUBLISHED phases but never select readable state. Replaying a published transaction UUID with identical generation inputs is idempotent even after later publications; reusing it with different inputs returns GF_IDEMPOTENCY_CONFLICT. A held writer lock returns GF_WRITER_BUSY immediately.

The protocol is limited to one local filesystem with one writer and multiple snapshot readers. Distributed transactions, object-store consensus, and caller-managed public transactions are outside the current-format contract.

Interrupted transaction recovery

Ordinary project open runs recovery as a first-class lifecycle step. open_or_initialize_project_with_recovery() (and therefore GraphForge::new / directory open) first resolves the exact committed generation, then uses bounded, link-safe inspection of journals, atomic temps, and trash. When candidate work exists it acquires locks/writer.lock non-blockingly in the established writer/transaction/checkpoint order, recovers idempotently, re-resolves CURRENT, and opens that pinned generation. Successful kernel lock acquisition is the only evidence that an earlier writer is gone. PID, owner text, timestamps, heartbeats, and file age never permit lock stealing.

When a live writer already owns the recovery locks, open still pins the validated CURRENT generation and reports recovery as deferred (live_writer_owns_kernel_lock) instead of failing the snapshot open or stealing locks. Explicit recover_project_transactions() retains the GF_WRITER_BUSY contract for callers that require exclusive cleanup.

While holding the recovery lock, cleanup classifies canonical transaction journals in bounded lexical order. A journal whose generation and manifest digest match CURRENT is repaired to PUBLISHED. A non-published journal for any other generation is marked ABORTED; its UUID-named private generation is removed only after the current generation and its two verified ancestors are retained, an exclusive generation lease is acquired when present, and CURRENT is rechecked. Published historical journals remain published. Missing journals never affect authority, while a torn or ambiguous journal returns GF_PROJECT_CORRUPT with restore guidance and preserves the evidence.

Cleanup atomically moves an authorized abandoned generation into root trash/, flushes both directories, then deletes and flushes the trash entry. A crash at either cleanup boundary is idempotently completed by the next recovery pass. Unknown, linked, noncanonical, or journal-less generation entries are counted and preserved rather than traversed or guessed about.

Initialization of an empty or resumable uninitialized container and read-only checkpoint opens remain semantically distinct: they do not run the recovery cleanup pass. Safe open evidence exposes selected generation class (committed_current or checkpoint_pinned), repaired/aborted journal counts, removed/quarantined entries, deferral reason, and elapsed phase without payload data or sensitive paths.

First-class retention and orphan garbage collection (inspect_project_reachability / preview_project_cleanup / execute_project_cleanup) share recovery’s verified reachability oracle under the same writer/checkpoint lock order. Dry-run and execute classify with that oracle before any move. CURRENT, retained ancestors, checkpoint roots, and live leases are never selected. Reports reconcile candidates, removed, skipped-live, quarantined, unknown, and remaining bytes within configured entry/byte/work bounds. Crash at project.after_gc_move / project.after_gc_delete remains idempotent on the next cleanup or recovery pass.

New-container interruption after FORMAT is durable leaves a supported uninitialized container. A later open_or_initialize_project() resumes that current-format initialization under the existing exclusive root lock. This is not migration or historical-format inference: only the exact current-format FORMAT contract is resumable.

The named failpoint suite terminates subprocess writers at every frozen container, staging, validation, durability, pointer, journal, and cleanup boundary. Each reopen asserts the selected UUID, manifest digest, participant bytes, commit-state envelope, kernel-lock release, and repeatable recovery. Graph-only resolution and recovery read no knowledge participant bytes, so a corrupt or future knowledge payload cannot block a graph-only operation after its committed manifest has been selected.

Versioned capability manifest

Every committed current-format generation manifest declares a canonical, strictly ID-ordered capability list. graph@1 and workspace@1 are mandatory from the first empty generation. workspace@1 explicitly records ontology absence and authoritative configuration. provenance@1 and knowledge@1 may be added through one complete generation publication; epistemic is reserved for epistemic. Capabilities are never inferred from directories or participant files.

project_capabilities() resolves only FORMAT, CURRENT, and the selected manifest. It reports unknown IDs or future versions as unsupported_future without opening their participants. A capability-specific API must separately require its supported version before opening any of its record families.

enable_capability() carries every verified existing participant into a complete replacement generation, adds the requested declaration and its registered initial participants, and publishes through the same writer lock, journal, durability, and recovery protocol. Its generation UUID is a deterministic UUIDv8 projection of the operation UUID and canonical request. The same operation/request is idempotent; conflicting versions and knowledge-layer attempts to enable epistemic fail before publication.

The capability declaration and participant inventory are separate manifest fields. A supported capability may initially have zero registered participants; that is an explicit empty ledger, not directory inference. As each owning domain schema lands, its implementation registers canonical empty participants for new enable operations and treats absent record families in an already enabled capability as empty. A present but corrupt or unsupported participant is never treated as empty.

Graph record families under graph@1

graph@1 remains the mandatory capability version. Two mutually exclusive record families may appear:

• snapshot (Arrow IPC) — legacy whole-workspace envelope (1 GiB/file, 2 GiB total). Existing projects remain readable through hydrate.
• files (canonical JSON inventory) — file-backed contract. Exact workspace files live under the generation-owned graph/ directory beside participants/. Open validates the inventory and either pins that tree (read-only) or materializes file-by-file into a private workspace. New publications use this path. For full-snapshot-v1 property fragments, open authenticates every canonical generation and ordinal in this inventory, then merges all fragments by UUID and descending (generation, ordinal) identity. Each newer row is a complete map or tombstone for that UUID; unchanged UUIDs remain live in older immutable fragments. New fragments authenticate an exact per-route live-key count summary. The newest summary selects the logical schema without scanning total rows/history; zero-owner historical keys read as Arrow Null, while old physical fields remain decodable. A missing summary denotes a legacy route and preserves its historical schema union rather than guessing from a targeted mutation. Unsupported inventory versions return GF_UNSUPPORTED_PROJECT_FORMAT.

Move supported current-format projects using the portable project workflow, which verifies and imports the selected package. Do not copy live project storage or bypass an unsupported-format refusal. Checkpoint inspection is separate from interchange.

There is no capability migration from unsupported layouts. A boolean manifest, missing current-format FORMAT, or any other pre-v1 root returns GF_UNSUPPORTED_PROJECT_FORMAT before mutation.

Security and diagnostics

Unsupported input is untrusted. The detector validates path containment and reads only the minimum bounded header/manifest bytes needed to return a structured classification. Diagnostics may contain the expected format version, detected format family/version when safely available, and sanitized error code. They never contain graph values, user content, unrestricted paths, or historical database records.

Last writer’s Core version

A generation manifest may carry producing_core_version: the exact Core version that committed that generation, so it names the project’s last writer. Every commit stamps it. The field is optional and the manifest version stays 2, so a generation written before the stamp existed still opens, and carries no version. Manifests still reject unknown fields, so an older Core refuses a newer manifest that adds one.

Opening a project passes the stamp, with the portable import and clone declarations, through one check point (graphforge_core::core_version), which #1936 added. A different version yields the warning GF_CORE_VERSION_MISMATCH, which names both versions and how to get a matching runtime; an absent stamp and a matching one yield none. The check never refuses: unreadable data fails through the format checks above. A future refusal or fix-up policy plugs in at the same check point.

Future compatibility

This decision does not define the v1.0 compatibility policy. A future format freeze requires its own versioning, support-window, and upgrade decision. No pre-v1 behavior becomes an implicit precedent.