Concurrency and recovery contract

Normative acknowledgement, filesystem scope, isolation tables, and anomaly classification live in ADR 0018. The machine- readable coverage matrix is tests/contracts/durability-isolation-matrix.json (graphforge-durability-isolation/1). This page is the architecture narrative that reconciles ADR 0013 publication, ADR 0014 checkpoints, and ADR 0015 write modes with that frozen public contract.

GraphForge supports concurrent, read-only operations on one facade, across facades in one process, and across sessions opened on the same project. Results are complete, canonically ordered Arrow data. A facade pins the generation it opened: a long-lived reader does not follow later commits; a newly opened facade does.

Dropping a stream or cancelling one request affects only that operation. A peer continues to a complete result, while cooperative cancellation reports GF_CANCELLED.

Acknowledged-durable writes

Caller-visible success means the write is acknowledged-durable only after participant and manifest file flushes, generation-tree platform-native namespace durability barriers, atomic CURRENT replacement or creation, and the project-root platform-native namespace durability barrier. Atomic CURRENT replacement is the visibility linearization point for new readers; acknowledgement against power loss additionally requires that final namespace barrier. Journals never select authority.

POSIX supplies the namespace barrier with directory fsync(2). Windows support is limited to fixed writable local NTFS whose storage honestly honors write-through completion: GraphForge flushes a FILE_FLAG_WRITE_THROUGH staging handle and renames through that same handle with SetFileInformationByHandle. Directory FlushFileBuffers is not claimed as a durability barrier, and ReFS remains unsupported/unproven. See ADR 0020.

A persistent project filesystem may be mounted beneath a different container or VM root filesystem. GraphForge admits the project parent as its storage boundary and requires the project root and durable contents to remain on that filesystem. Network, userspace, removable, symlink-mediated, ReFS, and unknown project filesystems return GF_UNSUPPORTED_FILESYSTEM before the project root or CURRENT changes. There is no best-effort durability mode.

Every durable lifecycle enters one Rust-owned parent-scoped admission before project mutation. A deterministic sibling lock file persists across unlock and process death; it coordinates absent-root initialization before FORMAT, generations/, project locks, or CURRENT exist. Existing roots retain opened parent/root identities. Ordinary publication, recovery, checkpoints and revert, delta publication and compaction spill, retention cleanup, portable import, and repository state initialization/removal all use that authority. Optimistic staging retains identity without serializing peer stagers, then readmits the same root before acquiring the commit writer lock. Python, Node, and CLI calls remain thin adapters over these Rust outcomes, including GF_UNSUPPORTED_FILESYSTEM.

Recovery resolves an exact valid CURRENT only. Journals and directory scans are advisory cleanup input. Corrupt or ambiguous pointers fail closed as GF_PROJECT_CORRUPT without electing a newest generation.

Mutable graph-workspace rewrites have a second, subordinate recovery boundary. One exclusive .graphforge-rewrite.lock protects the exact admitted project root while the engine derives prior/next topology and search generations, installs a bounded batch, and switches topology/generation.json last. The authenticated intent binds the retained root and parent-directory identities, every canonical staged/final basename, temporary identity, byte length and digest, and exactly one bounded generation-authority record. It cannot publish a project generation; CURRENT remains the sole project authority.

A preparing intent is safe only for identity-matched temporary cleanup. A durable intent is an unconditional roll-forward obligation: recovery accepts only the exact prior or next generation pair, authenticates an already-installed destination or its retained temporary, installs all data first, revalidates the root and named lock, and publishes generation authority last. Missing, substituted, truncated, non-canonical, cross-root, duplicate, or generation-divergent state fails closed without guessing or deleting ambiguous evidence; the named lock also has a single-link invariant. Reopen is idempotent before, during, and after the authority switch. The journal is bounded to 16,384 entries and 8 MiB, with a 4 KiB generation authority.

The same intent can bind one typed auxiliary receipt to one exact staged data entry by schema kind/version, length, and digest. #931 must use that participation point for its authenticated UUID-to-surrogate manifest/runs, ensuring topology and lookup authority recover together rather than through a later best-effort repair.

Write modes and isolation

Project mutation has three explicit embedded modes. Every mode gives readers pinned immutable snapshot isolation. The modes do not claim generic ACID, serializable isolation, or SSI.

Mode	Writer semantics	Isolation / conflict table
single_writer (default)	Competing writers receive GF_WRITER_BUSY before staging or publication	One serial writer; no concurrent publish races
queued_writer	Bounded FIFO per facade; snapshot reads bypass the queue; cancellation removes only unstarted work	One serial writer after dequeue; queue-limit and cancel errors are structured
optimistic_multi_writer	Distinct composite transaction identities may stage concurrently; only the CURRENT commit point is serialized; compatible work may rebase	Closed merge/conflict matrix in ADR 0015: merge, GF_WRITE_CONFLICT, GF_IDEMPOTENCY_CONFLICT, or GF_REBASE_EXHAUSTED

Optimistic merge rules are deliberately finite. Distinct creates and immutable ledger identities merge. Changes to different properties of the same graph object merge. Reusing an identity with changed content, changing the same property, losing a mutation target, and delete or administrative work do not merge. Only the composite publication API and the uniform GraphTransaction lifecycle are replayed optimistically in v0.5.x; other one-shot mutation APIs keep their established single-writer behavior even when the facade selects optimistic mode, unless staged through that lifecycle.

Scalar construction (add_edge) acquires the same graph visibility / write-admission coordinator as Cypher writes, bulk publication, and other mutations before selecting a mutation snapshot or touching the workspace. Endpoint validation, surrogate allocation, flush, publication, and rollback stay inside that one coherent transaction so concurrent same-instance callers cannot interleave partial edges or overlapping surrogates.

These modes coordinate embedded callers directly against the project directory. GraphForge core does not include an MCP or HTTP server. A separately packaged extension may expose one authenticated remote authority without changing the storage engine into a distributed database.

Write-skew witness

Optimistic mode is not SSI. Because different properties of one object may merge, the following history is admitted:

1. Object Account starts with credit=0, debit=0.
2. T1 reads both fields, sees debit=0, and stages credit=1.
3. T2 concurrently reads both fields, sees credit=0, and stages debit=1.
4. Both publish after rebase under optimistic_multi_writer.
5. The committed generation can contain credit=1 and debit=1, breaking an application invariant credit + debit <= 1 that neither writer observed the other violate.

Docs therefore classify optimistic mode as optimistic snapshot / conflict semantics. Preventing write-skew requires a separately approved SSI design outside Milestone 6.

Recovery and lifecycle

Recovery after a writer is killed selects either the previous complete generation or the newly published complete generation according to the durable publication phase. Graph, provenance, knowledge-layer, and epistemic state move together; mixed generations are unsupported and treated as corruption.

Within a private mutable graph workspace, the equivalent finite rule is: before durable rewrite intent the prior topology/search generation remains selected; at or after durable intent recovery rolls every staged destination forward and installs topology/generation.json last. That internal authority is fully recovered before the workspace can participate in a CURRENT publication.

Exact retry after acknowledgement returns the prior receipt without restaging. Same-identity content changes return GF_IDEMPOTENCY_CONFLICT. Failures before CURRENT replacement report committed: false. Failures after replacement reread validated CURRENT and report committed: true; the generation is not rolled back.

Import/export and other interchange surfaces that publish a generation reuse the same publication vocabulary: stage, validate, durable generation, linearize, acknowledge, publish, abort, and recover. See ADR 0018 and the repository integration guide.

Property snapshot publication and session isolation

Property publication stages authenticated full-map snapshots for the UUIDs changed in one write window. A generation may contain multiple ordinal fragments, but publication exposes the set atomically: the catalog replaces one immutable authority value containing both the generation UUID and its validated inventory. It never updates a provider list and inventory in separate steps.

Sessions and already-planned queries pin the prior immutable authority value. New sessions observe the replacement after the publication linearization point. Admission authenticates every fragment named by the selected graph-files inventory. Logical reads merge them by UUID and descending canonical fragment identity, so a newer full-map row or tombstone overrides that UUID only and an unchanged UUID remains live in its older immutable fragment. A partial or failed publication cannot enter the selected inventory or mix uncommitted ordinals, schemas, or metadata with the prior authority. The same immutable value authenticates the newest per-route live-schema summary. A write window applies old-complete-map to new-complete-map count deltas for touched UUIDs and publishes the resulting summary with its fragment. Later operations in that window consume the staged summary, so SET/REMOVE order cannot lose a newly introduced key or retain a last-owner deletion. Readers never combine a new fragment with an old summary; prior sessions retain both the prior fragments and prior logical schema.

Mutation windows also pin the authority generation used to resolve complete property maps. Commit rechecks that precondition while holding the publication lock. If another generation won first, the stale window fails with a typed conflict before any candidate becomes visible; it is never silently rebased or allowed to overwrite keys from the winner.

Before linearization, failure removes staged property files and leaves the catalog authority unchanged. After linearization, recovery selects the newly published complete generation through the ordinary durable publication rules; it does not rebuild authority by scanning filenames. Replayed property deltas publish new snapshots only for affected UUIDs and routes. Entity deletion emits authoritative tombstones on every authenticated route for that entity kind. Mutation preparation may use one bounded authenticated lookup for the touched UUID set; sealing does not reread historical fragments, and publication never rewrites them.

Correctness gates versus stress observations

There are four CI surfaces for concurrency and durability contracts:

1. Required concurrency tests — the deterministic Rust cases run in Test Suite / Rust Tests, and the Python and Node parity cases run with the full binding suites in Test Suite / Python and Node Bindings. Barriers/channels/failpoints coordinate interleavings. Timing sleeps, ignored tests, and probabilistic retries are forbidden. These tests run whenever Rust or binding surfaces change and must stay green for merge.
2. Scheduled/manual stress lane (Concurrency Stress Gate) — longer mixed workload with the published seed 2417, resource bounds (RSS and file descriptors), and cleanup checks for locks/staging. Stress retries are diagnostic only; they cannot turn a failed required short matrix green. Throughput or latency figures from stress are non-blocking performance observations, not correctness evidence.
3. Durability/isolation contract ledger — graphforge-durability-isolation/1 maps crash phases and anomalies to covered evidence. It is a reference ledger: the scheduled durability certification gate checks only its seeded-certification and write-skew rows. Persistent-media faults that process kill cannot express (torn CURRENT / manifest bytes, lost platform-native namespace durability barrier power-loss subsets) are modeled by the deterministic filesystem fault oracle in crates/graphforge-storage/src/project_fault_oracle.rs. Native POSIX and Windows subprocess failpoint matrices remain required for real API and handle behavior; the oracle is reusable by recovery, delta, compaction, and final certification. Authoritative graph delta runs (#752 / ADR 0019) publish only through the same CURRENT contract; they never recover by scanning newest logs. GFDR supports property operations only. Request admission precedes idempotent transaction shortcuts; persisted decoding and direct-run prevalidation reject topology records. Journal recovery repairs selected or abandoned publication attempts and never applies an arbitrary uncommitted topology record to CURRENT. See the publishing boundary. After CURRENT selects a generation, normal and checkpoint opens verify its inventory and complete GFDR chain, replay canonical Parquet plus typed records inside a private workspace, and publish no partial read view on corruption, unsupported versions, or resource-limit failure. Recovery-on-open (open_or_initialize_project_with_recovery / facade open) runs bounded inspection and idempotent cleanup when locks are free, and defers cleanup without blocking a valid CURRENT snapshot when a live writer holds them. Bounded snapshot retention and orphan GC (inspect_project_reachability, preview_project_cleanup, execute_project_cleanup) reuse the same verified reachability oracle as recovery: CURRENT, configured ancestors, and checkpoint roots. Live leases skip without waiting; concurrent publication returns GF_WRITER_BUSY; unknown or linked paths are quarantined and never deleted. Generation-entry classification and project graph-object sweeping have separate evidence. skipped_live counts only generation entries. The graph-object sweep reports whether it completed or did not run because the request was a dry run, generation cleanup remained bounded, the GC guard was busy, a graph-object publication was live, or a project publication attempt remained active. Completed sweeps also report objects marked, objects removed, and physical bytes removed; deferred sweeps retain objects and report zero physical-work counters. Graph delta compaction (compact_graph_delta / preview_graph_delta_compaction) publishes a new Parquet generation through the same CURRENT path and reclaims subsumed inputs only after acknowledgement via that shared oracle.
4. Seeded durability/isolation certification (Durability Certification Gate) — independent reference model in crates/graphforge-storage/src/project_certification.rs, compared at each certified boundary with production graphforge_api::GraphForge opens, transactions, queries, checkpoints, reachability, cleanup, delta replay, and compaction. Rust, Python, Node, and CLI probes normalize the same generation, query-state, checkpoint-root, and recovery observations; bindings never own a second model or storage implementation. (graphforge-durability-certification/1, published seed 7560). Required CI runs the bounded history budget; the scheduled/manual lane raises GRAPHFORGE_CERT_HISTORIES / GRAPHFORGE_CERT_OPS, records declared counts, minimized traces, commands, exact commit, runner/filesystem class, platform/tool versions, and artifact digests, and fails closed on the first untriaged invariant (no seed retries). Evidence and docs make no SSI, universal-filesystem, or distributed-durability claim; optimistic write-skew remains allowed_documented_not_ssi.

The deterministic fault oracle and native Linux/Windows subprocess kill lanes remain the process-death and persistent-media authority; the production history runner does not model a successful API call as proof of a process crash. M6 CPU-simulation, durable walltime, and peak RSS fallback evidence use the frozen storage-benchmarks-v1 / storage_io fixture contract documented in Benchmarking.

Local seeded certification (required budget):

python3 scripts/ci/durability-certification-gate.py validate
python3 scripts/ci/durability-certification-gate.py run \
  --output /tmp/gf-durability-cert

Scheduled-lane reproduction:

GRAPHFORGE_CERT_HISTORIES=64 GRAPHFORGE_CERT_OPS=32 \
  python3 scripts/ci/durability-certification-gate.py run \
  --histories 64 --ops 32 \
  --output /tmp/gf-durability-cert-scheduled

Local stress reproduction from an artifact uses the recorded command lines in reproduction.txt, or:

python3 scripts/ci/concurrency-stress-gate.py run \
  --seed 2417 \
  --iterations 24 \
  --output /tmp/gf-concurrency-stress