Testing Strategy & Infrastructure

Reader-facing strategy for v0.5.0 / release-prep (ownership, layered gates, what counts as proof): ../engineering/TESTING.md. This page keeps command recipes, suite layout, and local tooling detail.

Overview

GraphForge checks engine behavior and binding surfaces through these suites:

Suite	Location	What it tests
Rust tests (cargo test)	crates/*/src/	Each Rust crate in isolation and integration
Binding smoke	crates/graphforge-bindings-{py,node}/tests/	Native package loading, Arrow handoff, error mapping, type marshalling, lifecycle
Rust API/TCK BDD	crates/graphforge-api/tests/bdd/, tests/tck/	Public API and openCypher behavior

The testing principles are the same for both:

1. Spec-driven correctness — openCypher semantics verified via TCK
2. Fast feedback loops — unit tests run in milliseconds
3. Hermetic tests — no shared state between tests
4. Deterministic behavior — tests pass or fail consistently

Coverage floors are enforced by the Coverage workflow on pushes to main, not uploaded from CI; see codecov-integration.md. The rules for classifying behavior tests and binding smoke checks are in the test coverage classification.

---

Rust Tests

Structure

Each crate contains unit tests inline with the source and integration tests in tests/:

crates/graphforge-cypher/
├── src/
│   ├── lexer.rs        # #[cfg(test)] inline unit tests
│   ├── parser.rs       # #[cfg(test)] inline unit tests
│   └── lib.rs
└── tests/
    └── parse_corpus.rs # end-to-end parse tests against golden corpus

Running

# All crates
cargo test --workspace

# One crate
cargo test -p graphforge-cypher

# With output
cargo test --workspace -- --nocapture

# Only doctests
cargo test --doc --workspace

The CI Gate Rust lane runs the workspace under nextest; its exact commands are in agent-environment.md.

v0.5.0 non-Cypher release conformance

Part of the v0.5.0 testing strategy: the openCypher TCK proves execute() language semantics, but it does not cover construction, lifecycle, checkpoints, analyst verbs, search, or the knowledge/epistemic surfaces. The checked-in tests/contracts/non-cypher-rust-surface.json manifest classifies every public Rust receiver method, all 96 analyst-verb registry entries, and the supported find/index modes and policies. New public methods or registry values must be classified and linked to deterministic evidence before merge.

Run the omission gate and bounded public-facade matrix with:

python3 scripts/ci/non-cypher-surface-gate.py
python3 scripts/ci/test-non-cypher-surface-gate.py
cargo test -p graphforge-api \
  --test public_lifecycle_conformance \
  --test algorithm_public_surface \
  --test search_public_surface

No workflow dispatches the inventory gate on its own; the Python binding test crates/graphforge-bindings-py/tests/non_cypher_release.py imports it. A green TCK run cannot substitute for the surface inventory itself.

Rust test example

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn node_scan_roundtrip() {
        let op = GraphOp::NodeScan { var: VarId(0), ty: TypeId(1) };
        let json = serde_json::to_string(&op).unwrap();
        let back: GraphOp = serde_json::from_str(&json).unwrap();
        assert_eq!(op, back);
    }
}

Parser differential corpus

The parser migration strategy requires differential testing between the Python LALR(1) parser and the LALRPOP Rust parser. The corpus lives in tests/parser_corpus/ and includes:

• Valid queries (from the TCK and real-world examples)
• Invalid queries (error recovery cases)
• Precedence edge cases
• Unicode identifiers
• Parameter syntax
• Comments

Differential tests run both parsers on the same input and assert AST parity:

cargo test -p graphforge-cypher -- differential

Differential traversal oracle

Ordinary builds of graphforge-rel and graphforge-exec do not expose a reference-mode switch. The independent relational fixed-hop oracle requires the non-default differential-testing feature, and the differential_traversal target declares it as a required feature, so a default workspace run does not build it. Run its three-way corpus explicitly:

cargo test -p graphforge-exec --features differential-testing --test differential_traversal

It verifies indexed and scan-built provider execution against the relational oracle, including physical-plan differences and result equality.

Python test and extension build modes

Python Rust tests are executables and must link to libpython. Distribution extensions resolve Python symbols when loaded by the interpreter and must not link libpython on Unix. The same ABI3 (Python 3.10+) contract applies in both modes.

Build	Link mode
Ordinary cargo test / cargo llvm-cov	Default features retain ABI3 and link libpython
maturin build / maturin develop	The binding’s pyproject.toml explicitly selects its extension-module feature

Do not enable pyo3/extension-module in an unconditional Cargo dependency or default feature. Cargo unifies dependency features across the workspace, so that would break the binding’s Rust tests and the ordinary workspace coverage command. Do not export PYO3_BUILD_EXTENSION_MODULE globally: PyO3 treats its presence as enabled, even if its value is 0. A manual Cargo distribution build must opt in with --features graphforge-bindings-py/extension-module.

For native Rust tests, select a supported interpreter with its shared development library available. On Linux, a managed interpreter’s library directory also needs to be visible to the runtime loader:

uv python install 3.12
python_test_exe="$(uv python find 3.12)"
python_test_libdir="$("$python_test_exe" -c 'import sysconfig; print(sysconfig.get_config_var("LIBDIR"))')"
PYO3_PYTHON="$python_test_exe" \
  LD_LIBRARY_PATH="$python_test_libdir${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" \
  CARGO_TARGET_DIR=target/python-tests \
  cargo test -p graphforge-bindings-py --lib

scripts/ci/python-build-mode-check.py inspects the actual default and packaging Cargo feature graphs. It runs in make check (via scripts/ci/repo-checks.sh) and the CI Lint job before Clippy. Test builds, a real native import, and inspection of the distribution artifact’s dynamic dependencies remain required evidence when changing linkage; a successful cdylib build alone does not prove the Rust-test executable links.

---

Python Tests

Test Categories

1. Unit Tests (tests/unit/)

Test individual components in isolation.

tests/unit/
├── parser/
├── planner/
├── executor/
├── storage/
├── algorithms/
├── search/
└── recipes/

Characteristics: no I/O, < 1 ms per test, ≥90% coverage target.

2. Integration Tests (tests/integration/)

Test full query pipeline (parse → plan → execute), persistence, transactions, and the Python API surface.

Characteristics: may use temporary databases, < 100 ms per test.

3. openCypher TCK Tests (tests/tck/)

Official openCypher Technology Compatibility Kit. 3,885 scenarios; 100% passing on main. TCK is a hard merge gate for the rust-core branch too.

tests/tck/
├── conftest.py
├── coverage_matrix.json
└── features/

4. Property-Based Tests (tests/property/)

Hypothesis-driven generative tests for value semantics, expression evaluation, and storage consistency invariants.

5. Performance Benchmarks (tests/benchmarks/)

Real-dataset benchmarks tracked over time. Not part of the standard CI run. Measurement authority, inventory, and validation commands are documented in benchmarking.md.

Pytest Configuration

[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = ["-ra", "--strict-markers", "--tb=short", "-v"]

markers = [
    "unit: unit tests (fast, isolated)",
    "integration: integration tests (may use I/O)",
    "tck: openCypher TCK compliance tests",
    "property: property-based tests",
    "benchmark: performance benchmarks",
    "slow: tests that take >1s",
]

Running Python Tests

# All tests
make test

# By category
make test-unit
make test-integration
make test-tck

# With coverage (core Rust ≥85%; Rust adapters ≥80%; wrappers ≥85%)
make coverage            # all surfaces + thresholds
make coverage-rust       # core + same-SHA native adapter ledger
make coverage-python     # pytest-cov on graphforge-bindings-py/python/graphforge
make coverage-node       # c8 on @curatelabs/graphforge lib/ (needs *.node)
make coverage-report     # open Python HTML report
make coverage-diff       # changed Python wrapper files only

# Parallel (4× faster for TCK)
pytest tests/ -n auto

Core Fixtures (tests/conftest.py)

@pytest.fixture
def db():
    """Fresh in-memory GraphForge instance."""
    return GraphForge()

@pytest.fixture
def tmp_db(tmp_path):
    """GraphForge instance backed by a temporary Parquet directory."""
    return GraphForge(str(tmp_path / "graph"))

---

Quality Gates

Coverage Requirements

Local make coverage instruments the shipped surfaces independently. The Rust run builds PyO3 and napi-rs once with LLVM instrumentation and executes the real Python and Node native acceptance suites against those exact artifacts. The command defaults to an isolated target under build/coverage-rust/; set CARGO_TARGET_DIR to choose a different isolated location.

Surface	Tool	Scope	Threshold
Core Rust	cargo llvm-cov	workspace Rust excluding binding adapters	≥85% lines
Python adapter Rust	cargo llvm-cov + native acceptance	graphforge-bindings-py Rust	≥80% lines
Node adapter Rust	cargo llvm-cov + native acceptance	graphforge-bindings-node Rust	≥80% lines
Python	pytest-cov	crates/graphforge-bindings-py/python/graphforge	≥85% lines; ≥90% patch on changed wrapper files
Node	c8	hand-written @curatelabs/graphforge lib/**/*.mjs	≥85% lines

build/coverage-rust/ledger.json is the machine-readable source of truth. It records the exact source SHA, toolchain, artifact hashes, profile counts, and separate core, adapter, and merged-workspace totals. summary.txt is the human view and lcov.info is the merged report. A high workspace average never substitutes for a missing profile or a failed adapter floor; stale, empty, wrong-artifact, and wrong-SHA evidence fails closed.

Override floors with COVERAGE_FAIL_UNDER_RUST, COVERAGE_FAIL_UNDER_RUST_PYTHON_ADAPTER, COVERAGE_FAIL_UNDER_RUST_NODE_ADAPTER, COVERAGE_FAIL_UNDER_PYTHON, and COVERAGE_FAIL_UNDER_NODE.

Required Checks (all PRs)

1. cargo clippy --workspace -- -D warnings — zero warnings
2. cargo test --workspace — all Rust tests pass
3. pytest -m unit — all Python unit tests pass
4. pytest -m integration — all Python integration tests pass
5. pytest -m tck — all non-skipped TCK scenarios pass
6. make coverage — coverage thresholds met
7. make lint and make type-check — zero issues

---

TCK Coverage Matrix

Maintain tests/tck/coverage_matrix.json:

{
  "tck_version": "2024.2",
  "features": {
    "Match1_Nodes": {
      "status": "supported",
      "scenarios": {
        "Match single node": "pass",
        "Match node with label": "pass"
      }
    },
    "Match3_VariableLength": {
      "status": "supported"
    }
  }
}

When the Rust core implements a feature, verify the corresponding TCK scenarios pass end-to-end before marking "status": "supported".

---

CI/CD

GitHub Actions (.github/workflows/test.yml) runs the applicable jobs on every PR. Key jobs: Lint (Cargo fmt/Clippy + Python quality + policy checks), Rust Tests (nextest over the workspace), Rust Harness, Doc, and Feature Tests, Python and Node Bindings (full binding suites on Linux), Windows Storage, and the required CI Gate aggregate. See .github/workflows/README.md for the full job list and local equivalents.

Build-lane measurement method (ADR 0048)

ADR 0048 rests on a comparison of the Bazel CI lane with a Cargo lane on the same runner class. The dispatch-only harness that produced the Cargo samples, .github/workflows/build-lane-measurement.yml, was retired when the comparison finished (#1663). This section records how to reproduce it and the digests that identify its inputs. The results (per-run tables, run ids, logs and artifact hashes) are on #1618 and are not kept in the repository.

Method.

• Runner and toolchain: blacksmith-4vcpu-ubuntu-2404, Rust 1.96.0, cargo-nextest 0.9.145, CARGO_INCREMENTAL=0, and the dev/test profile with debug information off. Debug assertions and overflow checks stay on.
• Concurrent variant: the three Rust gate commands in agent-environment.md, timed as one interval that includes compilation. cargo fetch is timed separately.
• Serial variant: cargo test --workspace --locked.
• Keep the aggregate exit status. Run every constituent step even when an earlier one fails.
• Record the cache state before each sample, from the actual contents of target/. A cache key or label does not prove the cache is populated. Compilation caching and test-result caching are different inputs.
• Compare only compatible boundaries. Compare successful runs over the same timing boundary. The test interval and the full job wall time are different quantities. Label cancelled runs and failed collectors explicitly, even if a test step passed.
• Check cohort ancestry. Verify each main-history SHA with git merge-base --is-ancestor <measured-sha> <frozen-main-sha>. An event label does not establish ancestry.

Harness identity. A harness commit identifies the workflow, not the measured tree. Check out the measured SHA separately. To reproduce, run git show <harness-commit>:.github/workflows/build-lane-measurement.yml and check its SHA-256:

Harness commit	Workflow SHA-256
d05a1fcee10ca2a07cb75423a35954ba66d2cdc0	3d660ea31df1e5461cba0c22397cb22d878b8895044dab4fe3170af65d88ed82
6363d0c5b682347d7a53feab44975264c43bf2fe	816352ea82fc76914dcc228052253237781875c396c09c6094e2b762fd8b2ed8
3ba0ccf40c03140d8dbf0d3c96e03f9b1c7afaea	816352ea82fc76914dcc228052253237781875c396c09c6094e2b762fd8b2ed8
5bbf944e7204c54504f67236ba936e544945c345	a08ac97aeae44d5eb682db9576901fcdac934338090bfeb133193bf95ecc7978
59e9165970d08aeb20e94b6fd63dfa5bd3e9497b	d5d1d3ee5079d5018be1c74e3d47bccb2e6d9f7046d9bb347ef871fe08aa4763
1cf4548608a6d441808146b4268660f55f89d4d1	ee91e8d7bbfb14a9946bf2be494882ea49e3dbcb160b8e20525ea6244315a1ac
4ce5649c51bc9684eb9a83cc945a191d7a3d56ca	ee91e8d7bbfb14a9946bf2be494882ea49e3dbcb160b8e20525ea6244315a1ac
29a7b34ebe441a85ffb9274164d58aaeeb68dc8a	38d95bbe61096b904e5c16221d8f3e6a28235929109dacd1a564b18e843783d0

Samples from 5bbf944e failed before running tests: nextest could not list a custom-harness target. Samples from 59e91659 recompiled the custom-harness targets under per-package features. #1618 qualifies both groups.

Cohort identity. The twenty measured SHAs (ten main-history, ten pull request) and their Cargo.lock inputs have canonical digest 1661bfc7d281b6abcbc5edff33041f97502996d0492f80c888beb1cd174ab1d1. It is the SHA-256 of UTF-8 JSON for the ordered records {"cargo_lock_sha256", "lane", "measured_sha"}, with sorted keys and compact separators (, and :). It identifies inputs only and contains no timing or pass/fail results.

---

Known Issues

pytest-xdist + pytest-cov deadlock on macOS / Python 3.13

Symptom: make coverage hangs at the end of the test run — progress reaches ~100% then freezes. CPU drops to 0%. Only kill escapes it.

Root cause: pytest-cov collects coverage data from xdist workers via IPC sockets. When workers close their sockets, a coverage collection thread in the main process blocks on read(), deadlocking with the main thread. Reproduced on macOS (Darwin 25.x) + Python 3.13 + pytest-cov 7.0.0 + pytest-xdist 3.x.

Solution (current Makefile): Run coverage serially, skipping SNAP tests:

coverage:
    uv run pytest tests/unit tests/integration -m "not snap" \
        --cov=src --cov-branch \
        --cov-report=term-missing --cov-report=xml

The serial run is ~60 s slower than the parallel baseline but avoids the deadlock. If the upstream pytest-cov / pytest-xdist fix lands, re-evaluate.

---

References

• pytest documentation
• openCypher TCK
• Hypothesis documentation
• cargo test documentation