Skip to content
GraphForge

Dataset Integration

Status: Backlog — not shipped in v0.5.0. Canonical open-dataset catalogs and loaders (graphforge.datasets, SNAP / LDBC / NetworkRepository convenience APIs) are a planned extension, not part of the core product. Build graphs with the construction APIs or Cypher today. Pages below keep catalog and loading reference material for when that extension lands.

This reference describes the planned dataset catalogs and loading patterns for popular public graph repositories (experimentation, benchmarking, and learning).

Overview

Section titled “Overview”

When the extension ships, the dataset system is intended to:

  • Load pre-configured graph datasets with a single command
  • Explore standard benchmark datasets
  • Test queries on realistic data
  • Compare performance with other graph databases
  • Learn Cypher with meaningful examples

Planned catalog sources

Section titled “Planned catalog sources”

SNAP (Stanford Network Analysis Project)

Section titled “SNAP (Stanford Network Analysis Project)”

Real-world network datasets from Stanford, covering social networks, web graphs, citation networks, collaboration networks, and communication networks.

Use cases: Research, network analysis, graph algorithm development, academic projects

Example datasets:

  • snap-ego-facebook - Facebook social circles (4K nodes, 88K edges)
  • snap-email-enron - Enron email network (37K nodes, 184K edges)
  • snap-ca-astroph - Astrophysics collaboration (19K nodes, 198K edges)
  • snap-web-google - Google web graph (876K nodes, 5.1M edges)
  • snap-twitter-combined - Twitter social circles (81K nodes, 1.8M edges)

LDBC full suite

Section titled “LDBC full suite”

Official Graph Data Council / LDBC portfolio: SNB (Interactive + BI), Graphalytics, FinBench, and SPB — spec for datasets, workloads, and validation. Generators/drivers run in an external scale harness, not GraphForge core CI. Shared harness contract: Scale Evaluation.

Use cases: Standard DBMS and analytics workload completeness; SF → GSI crosswalk after load

NetworkRepository

Section titled “NetworkRepository”

Large collection of diverse network datasets.

Use cases: Network analysis, algorithm testing, research

Scale size ladder (Graph500 × GSI)

Section titled “Scale size ladder (Graph500 × GSI)”

Synthetic scale work uses two Graph500 tracks — not a separate dataset catalog page. Size labeling: GSI. Evaluation contract (Official + Derived tracks, first-fail, evidence schema): Scale Evaluation.

  1. Official Graph500 — standard parameters with normative ef=16 for GSI size notches / community comparability; progressive / first-fail in the harness.
  2. Graph500-derived SCALE×density matrix — parameterized edgefactor to hit GSI density tiers at feasible (usually XS) SCALEs; not official Graph500 submissions.

Both tracks execute in the external harness; this repo is spec only.

Quick Start

Section titled “Quick Start”

Loading a Dataset

Section titled “Loading a Dataset”
from graphforge import GraphForge
from graphforge.datasets import load_dataset


# Create a GraphForge instance
gf = GraphForge()


# Load a dataset by name
load_dataset(gf, "snap-ego-facebook")


# Query the loaded data
results = gf.execute("MATCH (n)-[r]->(m) RETURN n, r, m LIMIT 10")

Using the Convenience Method

Section titled “Using the Convenience Method”
from graphforge import GraphForge


# Load dataset during initialization
gf = GraphForge.from_dataset("snap-ego-facebook")


# Start querying immediately
results = gf.execute("MATCH (n) RETURN count(n) as node_count")

Listing Available Datasets

Section titled “Listing Available Datasets”
from graphforge.datasets import list_datasets


# Get all available datasets
datasets = list_datasets()


for ds in datasets:
    print(f"{ds.name}: {ds.description}")
    print(f"  Source: {ds.source}")
    print(f"  Nodes: {ds.nodes:,}, Edges: {ds.edges:,}")
    print(f"  Size: {ds.size_mb:.1f} MB")
    print()

Filtering by Source

Section titled “Filtering by Source”
from graphforge.datasets import list_datasets


# Get only SNAP datasets
snap_datasets = list_datasets(source="snap")


# Get only small datasets (< 10 MB)
small_datasets = [ds for ds in list_datasets() if ds.size_mb < 10]

CLI Usage

Section titled “CLI Usage”

GraphForge provides command-line tools for working with datasets:

Terminal window
# List all available datasets
graphforge list-datasets


# Show detailed information about a dataset
graphforge dataset-info snap-ego-facebook


# Load a dataset
graphforge load-dataset snap-ego-facebook


# List datasets by source
graphforge list-datasets --source snap


# Clear dataset cache
graphforge clear-cache

Dataset Caching

Section titled “Dataset Caching”

Datasets are automatically cached locally after the first download to improve load times:

  • Cache location: ~/.graphforge/datasets/
  • Cache behavior: Downloaded once, reused on subsequent loads
  • Cache management: Use graphforge clear-cache to remove cached datasets

Dataset Metadata

Section titled “Dataset Metadata”

Each dataset includes comprehensive metadata:

from graphforge.datasets import get_dataset_info


info = get_dataset_info("snap-ego-facebook")


print(f"Name: {info.name}")
print(f"Description: {info.description}")
print(f"Source: {info.source}")
print(f"URL: {info.url}")
print(f"Nodes: {info.nodes:,}")
print(f"Edges: {info.edges:,}")
print(f"Labels: {', '.join(info.labels)}")
print(f"Relationship Types: {', '.join(info.relationship_types)}")
print(f"Size: {info.size_mb:.1f} MB")
print(f"License: {info.license}")
print(f"Category: {info.category}")

Jupyter Notebook Integration

Section titled “Jupyter Notebook Integration”

Datasets work seamlessly in Jupyter notebooks:

# In a Jupyter notebook cell
from graphforge import GraphForge
from graphforge.datasets import load_dataset


gf = GraphForge()
load_dataset(gf, "snap-ego-facebook")


# Explore the data
gf.execute("MATCH (n) RETURN count(n) AS node_count")

Contributing Datasets

Section titled “Contributing Datasets”

To add a new dataset source or specific dataset:

  1. Create a loader in src/graphforge/datasets/
  2. Register the dataset in the registry
  3. Add tests in tests/integration/test_datasets.py
  4. Update documentation

See the development guide for details.

Troubleshooting

Section titled “Troubleshooting”

Download Failures

Section titled “Download Failures”

If a dataset download fails:

  • Check your internet connection
  • Try clearing the cache: graphforge clear-cache
  • Manually download from the source URL

Memory Issues

Section titled “Memory Issues”

Large datasets may require significant memory:

  • Start with smaller scale factors (e.g., LDBC SF0.001)
  • Use a machine with sufficient RAM
  • Consider a Parquet-backed project directory for large datasets

Import Errors

Section titled “Import Errors”

If dataset import fails:

  • Check the dataset format compatibility
  • Verify the source URL is accessible
  • Report issues on GitHub
Section titled “Related Documentation”

License Information

Section titled “License Information”

Each dataset has its own license. Always check the dataset metadata for licensing information before using in production or research.

Next Steps

Section titled “Next Steps”