Data Loading Tools and Bulk Ingestion

Neo4j 5.x exposes four distinct mechanisms for moving records into a graph, and picking the wrong one is the single most expensive early decision in a migration. An offline bulk importer can build a billion-relationship graph in an hour but refuses to touch a database that already holds data; a driver-side UNWIND batch can transform and validate every field in Python but will never match the raw throughput of a file-based build. The mechanisms are not interchangeable — each trades a different axis of downtime tolerance, source flexibility, and per-record transform power. This guide gives platform teams and Python engineers a decision framework for matching the ingestion tool to the job, then works through a concrete example of each so the trade-offs are grounded in running code rather than benchmarks in isolation. It is a stage inside the broader Automated Data Migration from Relational & JSON Sources reference, and it presumes you have already tuned the target instance for a cold load.

Prerequisite Concepts

Choosing an ingestion mechanism is a downstream decision — it depends on how the target is configured and how the source has been shaped upstream. Before applying the decision table below, the reader should be comfortable with:

• The end-to-end migration contract. Bulk loading is one phase of the extract → map → validate → load → verify flow defined by the parent Automated Data Migration from Relational & JSON Sources reference. The tool you pick has to fit the stages on either side of it.
• Cold-load tuning. The throughput ceiling of every online loader is set by heap, page cache, and index lifecycle. Size those first against Initial Load Performance Tuning; an under-provisioned instance turns a tool choice into a bottleneck no batching can hide.
• A stable target schema. Every loader here writes against fixed labels, relationship types, and uniqueness constraints. If the model is still moving, stabilise it against Neo4j Graph Schema Design & Architecture before committing to a load strategy.
• Idempotent write semantics. Any online loader that might run more than once must be MERGE-based and constraint-backed. That discipline is defined in Error Handling & Rollback Mechanisms and is assumed throughout the online examples below.

Conceptual Model

The four mechanisms partition cleanly along two questions: is the target database empty, and does the source need per-record transformation in application code? The answers route you to exactly one tool. The offline importer sits apart because it bypasses the transactional layer entirely — it writes store files directly and therefore only exists for the first, empty-database load.

Design Rules: Matching Volume, Source, and Downtime to a Tool

The four mechanisms are not ranked; each wins in a specific region of the trade space. The table below maps the three decisive inputs — data volume, source shape, and how much downtime the cutover can absorb — to the tool that dominates there.

Four rules follow from the table:

1. The offline importer is a first-load-only tool. neo4j-admin database import full refuses to run against a database that already contains data. It is the correct choice for the initial materialisation and nothing else; every subsequent top-up is an online load.
2. Never load millions of rows in one transaction. Whether the file is read by the server (LOAD CSV) or the client (driver batches), the write must be partitioned into committed sub-transactions. The 5.x mechanism for the server side is CALL {} IN TRANSACTIONS, which replaced the deprecated USING PERIODIC COMMIT.
3. Push transforms to the layer that owns them. If a value must be cleaned, validated, or enriched against an external service, do it in Python before it reaches Bolt — that is the LOAD CSV versus driver trade-off in one sentence.
4. Reach for APOC only when the shape demands it. apoc.load.json and apoc.periodic.iterate earn their keep on nested documents and long-running graph-refactor passes; for flat rows they add an operational dependency without a throughput gain over plain UNWIND.

Step-by-Step Implementation

Step 1 — Offline bulk build with neo4j-admin import

For the very first load into an empty database, the offline importer is unmatched. It runs while the database is stopped, reads header-typed CSV files, and constructs the store files directly. Because it never opens a transaction, it neither honours nor needs online constraints — you create those after the import completes.

# Database must be STOPPED and the target DB empty. Node and relationship
# files carry typed headers (e.g. id:ID, :LABEL, :START_ID, :END_ID, :TYPE).
neo4j-admin database import full neo4j \
  --nodes=Customer=import/customers_header.csv,import/customers.csv \
  --nodes=Transaction=import/tx_header.csv,import/tx.csv \
  --relationships=INITIATED=import/initiated_header.csv,import/initiated.csv \
  --skip-bad-relationships=true \
  --skip-duplicate-nodes=false \
  --high-parallel-io=on

Once the import finishes and the database restarts, apply the uniqueness constraints. Do not create them beforehand — the importer does not use them, and pre-existing schema forces the empty-database check to fail.

// Post-import: provision identity constraints so subsequent online loads
// (top-ups) can MERGE against an index-backed seek rather than a scan.
CREATE CONSTRAINT customer_id_unique IF NOT EXISTS
FOR (c:Customer) REQUIRE c.customer_id IS UNIQUE;

Step 2 — Server-side LOAD CSV with batched commits

When the database is already online and the source is a clean CSV the server process can read, LOAD CSV is the lowest-code path. In Neo4j 5.x the batching clause is CALL { … } IN TRANSACTIONS; the file is streamed row by row and the inner writes commit in fixed windows so the load never accumulates one giant transaction.

// Server reads the file from the import directory and streams rows.
// Each inner CALL commits every 10,000 rows — no client round trip per row.
LOAD CSV WITH HEADERS FROM 'file:///transactions.csv' AS row
CALL {
  WITH row
  MERGE (c:Customer {customer_id: row.customer_id})
  MERGE (t:Transaction {tx_id: row.tx_id})
    SET t.amount = toFloat(row.amount),
        t.processed_at = datetime(row.processed_at)
  MERGE (c)-[:INITIATED]->(t)
} IN TRANSACTIONS OF 10000 ROWS;

Casting is explicit — toFloat, datetime — because every CSV field arrives as a string. That casting ceiling is exactly why richer transforms belong in the driver path of Step 3.

Step 3 — Programmatic loading with driver UNWIND batches

When records need validation, enrichment, or type coercion that Cypher cannot express cleanly, the Python driver’s UNWIND $rows pattern is the workhorse. You shape and check each row in Python, then send a whole batch as one parameter so a single transaction upserts the entire list — never one round trip per row.

from neo4j import GraphDatabase, ManagedTransaction

UPSERT = """
UNWIND $rows AS row
MERGE (c:Customer {customer_id: row.customer_id})
MERGE (t:Transaction {tx_id: row.tx_id})
  SET t.amount = row.amount, t.processed_at = datetime(row.processed_at)
MERGE (c)-[:INITIATED]->(t)
"""

def _load_batch(tx: ManagedTransaction, rows: list[dict]) -> int:
    # The batch is one transaction: the whole list commits or rolls back
    # together, and execute_write retries the entire function on a
    # transient error — safe because every write is an idempotent MERGE.
    return tx.run(UPSERT, rows=rows).consume().counters.nodes_created

def load(uri, auth, records: list[dict], batch_size: int = 10_000) -> None:
    with GraphDatabase.driver(uri, auth=auth) as driver:
        driver.verify_connectivity()
        with driver.session(database="neo4j") as session:
            for start in range(0, len(records), batch_size):
                batch = [validate(r) for r in records[start:start + batch_size]]
                session.execute_write(_load_batch, batch)

The validate(r) call is the entire reason to choose this path: rejecting a malformed row, casting a field, or looking up an enrichment value happens in code before the write ever opens. Connection-pool sizing and retry policy for this pattern are covered in Python Driver Integration Patterns.

Step 4 — Streaming complex sources with APOC

For deeply nested JSON or a driving query too large to hold in one transaction, APOC procedures stream the outer iterable and batch the inner writes on the server. apoc.load.json parses a document (local or remote) into Cypher maps, and apoc.periodic.iterate runs an inner statement in committed batches with configurable parallelism.

// apoc.load.json unwinds a nested document; apoc.periodic.iterate runs the
// inner MERGE in committed batches of 5,000 with 4 parallel workers.
CALL apoc.periodic.iterate(
  "CALL apoc.load.json('file:///orders.json') YIELD value AS order
   UNWIND order.lines AS line RETURN order, line",
  "MERGE (o:Order {order_id: order.id})
   MERGE (p:Product {sku: line.sku})
   MERGE (o)-[r:CONTAINS]->(p) SET r.qty = line.qty",
  {batchSize: 5000, parallel: true, concurrency: 4}
);

Set parallel: true only when the batches touch disjoint nodes; overlapping writes (shared reference data, hub nodes) will deadlock under concurrency. Unwinding nested arrays into relationships is developed further in handling nested JSON arrays during graph ingestion.

Constraint & Validation Layer

Every online loader in Steps 2–4 relies on MERGE being race-safe, which is only true when a uniqueness constraint backs the merge key with an ONLINE index. The offline importer is the exception — it enforces uniqueness through its own --skip-duplicate-nodes flag and you add constraints afterward. For all online paths, create the constraint first and gate the load on index readiness:

// Create before any online MERGE; a MERGE issued before the backing index
// is ONLINE falls back to a label scan and can create duplicates under load.
CREATE CONSTRAINT tx_id_unique IF NOT EXISTS
FOR (t:Transaction) REQUIRE t.tx_id IS UNIQUE;

// Gate the load: block until every index reports ONLINE.
CALL db.awaitIndexes(300);

Two validation gates keep bad records out of the write path regardless of tool. Structural failures — mismatched keys, null identity columns, uncastable types — should be caught where the mapping is defined in Relational Schema Mapping Strategies and routed to quarantine before a transaction opens. Contract failures across the whole run belong to Data Validation & Integrity Checks. The driver path in Step 3 is the only one that can enforce these gates inline; the CSV and APOC paths depend on the file already being clean.

Performance & Scale Considerations

Tool choice sets the throughput ceiling before any tuning knob is turned.

• The offline importer is an order of magnitude faster — and a cliff. Because it writes store files directly and never touches the transaction log, neo4j-admin import sustains rates no online loader approaches. But it is all-or-nothing per invocation and cannot resume mid-file, so a bad-data failure restarts the whole build. Validate the CSV headers and referential integrity before you commit a multi-hour import.
• Server-side reads beat client round trips for clean data. LOAD CSV avoids serialising every row over Bolt, so for pre-shaped files it out-throughputs the driver. The moment records need per-row Python, that advantage inverts — the transform cost dwarfs the transport saving.
• Batch size is the universal lever. Whether the batching happens server-side via IN TRANSACTIONS OF n ROWS or client-side via slicing the row list, windows in the 5,000–10,000 range balance commit overhead against transaction-log and heap pressure. Oversized batches risk OutOfMemoryError; undersized ones drown in commit overhead. This trades directly against Initial Load Performance Tuning.
• Parallelism scales with contention, not cores. APOC parallel: true and multi-worker driver loads only speed up when batches touch disjoint regions of the graph. Overlapping writes convert extra workers into lock-contention and deadlock — an effect examined in resolving duplicate nodes during parallel batch loads.

Known Pitfalls

1. Using the offline importer on a non-empty database. Symptom: neo4j-admin database import full aborts with an error that the database is not empty, or --overwrite-destination silently discards existing data. Root cause: the importer is a first-load tool that builds store files from scratch. Fix: use it only for the initial materialisation; run every top-up through an online loader (LOAD CSV or the driver).

2. Reaching for the deprecated USING PERIODIC COMMIT. Symptom: a LOAD CSV script written for Neo4j 3.x/4.x throws a syntax error or a deprecation warning on 5.x. Root cause: PERIODIC COMMIT was removed in favour of CALL { … } IN TRANSACTIONS. Fix: wrap the per-row work in a subquery and add IN TRANSACTIONS OF n ROWS, as in Step 2 and the dedicated CALL IN TRANSACTIONS guide.

3. One giant transaction disguised as a bulk load. Symptom: OutOfMemoryError, long GC pauses, or a bloated transaction log midway through a driver load. Root cause: the whole dataset was sent as a single UNWIND without slicing, or a LOAD CSV ran without the IN TRANSACTIONS clause. Fix: partition into committed windows; recoverability comes from small committed units plus checkpoints, never from atomicity of the whole load.

4. Server-unreachable file paths in LOAD CSV. Symptom: Couldn't load the external resource even though the file exists on your workstation. Root cause: LOAD CSV and apoc.load.json read from the server’s filesystem and import directory, not the client’s. Fix: stage the file in the server import directory, or switch to the driver path where the client owns the data.

Related

• Automated Data Migration from Relational & JSON Sources — the parent reference this loading stage plugs into.
• LOAD CSV vs Python Driver for Bulk Import — the server-side versus client-driven decision in full.
• Using CALL IN TRANSACTIONS for Batched Writes — the 5.x replacement for PERIODIC COMMIT.
• Python Driver Integration Patterns — pooling, retries, and async for the programmatic path.
• Initial Load Performance Tuning — heap, cache, and chunk sizing that set every loader’s ceiling.
• Batch Processing & Chunking Workflows — commit boundaries and parallel-load contention.