Graph Data Modeling

Python Driver Integration Patterns

A migration pipeline is only as reliable as the object model it drives Neo4j through, and most integrity incidents that surface during a bulk load trace back to a handful of driver-level mistakes: a driver constructed per request, a session shared across threads, a bare session.run where a managed transaction belonged, or reads pinned to a stale cluster member. When automating Automated Data Migration from Relational & JSON Sources, the neo4j 5.x Python driver is the component that turns validated, chunked records into committed graph state, so its lifecycle, concurrency model, and transaction primitives must be engineered deliberately rather than copied from a quickstart. This reference defines how the driver, its connection pool, sessions, and transaction functions fit together, how routing and bookmarks preserve causal consistency across a Neo4j cluster, and how to select the target database explicitly — the load-bearing decisions every migration script depends on.

Prerequisite Concepts

The driver executes work that upstream stages have already shaped; it does not repair bad input. Before applying the patterns below, the reader should be comfortable with:

  • The end-to-end pipeline contract. The driver sits at the load stage of the flow defined by the parent Automated Data Migration from Relational & JSON Sources reference — extract, map, validate, chunk, load, verify. Everything here concerns the last two steps.
  • A stable target schema. Managed transactions retry by re-running the whole unit of work, so the labels, relationship types, and constraints they write into must be fixed first. Stabilise the model against Neo4j Graph Schema Design & Architecture before tuning the driver around it.
  • Chunk boundaries. A session’s transaction is the commit boundary, and recovery granularity can never be finer than that boundary. How work is partitioned into chunks is covered in Batch Processing & Chunking Workflows.
  • Failure classification. The driver raises a typed exception hierarchy that the pipeline must route deterministically; the decision table lives in Error Handling & Rollback Mechanisms.

Conceptual Model: Driver, Pool, Sessions, Transactions

The four objects form a strict ownership hierarchy. Exactly one Driver exists per process. It is thread-safe and immutable after construction, and it owns a bounded connection pool — the expensive, long-lived resource. Sessions are cheap, lightweight, and explicitly not thread-safe: each one borrows a connection from the pool for the duration of a transaction, then returns it. Transactions are the unit of retry and the unit of commit. Getting the cardinality right — one driver, many short-lived sessions, one transaction function per unit of work — is the whole game.

Driver, pool, sessions and transaction routing Left: a single Driver box, labelled one per process, thread-safe. It feeds a bounded Connection pool box. The pool lends connections to three stacked Session boxes, labelled cheap and not thread-safe. From the sessions, an execute_write arrow routes to a Leader member and an execute_read arrow routes to a Follower member of the database cluster, with the neo4j:// routing scheme choosing the target. Driver one per process thread-safe Connection pool bounded, shared Session borrows a connection Session cheap, not thread-safe Session one unit of work Leader writes Follower reads execute_write execute_read
One driver owns the pool; sessions are disposable; routing sends writes to the leader and reads to a follower.

Design Rules

The rules below encode the driver’s contract. Every migration script should hold all of them; each maps to a mechanical decision, not a judgement call.

Concern Rule Rationale
Driver lifetime Construct one Driver per process and share it The driver owns the pool and is thread-safe; a per-request driver leaks connections and defeats pooling
Session lifetime One session per unit of work, always in a with block Sessions are cheap and not thread-safe; a leaked or shared session corrupts result cursors
Write path session.execute_write(fn) for all writes Managed transactions retry the whole function on transient cluster errors; session.run does not
Read path session.execute_read(fn) for all reads Enables read routing to followers and the same managed retry semantics
Auto-commit Reserve session.run for CALL {} IN TRANSACTIONS and admin DDL Those clauses cannot run inside an explicit transaction and must not be retried blindly
Idempotency The transaction function body must be a constraint-backed MERGE The function may run more than once; only an idempotent body is retry-safe
Routing scheme Use neo4j:// in a Neo4j cluster, bolt:// for a single instance neo4j:// performs server-side routing; bolt:// pins to one address and cannot fail over
Consistency Chain bookmarks when a later read must see an earlier write Bookmarks give causal (read-your-writes) consistency across sessions and members
Target database Pass database= explicitly on every session Skips a round-trip to resolve the home database and prevents writing to the wrong one

Step-by-Step Implementation

Step 1 — One driver per process, verified at startup

The Driver is created once, held for the process lifetime, and closed on shutdown. Because it is thread-safe and internally pooled, sharing the single instance across every worker is not just allowed — it is the intended design. Call verify_connectivity() once at startup so a bad URI, expired credentials, or an unreachable cluster fails loudly before the migration begins instead of on the first chunk.

python
from neo4j import GraphDatabase

# One driver, constructed once and reused. `neo4j://` selects the routing
# driver so writes and reads reach the right cluster member automatically.
driver = GraphDatabase.driver(
    "neo4j://cluster.internal:7687",
    auth=("migrator", "…"),
    max_connection_pool_size=50,        # bound the pool to your worker count
)

# Fail fast at startup: a bad URI or credential should never surface mid-load.
driver.verify_connectivity()

The pool sizing above is deliberate, not decorative — it must track your concurrency. Undersizing it starves workers with acquisition timeouts; oversizing it pushes thread pressure onto the server. That trade-off has its own guide: configuring connection pool size for the Neo4j Python driver.

Step 2 — Sessions and the context-manager lifecycle

A session is a short-lived, single-threaded logical container for a sequence of transactions against one database. Open it in a with block so the connection is returned to the pool and the result buffer is released even if the body raises. Never cache a session on an object, never pass one between threads, and never keep one open across an idle wait — open it, do the work, let the context manager close it.

python
# A session is disposable. Open it, run one unit of work, let `with` close it.
with driver.session(database="neo4j") as session:
    count = session.execute_write(upsert_chunk, chunk)
# Connection is back in the pool here; the session object is dead.

Step 3 — Managed transactions: execute_write and execute_read

The driver offers three transaction styles, and a migration should reach for exactly one of them by default. execute_write and execute_read are managed transactions: you hand them a function, and the driver opens a transaction, runs the function, commits it, and — crucially — re-runs the entire function on a transient error such as a leader switch. That retry is why the function must be idempotent and side-effect-free. Explicit transactions (session.begin_transaction()) give you manual commit/rollback control but no automatic retry, and are reserved for cases where you must interleave reads and conditional writes in one transaction. Auto-commit (session.run) commits each statement immediately with no retry and is reserved for statements that cannot run inside a transaction, chiefly CALL {} IN TRANSACTIONS.

python
from neo4j import ManagedTransaction

def upsert_chunk(tx: ManagedTransaction, chunk: list[dict]) -> int:
    # This function may execute MORE THAN ONCE (managed retry re-runs it whole),
    # so its body must be idempotent: parameterised and MERGE-based, with NO
    # external side effects (no logging counters, no queue publishes here).
    result = tx.run(
        """
        UNWIND $rows AS row
        MERGE (n:Entity {id: row.id})
        ON CREATE SET n += row.properties
        ON MATCH  SET n += row.properties
        RETURN count(n) AS upserted
        """,
        rows=chunk,
    )
    return result.single()["upserted"]


with driver.session(database="neo4j") as session:
    # execute_write retries the whole function on ServiceUnavailable,
    # SessionExpired, or a retryable TransientError — bare session.run will not.
    upserted = session.execute_write(upsert_chunk, chunk)

Read-only verification queries should go through execute_read so the driver may route them to a follower and lift read load off the leader. The exception hierarchy those managed transactions retry on — and when to layer your own bounded retry on top — is detailed in retrying transient errors with managed transactions.

Step 4 — Routing, bookmarks, and database selection

The connection scheme decides how the driver reaches the database. neo4j:// is the routing scheme: the driver fetches the database cluster’s routing table and sends writes to the leader and reads to an available follower, transparently following a failover. bolt:// opens a direct connection to a single address with no routing — correct for a standalone instance or when you deliberately target one member, wrong for a clustered deployment because it cannot fail over. Always name the target database with database=; omitting it forces a server round-trip to resolve the home database and risks landing writes in the wrong one.

Causal consistency across sessions is handled with bookmarks. Each committed transaction returns a bookmark; passing the bookmarks from one session into the next guarantees the second session sees the first’s writes, even if routing sends it to a different member. The driver chains bookmarks automatically within a single session — you only manage them explicitly when a subsequent, separate session must read your earlier writes.

python
# Load in one session, then verify in another that must see those writes.
with driver.session(database="neo4j") as load_session:
    load_session.execute_write(upsert_chunk, chunk)
    bookmarks = load_session.last_bookmarks()   # capture causal token

# The verify session inherits the bookmark, so its read-routed query is
# guaranteed to observe the committed load even on a different member.
with driver.session(database="neo4j", bookmarks=bookmarks) as verify_session:
    total = verify_session.execute_read(
        lambda tx: tx.run("MATCH (n:Entity) RETURN count(n) AS c").single()["c"]
    )

Constraint & Validation Layer

Managed retry is only safe because the transaction function is idempotent, and idempotency depends on a schema-level uniqueness constraint. Without one, a MERGE under concurrency can create duplicates because two transactions each find no match and both insert. Create the backing constraint before the first write and gate the load on it being ONLINE:

cypher
// Neo4j 5.x — the constraint that makes the MERGE in the tx function
// race-safe under managed retry and parallel sessions.
CREATE CONSTRAINT entity_id_unique IF NOT EXISTS
FOR (n:Entity) REQUIRE n.id IS UNIQUE;

Because CREATE CONSTRAINT can return before its backing index is ONLINE on a non-empty database, block the pipeline until the index is live, then proceed:

python
with driver.session(database="neo4j") as session:
    session.run(
        "CREATE CONSTRAINT entity_id_unique IF NOT EXISTS "
        "FOR (n:Entity) REQUIRE n.id IS UNIQUE"
    ).consume()
    session.run("CALL db.awaitIndexes(300)").consume()   # gate on ONLINE

Parameterising every query — passing $rows rather than string-formatting values into Cypher — is not only an injection defence; it lets the planner reuse a cached plan across chunks instead of recompiling per literal. That interaction is the subject of query parameterization and plan caching. Confirm the constraint is genuinely backing your MERGE with SHOW CONSTRAINTS, and consult the Neo4j Python Manual: Transactions for the authoritative lifecycle semantics.

Performance & Scale Considerations

Driver configuration shapes throughput as directly as chunk size does, because every session competes for the same bounded pool.

  • Pool size gates concurrency. The pool caps how many sessions can hold a connection at once. If more workers run than the pool has connections, the surplus blocks on connection_acquisition_timeout. Match max_connection_pool_size to your worker count with a small margin, and treat acquisition timeouts as the primary signal to widen the pool or narrow the fan-out.
  • Batch inside the transaction, not around it. One UNWIND $rows transaction over 5,000 rows is far cheaper than 5,000 auto-commit round-trips. The network and transaction-setup cost amortises across the batch, and the managed retry re-runs one bounded unit rather than thousands.
  • Route reads off the leader. Sending verification and reconciliation reads through execute_read over a neo4j:// driver lets followers absorb them, freeing the leader’s capacity for the write path during a load.
  • Reuse the driver, never the session. Constructing a driver performs DNS resolution, TLS negotiation, and routing-table discovery; doing that per chunk collapses throughput. The session, by contrast, is meant to be discarded after each unit of work.
  • Async overlaps I/O, it does not add cores. When a load is dominated by many small independent round-trips, the async driver overlaps their wait time; when it is CPU- or lock-bound, server-side batching serves better. That decision is developed in async Neo4j driver patterns for concurrent loads.

Known Pitfalls

  1. A driver per request. Symptom: connection counts climb, latency degrades, and the server reports pool exhaustion despite light logical load. Root cause: a new GraphDatabase.driver(...) is built inside the request/chunk loop, so pooling never engages and TLS/routing setup repeats. Fix: construct one driver at process start, share it across all workers, and close it once on shutdown.

  2. Sharing a session across threads. Symptom: intermittent ResultConsumedError, interleaved or corrupted results, and non-deterministic failures under parallelism. Root cause: a single session — which is not thread-safe — was passed to multiple workers. Fix: open one session per thread per unit of work inside a with block; share only the driver.

  3. Side effects inside the transaction function. Symptom: duplicate log lines, inflated metrics, or a checkpoint that advances after a rollback. Root cause: a counter increment or queue publish lives inside the function passed to execute_write, so it fires on every managed retry. Fix: keep the function pure; perform side effects only after the managed call returns.

  4. bolt:// against a Neo4j cluster. Symptom: writes fail after a leader election, or reads never offload to followers. Root cause: the direct bolt:// scheme pins to one member and performs no routing. Fix: use neo4j:// so the driver follows the routing table across failovers, reserving bolt:// for a genuine single instance.

Explore this section