Graph Data Modeling

Retrying transient errors with managed transactions

You are loading into a Neo4j cluster where leader elections, connection resets, and lock timeouts are routine facts of operation, and you need each chunk to survive them without corrupting the graph or aborting the run. The Neo4j 5.x Python driver already solves most of this: execute_write and execute_read wrap your unit of work in a managed transaction that classifies the exception on failure and re-runs the whole function on a retryable one, backing off between attempts. This task explains exactly which exceptions the managed retry catches, why that retry forces your transaction function to be idempotent, how to tell a retryable fault from a permanent one, and when a bounded application-level retry belongs on top. This task is part of Python Driver Integration Patterns.

Prerequisites

What the managed retry actually does

When you call session.execute_write(fn, ...), the driver invokes fn, and if fn raises an exception it inspects the type. If the exception is retryable — ServiceUnavailable (the member is unreachable), SessionExpired (the connection’s member left the database cluster, typically a leader switch), or a TransientError that reports itself as retryable (deadlock, lock-acquisition timeout) — the driver sleeps for an increasing, jittered interval and calls fn again from the top. It keeps doing this until the call succeeds or a total retry-time budget (max_transaction_retry_time, default 30 s) is exhausted, then re-raises. A non-retryable exception — ClientError and its subclasses such as ConstraintValidationFailed and CypherSyntaxError — is raised immediately with no retry, because re-running deterministic-failing work only wastes locks.

The critical consequence: because fn re-runs whole, everything it does must be safe to repeat. A MERGE on a stable key converges; a CREATE duplicates on every retry; a counter increment or queue publish inside fn fires once per attempt.

Retry loop and exception classification for a managed transaction Flow. Run function feeds an Outcome decision. Success flows left to Commit. Raise flows down to a Retryable? decision. Yes checks a Budget left? decision: within budget flows to Backoff and jitter, which loops back up to Run function; budget exhausted flows to Re-raise. No flows right from Retryable directly to Re-raise, tagged ClientError and ConstraintValidationFailed. Run tx function Outcome Commit success Retryable? raise Budget left? yes Backoff + jitter within re-run whole fn Re-raise ClientError / constraint no budget spent
Retryable faults loop through backoff and re-run the whole function; client and constraint errors re-raise at once.

Core implementation

The unit of work below is safe to re-run because it is a constraint-backed MERGE with no external side effects. The bounded application-level wrapper adds one thing the driver cannot: a policy for what to do after the managed budget is finally exhausted — smaller chunk, or dead-letter.

python
import time
from neo4j import GraphDatabase, ManagedTransaction
from neo4j.exceptions import (
    TransientError, ServiceUnavailable, SessionExpired, ClientError,
)

def upsert_chunk(tx: ManagedTransaction, rows: list[dict]) -> int:
    # IDEMPOTENT UNIT OF WORK. The managed retry re-runs this whole function on
    # a transient fault, so it must converge on replay: MERGE on a stable key,
    # parameterised, and with NO side effects (no metrics, no logging counters).
    # A CREATE here would duplicate every retry; a counter here would over-count.
    result = tx.run(
        """
        UNWIND $rows AS row
        MERGE (n:Entity {id: row.id})
        ON CREATE SET n += row.properties
        ON MATCH  SET n += row.properties
        RETURN count(n) AS upserted
        """,
        rows=rows,
    )
    return result.single()["upserted"]


def load_with_policy(driver, rows: list[dict], *, max_app_attempts: int = 3) -> int:
    # APPLICATION-LEVEL RETRY on top of the driver's managed retry. The driver
    # already handles in-budget transient blips; this outer loop decides what to
    # do once the driver GIVES UP (budget exhausted → it re-raises transient).
    for attempt in range(1, max_app_attempts + 1):
        try:
            with driver.session(database="neo4j") as session:
                # Managed retry: classifies + backs off on retryable faults.
                return session.execute_write(upsert_chunk, rows)
        except (ServiceUnavailable, SessionExpired, TransientError) as exc:
            # RETRYABLE but the driver's budget was spent. Bounded outer retry;
            # optionally halve the chunk to relieve lock contention, then fail.
            if attempt == max_app_attempts:
                raise                              # hand off to dead-letter
            time.sleep(min(30.0, 2 ** attempt))    # coarse outer backoff
        except ClientError:
            # NON-RETRYABLE: constraint violation, syntax error, auth failure.
            # Never loop on these — they fail identically every time.
            raise
    raise RuntimeError("unreachable")

The division of labour is the point. The driver owns fast, in-budget retries of infrastructure blips so your code never sees a leader election. Your wrapper owns the coarse policy — how many times to re-attempt after the driver gives up, whether to shrink the chunk, and when to quarantine — and it must branch on exception type so it never loops a ClientError.

Validation & verification

Prove idempotency behaviourally, because that is the property retry safety rests on. Record the node count, replay the identical chunk, and confirm the total is unchanged:

cypher
// Run the load, capture this, replay the SAME chunk, and re-run: a truly
// idempotent unit of work leaves the count identical on the second pass.
MATCH (n:Entity) RETURN count(n) AS total;

Then confirm the retried MERGE seeks an index rather than scanning — a scan is the fingerprint of a missing or not-yet-online constraint, and a scanning MERGE can still race under concurrency:

cypher
// Expect NodeUniqueIndexSeek at the leaf, never NodeByLabelScan + Filter.
EXPLAIN
UNWIND $rows AS row
MERGE (n:Entity {id: row.id})
RETURN count(n);

Edge cases & gotchas

1. A non-idempotent body under retry. If the transaction function contains a CREATE, or a MERGE anchored on a mutable property, each managed retry adds duplicates instead of converging. Anchor on the stable key and set everything else:

cypher
// WRONG — CREATE (or MERGE on a mutable prop) duplicates on every retry
CREATE (n:Entity {id: row.id})
// RIGHT — MERGE on the immutable key converges no matter how many replays
MERGE (n:Entity {id: row.id}) SET n += row.properties

2. Treating a constraint violation as transient. ConstraintValidationFailed is a ClientError — deterministic and non-retryable. Catching it in a generic transient handler burns attempts and holds locks while failing identically each time. Route it straight to quarantine, as the error-handling reference prescribes.

3. Side effects inside the transaction function. A metric increment or queue publish placed inside the function passed to execute_write fires once per attempt, over-counting under retry. Keep the function pure and perform side effects only after execute_write returns.

Related pages