Refactoring a supernode with intermediate nodes
You already have a supernode. One vertex — a busy User, a shared Tenant, a popular Product — has accumulated millions of incident relationships, and every traversal that passes through it now pages its entire adjacency list into the heap, so a query that should touch a few hundred edges reads a few million. This page is the remediation runbook: how to interpose intermediate bucket nodes between the supernode and its neighbours so each expansion touches a bounded fan-out, how to reroute the existing edges in batches without one catastrophic transaction, and how to prove the fix with PROFILE db-hits before and after. It assumes you have already decided the concentration is wrong — the threshold question of when a dense node is acceptable is settled separately in when to use dense nodes vs sparse relationships. Here the node is already too dense and must be reshaped in place.
Prerequisites
What the intermediate node buys you
Traversal cost in Neo4j scales with the degree at each hop, not with the size of the graph. A supernode with degree N forces every expansion to walk an N-length relationship chain, so cost grows linearly with how popular the node has become. Interposing a bucket node partitions that one chain into many short ones: the supernode now points at a small set of bucket nodes (one per month, one per category), and each bucket holds only the slice of edges that belong to it. A query that already knows which month it wants seeks the right bucket and expands a chain of a few thousand, never the full millions. The degree at every hop stays bounded, which is what pulls the Expand(All) db-hits down by orders of magnitude.
The diagram contrasts the star topology you start from with the bucketed topology you migrate to.
Core implementation
Reroute the existing edges in bounded batches. The migration MERGEs the bucket derived from each edge’s own property, hangs the leaf off the bucket, and deletes the old direct edge — all inside CALL { … } IN TRANSACTIONS so it commits in inner batches and never opens one heap-exhausting transaction. The comments mark the three load-bearing steps.
// 0) BUCKET IDENTITY FIRST. A composite NODE KEY makes each (user, period)
// bucket unique, so the reroute MERGE seeks it instead of scanning and a
// replayed batch reuses the same bucket rather than forking a duplicate.
CREATE CONSTRAINT month_bucket_key IF NOT EXISTS
FOR (b:MonthBucket) REQUIRE (b.user_id, b.period) IS NODE KEY;
// 1) REROUTE IN BATCHES. Process only edges not yet migrated (still direct),
// so the migration is idempotent and resumable after a failure. The bucket
// key is DERIVED from the edge's own timestamp — no external lookup.
MATCH (u:User)-[old:GENERATED]->(e:Event)
CALL (u, old, e) {
WITH u, old, e, date.truncate('month', e.occurred_at) AS period
MERGE (b:MonthBucket {user_id: u.user_id, period: period})
ON CREATE SET b.month = period
MERGE (u)-[:HAS_MONTH]->(b) // supernode now points at a few buckets
MERGE (b)-[:CONTAINS]->(e) // bucket holds only its own slice
DELETE old // drop the direct edge that caused the fan-out
} IN TRANSACTIONS OF 10000 ROWS; // bounded commit == bounded heap and lock set
Three properties make this safe to run against a live database. It is idempotent: the driving MATCH selects only surviving :GENERATED edges, so a re-run after a partial failure resumes where it stopped instead of double-processing. It is bounded: IN TRANSACTIONS OF 10000 ROWS caps the undo log and lock set per commit, the same batch discipline the property graph anti-patterns reference applies to any large rewrite. And it is key-derived: the bucket comes from each edge’s own occurred_at, so no lookup table or ordering is needed and buckets materialize on demand. Because CALL { … } IN TRANSACTIONS is an auto-commit clause, issue it through a plain session.run(...), never inside a managed execute_write — the same rule that governs every server-side batched write.
Rewrite the read queries to route through the bucket. A query that filters by month now seeks the bucket and expands a bounded CONTAINS chain: MATCH (u:User {user_id: $uid})-[:HAS_MONTH]->(b:MonthBucket {period: $month})-[:CONTAINS]->(e:Event). The supernode is never scanned again.
Validation & verification
The whole point is a measured reduction in traversal cost, so measure it. PROFILE the original access pattern against the supernode and record the Expand(All) db-hits, then PROFILE the bucketed pattern and compare:
// AFTER: the expansion is bounded to one month's slice, not the whole node.
PROFILE
MATCH (u:User {user_id: $uid})-[:HAS_MONTH]->(b:MonthBucket {period: date($month)})
MATCH (b)-[:CONTAINS]->(e:Event)
RETURN count(e);
The bucketed plan should show a small Expand(All) db-hits total proportional to one bucket’s degree, where the direct-fan-out plan showed a total proportional to the full node degree. Next, confirm no direct edges survived the migration — a non-zero count is unmigrated data:
MATCH (:User)-[old:GENERATED]->(:Event) RETURN count(old) AS unmigrated_edges;
Finally, reconcile the leaf count: count { (:MonthBucket)-[:CONTAINS]->(:Event) } must equal the number of :GENERATED edges you started with, so no event was orphaned or duplicated by the reroute.
Edge cases & gotchas
1. Rerouting the whole node in one transaction. A single MATCH … DELETE … CREATE over millions of edges reintroduces exactly the heap exhaustion you are escaping. The IN TRANSACTIONS OF N ROWS batching is not optional; without it the migration itself becomes a supernode-sized transaction.
2. Choosing a bucket key queries do not filter on. If you bucket by month but every query filters by category, the traversal must fan out across all buckets to find its category — you have added a hop without pruning anything. Bucket by the dimension the hot queries already constrain, or add a second bucket layer for a second dimension.
3. Skew inside a single bucket. One month with tens of millions of events is still a supernode wearing a bucket label. When a period is itself too dense, sub-partition it — bucket by day within that month, or by day and category — so no single bucket’s degree stays unbounded:
// Sub-bucket a hot period by day when one month is still too dense.
WITH date.truncate('day', e.occurred_at) AS day
MERGE (d:DayBucket {user_id: u.user_id, day: day});
This task is one remediation in the property graph anti-patterns catalogue; the degree-threshold analysis that tells you whether a node needs it at all lives in the dense-versus-sparse guide.
Related
- Up: Property Graph Anti-Patterns — the failure-mode catalogue this refactor resolves.
- When to Use Dense Nodes vs Sparse Relationships in Neo4j — the degree thresholds that decide whether to refactor.
- Graph Partitioning Strategies — bounding per-node degree at cluster scale once buckets are in place.