Skip to content

perf: remove blocking calls from POI queries#6300

Open
MoonBoi9001 wants to merge 3 commits intomasterfrom
feat/async-poi-resolver
Open

perf: remove blocking calls from POI queries#6300
MoonBoi9001 wants to merge 3 commits intomasterfrom
feat/async-poi-resolver

Conversation

@MoonBoi9001
Copy link
Member

@MoonBoi9001 MoonBoi9001 commented Jan 24, 2026

Summary

  • Makes the proofOfIndexing resolver async, removing graph::block_on calls that blocked the Tokio runtime
  • Adds a new blockForPoi query that finds which block number produced a given proof of indexing, supporting dispute investigation where an indexer may have submitted a POI for block N that actually corresponds to block X < N

blockForPoi design

The resolver queries all historical POI digest entries from poi2$ in a single DB call, then searches backwards from endBlock in 1M-block chunks. Block hash fetching is pipelined (the next chunk is prefetched while the current one is being processed), and POI computation within each chunk is parallelized across all CPU cores via rayon's find_map_any.

Key decisions:

  • Searches backwards from endBlock towards startBlock
  • indexer is required so the caller specifies which address to use in the POI computation
  • No artificial cap on the search range; internally processes in 1M-block chunks

Generated with Claude Code

@MoonBoi9001 MoonBoi9001 requested a review from lutter January 26, 2026 17:50
@MoonBoi9001 MoonBoi9001 force-pushed the feat/async-poi-resolver branch from 8045272 to b6cd094 Compare January 26, 2026 19:57
Remove synchronous `block_on` call in `resolve_proof_of_indexing` which
was blocking tokio worker threads while waiting for database queries.

Before this change, each POI query blocked an entire tokio worker thread.

After this change, POI queries properly yield to the async runtime while
waiting for database I/O, allowing the connection pool to be fully utilized.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@MoonBoi9001 MoonBoi9001 force-pushed the feat/async-poi-resolver branch from b6cd094 to 888a12e Compare January 28, 2026 03:27
@lutter
Copy link
Collaborator

lutter commented Jan 28, 2026

The first commit is great and something that absolutely needs to happen. For the second commit, I am not quite clear on what the motivation is - are PoI requests slow? The worry I have is that when graph-node is under a lot of load, parallelizing these requests makes the situation worse. The code limits the number of requests to 10, which is good, but I'd still like to understand what the motivation for this change is.

@MoonBoi9001
Copy link
Member Author

The motivation is POI verification tooling for dispute investigation.

When investigating POI discrepancies (e.g., for arbitration disputes), we need to compute POIs across large block ranges to identify where divergence occurred.

With sequential processing, throughput is bottlenecked at ~2k POIs/second even though the database connection pool has capacity. The publicProofsOfIndexing endpoint accepts batch requests (up to 10), but processing them sequentially means we're not utilizing available parallelism. Checking every block sequentially on a network with 100m blocks takes 13+ hours at ~2k blocks/second.

@MoonBoi9001 MoonBoi9001 force-pushed the feat/async-poi-resolver branch from 8b2a3fc to 4f89b3f Compare January 29, 2026 02:10
@MoonBoi9001
Copy link
Member Author

The worry I have is that when graph-node is under a lot of load, parallelizing these requests makes the situation worse.

The third commit addresses this, now pre-fetch all block hashes in a single batch query before parallel processing, reducing DB round-trips.

@lutter
Copy link
Collaborator

lutter commented Jan 29, 2026

With sequential processing, throughput is bottlenecked at ~2k POIs/second even though the database connection pool has capacity. The publicProofsOfIndexing endpoint accepts batch requests (up to 10), but processing them sequentially means we're not utilizing available parallelism. Checking every block sequentially on a network with 100m blocks takes 13+ hours at ~2k blocks/second.

Have you tried a binary search for PoI's? I assume you are looking for the point where the PoI's for some subgraph diverge between two indexers. Instead of a binary search, you could actually request multiple PoI's across some range (start with [0, current subgraph head]) and then refine the range so that the point of divergence is always in the range. That should yield results very rapidly.

@MoonBoi9001
Copy link
Member Author

Binary search works well for finding divergence points between two indexers. This PR speeds up computations for finding which block a given POI actually corresponds to if it doesnt correspond to the block that it is claimed to (can be used to prove an indexer claimed rewards for block N, but only synced to block X). Under the assumption that an indexer force closes with a valid POI for a block x < n, but submits that block x POI for block n.

@lutter
Copy link
Collaborator

lutter commented Jan 30, 2026

Thanks for the explanation - if you use this to find the block for which a given POI was generated, how about we add a route in the resolver that just does that? That might be much faster than just speeding up POI lookup.

@MoonBoi9001 MoonBoi9001 force-pushed the feat/async-poi-resolver branch 3 times, most recently from 6620768 to bae0c11 Compare February 6, 2026 03:14
Adds a new index-node GraphQL query that searches a block range to find
which block produced a given POI. This supports dispute investigation
where an indexer may have submitted a POI for block N that actually
corresponds to block X < N.

The resolver fetches all poi2$ digest entries in a single DB call, batch
fetches block hashes in 50k chunks, and runs ProofOfIndexingFinisher for
each block until a match is found.

Also adds network_for_deployment to the StatusStore trait and its
implementation chain (SubgraphStore, Store) as a supporting method
for resolving the chain store from a deployment hash.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@MoonBoi9001 MoonBoi9001 force-pushed the feat/async-poi-resolver branch from bae0c11 to b8bd47d Compare February 6, 2026 03:20
@MoonBoi9001
Copy link
Member Author

Great suggestion @lutter I dropped those last two commits and implemented a route in b8bd47d

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants