Add extensible compression support for RPC requests#6084
Conversation
a2eba9c to
4917aef
Compare
4fb2574 to
b8939aa
Compare
8dc6615 to
acb0979
Compare
|
@lutter This is also building now ... |
8bab643 to
cf7477b
Compare
|
Rebased to latest master (and cleaned up a little) |
|
I just looked at this, and seeing how we do similar things for other providers, why do we use a separate |
|
This pull request hasn't had any activity for the last 90 days. If there's no more activity over the course of the next 14 days, it will automatically be closed. |
b7f93ce to
e201a75
Compare
@lutter the PR has been refactored and the branch rebased. The format is now NOTE: The PR went stale and automatically closed due to no activity for months. It would be good to have this reviewed and merged into master. |
339e484 to
8e01d4c
Compare
f42dee2 to
9b0ee4e
Compare
- Replace boolean compression_enabled with Compression enum (None, Gzip) - Support per-provider compression configuration via "compression" field - Add placeholders for future compression methods (Brotli, Deflate) - Update transport layer to handle compression enum with match statement - Add comprehensive unit tests for compression configuration parsing - Update example configuration and documentation Configuration examples: compression = "gzip" # Enable gzip compression compression = "none" # Disable compression (default) Addresses issue #5671 with future-extensible design.
When reqwest gzip feature is enabled globally, Client::default() enables automatic gzip compression which removes content-length headers from responses. ArweaveClient needs content-length to check file sizes, so explicitly disable gzip for Arweave requests. Fixes test: polling_monitor::ipfs_service::test::arweave_get 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
9b0ee4e to
8c7cc8b
Compare
Summary
Adds configurable compression support for Graph Node's outgoing JSON-RPC requests to upstream providers, supporting gzip, brotli, and deflate methods.
Goal
Allow node operators to enable compression on a per-provider basis for outgoing RPC requests, reducing bandwidth usage between Graph Node and upstream providers.
Design
Compression is configured through the existing features array on providers using
compression/gzip,compression/brotli, orcompression/deflate. ACompressionenum is used internally to represent the selected method. At most onecompression/*feature may be set per provider — this is validated at config load time.Changes
Transport layer (
chain/ethereum/src/transport.rs)Compressionenum withNone(default),Gzip,Brotli,DeflatevariantsTransport::new_rpc()accepts aCompressionparameter and calls the appropriate reqwest builder methodConfig layer (
node/src/config.rs)compression/gzip,compression/brotli, andcompression/deflatetoPROVIDER_FEATUREScompression() -> CompressiononWeb3Providerthat maps features to the enumcompression/*feature is present per providerWiring layer (
node/src/chain.rs)web3.compression()and passes it toTransport::new_rpc()Network tests (
chain/ethereum/src/network.rs)Transport::new_rpc()test call sites to useCompression::NonePublic exports (
chain/ethereum/src/lib.rs)CompressionalongsideTransportDependencies (
graph/Cargo.toml)"gzip","brotli", and"deflate"to reqwest featuresArweaveClient fix (
graph/src/components/link_resolver/arweave.rs)gzipfeature makesClient::default()automatically negotiate gzip, which stripscontent-lengthheaders that ArweaveClient depends on for file size checks.Configuration
Omitting any
compression/*feature means no compression (the default).Full example
Test plan
cargo check --releasepassesjust lintpasses with zero warnings