From a6363ffa5f7bb39e991e95607b6e50ace332a80e Mon Sep 17 00:00:00 2001 From: bumblefudge Date: Mon, 12 Jan 2026 15:41:46 +0100 Subject: [PATCH 1/6] editorial tweaks to ipni page --- docs/concepts/ipni.md | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/docs/concepts/ipni.md b/docs/concepts/ipni.md index 0572f3cca..df7df8375 100644 --- a/docs/concepts/ipni.md +++ b/docs/concepts/ipni.md @@ -8,30 +8,36 @@ description: Learn about mutability in IPFS, InterPlanetary Name System (IPNS), [InterPlanetary Network Indexer (IPNI)](https://github.com/ipni), also referred to as _Network Indexer_, _indexer_ and _IPNI_, enables quick and efficient search of content-addressable data available on the InterPlanetary File System (IPFS) and Filecoin networks. Using IPNI, IPFS nodes can publish the content IDs (CIDs) of their data to an indexer, and clients can query the indexer to learn where to retrieve the content associated with those CIDs. -To publish CIDs, IPNI uses: +Opting into indexing by IPNI can be done from: - [Lotus](https://lotus.filecoin.io/), the reference implementation for the [Filecoin network](https://docs.filecoin.io/). - [Boost](https://boost.filecoin.io/), a tool for Filecoin storage providers to manage Filecoin data onboarding and retrieval. +- A self-hosted IPFS server with the [someguy](https://github.com/ipfs/someguy) sidecar integrated. +- A production-grade IPFS deployment configured to support the [IPNI provider sidecar](https://github.com/ipni/index-provider) (configuration may be more involved) -IPNI is designed to improve the performance and efficiency of IPFS by providing an alternate method of content routing to the [Kademlia Distributed Hash Table (DHT)](../concepts/dht.md#kademlia). +IPNI is designed to create an alternate routing and discovery infrastructure outside of and independent from the [Kademlia Distributed Hash Table (DHT)](../concepts/dht.md#kademlia). :::callout For a deeper dive into the technical specification for IPNI, see [https://github.com/ipni/specs/blob/main/IPNI.md](https://github.com/ipni/specs/blob/main/IPNI.md). ::: -## Design rationale +## What use-cases IPNI serves -To support retrievals of unsealed Filecoin and IPFS pinned data with a speed comparable to a CDN, a reliable, distributed index of all data and its associated peer must be maintained geographically near the lookups. This is necessary to fulfill lookups that cannot be quickly fulfilled using the DHT. +While in-protocol routing and discovery have [advanced leaps and bounds in recent versions of Kubo and Helia](https://ipshipyard.com/blog/2025-dht-provide-sweep/), there continue to be use-cases where keeping announcements live on the DHT is onerous. +By comparison, announcements to an IPNI indexer only need to be made once, making them particularly attractive to announcers of large volumes of infrequently-sought CIDs, like large-scale providers of "cold storage" in the Filecoin economy or archivers of public open data. -With this in mind, the Network Indexer was created as an alternative content routing system to the Kadmelia DHT used by IPFS. While the DHT is a key component of the IPFS ecosystem, IPNI can support content routing at a much larger scale and speed using Lotus and Boost. +To support performant _retrievals_ of unsealed Filecoin and IPFS pinned data with a speed comparable to a CDN, a reliable, distributed index of all data and the peer(s) hosting and/or caching it must be assembled, and this index must be replicated to be geographically near the lookups. Comparable lookup and time-to-first-byte metrics are quite difficult to achieve on the DHT. + +One advantage to being completely orthogonal to DHT-based announcing and discovery is that one or more IPNI-style indexer systems can be opted into or out of over time, without affecting DHT performance. +Many users have found value in both opting into IPNI indexing for long-tail discovery, but still announcing all new content to peers in the DHT at time of publication to allow for resilience and caching in realtime. This mixed approach also sidesteps the complexity of re-announcing and keeping DHT announcements circulating over time for content that is not expected to be cached and dynamically distributed peer-to-peer over the course of its lifecycle. ### How IPNI benefits IPFS The indexer offers several benefits to IPFS, including: -- **Faster data retrieval**: By maintaining an additional layer of information on top of the DHT, the indexer can help speed up data location and retrieval in IPFS. -- **Reduced resource consumption**: The indexer can help reduce the amount of bandwidth and processing power needed to locate and retrieve data, improving the performance of individual nodes and the overall network. -- **Improved scalability**: With the indexer, IPFS can better handle growth in user base and data volume, allowing it to scale more effectively and support larger networks. +- **Faster data retrieval**: By maintaining an additional layer of information on top of the DHT, the indexer can help speed up data location and retrieval. +- **Reduced resource consumption**: The indexer can help reduce the amount of bandwidth and processing power expended circulating and re-announcing CID location records, particularly for individual nodes for whom this peer-to-peer network traffic is costly (i.e. in commercial data centers). +- **Improved scalability**: With the indexer, IPFS can reduce the portion of network traffic used by large-volume publishers, allowing the rest of the network to scale more effectively and support broader and heterogeneous network topologies. ## The IPNI ecosystem From 9f4df80a90ab9dcadeec4dd644d9152cd922d015 Mon Sep 17 00:00:00 2001 From: bumblefudge Date: Mon, 19 Jan 2026 15:52:36 +0100 Subject: [PATCH 2/6] minor wordsmithing --- docs/concepts/ipni.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/concepts/ipni.md b/docs/concepts/ipni.md index df7df8375..a96b23bcf 100644 --- a/docs/concepts/ipni.md +++ b/docs/concepts/ipni.md @@ -28,7 +28,7 @@ By comparison, announcements to an IPNI indexer only need to be made once, makin To support performant _retrievals_ of unsealed Filecoin and IPFS pinned data with a speed comparable to a CDN, a reliable, distributed index of all data and the peer(s) hosting and/or caching it must be assembled, and this index must be replicated to be geographically near the lookups. Comparable lookup and time-to-first-byte metrics are quite difficult to achieve on the DHT. -One advantage to being completely orthogonal to DHT-based announcing and discovery is that one or more IPNI-style indexer systems can be opted into or out of over time, without affecting DHT performance. +One advantage to being completely orthogonal to DHT-based announcing and discovery is that over the life of a gateway or other data provider service, one or more IPNI-style indexer systems can be opted into or out of, without affecting DHT performance that can similarly be taken down and brought back up. Many users have found value in both opting into IPNI indexing for long-tail discovery, but still announcing all new content to peers in the DHT at time of publication to allow for resilience and caching in realtime. This mixed approach also sidesteps the complexity of re-announcing and keeping DHT announcements circulating over time for content that is not expected to be cached and dynamically distributed peer-to-peer over the course of its lifecycle. ### How IPNI benefits IPFS @@ -36,7 +36,7 @@ Many users have found value in both opting into IPNI indexing for long-tail disc The indexer offers several benefits to IPFS, including: - **Faster data retrieval**: By maintaining an additional layer of information on top of the DHT, the indexer can help speed up data location and retrieval. -- **Reduced resource consumption**: The indexer can help reduce the amount of bandwidth and processing power expended circulating and re-announcing CID location records, particularly for individual nodes for whom this peer-to-peer network traffic is costly (i.e. in commercial data centers). +- **Reduced resource consumption**: The indexer can help reduce the amount of bandwidth and processing power expended circulating and re-announcing CID location records, particularly for individual nodes for whom this peer-to-peer network traffic is costly (i.e. in commercial data centers). While the egress/networking and compute needed to periodically reannounce can now be delegated, it still needs to be done every 24 hours. - **Improved scalability**: With the indexer, IPFS can reduce the portion of network traffic used by large-volume publishers, allowing the rest of the network to scale more effectively and support broader and heterogeneous network topologies. ## The IPNI ecosystem @@ -69,7 +69,7 @@ The indexer works in conjunction with the existing DHT to improve data location When a user searches for a piece of data using a CID or multihash, the indexer is consulted first. If the data is found in the index, the user is directly connected to the node hosting the data, resulting in faster retrieval. If the data is not found in the index, the user falls back to the traditional DHT-based search, ensuring that the data can still be located even if it's not in the indexer. -By providing this additional layer of information, the indexer helps to speed up data location and retrieval, reduce resource consumption, and improve the overall scalability of IPFS. +By providing this additional layer of information, the indexer helps to speed up data location and retrieval, reduce resource consumption, and improve the overall scalability of IPFS for use-cases further from the DHT. ### Example: finding providers via `/routing/v1` From 97f9ec42951d47463f3870aeafd791f39e501b37 Mon Sep 17 00:00:00 2001 From: bumblefudge Date: Mon, 19 Jan 2026 15:57:39 +0100 Subject: [PATCH 3/6] add gui's benchmark --- docs/concepts/ipni.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/concepts/ipni.md b/docs/concepts/ipni.md index a96b23bcf..4bd3410ba 100644 --- a/docs/concepts/ipni.md +++ b/docs/concepts/ipni.md @@ -23,7 +23,7 @@ For a deeper dive into the technical specification for IPNI, see [https://github ## What use-cases IPNI serves -While in-protocol routing and discovery have [advanced leaps and bounds in recent versions of Kubo and Helia](https://ipshipyard.com/blog/2025-dht-provide-sweep/), there continue to be use-cases where keeping announcements live on the DHT is onerous. +While in-protocol routing and discovery have [advanced leaps and bounds in recent versions of Kubo and Helia](https://ipshipyard.com/blog/2025-dht-provide-sweep/), with well-tuned test server performance benchmarked in the high tens of millions of CIDs announce-able within the 24th recursion window, there continue to be use-cases where keeping announcements live on the DHT is onerous, or swarms where that volume of message would not be propagated before the cycle recursed and began again. By comparison, announcements to an IPNI indexer only need to be made once, making them particularly attractive to announcers of large volumes of infrequently-sought CIDs, like large-scale providers of "cold storage" in the Filecoin economy or archivers of public open data. To support performant _retrievals_ of unsealed Filecoin and IPFS pinned data with a speed comparable to a CDN, a reliable, distributed index of all data and the peer(s) hosting and/or caching it must be assembled, and this index must be replicated to be geographically near the lookups. Comparable lookup and time-to-first-byte metrics are quite difficult to achieve on the DHT. From c263d31fbda93d23b55adb7c1a5d3318e6691793 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Wed, 21 Jan 2026 05:33:16 +0100 Subject: [PATCH 4/6] docs: clarify someguy vs index-provider roles in IPNI someguy is a caching proxy for routing lookups (read path), index-provider is a sidecar for publishing (write path) --- docs/concepts/ipni.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/concepts/ipni.md b/docs/concepts/ipni.md index 4bd3410ba..e276ff678 100644 --- a/docs/concepts/ipni.md +++ b/docs/concepts/ipni.md @@ -12,8 +12,8 @@ Opting into indexing by IPNI can be done from: - [Lotus](https://lotus.filecoin.io/), the reference implementation for the [Filecoin network](https://docs.filecoin.io/). - [Boost](https://boost.filecoin.io/), a tool for Filecoin storage providers to manage Filecoin data onboarding and retrieval. -- A self-hosted IPFS server with the [someguy](https://github.com/ipfs/someguy) sidecar integrated. -- A production-grade IPFS deployment configured to support the [IPNI provider sidecar](https://github.com/ipni/index-provider) (configuration may be more involved) +- A self-hosted IPFS server with the [someguy](https://github.com/ipfs/someguy) caching proxy for routing lookups to IPNI (read path). +- A production-grade IPFS deployment configured to support the [IPNI index-provider](https://github.com/ipni/index-provider) sidecar for publishing to IPNI (write path). IPNI is designed to create an alternate routing and discovery infrastructure outside of and independent from the [Kademlia Distributed Hash Table (DHT)](../concepts/dht.md#kademlia). From da618ac69b814f89f9fa45acec5af490e3fc55ed Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Wed, 21 Jan 2026 05:43:29 +0100 Subject: [PATCH 5/6] docs: editorial cleanup of IPNI page - break up dense paragraphs - simplify sentence structure - clarify DHT re-announcement timing - reword benefits section for clarity --- docs/concepts/ipni.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/concepts/ipni.md b/docs/concepts/ipni.md index e276ff678..586890bbe 100644 --- a/docs/concepts/ipni.md +++ b/docs/concepts/ipni.md @@ -23,20 +23,21 @@ For a deeper dive into the technical specification for IPNI, see [https://github ## What use-cases IPNI serves -While in-protocol routing and discovery have [advanced leaps and bounds in recent versions of Kubo and Helia](https://ipshipyard.com/blog/2025-dht-provide-sweep/), with well-tuned test server performance benchmarked in the high tens of millions of CIDs announce-able within the 24th recursion window, there continue to be use-cases where keeping announcements live on the DHT is onerous, or swarms where that volume of message would not be propagated before the cycle recursed and began again. +While in-protocol routing and discovery have [advanced leaps and bounds in recent versions of Kubo and Helia](https://ipshipyard.com/blog/2025-dht-provide-sweep/), with well-tuned test server performance benchmarked in the high tens of millions of CIDs announce-able within the 24-hour re-announcement window, there continue to be use-cases where keeping announcements live on the DHT is onerous, or swarms where that volume of messages would not be propagated before the next re-announcement cycle. + By comparison, announcements to an IPNI indexer only need to be made once, making them particularly attractive to announcers of large volumes of infrequently-sought CIDs, like large-scale providers of "cold storage" in the Filecoin economy or archivers of public open data. -To support performant _retrievals_ of unsealed Filecoin and IPFS pinned data with a speed comparable to a CDN, a reliable, distributed index of all data and the peer(s) hosting and/or caching it must be assembled, and this index must be replicated to be geographically near the lookups. Comparable lookup and time-to-first-byte metrics are quite difficult to achieve on the DHT. +To support performant _retrievals_ of unsealed Filecoin and IPFS pinned data with a speed comparable to a CDN, a reliable, distributed index must be assembled that maps data to the peer(s) hosting or caching it, and this index must be replicated to be geographically near the lookups. Comparable lookup and time-to-first-byte metrics are quite difficult to achieve on the DHT. -One advantage to being completely orthogonal to DHT-based announcing and discovery is that over the life of a gateway or other data provider service, one or more IPNI-style indexer systems can be opted into or out of, without affecting DHT performance that can similarly be taken down and brought back up. -Many users have found value in both opting into IPNI indexing for long-tail discovery, but still announcing all new content to peers in the DHT at time of publication to allow for resilience and caching in realtime. This mixed approach also sidesteps the complexity of re-announcing and keeping DHT announcements circulating over time for content that is not expected to be cached and dynamically distributed peer-to-peer over the course of its lifecycle. +One advantage to being completely orthogonal to DHT-based announcing and discovery is operational flexibility. Over the life of a gateway or other data provider service, one or more IPNI-style indexer systems can be opted into or out of without affecting DHT performance. The reverse is also true: DHT announcing can be taken down and brought back up independently. +Many users have found value in opting into IPNI indexing for long-tail discovery while still announcing all new content to peers in the DHT at time of publication to allow for resilience and caching in realtime. This mixed approach also sidesteps the complexity of re-announcing and keeping DHT announcements circulating over time for content that is not expected to be cached and dynamically distributed peer-to-peer over the course of its lifecycle. ### How IPNI benefits IPFS The indexer offers several benefits to IPFS, including: - **Faster data retrieval**: By maintaining an additional layer of information on top of the DHT, the indexer can help speed up data location and retrieval. -- **Reduced resource consumption**: The indexer can help reduce the amount of bandwidth and processing power expended circulating and re-announcing CID location records, particularly for individual nodes for whom this peer-to-peer network traffic is costly (i.e. in commercial data centers). While the egress/networking and compute needed to periodically reannounce can now be delegated, it still needs to be done every 24 hours. +- **Reduced resource consumption**: The indexer can help reduce the amount of bandwidth and processing power expended circulating and re-announcing CID location records. This is particularly beneficial for nodes in commercial data centers where peer-to-peer network traffic is costly. Unlike DHT announcements which must be repeated every 24 hours, IPNI announcements only need to be made once. - **Improved scalability**: With the indexer, IPFS can reduce the portion of network traffic used by large-volume publishers, allowing the rest of the network to scale more effectively and support broader and heterogeneous network topologies. ## The IPNI ecosystem @@ -69,7 +70,7 @@ The indexer works in conjunction with the existing DHT to improve data location When a user searches for a piece of data using a CID or multihash, the indexer is consulted first. If the data is found in the index, the user is directly connected to the node hosting the data, resulting in faster retrieval. If the data is not found in the index, the user falls back to the traditional DHT-based search, ensuring that the data can still be located even if it's not in the indexer. -By providing this additional layer of information, the indexer helps to speed up data location and retrieval, reduce resource consumption, and improve the overall scalability of IPFS for use-cases further from the DHT. +By providing this additional layer of information, the indexer helps to speed up data location and retrieval, reduce resource consumption, and improve the overall scalability of IPFS for use-cases where DHT performance, resiliency, or security characteristics alone are insufficient. ### Example: finding providers via `/routing/v1` From 6b861de8440bb40d92f724ebf61fcaa945fc59f0 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Wed, 21 Jan 2026 05:48:45 +0100 Subject: [PATCH 6/6] docs: grammar fix and add words to spellcheck ignore MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - "outside of and independent from" → "outside and independent of" - add archivers, benchmarked, realtime to pln-ignore.txt --- .github/styles/pln-ignore.txt | 3 +++ docs/concepts/ipni.md | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/styles/pln-ignore.txt b/.github/styles/pln-ignore.txt index 28a71cd2f..081466fcc 100644 --- a/.github/styles/pln-ignore.txt +++ b/.github/styles/pln-ignore.txt @@ -9,6 +9,7 @@ apis arbol arbol's arbol('s) +archivers arg astro atcute @@ -16,6 +17,7 @@ auditable audius auspinner Bacalhau +benchmarked bitswap bit[ss]wap blockchain @@ -235,6 +237,7 @@ qm rabin rasterio reachability +realtime readmes referenceable repo diff --git a/docs/concepts/ipni.md b/docs/concepts/ipni.md index 586890bbe..07b5de912 100644 --- a/docs/concepts/ipni.md +++ b/docs/concepts/ipni.md @@ -15,7 +15,7 @@ Opting into indexing by IPNI can be done from: - A self-hosted IPFS server with the [someguy](https://github.com/ipfs/someguy) caching proxy for routing lookups to IPNI (read path). - A production-grade IPFS deployment configured to support the [IPNI index-provider](https://github.com/ipni/index-provider) sidecar for publishing to IPNI (write path). -IPNI is designed to create an alternate routing and discovery infrastructure outside of and independent from the [Kademlia Distributed Hash Table (DHT)](../concepts/dht.md#kademlia). +IPNI is designed to create an alternate routing and discovery infrastructure outside and independent of the [Kademlia Distributed Hash Table (DHT)](../concepts/dht.md#kademlia). :::callout For a deeper dive into the technical specification for IPNI, see [https://github.com/ipni/specs/blob/main/IPNI.md](https://github.com/ipni/specs/blob/main/IPNI.md).