From 1fff4187869578546a28f317943424e89f9050ca Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 20 Sep 2023 13:25:24 +0200 Subject: [PATCH 01/46] chore: move UNIXFS.md (preserve history) --- UNIXFS.md => src/architecture/unixfs.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename UNIXFS.md => src/architecture/unixfs.md (100%) diff --git a/UNIXFS.md b/src/architecture/unixfs.md similarity index 100% rename from UNIXFS.md rename to src/architecture/unixfs.md From 86b93cf01d11628b89c8094e830cdfe91fe05b7a Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 20 Sep 2023 13:26:31 +0200 Subject: [PATCH 02/46] chore: add UNIXFS.md to link to new website --- UNIXFS.md | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 UNIXFS.md diff --git a/UNIXFS.md b/UNIXFS.md new file mode 100644 index 000000000..00444fe49 --- /dev/null +++ b/UNIXFS.md @@ -0,0 +1,3 @@ +# UnixFS + +Moved to https://specs.ipfs.tech/architecture/unixfs/ From 97abffc67a5ad8a169babad93d436cfef55e9fae Mon Sep 17 00:00:00 2001 From: Jorropo Date: Mon, 10 Oct 2022 17:05:17 +0200 Subject: [PATCH 03/46] docs: Write UNIXFSv1 spec --- src/architecture/unixfs.md | 382 +++++++++++++++++++++++++++++++------ 1 file changed, 324 insertions(+), 58 deletions(-) diff --git a/src/architecture/unixfs.md b/src/architecture/unixfs.md index a53c7af2c..4130ac6fc 100644 --- a/src/architecture/unixfs.md +++ b/src/architecture/unixfs.md @@ -7,9 +7,7 @@ **Abstract** -UnixFS is a [protocol-buffers](https://developers.google.com/protocol-buffers/) based format for describing files, directories, and symlinks in IPFS. The current implementation of UnixFS has grown organically and does not have a clear specification document. See [“implementations”](#implementations) below for reference implementations you can examine to understand the format. - -Draft work and discussion on a specification for the upcoming version 2 of the UnixFS format is happening in the [`ipfs/unixfs-v2` repo](https://github.com/ipfs/unixfs-v2). Please see the issues there for discussion and PRs for drafts. When the specification is completed there, it will be copied back to this repo and replace this document. +UnixFS is a [protocol-buffers](https://developers.google.com/protocol-buffers/) based format for describing files, directories, and symlinks as merkle-dags in IPFS. ## Table of Contents @@ -29,19 +27,41 @@ Draft work and discussion on a specification for the upcoming version 2 of the U - [Side trees](#side-trees) - [Side database](#side-database) -## Implementations +## How to read a Node -- JavaScript - - Data Formats - [unixfs](https://github.com/ipfs/js-ipfs-unixfs) - - Importer - [unixfs-importer](https://github.com/ipfs/js-ipfs-unixfs-importer) - - Exporter - [unixfs-exporter](https://github.com/ipfs/js-ipfs-unixfs-exporter) -- Go - - [`ipfs/go-ipfs/unixfs`](https://github.com/ipfs/go-ipfs/tree/b3faaad1310bcc32dc3dd24e1919e9edf51edba8/unixfs) - - Protocol Buffer Definitions - [`ipfs/go-ipfs/unixfs/pb`](https://github.com/ipfs/go-ipfs/blob/b3faaad1310bcc32dc3dd24e1919e9edf51edba8/unixfs/pb/unixfs.proto) +To read a node, first get a CID. This is what we will decode. + +To recap, every [CID](https://github.com/multiformats/cid) includes: +1. A [multicodec](https://github.com/multiformats/multicodec), also called codec. +1. A [Multihash](https://github.com/multiformats/multihash) used to specify a hashing algorithm, the hashing parameters and the hash digest. + +The first step is to get the block, that means the actual bytes which when hashed (using the hash function specified in the multihash) gives you the same multihash value back. + +### Multicodecs -## Data Format +With Unixfs we deal with two codecs which will be decoded differently: +- `Raw`, single block files +- `dag-pb`, can be any nodes -The UnixfsV1 data format is represented by this protobuf: +#### `Raw` blocks + +The simplest nodes use `Raw` encoding. + +They are always implicitly of type `file`. + +They can be recognized because their CIDs have `Raw` codec. + +The file content is purely the block body. + +They never have any children nodes, and thus are also known as single block files. + +Their sizes (both `dagsize` and `blocksize`) is the length of the block body. + +#### `dag-pb` nodes + +##### Data Format + +The UnixfsV1 `Data` message format is represented by this protobuf: ```protobuf message Data { @@ -74,15 +94,235 @@ message UnixTime { } ``` -This `Data` object is used for all non-leaf nodes in Unixfs. +##### IPLD `dag-pb` + +A very important other spec for unixfs is the [`dag-pb`](https://ipld.io/specs/codecs/dag-pb/spec/) IPLD spec: + +```protobuf +message PBLink { + // binary CID (with no multibase prefix) of the target object + optional bytes Hash = 1; + + // UTF-8 string name + optional string Name = 2; + + // cumulative size of target object + optional uint64 Tsize = 3; // also known as dagsize +} + +message PBNode { + // refs to other objects + repeated PBLink Links = 2; + + // opaque user data + optional bytes Data = 1; +} +``` + +The two different schemas plays together and it is important to understand their different effect, +- The `dag-pb` / `PBNode` protobuf is the "outside" protobuf message; in other words, it is the first message decoded. This protobuf contains the list of links and some "opaque user data". +- The `Data` message is the "inside" protobuf message. After the "outside" `dag-pb` (also known as `PBNode`) object is decoded, `Data` is decoded from the bytes inside the `PBNode.Data` field. This contains the rest of information. + +In other words, we have a serialized protobuf message stored inside another protobuf message. +For clarity, the spec document may represents these nested protobufs as one object. In this representation, it is implied that the `PBNode.Data` field is encoded in a prototbuf. + +##### Different Data types + +`dag-pb` nodes supports many different types, which can be found in `decodeData(PBNode.Data).Type`. Every type is handled differently. + +###### `File` type + +####### The _sister-lists_ `PBNode.Links` and `decodeMessage(PBNode.Data).blocksizes` + +The _sister-lists_ are the key point of why `dag-pb` is important for files. + +This allows us to concatenate smaller files together. + +Linked files would be loaded recursively with the same process following a DFS (Depth-First-Search) order. + +Child nodes must be of type file (so `dag-pb` where type is `File` or `Raw`) + +For example this example pseudo-json block: +```json +{ + "Links": [{"Hash":"Qmfoo"}, {"Hash":"Qmbar"}], + "Data": { + "Type": "File", + "blocksizes": [20, 30] + } +} +``` + +This indicates that this file is the concatenation of the `Qmfoo` and `Qmbar` files. + +When reading a file represented with `dag-pb`, the `blocksizes` array gives us the size in bytes of the partial file content present in child DAGs. +Each index in `PBNode.Links` MUST have a corresponding chunk size stored at the same index in `decodeMessage(PBNode.Data).blocksizes`. + +Implementers need to be extra careful to ensure the values in `Data.blocksizes` are calculated by following the definition from [Blocksize](#blocksize) section. + +This allows to do fast indexing into the file, for example if someone is trying to read bytes 25 to 35 we can compute an offset list by summing all previous indexes in `blocksizes`, then do a search to find which indexes contain the range we are intrested in. + +For example here the offset list would be `[0, 20]` and thus we know we only need to download `Qmbar` to get the range we are intrested in. + +UnixFS parser MUST error if `blocksizes` or `Links` are not of the same length. + +####### `decodeMessage(PBNode.Data).Data` + +This field is an array of bytes, it is file content and is appended before the links. + +This must be taken into a count when doing offsets calculations (the len of the `Data.Data` field define the value of the zeroth element of the offset list when computing offsets). + +####### `PBNode.Links[].Name` with Files + +This field makes sense only in directory contexts and MUST be absent when creating a new file `PBNode`. +For historic reasons, implementations parsing third-party data SHOULD accept empty value here. + +If this field is present and non empty, the file is invalid and parser MUST error. + +####### `Blocksize` of a dag-pb file + +This is not a field present in the block directly, but rather a computable property of `dag-pb` which would be used in parent node in `decodeMessage(PBNode.Data).blocksizes`. +It is the sum of the length of the `Data.Data` field plus the sum of all link's blocksizes. + +####### `PBNode.Data.Filesize` + +If present, this field must be equal to the `Blocksize` computation above, else the file is invalid. + +####### Path resolution + +A file terminates UnixFS content path. + +Any attempt of path resolution on `File` type MUST error. + +###### `Directory` Type + +A directory node is a named collection of nodes. + +The minimum valid `PBNode.Data` field for a directory is (pseudo-json): `{"Type":"Directory"}`, other values are covered in Metadata. + +Every link in the Links list is an entry (children) of the directory, and the `PBNode.Links[].Name` field give you the name. + +####### Link ordering + +The cannonical sorting order is lexicographical over the names. + +In theory there is no reason an encoder couldn't use an other ordering, however this lose some of it's meaning when mapped into most file systems today (most file systems consider directories are unordered-key-value objects). + +A decoder SHOULD if it can, preserve the order of the original files in however it consume thoses names. + +However when some implementation decode, modify then reencode some, the orignal links order fully lose it's meaning. (given that there is no way to indicate which sorting was used originally) + +####### Path Resolution + +Pop the left most component of the path, and try to match it to one of the Name in Links. + +If you find a match you can then remember the CID. You MUST continue your search, however if you find a match again you MUST error. + +Assuming no errors were raised, you can continue to the path resolution on the mainaing component and on the CID you poped. + +####### Duplicate names + +Duplicate names are not allowed, if two identical names are present in an directory, the decoder MUST error. -For files that are comprised of more than a single block, the 'Type' field will be set to 'File', the 'filesize' field will be set to the total number of bytes in the file (not the graph structure) represented by this node, and 'blocksizes' will contain a list of the filesizes of each child node. +###### `Symlink` type -This data is serialized and placed inside the 'Data' field of the outer merkledag protobuf, which also contains the actual links to the child nodes of this object. +Symlinks MUST NOT have childs. -For files comprised of a single block, the 'Type' field will be set to 'File', 'filesize' will be set to the total number of bytes in the file and the file data will be stored in the 'Data' field. +Their Data.Data field is a POSIX path that maybe appended in front of the currently remaining path component stack. -## Metadata +####### Path resolution on symlinks + +There is no current consensus on how pathing over symlinks should behave. +Some implementations return symlinks objects and fail if a consumer tries to follow it through. + +Following the POSIX spec over the current unixfs path context is probably fine. + +###### `HAMTDirectory` + +Thoses nodes are also sometimes called sharded directories, they allow to split directories into many blocks when they are so big that they don't fit into one single block anymore. + +- `node.Data.hashType` indicates a multihash function to use to digest path components used for sharding. +It MUST be murmur3-x64-64 (multihash `0x22`). +- `node.Data.Data` is some bitfield, ones indicates whether or not the links are part of this HAMT or leaves of the HAMT. +The usage of this field is unknown given you can deduce the same information from the links names. +- `node.Data.fanout` MUST be a power of two. This encode the number of hash permutations that will be used on each resolution step. +The log base 2 of the fanout indicate how wide the bitmask will be on the hash at for that step. `fanout` MUST be between 8 and probably 65536. + +####### `node.Links[].Name` on HAMTs + +They start by some uppercase hex encoded prefix which is `log2(fanout)` bits wide + +####### Path resolution on HAMTs + +Steps: +1. Take the current path component then hash it using the multihash id provided in `Data.hashType`. +2. Pop the `log2(fanout)` lowest bits from the path component hash digest, then hex encode (using 0-F) thoses bits using little endian thoses bits and find the link that starts with this hex encoded path. +3. If the link name is exactly as long as the hex encoded representation, follow the link and repeat step 2 with the child node and the remaining bit stack. The child node MUST be a hamt directory else the directory is invalid, else continue. +4. Compare the remaining part of the last name you found, if it match the original name you were trying to resolve you successfully resolved a path component, everything past the hex encoded prefix is the name of that element (usefull when listing childs of this directory). + + +###### `TSize` / `DagSize` + +This is an optional field for Links of `dag-pb` nodes, **it does not represent any meaningfull information of the underlying structure** and no known usage of it to this day (altho some implementation emit thoses). + +To compute the `dagsize` of a node (which would be stored in the parents) you sum the length of the dag-pb outside message binary length, plus the blocksizes of all child files. + +An example of where this could be usefull is as a hint to smart download clients, for example if you are downloading a file concurrently from two sources that have radically different speeds, it would probably be more efficient to download bigger links from the fastest source, and smaller ones from the slowest source. + + +There is no failure mode known for this field, so your implementation should be able to decode nodes where this field is wrong (not the value you expect), partially or completely missing. This also allows smarter encoder to give a more accurate picture (for example don't count duplicate blocks, ...). + +### Paths + +Paths first start with `/`or `/ipfs//` where `` is a [multibase](https://github.com/multiformats/multibase) encoded [CID](https://github.com/multiformats/cid). +The CID encoding MUST NOT use a multibase alphabet that have `/` (`0x2f`) unicode codepoints however CIDs may use a multibase encoding with a `/` in the alphabet if the encoded CID does not contain `/` once encoded. + +Everything following the CID is a collection of path component (some bytes) seperated by `/` (`0x2f`), read from left to right. +This is inspired by POSIX paths. + +- Components MUST NOT contain `/` unicode codepoints because else it would break the path into two components. +- Components SHOULD be UTF8 unicode. +- Components are case sensitive. + +#### Escaping + +The `\` may be supposed to trigger an escape sequence. + +This might be a thing, but is broken and inconsistent current implementations. +So until we agree on a new spec for this, you SHOULD NOT use any escape sequence and non ascii character. + +#### Relative path components + +Thoses path components must be resolved before trying to work on the path. + +- `.` points to the current node, those path components must be removed. +- `..` points to the parent, they must be removed first to last however when you remove a `..` you also remove the previous component on the left. If there is no component on the left to remove leave the `..` as-is however this is an attempt for an out-of-bound path resolution which mean you MUST error. + +#### Restricted names + +Thoses names SHOULD NOT be used: + +- The `.` string. This represents the self node in POSIX pathing. +- The `..` string. This represents the parent node in POSIX pathing. +- nothing (the empty string) We don't actually know the failure mode for this, but it really feels like this shouldn't be a thing. +- Any string containing a NUL (0x00) byte, this is often used to signify string terminations in some systems (such as most C compatible systems), and many unix file systems don't accept this character in path components. + +### Glossary + +- Node, Block + A node is a word from graph theory, this is the smallest unit present in the graph. + Due to how unixfs work, there is a 1 to 1 mapping between nodes and blocks. +- File + A file is some container over an arbitrary sized amounts of bytes. + Files can be said to be single block, or multi block, in the later case they are the concatenation of multiple children files. +- Directory, Folder + A named collection of child nodes. +- HAMT Directory + This is a [Hashed-Array-Mapped-Trie](https://en.wikipedia.org/wiki/Hash_array_mapped_trie) data structure representing a Directory, those may be used to split directories into multiple blocks when they get too big, and the list of children does not fit in a single block. +- Symlink + This represents a POSIX Symlink. + +### Metadata UnixFS currently supports two optional metadata fields: @@ -112,45 +352,9 @@ UnixFS currently supports two optional metadata fields: - When no `mtime` is specified or the resulting `UnixTime` is negative: implementations must assume `0`/`1970-01-01T00:00:00Z` ( note that such values are not merely academic: e.g. the OpenVMS epoch is `1858-11-17T00:00:00Z` ) - When the resulting `UnixTime` is larger than the targets range ( e.g. 32bit vs 64bit mismatch ) implementations must assume the highest possible value in the targets range ( in most cases that would be `2038-01-19T03:14:07Z` ) -### Deduplication and inlining - -Where the file data is small it would normally be stored in the `Data` field of the UnixFS `File` node. - -To aid in deduplication of data even for small files, file data can be stored in a separate node linked to from the `File` node in order for the data to have a constant [CID] regardless of the metadata associated with it. - -As a further optimization, if the `File` node's serialized size is small, it may be inlined into its v1 [CID] by using the [`identity`](https://github.com/multiformats/multicodec/blob/master/table.csv) [multihash]. - -## Importing - -Importing a file into unixfs is split up into two parts. The first is chunking, the second is layout. - -### Chunking - -Chunking has two main parameters, chunking strategy and leaf format. - -Leaf format should always be set to 'raw', this is mainly configurable for backwards compatibility with earlier formats that used a Unixfs Data object with type 'Raw'. Raw leaves means that the nodes output from chunking will be just raw data from the file with a CID type of 'raw'. - -Chunking strategy currently has two different options, 'fixed size' and 'rabin'. Fixed size chunking will chunk the input data into pieces of a given size. Rabin chunking will chunk the input data using rabin fingerprinting to determine the boundaries between chunks. - - -### Layout - -Layout defines the shape of the tree that gets built from the chunks of the input file. - -There are currently two options for layout, balanced, and trickle. -Additionally, a 'max width' must be specified. The default max width is 174. - -The balanced layout creates a balanced tree of width 'max width'. The tree is formed by taking up to 'max width' chunks from the chunk stream, and creating a unixfs file node that links to all of them. This is repeated until 'max width' unixfs file nodes are created, at which point a unixfs file node is created to hold all of those nodes, recursively. The root node of the resultant tree is returned as the handle to the newly imported file. - -If there is only a single chunk, no intermediate unixfs file nodes are created, and the single chunk is returned as the handle to the file. - -## Exporting - -To read the file data out of the unixfs graph, perform an in order traversal, emitting the data contained in each of the leaves. - ## Design decision rationale -### Metadata +### `mtime` and `mode` metadata support in UnixFSv1.5 Metadata support in UnixFSv1.5 has been expanded to increase the number of possible use cases. These include rsync and filesystem based package managers. @@ -227,7 +431,69 @@ Fractional values are effectively a random number in the range 1 ~ 999,999,999. 2^28 nanoseconds ( 268,435,456 ) in most cases. Therefore, the fractional part is represented as a 4-byte `fixed32`, [as per Google's recommendation](https://developers.google.com/protocol-buffers/docs/proto#scalar). -[multihash]: https://tools.ietf.org/html/draft-multiformats-multihash-00 -[CID]: https://docs.ipfs.io/guides/concepts/cid/ +## References + +[multihash]: https://tools.ietf.org/html/draft-multiformats-multihash-05 +[CID]: https://github.com/multiformats/cid/ [Bitswap]: https://github.com/ipfs/specs/blob/master/BITSWAP.md -[MFS]: https://docs.ipfs.io/guides/concepts/mfs/ + +# Notes for Implementers + +This section and included subsections are not authoritative. + +## Implementations + +- JavaScript + - Data Formats - [unixfs](https://github.com/ipfs/js-ipfs-unixfs) + - Importer - [unixfs-importer](https://github.com/ipfs/js-ipfs-unixfs-importer) + - Exporter - [unixfs-exporter](https://github.com/ipfs/js-ipfs-unixfs-exporter) +- Go + - Protocol Buffer Definitions - [`ipfs/go-unixfs/pb`](https://github.com/ipfs/go-unixfs/blob/707110f05dac4309bdcf581450881fb00f5bc578/pb/unixfs.proto) + - [`ipfs/go-unixfs`](https://github.com/ipfs/go-unixfs/) + - `go-ipld-prime` implementation [`ipfs/go-unixfsnode`](https://github.com/ipfs/go-unixfsnode) +- Rust + - [`iroh-unixfs`](https://github.com/n0-computer/iroh/tree/b7a4dd2b01dbc665435659951e3e06d900966f5f/iroh-unixfs) + - [`unixfs-v1`](https://github.com/ipfs-rust/unixfsv1) + +## Simple `Raw` Example + +In this example, we will build a `Raw` file with the string `test` as its content. + +1. First hash the data: +```console +$ echo -n "test" | sha256sum +9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 - +``` + +2. Add the CID prefix: +``` +f this is the multibase prefix, we need it because we are working with a hex CID, this is omitted for binary CIDs + 01 the CID version, here one + 55 the codec, here we MUST use Raw because this is a Raw file + 12 the hashing function used, here sha256 + 20 the digest length 32 bytes + 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 the digest we computed earlier +``` + +3. Profit +Assuming we stored this block in some implementation of our choice which makes it accessible to our client, we can try to decode it: +```console +$ ipfs cat f015512209f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 +test +``` + + +### Offset list + +The offset list isn't the only way to use blocksizes and reach a correct implementation, it is a simple cannonical one, python pseudo code to compute it looks like this: +```python +def offsetlist(node): + unixfs = decodeDataField(node.Data) + if len(node.Links) != len(unixfs.Blocksizes): + raise "unmatched sister-lists" # error messages are implementation details + + cursor = len(unixfs.Data) if unixfs.Data else 0 + return [cursor] + [cursor := cursor + size for size in unixfs.Blocksizes[:-1]] +``` + +This will tell you which offset inside this node the children at the corresponding index starts to cover. (using `[x,y)` ranging) From c4e812a74f7ea44438d9359f1b4682f63e8b7393 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 20 Sep 2023 13:36:52 +0200 Subject: [PATCH 04/46] chore: editorial fixes --- src/architecture/unixfs.md | 155 ++++++++++++++++++++----------------- 1 file changed, 83 insertions(+), 72 deletions(-) diff --git a/src/architecture/unixfs.md b/src/architecture/unixfs.md index 4130ac6fc..2d6dec8f2 100644 --- a/src/architecture/unixfs.md +++ b/src/architecture/unixfs.md @@ -1,32 +1,43 @@ -# ![](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square) UnixFS - -**Author(s)**: -- NA - -* * * - -**Abstract** +--- +title: UnixFS +description: > + UnixFS is a Protocol Buffers-based format for describing files, directories, + and symlinks as DAGs in IPFS. +date: 2022-10-10 +maturity: reliable +editors: + - name: David Dias + github: daviddias + affiliation: + name: Protocol Labs + url: https://protocol.ai/ + - name: Jeromy Johnson + github: whyrusleeping + affiliation: + name: Protocol Labs + url: https://protocol.ai/ + - name: Alex Potsides + github: achingbrain + affiliation: + name: Protocol Labs + url: https://protocol.ai/ + - name: Peter Rabbitson + github: ribasushi + affiliation: + name: Protocol Labs + url: https://protocol.ai/ + - name: Hugo Valtier + github: jorropo + affiliation: + name: Protocol Labs + url: https://protocol.ai/ + +tags: ['architecture'] +order: 1 +--- UnixFS is a [protocol-buffers](https://developers.google.com/protocol-buffers/) based format for describing files, directories, and symlinks as merkle-dags in IPFS. -## Table of Contents - -- [Implementations](#implementations) -- [Data Format](#data-format) -- [Metadata](#metadata) - - [Deduplication and inlining](#deduplication-and-inlining) -- [Importing](#importing) - - [Chunking](#chunking) - - [Layout](#layout) -- [Exporting](#exporting) -- [Design decision rationale](#design-decision-rationale) - - [Metadata](#metadata-1) - - [Separate Metadata node](#separate-metadata-node) - - [Metadata in the directory](#metadata-in-the-directory) - - [Metadata in the file](#metadata-in-the-file) - - [Side trees](#side-trees) - - [Side database](#side-database) - ## How to read a Node To read a node, first get a CID. This is what we will decode. @@ -65,32 +76,32 @@ The UnixfsV1 `Data` message format is represented by this protobuf: ```protobuf message Data { - enum DataType { - Raw = 0; - Directory = 1; - File = 2; - Metadata = 3; - Symlink = 4; - HAMTShard = 5; - } - - required DataType Type = 1; - optional bytes Data = 2; - optional uint64 filesize = 3; - repeated uint64 blocksizes = 4; - optional uint64 hashType = 5; - optional uint64 fanout = 6; - optional uint32 mode = 7; - optional UnixTime mtime = 8; + enum DataType { + Raw = 0; + Directory = 1; + File = 2; + Metadata = 3; + Symlink = 4; + HAMTShard = 5; + } + + required DataType Type = 1; + optional bytes Data = 2; + optional uint64 filesize = 3; + repeated uint64 blocksizes = 4; + optional uint64 hashType = 5; + optional uint64 fanout = 6; + optional uint32 mode = 7; + optional UnixTime mtime = 8; } message Metadata { - optional string MimeType = 1; + optional string MimeType = 1; } message UnixTime { - required int64 Seconds = 1; - optional fixed32 FractionalNanoseconds = 2; + required int64 Seconds = 1; + optional fixed32 FractionalNanoseconds = 2; } ``` @@ -100,22 +111,22 @@ A very important other spec for unixfs is the [`dag-pb`](https://ipld.io/specs/c ```protobuf message PBLink { - // binary CID (with no multibase prefix) of the target object - optional bytes Hash = 1; + // binary CID (with no multibase prefix) of the target object + optional bytes Hash = 1; - // UTF-8 string name - optional string Name = 2; + // UTF-8 string name + optional string Name = 2; - // cumulative size of target object - optional uint64 Tsize = 3; // also known as dagsize + // cumulative size of target object + optional uint64 Tsize = 3; // also known as dagsize } message PBNode { - // refs to other objects - repeated PBLink Links = 2; + // refs to other objects + repeated PBLink Links = 2; - // opaque user data - optional bytes Data = 1; + // opaque user data + optional bytes Data = 1; } ``` @@ -145,11 +156,11 @@ Child nodes must be of type file (so `dag-pb` where type is `File` or `Raw`) For example this example pseudo-json block: ```json { - "Links": [{"Hash":"Qmfoo"}, {"Hash":"Qmbar"}], - "Data": { - "Type": "File", - "blocksizes": [20, 30] - } + "Links": [{"Hash":"Qmfoo"}, {"Hash":"Qmbar"}], + "Data": { + "Type": "File", + "blocksizes": [20, 30] + } } ``` @@ -368,7 +379,7 @@ This was ultimately rejected for a number of reasons: 1. You would always need to retrieve an additional node to access file data which limits the kind of optimizations that are possible. - For example many files are under the 256KiB block size limit, so we tend to inline them into the describing UnixFS `File` node. This would not be possible with an intermediate `Metadata` node. + For example many files are under the 256KiB block size limit, so we tend to inline them into the describing UnixFS `File` node. This would not be possible with an intermediate `Metadata` node. 2. The `File` node already contains some metadata (e.g. the file size) so metadata would be stored in multiple places which complicates forwards compatibility with UnixFSv2 as to map between metadata formats potentially requires multiple fetch operations @@ -398,7 +409,7 @@ Downsides to this approach are: 1. Two users adding the same file to IPFS at different times will have different [CID]s due to the `mtime`s being different. - If the content is stored in another node, its [CID] will be constant between the two users but you can't navigate to it unless you have the parent node which will be less available due to the proliferation of [CID]s. + If the content is stored in another node, its [CID] will be constant between the two users but you can't navigate to it unless you have the parent node which will be less available due to the proliferation of [CID]s. 2. Metadata is also impossible to remove without changing the [CID], so metadata becomes part of the content. @@ -448,12 +459,12 @@ This section and included subsections are not authoritative. - Importer - [unixfs-importer](https://github.com/ipfs/js-ipfs-unixfs-importer) - Exporter - [unixfs-exporter](https://github.com/ipfs/js-ipfs-unixfs-exporter) - Go - - Protocol Buffer Definitions - [`ipfs/go-unixfs/pb`](https://github.com/ipfs/go-unixfs/blob/707110f05dac4309bdcf581450881fb00f5bc578/pb/unixfs.proto) + - Protocol Buffer Definitions - [`ipfs/go-unixfs/pb`](https://github.com/ipfs/go-unixfs/blob/707110f05dac4309bdcf581450881fb00f5bc578/pb/unixfs.proto) - [`ipfs/go-unixfs`](https://github.com/ipfs/go-unixfs/) - - `go-ipld-prime` implementation [`ipfs/go-unixfsnode`](https://github.com/ipfs/go-unixfsnode) + - `go-ipld-prime` implementation [`ipfs/go-unixfsnode`](https://github.com/ipfs/go-unixfsnode) - Rust - - [`iroh-unixfs`](https://github.com/n0-computer/iroh/tree/b7a4dd2b01dbc665435659951e3e06d900966f5f/iroh-unixfs) - - [`unixfs-v1`](https://github.com/ipfs-rust/unixfsv1) + - [`iroh-unixfs`](https://github.com/n0-computer/iroh/tree/b7a4dd2b01dbc665435659951e3e06d900966f5f/iroh-unixfs) + - [`unixfs-v1`](https://github.com/ipfs-rust/unixfsv1) ## Simple `Raw` Example @@ -488,12 +499,12 @@ test The offset list isn't the only way to use blocksizes and reach a correct implementation, it is a simple cannonical one, python pseudo code to compute it looks like this: ```python def offsetlist(node): - unixfs = decodeDataField(node.Data) - if len(node.Links) != len(unixfs.Blocksizes): - raise "unmatched sister-lists" # error messages are implementation details + unixfs = decodeDataField(node.Data) + if len(node.Links) != len(unixfs.Blocksizes): + raise "unmatched sister-lists" # error messages are implementation details - cursor = len(unixfs.Data) if unixfs.Data else 0 - return [cursor] + [cursor := cursor + size for size in unixfs.Blocksizes[:-1]] + cursor = len(unixfs.Data) if unixfs.Data else 0 + return [cursor] + [cursor := cursor + size for size in unixfs.Blocksizes[:-1]] ``` This will tell you which offset inside this node the children at the corresponding index starts to cover. (using `[x,y)` ranging) From d2d9f670d813a028350dd67c7a031038305e4d4e Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Wed, 20 Sep 2023 15:31:41 +0200 Subject: [PATCH 05/46] chore: further editorial changes --- src/architecture/unixfs.md | 602 ++++++++++++++++++++++--------------- 1 file changed, 355 insertions(+), 247 deletions(-) diff --git a/src/architecture/unixfs.md b/src/architecture/unixfs.md index 2d6dec8f2..0b7887f71 100644 --- a/src/architecture/unixfs.md +++ b/src/architecture/unixfs.md @@ -36,43 +36,71 @@ tags: ['architecture'] order: 1 --- -UnixFS is a [protocol-buffers](https://developers.google.com/protocol-buffers/) based format for describing files, directories, and symlinks as merkle-dags in IPFS. +UnixFS is a [protocol-buffers][protobuf]-based format for describing files, +directories and symlinks as DAGs in IPFS. -## How to read a Node +## Nodes -To read a node, first get a CID. This is what we will decode. +A :dfn[Node] is the smallest unit present in a graph, and it comes from graph +theory. In UnixFS, there is a 1 to 1 mapping between nodes and blocks. Therefore, +they are used interchangeably in this document. -To recap, every [CID](https://github.com/multiformats/cid) includes: -1. A [multicodec](https://github.com/multiformats/multicodec), also called codec. -1. A [Multihash](https://github.com/multiformats/multihash) used to specify a hashing algorithm, the hashing parameters and the hash digest. +A node is addressed by a [CID]. In order to be able to read a node, its [CID] is +required. A [CID] includes two important information: -The first step is to get the block, that means the actual bytes which when hashed (using the hash function specified in the multihash) gives you the same multihash value back. +1. A [multicodec], also known as simply codec. +2. A [multihash] used to specify the hashing algorithm, the hash parameters and + the hash digest. -### Multicodecs +Thus, the block must be retrieved, that is, the bytes which when hashed using the +hash function specified in the multihash gives us the same multihash value back. -With Unixfs we deal with two codecs which will be decoded differently: -- `Raw`, single block files -- `dag-pb`, can be any nodes +In UnixFS, a node can be encoded using two different multicodecs, which we give +more details about in the following sections: -#### `Raw` blocks +- `raw` (`0x55`), which are single block :ref[Files]. +- `dag-pb` (`0x70`), which can be of any other type. -The simplest nodes use `Raw` encoding. +## `Raw` Nodes -They are always implicitly of type `file`. +The simplest nodes use `raw` encoding and are implicitly a :ref[File]. They can +be recognized because their CIDs are encoded using the `raw` codec: -They can be recognized because their CIDs have `Raw` codec. +- The file content is purely the block body. +- They never have any children nodes, and thus are also known as single block files. +- Their size (both `dagsize` and `blocksize`) is the length of the block body. -The file content is purely the block body. +## `dag-pb` Nodes -They never have any children nodes, and thus are also known as single block files. +More complex nodes use the `dag-pb` encoding. These nodes require two steps of +decoding. The first step is to decode the outer container of the block, which +is encoded using the IPLD [`dag-pb`][ipld-dag-pb] specification, which can be +summarized as follows: -Their sizes (both `dagsize` and `blocksize`) is the length of the block body. +```protobuf +message PBLink { + // binary CID (with no multibase prefix) of the target object + optional bytes Hash = 1; + + // UTF-8 string name + optional string Name = 2; + + // cumulative size of target object + optional uint64 Tsize = 3; +} -#### `dag-pb` nodes +message PBNode { + // refs to other objects + repeated PBLink Links = 2; -##### Data Format + // opaque user data + optional bytes Data = 1; +} +``` -The UnixfsV1 `Data` message format is represented by this protobuf: +After decoding the node, we obtain a `PBNode`. This `PBNode` contains a field +`Data` that contains the bytes that require the second decoding. These are also +a protobuf message specified in the UnixFSV1 format: ```protobuf message Data { @@ -105,55 +133,35 @@ message UnixTime { } ``` -##### IPLD `dag-pb` - -A very important other spec for unixfs is the [`dag-pb`](https://ipld.io/specs/codecs/dag-pb/spec/) IPLD spec: - -```protobuf -message PBLink { - // binary CID (with no multibase prefix) of the target object - optional bytes Hash = 1; - - // UTF-8 string name - optional string Name = 2; - - // cumulative size of target object - optional uint64 Tsize = 3; // also known as dagsize -} - -message PBNode { - // refs to other objects - repeated PBLink Links = 2; - - // opaque user data - optional bytes Data = 1; -} -``` - -The two different schemas plays together and it is important to understand their different effect, -- The `dag-pb` / `PBNode` protobuf is the "outside" protobuf message; in other words, it is the first message decoded. This protobuf contains the list of links and some "opaque user data". -- The `Data` message is the "inside" protobuf message. After the "outside" `dag-pb` (also known as `PBNode`) object is decoded, `Data` is decoded from the bytes inside the `PBNode.Data` field. This contains the rest of information. - -In other words, we have a serialized protobuf message stored inside another protobuf message. -For clarity, the spec document may represents these nested protobufs as one object. In this representation, it is implied that the `PBNode.Data` field is encoded in a prototbuf. +Summarizing, a `dag-pb` UnixFS node is an IPLD [`dag-pb`][ipld-dag-pb] protobuf, +whose `Data` field is a UnixFSV1 Protobuf message. For clarity, the specification +document may represent these nested Protobufs as one object. In this representation, +it is implied that the `PBNode.Data` field is encoded in a protobuf. -##### Different Data types +### Data Types -`dag-pb` nodes supports many different types, which can be found in `decodeData(PBNode.Data).Type`. Every type is handled differently. +A `dag-pb` UnixFS node supports different types, which are defined in +`decode(PBNode.Data).Type`. Every type is handled differently. -###### `File` type +#### `File` type -####### The _sister-lists_ `PBNode.Links` and `decodeMessage(PBNode.Data).blocksizes` +A :dfn[File] is a container over an arbitrary sized amount of bytes. Files can be +said to be either single block or multi block. When multi block, a File is then a +concatenation of multiple children files -The _sister-lists_ are the key point of why `dag-pb` is important for files. +##### The _sister-lists_ `PBNode.Links` and `decode(PBNode.Data).blocksizes` -This allows us to concatenate smaller files together. +The _sister-lists_ are the key point of why IPLD `dag-pb` is important for files. They +allow us to concatenate smaller files together. -Linked files would be loaded recursively with the same process following a DFS (Depth-First-Search) order. +Linked files would be loaded recursively with the same process following a DFS +(Depth-First-Search) order. -Child nodes must be of type file (so `dag-pb` where type is `File` or `Raw`) +Child nodes must be of type file, so either a [`dag-pb` File](#file-type), or a +[`raw` block](#raw-blocks). For example this example pseudo-json block: + ```json { "Links": [{"Hash":"Qmfoo"}, {"Hash":"Qmbar"}], @@ -166,287 +174,377 @@ For example this example pseudo-json block: This indicates that this file is the concatenation of the `Qmfoo` and `Qmbar` files. -When reading a file represented with `dag-pb`, the `blocksizes` array gives us the size in bytes of the partial file content present in child DAGs. -Each index in `PBNode.Links` MUST have a corresponding chunk size stored at the same index in `decodeMessage(PBNode.Data).blocksizes`. +When reading a file represented with `dag-pb`, the `blocksizes` array gives us the +size in bytes of the partial file content present in children DAGs. Each index in +`PBNode.Links` MUST have a corresponding chunk size stored at the same index +in `decode(PBNode.Data).blocksizes`. -Implementers need to be extra careful to ensure the values in `Data.blocksizes` are calculated by following the definition from [Blocksize](#blocksize) section. +Implementers need to be extra careful to ensure the values in `Data.blocksizes` +are calculated by following the definition from [`Blocksize`](#decodepbnodedatablocksize). -This allows to do fast indexing into the file, for example if someone is trying to read bytes 25 to 35 we can compute an offset list by summing all previous indexes in `blocksizes`, then do a search to find which indexes contain the range we are intrested in. +This allows to do fast indexing into the file, for example if someone is trying +to read bytes 25 to 35 we can compute an offset list by summing all previous +indexes in `blocksizes`, then do a search to find which indexes contain the +range we are interested in. For example here the offset list would be `[0, 20]` and thus we know we only need to download `Qmbar` to get the range we are intrested in. UnixFS parser MUST error if `blocksizes` or `Links` are not of the same length. -####### `decodeMessage(PBNode.Data).Data` - -This field is an array of bytes, it is file content and is appended before the links. - -This must be taken into a count when doing offsets calculations (the len of the `Data.Data` field define the value of the zeroth element of the offset list when computing offsets). - -####### `PBNode.Links[].Name` with Files - -This field makes sense only in directory contexts and MUST be absent when creating a new file `PBNode`. -For historic reasons, implementations parsing third-party data SHOULD accept empty value here. +##### `decode(PBNode.Data).Data` -If this field is present and non empty, the file is invalid and parser MUST error. +This field is an array of bytes, it is the file content and is appended before +the links. This must be taken into account when doing offset calculations, that is +the length of `decode(PBNode.Data).Data` defines the value of the zeroth element +of the offset list when computing offsets. -####### `Blocksize` of a dag-pb file +##### `PBNode.Links[].Name` -This is not a field present in the block directly, but rather a computable property of `dag-pb` which would be used in parent node in `decodeMessage(PBNode.Data).blocksizes`. -It is the sum of the length of the `Data.Data` field plus the sum of all link's blocksizes. +This field makes sense only in :ref[Directories] contexts and MUST be absent +when creating a new file. For historical reasons, implementations parsing +third-party data SHOULD accept empty values here. -####### `PBNode.Data.Filesize` +If this field is present and non-empty, the file is invalid and the parser MUST +error. -If present, this field must be equal to the `Blocksize` computation above, else the file is invalid. +##### `decode(PBNode.Data).Blocksize` -####### Path resolution +This field is not directly present in the block, but rather a computable property +of a `dag-pb`, which would be used in the parent node in `decode(PBNode.Data).blocksizes`. +It is the sum of the length of `decode(PBNode.Data).Data` field plus the sum +of all link's `blocksizes`. -A file terminates UnixFS content path. +##### `decode(PBNode.Data).filesize` -Any attempt of path resolution on `File` type MUST error. +If present, this field MUST be equal to the `Blocksize` computation above. +Otherwise, this file is invalid. -###### `Directory` Type +##### Path Resolution -A directory node is a named collection of nodes. +A file terminates a UnixFS content path. Any attempt to resolve a path past a +file MUST error. -The minimum valid `PBNode.Data` field for a directory is (pseudo-json): `{"Type":"Directory"}`, other values are covered in Metadata. +#### `Directory` Type -Every link in the Links list is an entry (children) of the directory, and the `PBNode.Links[].Name` field give you the name. +A :dfn[Directory], also known as folder, is a named collection of child :ref[Nodes]: -####### Link ordering +- Every link in `PBNode.Links` is an entry (child) of the directory, and + `PBNode.Links[].Name` gives you the name of such child. +- Duplicate names are not allowed. Therefore, two elements of `PBNode.Link` CANNOT + have the same `Name`. if two identical names are present in a directory, the + decoder MUST fail. -The cannonical sorting order is lexicographical over the names. +The minimum valid `PBNode.Data` field for a directory is as follows: -In theory there is no reason an encoder couldn't use an other ordering, however this lose some of it's meaning when mapped into most file systems today (most file systems consider directories are unordered-key-value objects). +```json +{ + "Type": "Directory" +} +``` -A decoder SHOULD if it can, preserve the order of the original files in however it consume thoses names. +The remaining relevant values are covered in [Metadata](#metadata). -However when some implementation decode, modify then reencode some, the orignal links order fully lose it's meaning. (given that there is no way to indicate which sorting was used originally) +##### Link Ordering -####### Path Resolution +The canonical sorting order is lexicographical over the names. -Pop the left most component of the path, and try to match it to one of the Name in Links. +In theory there is no reason an encoder couldn't use an other ordering, however +this lose some of its meaning when mapped into most file systems today (most file +systems consider directories are unordered-key-value objects). -If you find a match you can then remember the CID. You MUST continue your search, however if you find a match again you MUST error. +A decoder SHOULD, if it can, preserve the order of the original files in however +it consume those names. However when, some implementation decode, modify then +re-encode some, the original links order fully lose it's meaning (given that there +is no way to indicate which sorting was used originally). -Assuming no errors were raised, you can continue to the path resolution on the mainaing component and on the CID you poped. +##### Path Resolution -####### Duplicate names +Pop the left-most component of the path, and try to match it to the `Name` of +a child under `PBNode.Links`. If you find a match, you can then remember the CID. +You MUST continue the search. If you find another match, you MUST error since +duplicate names are not allowed. -Duplicate names are not allowed, if two identical names are present in an directory, the decoder MUST error. +Assuming no errors were raised, you can continue to the path resolution on the +remaining components and on the CID you popped. -###### `Symlink` type -Symlinks MUST NOT have childs. +#### `Symlink` type -Their Data.Data field is a POSIX path that maybe appended in front of the currently remaining path component stack. +A :dfn[Symlink] represents a POSIX [symbolic link](https://pubs.opengroup.org/onlinepubs/9699919799/functions/symlink.html). +A symlink MUST NOT have children. -####### Path resolution on symlinks +The `PBNode.Data.Data` field is a POSIX path that MAY be appended in front of the +currently remaining path component stack. -There is no current consensus on how pathing over symlinks should behave. -Some implementations return symlinks objects and fail if a consumer tries to follow it through. +##### Path Resolution -Following the POSIX spec over the current unixfs path context is probably fine. +There is no current consensus on how pathing over symlinks should behave. Some +implementations return symlink objects and fail if a consumer tries to follow them +through. -###### `HAMTDirectory` +Following the POSIX specification over the current UnixFS path context is probably fine. -Thoses nodes are also sometimes called sharded directories, they allow to split directories into many blocks when they are so big that they don't fit into one single block anymore. +#### `HAMTDirectory` -- `node.Data.hashType` indicates a multihash function to use to digest path components used for sharding. -It MUST be murmur3-x64-64 (multihash `0x22`). -- `node.Data.Data` is some bitfield, ones indicates whether or not the links are part of this HAMT or leaves of the HAMT. -The usage of this field is unknown given you can deduce the same information from the links names. -- `node.Data.fanout` MUST be a power of two. This encode the number of hash permutations that will be used on each resolution step. -The log base 2 of the fanout indicate how wide the bitmask will be on the hash at for that step. `fanout` MUST be between 8 and probably 65536. +A :dfn[HAMT Directory] is a [Hashed-Array-Mapped-Trie](https://en.wikipedia.org/wiki/Hash_array_mapped_trie) +data structure representing a :ref[Directory]. It is generally used to represent +directories that cannot fit inside a single block. They are also known as "sharded +directories" since they allow to split large directories into multiple blocks, the "shards". -####### `node.Links[].Name` on HAMTs +- `decode(PBNode.Data).hashType` indicates the [multihash] function to use to digest + the path components used for sharding. It MUST be `murmur3-x64-64` (`0x22`). +- `decode(PBNode.Data).Data.Data` is a bit field, which indicates whether or not + links are part of this HAMT, or its leaves. The usage of this field is unknown given + that you can deduce the same information from the link names. +- `decode(PBNode.Data).Data.fanout` MUST be a power of two. This encodes the number + of hash permutations that will be used on each resolution step. The log base 2 + of the `fanout` indicate how wide the bitmask will be on the hash at for that step. + `fanout` MUST be between 8 and probably 65536. . -They start by some uppercase hex encoded prefix which is `log2(fanout)` bits wide +The field `Name` of an element of `PBNode.Links` for a HAMT starts with an +uppercase hex-encoded prefix, which is `log2(fanout)` bits wide. -####### Path resolution on HAMTs +##### Path Resolution -Steps: -1. Take the current path component then hash it using the multihash id provided in `Data.hashType`. -2. Pop the `log2(fanout)` lowest bits from the path component hash digest, then hex encode (using 0-F) thoses bits using little endian thoses bits and find the link that starts with this hex encoded path. -3. If the link name is exactly as long as the hex encoded representation, follow the link and repeat step 2 with the child node and the remaining bit stack. The child node MUST be a hamt directory else the directory is invalid, else continue. -4. Compare the remaining part of the last name you found, if it match the original name you were trying to resolve you successfully resolved a path component, everything past the hex encoded prefix is the name of that element (usefull when listing childs of this directory). +To resolve the path inside a HAMT: +1. Take the current path component then hash it using the [multihash] represented + by the value of `decode(PBNode.Data).hashType`. +2. Pop the `log2(fanout)` lowest bits from the path component hash digest, then + hex encode (using 0-F) those bits using little endian. Find the link that starts + with this hex encoded path. +3. If the link `Name` is exactly as long as the hex encoded representation, follow + the link and repeat step 2 with the child node and the remaining bit stack. + The child node MUST be a HAMT directory else the directory is invalid, else continue. +4. Compare the remaining part of the last name you found, if it match the original + name you were trying to resolve you successfully resolved a path component, + everything past the hex encoded prefix is the name of that element + (useful when listing children of this directory). -###### `TSize` / `DagSize` +### `TSize` / `DagSize` -This is an optional field for Links of `dag-pb` nodes, **it does not represent any meaningfull information of the underlying structure** and no known usage of it to this day (altho some implementation emit thoses). +This is an option field of `PBNode.Links[]`. It **does not** represent any +meaningful information of the underlying structure, and there is no known +usage of it to this day, although some implementations emit these. -To compute the `dagsize` of a node (which would be stored in the parents) you sum the length of the dag-pb outside message binary length, plus the blocksizes of all child files. +To compute the `DagSize` of a node, which would be store in the parents, you have +to sum the length of the `dag-pb` outside message binary length, plus the +`blocksizes` of all child files. -An example of where this could be usefull is as a hint to smart download clients, for example if you are downloading a file concurrently from two sources that have radically different speeds, it would probably be more efficient to download bigger links from the fastest source, and smaller ones from the slowest source. +An example of where this could be useful is as a hint to smart download clients, +for example if you are downloading a file concurrently from two sources that have +radically different speeds, it would probably be more efficient to download bigger +links from the fastest source, and smaller ones from the slowest source. -There is no failure mode known for this field, so your implementation should be able to decode nodes where this field is wrong (not the value you expect), partially or completely missing. This also allows smarter encoder to give a more accurate picture (for example don't count duplicate blocks, ...). - -### Paths +There is no failure mode known for this field, so your implementation should be +able to decode nodes where this field is wrong (not the value you expect), +partially or completely missing. This also allows smarter encoder to give a +more accurate picture (for example don't count duplicate blocks, ...). -Paths first start with `/`or `/ipfs//` where `` is a [multibase](https://github.com/multiformats/multibase) encoded [CID](https://github.com/multiformats/cid). -The CID encoding MUST NOT use a multibase alphabet that have `/` (`0x2f`) unicode codepoints however CIDs may use a multibase encoding with a `/` in the alphabet if the encoded CID does not contain `/` once encoded. - -Everything following the CID is a collection of path component (some bytes) seperated by `/` (`0x2f`), read from left to right. -This is inspired by POSIX paths. +### Metadata -- Components MUST NOT contain `/` unicode codepoints because else it would break the path into two components. +UnixFS currently supports two optional metadata fields. + +#### `mode` + +The `mode` is for persisting the file permissions in [numeric notation](https://en.wikipedia.org/wiki/File_system_permissions#Numeric_notation) +\[[spec](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html)\]. + +- If unspecified this defaults to + - `0755` for directories/HAMT shards + - `0644` for all other types where applicable +- The nine least significant bits represent `ugo-rwx` +- The next three least significant bits represent `setuid`, `setgid` and the `sticky bit` +- The remaining 20 bits are reserved for future use, and are subject to change. Spec implementations **MUST** handle bits they do not expect as follows: + - For future-proofing the (de)serialization layer must preserve the entire uint32 value during clone/copy operations, modifying only bit values that have a well defined meaning: `clonedValue = ( modifiedBits & 07777 ) | ( originalValue & 0xFFFFF000 )` + - Implementations of this spec must proactively mask off bits without a defined meaning in the implemented version of the spec: `interpretedValue = originalValue & 07777` + +#### `mtime` + +A two-element structure ( `Seconds`, `FractionalNanoseconds` ) representing the +modification time in seconds relative to the unix epoch `1970-01-01T00:00:00Z`. +The two fields are: + +1. `Seconds` ( always present, signed 64bit integer ): represents the amount of seconds after **or before** the epoch. +2. `FractionalNanoseconds` ( optional, 32bit unsigned integer ): when specified represents the fractional part of the mtime as the amount of nanoseconds. The valid range for this value are the integers `[1, 999999999]`. + +Implementations encoding or decoding wire-representations MUST observe the following: + +- An `mtime` structure with `FractionalNanoseconds` outside of the on-wire range + `[1, 999999999]` is **not** valid. This includes a fractional value of `0`. + Implementations encountering such values should consider the entire enclosing + metadata block malformed and abort processing the corresponding DAG. +- The `mtime` structure is optional - its absence implies `unspecified`, rather + than `0` +- For ergonomic reasons a surface API of an encoder MUST allow fractional 0 as + input, while at the same time MUST ensure it is stripped from the final structure + before encoding, satisfying the above constraints. + +Implementations interpreting the mtime metadata in order to apply it within a +non-IPFS target MUST observe the following: + +- If the target supports a distinction between `unspecified` and `0`/`1970-01-01T00:00:00Z`, + the distinction must be preserved within the target. E.g. if no `mtime` structure + is available, a web gateway must **not** render a `Last-Modified:` header. +- If the target requires an mtime ( e.g. a FUSE interface ) and no `mtime` is + supplied OR the supplied `mtime` falls outside of the targets accepted range: + - When no `mtime` is specified or the resulting `UnixTime` is negative: + implementations must assume `0`/`1970-01-01T00:00:00Z` (note that such values + are not merely academic: e.g. the OpenVMS epoch is `1858-11-17T00:00:00Z`) + - When the resulting `UnixTime` is larger than the targets range ( e.g. 32bit + vs 64bit mismatch) implementations must assume the highest possible value + in the targets range (in most cases that would be `2038-01-19T03:14:07Z`) + +## Paths + +Paths first start with `/` or `/ipfs//` where `` is a [multibase] +encoded [CID]. The CID encoding MUST NOT use a multibase alphabet that have +`/` (`0x2f`) unicode codepoints however CIDs may use a multibase encoding with +a `/` in the alphabet if the encoded CID does not contain `/` once encoded. + +Everything following the CID is a collection of path component (some bytes) +separated by `/` (`0x2F`). UnixFS paths read from left to right, and are +inspired by POSIX paths. + +- Components MUST NOT contain `/` unicode codepoints because else it would break + the path into two components. - Components SHOULD be UTF8 unicode. - Components are case sensitive. -#### Escaping - -The `\` may be supposed to trigger an escape sequence. - -This might be a thing, but is broken and inconsistent current implementations. -So until we agree on a new spec for this, you SHOULD NOT use any escape sequence and non ascii character. - -#### Relative path components +### Escaping -Thoses path components must be resolved before trying to work on the path. +The `\` may be supposed to trigger an escape sequence. However, it is currently +broken and inconsistent across implementations. Until we agree on a specification +for this, you SHOULD NOT use any escape sequences and non-ASCII characters. -- `.` points to the current node, those path components must be removed. -- `..` points to the parent, they must be removed first to last however when you remove a `..` you also remove the previous component on the left. If there is no component on the left to remove leave the `..` as-is however this is an attempt for an out-of-bound path resolution which mean you MUST error. +### Relative Path Components -#### Restricted names +Relative path components MUST be resolved before trying to work on the path: -Thoses names SHOULD NOT be used: - -- The `.` string. This represents the self node in POSIX pathing. -- The `..` string. This represents the parent node in POSIX pathing. -- nothing (the empty string) We don't actually know the failure mode for this, but it really feels like this shouldn't be a thing. -- Any string containing a NUL (0x00) byte, this is often used to signify string terminations in some systems (such as most C compatible systems), and many unix file systems don't accept this character in path components. - -### Glossary - -- Node, Block - A node is a word from graph theory, this is the smallest unit present in the graph. - Due to how unixfs work, there is a 1 to 1 mapping between nodes and blocks. -- File - A file is some container over an arbitrary sized amounts of bytes. - Files can be said to be single block, or multi block, in the later case they are the concatenation of multiple children files. -- Directory, Folder - A named collection of child nodes. -- HAMT Directory - This is a [Hashed-Array-Mapped-Trie](https://en.wikipedia.org/wiki/Hash_array_mapped_trie) data structure representing a Directory, those may be used to split directories into multiple blocks when they get too big, and the list of children does not fit in a single block. -- Symlink - This represents a POSIX Symlink. - -### Metadata +- `.` points to the current node, those path components MUST be removed. +- `..` points to the parent node, they MUST be removed left to right. When removing + a `..`, the path component on the left MUST also be removed. If there is no path + component on the left, you MUST error since it is an attempt of out-of-bounds + path resolution. -UnixFS currently supports two optional metadata fields: +### Restricted Names -* `mode` -- The `mode` is for persisting the file permissions in [numeric notation](https://en.wikipedia.org/wiki/File_system_permissions#Numeric_notation) \[[spec](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html)\]. - - If unspecified this defaults to - - `0755` for directories/HAMT shards - - `0644` for all other types where applicable - - The nine least significant bits represent `ugo-rwx` - - The next three least significant bits represent `setuid`, `setgid` and the `sticky bit` - - The remaining 20 bits are reserved for future use, and are subject to change. Spec implementations **MUST** handle bits they do not expect as follows: - - For future-proofing the (de)serialization layer must preserve the entire uint32 value during clone/copy operations, modifying only bit values that have a well defined meaning: `clonedValue = ( modifiedBits & 07777 ) | ( originalValue & 0xFFFFF000 )` - - Implementations of this spec must proactively mask off bits without a defined meaning in the implemented version of the spec: `interpretedValue = originalValue & 07777` +The following names SHOULD NOT be used: -* `mtime` -- A two-element structure ( `Seconds`, `FractionalNanoseconds` ) representing the modification time in seconds relative to the unix epoch `1970-01-01T00:00:00Z`. - - The two fields are: - 1. `Seconds` ( always present, signed 64bit integer ): represents the amount of seconds after **or before** the epoch. - 2. `FractionalNanoseconds` ( optional, 32bit unsigned integer ): when specified represents the fractional part of the mtime as the amount of nanoseconds. The valid range for this value are the integers `[1, 999999999]`. +- The `.` string: represents the self node in POSIX pathing. +- The `..` string: represents the parent node in POSIX pathing. +- The empty string. +- Any string containing a `NUL` (`0x00`) byte: this is often used to signify string + terminations in some systems (such as most C compatible systems), and many unix + file systems do not accept this character in path components. - - Implementations encoding or decoding wire-representations must observe the following: - - An `mtime` structure with `FractionalNanoseconds` outside of the on-wire range `[1, 999999999]` is **not** valid. This includes a fractional value of `0`. Implementations encountering such values should consider the entire enclosing metadata block malformed and abort processing the corresponding DAG. - - The `mtime` structure is optional - its absence implies `unspecified`, rather than `0` - - For ergonomic reasons a surface API of an encoder must allow fractional 0 as input, while at the same time must ensure it is stripped from the final structure before encoding, satisfying the above constraints. +## Design Decision Rationale - - Implementations interpreting the mtime metadata in order to apply it within a non-IPFS target must observe the following: - - If the target supports a distinction between `unspecified` and `0`/`1970-01-01T00:00:00Z`, the distinction must be preserved within the target. E.g. if no `mtime` structure is available, a web gateway must **not** render a `Last-Modified:` header. - - If the target requires an mtime ( e.g. a FUSE interface ) and no `mtime` is supplied OR the supplied `mtime` falls outside of the targets accepted range: - - When no `mtime` is specified or the resulting `UnixTime` is negative: implementations must assume `0`/`1970-01-01T00:00:00Z` ( note that such values are not merely academic: e.g. the OpenVMS epoch is `1858-11-17T00:00:00Z` ) - - When the resulting `UnixTime` is larger than the targets range ( e.g. 32bit vs 64bit mismatch ) implementations must assume the highest possible value in the targets range ( in most cases that would be `2038-01-19T03:14:07Z` ) +### `mtime` and `mode` Metadata Support in UnixFSv1.5 -## Design decision rationale +Metadata support in UnixFSv1.5 has been expanded to increase the number of possible +use cases. These include rsync and filesystem based package managers. -### `mtime` and `mode` metadata support in UnixFSv1.5 +Several metadata systems were evaluated, as discussed in the following sections. -Metadata support in UnixFSv1.5 has been expanded to increase the number of possible use cases. These include rsync and filesystem based package managers. +#### Separate Metadata Node -Several metadata systems were evaluated: - -#### Separate Metadata node - -In this scheme, the existing `Metadata` message is expanded to include additional metadata types (`mtime`, `mode`, etc). It then contains links to the actual file data but never the file data itself. +In this scheme, the existing `Metadata` message is expanded to include additional +metadata types (`mtime`, `mode`, etc). It contains links to the actual file data +but never the file data itself. This was ultimately rejected for a number of reasons: -1. You would always need to retrieve an additional node to access file data which limits the kind of optimizations that are possible. - - For example many files are under the 256KiB block size limit, so we tend to inline them into the describing UnixFS `File` node. This would not be possible with an intermediate `Metadata` node. - -2. The `File` node already contains some metadata (e.g. the file size) so metadata would be stored in multiple places which complicates forwards compatibility with UnixFSv2 as to map between metadata formats potentially requires multiple fetch operations - -#### Metadata in the directory +1. You would always need to retrieve an additional node to access file data which + limits the kind of optimizations that are possible. For example many files are + under the 256 KiB block size limit, so we tend to inline them into the describing + UnixFS `File` node. This would not be possible with an intermediate `Metadata` node. +2. The `File` node already contains some metadata (e.g. the file size) so metadata + would be stored in multiple places which complicates forwards compatibility with + UnixFSv2 as to map between metadata formats potentially requires multiple fetch + operations. -Repeated `Metadata` messages are added to UnixFS `Directory` and `HAMTShard` nodes, the index of which indicates which entry they are to be applied to. +#### Metadata in the Directory -Where entries are `HAMTShard`s, an empty message is added. +Repeated `Metadata` messages are added to UnixFS `Directory` and `HAMTShard` nodes, +the index of which indicates which entry they are to be applied to. Where entries are +`HAMTShard`s, an empty message is added. -One advantage of this method is that if we expand stored metadata to include entry types and sizes we can perform directory listings without needing to fetch further entry nodes (excepting `HAMTShard` nodes), though without removing the storage of these datums elsewhere in the spec we run the risk of having non-canonical data locations and perhaps conflicting data as we traverse through trees containing both UnixFS v1 and v1.5 nodes. +One advantage of this method is that if we expand stored metadata to include entry +types and sizes we can perform directory listings without needing to fetch further +entry nodes (excepting `HAMTShard` nodes), though without removing the storage of +these datums elsewhere in the spec we run the risk of having non-canonical data +locations and perhaps conflicting data as we traverse through trees containing +both UnixFS v1 and v1.5 nodes. This was rejected for the following reasons: -1. When creating a UnixFS node there's no way to record metadata without wrapping it in a directory. +1. When creating a UnixFS node there's no way to record metadata without wrapping + it in a directory. +2. If you access any UnixFS node directly by its [CID], there is no way of recreating + the metadata which limits flexibility. +3. In order to list the contents of a directory including entry types and sizes, + you have to fetch the root node of each entry anyway so the performance benefit + of including some metadata in the containing directory is negligible in this + use case. -2. If you access any UnixFS node directly by its [CID], there is no way of recreating the metadata which limits flexibility. - -3. In order to list the contents of a directory including entry types and sizes, you have to fetch the root node of each entry anyway so the performance benefit of including some metadata in the containing directory is negligible in this use case. - -#### Metadata in the file +#### Metadata in the File This adds new fields to the UnixFS `Data` message to represent the various metadata fields. -It has the advantage of being simple to implement, metadata is maintained whether the file is accessed directly via its [CID] or via an IPFS path that includes a containing directory, and by keeping the metadata small enough we can inline root UnixFS nodes into their CIDs so we can end up fetching the same number of nodes if we decide to keep file data in a leaf node for deduplication reasons. +It has the advantage of being simple to implement, metadata is maintained whether +the file is accessed directly via its [CID] or via an IPFS path that includes a +containing directory, and by keeping the metadata small enough we can inline root +UnixFS nodes into their CIDs so we can end up fetching the same number of nodes if +we decide to keep file data in a leaf node for deduplication reasons. Downsides to this approach are: -1. Two users adding the same file to IPFS at different times will have different [CID]s due to the `mtime`s being different. - - If the content is stored in another node, its [CID] will be constant between the two users but you can't navigate to it unless you have the parent node which will be less available due to the proliferation of [CID]s. - -2. Metadata is also impossible to remove without changing the [CID], so metadata becomes part of the content. - -3. Performance may be impacted as well as if we don't inline UnixFS root nodes into [CID]s, additional fetches will be required to load a given UnixFS entry. +1. Two users adding the same file to IPFS at different times will have different + [CID]s due to the `mtime`s being different. If the content is stored in another + node, its [CID] will be constant between the two users but you can't navigate + to it unless you have the parent node which will be less available due to the + proliferation of [CID]s. +1. Metadata is also impossible to remove without changing the [CID], so + metadata becomes part of the content. +2. Performance may be impacted as well as if we don't inline UnixFS root nodes + into [CID]s, additional fetches will be required to load a given UnixFS entry. -#### Side trees +#### Side Trees -With this approach we would maintain a separate data structure outside of the UnixFS tree to hold metadata. +With this approach we would maintain a separate data structure outside of the +UnixFS tree to hold metadata. -This was rejected due to concerns about added complexity, recovery after system crashes while writing, and having to make extra requests to fetch metadata nodes when resolving [CID]s from peers. +This was rejected due to concerns about added complexity, recovery after system +crashes while writing, and having to make extra requests to fetch metadata nodes +when resolving [CID]s from peers. -#### Side database +#### Side Database This scheme would see metadata stored in an external database. -The downsides to this are that metadata would not be transferred from one node to another when syncing as [Bitswap] is not aware of the database, and in-tree metadata +The downsides to this are that metadata would not be transferred from one node +to another when syncing as [Bitswap] is not aware of the database, and in-tree +metadata. -### UnixTime protobuf datatype rationale +### UnixTime Protobuf Datatype Rationale #### Seconds -The integer portion of UnixTime is represented on the wire using a varint encoding. While this is -inefficient for negative values, it avoids introducing zig-zag encoding. Values before the year 1970 -will be exceedingly rare, and it would be handy having such cases stand out, while at the same keeping -the "usual" positive values easy to eyeball. The varint representing the time of writing this text is -5 bytes long. It will remain so until October 26, 3058 ( 34,359,738,367 ) +The integer portion of UnixTime is represented on the wire using a `varint` encoding. +While this is inefficient for negative values, it avoids introducing zig-zag encoding. +Values before the year 1970 will be exceedingly rare, and it would be handy having +such cases stand out, while at the same keeping the "usual" positive values easy +to eyeball. The `varint` representing the time of writing this text is 5 bytes +long. It will remain so until October 26, 3058 (34,359,738,367). #### FractionalNanoseconds -Fractional values are effectively a random number in the range 1 ~ 999,999,999. Such values will exceed -2^28 nanoseconds ( 268,435,456 ) in most cases. Therefore, the fractional part is represented as a 4-byte -`fixed32`, [as per Google's recommendation](https://developers.google.com/protocol-buffers/docs/proto#scalar). -## References - -[multihash]: https://tools.ietf.org/html/draft-multiformats-multihash-05 -[CID]: https://github.com/multiformats/cid/ -[Bitswap]: https://github.com/ipfs/specs/blob/master/BITSWAP.md +Fractional values are effectively a random number in the range 1 ~ 999,999,999. +Such values will exceed 2^28 nanoseconds (268,435,456) in most cases. Therefore, +the fractional part is represented as a 4-byte `fixed32`, +[as per Google's recommendation](https://developers.google.com/protocol-buffers/docs/proto#scalar). # Notes for Implementers @@ -471,12 +569,14 @@ This section and included subsections are not authoritative. In this example, we will build a `Raw` file with the string `test` as its content. 1. First hash the data: + ```console $ echo -n "test" | sha256sum 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 - ``` 2. Add the CID prefix: + ``` f this is the multibase prefix, we need it because we are working with a hex CID, this is omitted for binary CIDs 01 the CID version, here one @@ -486,17 +586,18 @@ f this is the multibase prefix, we need it because we are working with a hex CID 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 the digest we computed earlier ``` -3. Profit -Assuming we stored this block in some implementation of our choice which makes it accessible to our client, we can try to decode it: +3. Profit: assuming we stored this block in some implementation of our choice which makes it accessible to our client, we can try to decode it. + ```console $ ipfs cat f015512209f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 test ``` +## Offset List -### Offset list +The offset list isn't the only way to use blocksizes and reach a correct implementation, +it is a simple canonical one, python pseudo code to compute it looks like this: -The offset list isn't the only way to use blocksizes and reach a correct implementation, it is a simple cannonical one, python pseudo code to compute it looks like this: ```python def offsetlist(node): unixfs = decodeDataField(node.Data) @@ -508,3 +609,10 @@ def offsetlist(node): ``` This will tell you which offset inside this node the children at the corresponding index starts to cover. (using `[x,y)` ranging) + +[protobuf]: https://developers.google.com/protocol-buffers/ +[CID]: https://github.com/multiformats/cid/ +[multicodec]: https://github.com/multiformats/multicodec +[multihash]: https://github.com/multiformats/multihash +[Bitswap]: https://github.com/ipfs/specs/blob/master/BITSWAP.md +[ipld-dag-pb]: https://ipld.io/specs/codecs/dag-pb/spec/ From e2cf0af3b6868ae59bbb3a23318b0cb274085485 Mon Sep 17 00:00:00 2001 From: Henrique Dias Date: Mon, 30 Oct 2023 11:02:58 +0100 Subject: [PATCH 06/46] chore: apply @ElPaisano suggestions --- src/architecture/unixfs.md | 201 ++++++++++++++++++------------------- 1 file changed, 97 insertions(+), 104 deletions(-) diff --git a/src/architecture/unixfs.md b/src/architecture/unixfs.md index 0b7887f71..a9d2f4663 100644 --- a/src/architecture/unixfs.md +++ b/src/architecture/unixfs.md @@ -37,26 +37,25 @@ order: 1 --- UnixFS is a [protocol-buffers][protobuf]-based format for describing files, -directories and symlinks as DAGs in IPFS. +directories and symlinks as Directed Acyclic Graphs (DAGs) in IPFS. ## Nodes A :dfn[Node] is the smallest unit present in a graph, and it comes from graph -theory. In UnixFS, there is a 1 to 1 mapping between nodes and blocks. Therefore, +theory. In UnixFS, there is a 1-to-1 mapping between nodes and blocks. Therefore, they are used interchangeably in this document. A node is addressed by a [CID]. In order to be able to read a node, its [CID] is -required. A [CID] includes two important information: +required. A [CID] includes two important pieces of information: -1. A [multicodec], also known as simply codec. +1. A [multicodec], simply known as a codec. 2. A [multihash] used to specify the hashing algorithm, the hash parameters and the hash digest. -Thus, the block must be retrieved, that is, the bytes which when hashed using the -hash function specified in the multihash gives us the same multihash value back. +Thus, the block must be retrieved; that is, the bytes which ,when hashed using the +hash function specified in the multihash, gives us the same multihash value back. -In UnixFS, a node can be encoded using two different multicodecs, which we give -more details about in the following sections: +In UnixFS, a node can be encoded using two different multicodecs, listed below. More details are provided in the following sections: - `raw` (`0x55`), which are single block :ref[Files]. - `dag-pb` (`0x70`), which can be of any other type. @@ -73,8 +72,7 @@ be recognized because their CIDs are encoded using the `raw` codec: ## `dag-pb` Nodes More complex nodes use the `dag-pb` encoding. These nodes require two steps of -decoding. The first step is to decode the outer container of the block, which -is encoded using the IPLD [`dag-pb`][ipld-dag-pb] specification, which can be +decoding. The first step is to decode the outer container of the block. This is encoded using the IPLD [`dag-pb`][ipld-dag-pb] specification, which can be summarized as follows: ```protobuf @@ -145,9 +143,8 @@ A `dag-pb` UnixFS node supports different types, which are defined in #### `File` type -A :dfn[File] is a container over an arbitrary sized amount of bytes. Files can be -said to be either single block or multi block. When multi block, a File is then a -concatenation of multiple children files +A :dfn[File] is a container over an arbitrary sized amount of bytes. Files are either +single block or multi-block. A multi-block file is a concatenation of multiple child files. ##### The _sister-lists_ `PBNode.Links` and `decode(PBNode.Data).blocksizes` @@ -157,10 +154,10 @@ allow us to concatenate smaller files together. Linked files would be loaded recursively with the same process following a DFS (Depth-First-Search) order. -Child nodes must be of type file, so either a [`dag-pb` File](#file-type), or a +Child nodes must be of type File; either a `dag-pb`:ref[File], or a [`raw` block](#raw-blocks). -For example this example pseudo-json block: +For example, consider this pseudo-json block: ```json { @@ -182,19 +179,19 @@ in `decode(PBNode.Data).blocksizes`. Implementers need to be extra careful to ensure the values in `Data.blocksizes` are calculated by following the definition from [`Blocksize`](#decodepbnodedatablocksize). -This allows to do fast indexing into the file, for example if someone is trying -to read bytes 25 to 35 we can compute an offset list by summing all previous +This allows for fast indexing into the file. For example, if someone is trying +to read bytes 25 to 35, we can compute an offset list by summing all previous indexes in `blocksizes`, then do a search to find which indexes contain the range we are interested in. -For example here the offset list would be `[0, 20]` and thus we know we only need to download `Qmbar` to get the range we are intrested in. +In the example above, the offset list would be `[0, 20]`. Thus, we know we only need to download `Qmbar` to get the range we are interested in. UnixFS parser MUST error if `blocksizes` or `Links` are not of the same length. ##### `decode(PBNode.Data).Data` -This field is an array of bytes, it is the file content and is appended before -the links. This must be taken into account when doing offset calculations, that is +An array of bytes that is the file content and is appended before +the links. This must be taken into account when doing offset calculations; that is, the length of `decode(PBNode.Data).Data` defines the value of the zeroth element of the offset list when computing offsets. @@ -229,9 +226,9 @@ file MUST error. A :dfn[Directory], also known as folder, is a named collection of child :ref[Nodes]: - Every link in `PBNode.Links` is an entry (child) of the directory, and - `PBNode.Links[].Name` gives you the name of such child. + `PBNode.Links[].Name` gives you the name of that child. - Duplicate names are not allowed. Therefore, two elements of `PBNode.Link` CANNOT - have the same `Name`. if two identical names are present in a directory, the + have the same `Name`. If two identical names are present in a directory, the decoder MUST fail. The minimum valid `PBNode.Data` field for a directory is as follows: @@ -248,14 +245,14 @@ The remaining relevant values are covered in [Metadata](#metadata). The canonical sorting order is lexicographical over the names. -In theory there is no reason an encoder couldn't use an other ordering, however -this lose some of its meaning when mapped into most file systems today (most file -systems consider directories are unordered-key-value objects). +In theory, there is no reason an encoder couldn't use an other ordering. However, +this loses some of its meaning when mapped into most file systems today, as most file +systems consider directories to be unordered key-value objects. -A decoder SHOULD, if it can, preserve the order of the original files in however -it consume those names. However when, some implementation decode, modify then -re-encode some, the original links order fully lose it's meaning (given that there -is no way to indicate which sorting was used originally). +A decoder SHOULD, if it can, preserve the order of the original files in the same way +it consumed those names. However, when some implementations decode, modify and then +re-encode, the original link order loses it's original meaning, given that there +is no way to indicate which sorting was used originally. ##### Path Resolution @@ -288,13 +285,13 @@ Following the POSIX specification over the current UnixFS path context is probab A :dfn[HAMT Directory] is a [Hashed-Array-Mapped-Trie](https://en.wikipedia.org/wiki/Hash_array_mapped_trie) data structure representing a :ref[Directory]. It is generally used to represent -directories that cannot fit inside a single block. They are also known as "sharded -directories" since they allow to split large directories into multiple blocks, the "shards". +directories that cannot fit inside a single block. These are also known as "sharded +directories:, since they allow you to split large directories into multiple blocks, known as "shards". - `decode(PBNode.Data).hashType` indicates the [multihash] function to use to digest the path components used for sharding. It MUST be `murmur3-x64-64` (`0x22`). - `decode(PBNode.Data).Data.Data` is a bit field, which indicates whether or not - links are part of this HAMT, or its leaves. The usage of this field is unknown given + links are part of this HAMT, or its leaves. The usage of this field is unknown, given that you can deduce the same information from the link names. - `decode(PBNode.Data).Data.fanout` MUST be a power of two. This encodes the number of hash permutations that will be used on each resolution step. The log base 2 @@ -308,39 +305,36 @@ uppercase hex-encoded prefix, which is `log2(fanout)` bits wide. To resolve the path inside a HAMT: -1. Take the current path component then hash it using the [multihash] represented +1. Take the current path component, then hash it using the [multihash] represented by the value of `decode(PBNode.Data).hashType`. 2. Pop the `log2(fanout)` lowest bits from the path component hash digest, then hex encode (using 0-F) those bits using little endian. Find the link that starts with this hex encoded path. 3. If the link `Name` is exactly as long as the hex encoded representation, follow the link and repeat step 2 with the child node and the remaining bit stack. - The child node MUST be a HAMT directory else the directory is invalid, else continue. -4. Compare the remaining part of the last name you found, if it match the original - name you were trying to resolve you successfully resolved a path component, - everything past the hex encoded prefix is the name of that element - (useful when listing children of this directory). + The child node MUST be a HAMT directory, or else the directory is invalid. Otherwise, continue. +4. Compare the remaining part of the last name you found. If it matches the original + name you were trying to resolve, you have successfully resolved a path component. + Everything past the hex encoded prefix is the name of that element, which is useful when listing children of this directory. ### `TSize` / `DagSize` -This is an option field of `PBNode.Links[]`. It **does not** represent any +This is an optional field in `PBNode.Links[]`. It **does not** represent any meaningful information of the underlying structure, and there is no known -usage of it to this day, although some implementations emit these. +usage of it to this day, although some implementations omit these. -To compute the `DagSize` of a node, which would be store in the parents, you have -to sum the length of the `dag-pb` outside message binary length, plus the -`blocksizes` of all child files. +To compute the `DagSize` of a node, which is stored in the parents, sum the length of the `dag-pb` outside message binary length and the `blocksizes` of all child files. -An example of where this could be useful is as a hint to smart download clients, -for example if you are downloading a file concurrently from two sources that have +An example of where this could be useful is as a hint to smart download clients. +For example, if you are downloading a file concurrently from two sources that have radically different speeds, it would probably be more efficient to download bigger links from the fastest source, and smaller ones from the slowest source. There is no failure mode known for this field, so your implementation should be -able to decode nodes where this field is wrong (not the value you expect), -partially or completely missing. This also allows smarter encoder to give a -more accurate picture (for example don't count duplicate blocks, ...). +able to decode nodes where this field is wrong (not the value you expect), or +partially or completely missing. This also allows smarter encoders to give a +more accurate picture (Don't count duplicate blocks, etc.). ### Metadata @@ -351,13 +345,13 @@ UnixFS currently supports two optional metadata fields. The `mode` is for persisting the file permissions in [numeric notation](https://en.wikipedia.org/wiki/File_system_permissions#Numeric_notation) \[[spec](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html)\]. -- If unspecified this defaults to +- If unspecified, this defaults to - `0755` for directories/HAMT shards - `0644` for all other types where applicable - The nine least significant bits represent `ugo-rwx` - The next three least significant bits represent `setuid`, `setgid` and the `sticky bit` - The remaining 20 bits are reserved for future use, and are subject to change. Spec implementations **MUST** handle bits they do not expect as follows: - - For future-proofing the (de)serialization layer must preserve the entire uint32 value during clone/copy operations, modifying only bit values that have a well defined meaning: `clonedValue = ( modifiedBits & 07777 ) | ( originalValue & 0xFFFFF000 )` + - For future-proofing, the (de)serialization layer must preserve the entire uint32 value during clone/copy operations, modifying only bit values that have a well defined meaning: `clonedValue = ( modifiedBits & 07777 ) | ( originalValue & 0xFFFFF000 )` - Implementations of this spec must proactively mask off bits without a defined meaning in the implemented version of the spec: `interpretedValue = originalValue & 07777` #### `mtime` @@ -367,76 +361,76 @@ modification time in seconds relative to the unix epoch `1970-01-01T00:00:00Z`. The two fields are: 1. `Seconds` ( always present, signed 64bit integer ): represents the amount of seconds after **or before** the epoch. -2. `FractionalNanoseconds` ( optional, 32bit unsigned integer ): when specified represents the fractional part of the mtime as the amount of nanoseconds. The valid range for this value are the integers `[1, 999999999]`. +2. `FractionalNanoseconds` ( optional, 32bit unsigned integer ): when specified, represents the fractional part of the `mtime` as the amount of nanoseconds. The valid range for this value are the integers `[1, 999999999]`. Implementations encoding or decoding wire-representations MUST observe the following: - An `mtime` structure with `FractionalNanoseconds` outside of the on-wire range `[1, 999999999]` is **not** valid. This includes a fractional value of `0`. Implementations encountering such values should consider the entire enclosing - metadata block malformed and abort processing the corresponding DAG. -- The `mtime` structure is optional - its absence implies `unspecified`, rather - than `0` -- For ergonomic reasons a surface API of an encoder MUST allow fractional 0 as + metadata block malformed and abort the processing of the corresponding DAG. +- The `mtime` structure is optional. Its absence implies `unspecified` rather + than `0`. +- For ergonomic reasons, a surface API of an encoder MUST allow fractional `0` as input, while at the same time MUST ensure it is stripped from the final structure before encoding, satisfying the above constraints. -Implementations interpreting the mtime metadata in order to apply it within a +Implementations interpreting the `mtime` metadata in order to apply it within a non-IPFS target MUST observe the following: - If the target supports a distinction between `unspecified` and `0`/`1970-01-01T00:00:00Z`, - the distinction must be preserved within the target. E.g. if no `mtime` structure + the distinction must be preserved within the target. For example, if no `mtime` structure is available, a web gateway must **not** render a `Last-Modified:` header. -- If the target requires an mtime ( e.g. a FUSE interface ) and no `mtime` is +- If the target requires an `mtime` ( e.g. a FUSE interface ) and no `mtime` is supplied OR the supplied `mtime` falls outside of the targets accepted range: - When no `mtime` is specified or the resulting `UnixTime` is negative: implementations must assume `0`/`1970-01-01T00:00:00Z` (note that such values are not merely academic: e.g. the OpenVMS epoch is `1858-11-17T00:00:00Z`) - When the resulting `UnixTime` is larger than the targets range ( e.g. 32bit - vs 64bit mismatch) implementations must assume the highest possible value - in the targets range (in most cases that would be `2038-01-19T03:14:07Z`) + vs 64bit mismatch), implementations must assume the highest possible value + in the targets range. In most cases, this would be `2038-01-19T03:14:07Z`. ## Paths -Paths first start with `/` or `/ipfs//` where `` is a [multibase] -encoded [CID]. The CID encoding MUST NOT use a multibase alphabet that have -`/` (`0x2f`) unicode codepoints however CIDs may use a multibase encoding with +Paths begin with a `/` or `/ipfs//`, where `` is a [multibase] +encoded [CID]. The CID encoding MUST NOT use a multibase alphabet that contains +`/` (`0x2f`) unicode codepoints. However, CIDs may use a multibase encoding with a `/` in the alphabet if the encoded CID does not contain `/` once encoded. -Everything following the CID is a collection of path component (some bytes) +Everything following the CID is a collection of path components (some bytes) separated by `/` (`0x2F`). UnixFS paths read from left to right, and are inspired by POSIX paths. -- Components MUST NOT contain `/` unicode codepoints because else it would break +- Components MUST NOT contain `/` unicode codepoints because it would break the path into two components. - Components SHOULD be UTF8 unicode. -- Components are case sensitive. +- Components are case-sensitive. ### Escaping -The `\` may be supposed to trigger an escape sequence. However, it is currently +The `\` may be used to trigger an escape sequence. However, it is currently broken and inconsistent across implementations. Until we agree on a specification -for this, you SHOULD NOT use any escape sequences and non-ASCII characters. +for this, you SHOULD NOT use any escape sequences and/or non-ASCII characters. ### Relative Path Components Relative path components MUST be resolved before trying to work on the path: -- `.` points to the current node, those path components MUST be removed. -- `..` points to the parent node, they MUST be removed left to right. When removing +- `.` points to the current node and MUST be removed. +- `..` points to the parent node and MUST be removed left to right. When removing a `..`, the path component on the left MUST also be removed. If there is no path - component on the left, you MUST error since it is an attempt of out-of-bounds + component on the left, you MUST error to avoid out-of-bounds path resolution. ### Restricted Names The following names SHOULD NOT be used: -- The `.` string: represents the self node in POSIX pathing. -- The `..` string: represents the parent node in POSIX pathing. +- The `.` string, as it represents the self node in POSIX pathing. +- The `..` string, as it represents the parent node in POSIX pathing. - The empty string. -- Any string containing a `NUL` (`0x00`) byte: this is often used to signify string - terminations in some systems (such as most C compatible systems), and many unix +- Any string containing a `NULL` (`0x00`) byte, as this is often used to signify string + terminations in some systems, such as C-compatible systems. Many unix file systems do not accept this character in path components. ## Design Decision Rationale @@ -444,25 +438,25 @@ The following names SHOULD NOT be used: ### `mtime` and `mode` Metadata Support in UnixFSv1.5 Metadata support in UnixFSv1.5 has been expanded to increase the number of possible -use cases. These include rsync and filesystem based package managers. +use cases. These include `rsync` and filesystem-based package managers. Several metadata systems were evaluated, as discussed in the following sections. #### Separate Metadata Node In this scheme, the existing `Metadata` message is expanded to include additional -metadata types (`mtime`, `mode`, etc). It contains links to the actual file data +metadata types (`mtime`, `mode`, etc). It contains links to the actual file data, but never the file data itself. This was ultimately rejected for a number of reasons: -1. You would always need to retrieve an additional node to access file data which - limits the kind of optimizations that are possible. For example many files are +1. You would always need to retrieve an additional node to access file data, which + limits the kind of optimizations that are possible. For example, many files are under the 256 KiB block size limit, so we tend to inline them into the describing UnixFS `File` node. This would not be possible with an intermediate `Metadata` node. -2. The `File` node already contains some metadata (e.g. the file size) so metadata - would be stored in multiple places which complicates forwards compatibility with - UnixFSv2 as to map between metadata formats potentially requires multiple fetch +2. The `File` node already contains some metadata (e.g. the file size), so metadata + would be stored in multiple places. This complicates forwards compatibility with + UnixFSv2, as mapping between metadata formats potentially requires multiple fetch operations. #### Metadata in the Directory @@ -471,21 +465,21 @@ Repeated `Metadata` messages are added to UnixFS `Directory` and `HAMTShard` nod the index of which indicates which entry they are to be applied to. Where entries are `HAMTShard`s, an empty message is added. -One advantage of this method is that if we expand stored metadata to include entry -types and sizes we can perform directory listings without needing to fetch further -entry nodes (excepting `HAMTShard` nodes), though without removing the storage of -these datums elsewhere in the spec we run the risk of having non-canonical data +One advantage of this method is that, if we expand stored metadata to include entry +types and sizes, we can perform directory listings without needing to fetch further +entry nodes (excepting `HAMTShard` nodes). However, without removing the storage of +these datums elsewhere in the spec, we run the risk of having non-canonical data locations and perhaps conflicting data as we traverse through trees containing both UnixFS v1 and v1.5 nodes. This was rejected for the following reasons: -1. When creating a UnixFS node there's no way to record metadata without wrapping +1. When creating a UnixFS node, there's no way to record metadata without wrapping it in a directory. 2. If you access any UnixFS node directly by its [CID], there is no way of recreating the metadata which limits flexibility. 3. In order to list the contents of a directory including entry types and sizes, - you have to fetch the root node of each entry anyway so the performance benefit + you have to fetch the root node of each entry, so the performance benefit of including some metadata in the containing directory is negligible in this use case. @@ -493,27 +487,27 @@ This was rejected for the following reasons: This adds new fields to the UnixFS `Data` message to represent the various metadata fields. -It has the advantage of being simple to implement, metadata is maintained whether +It has the advantage of being simple to implement. Metadata is maintained whether the file is accessed directly via its [CID] or via an IPFS path that includes a -containing directory, and by keeping the metadata small enough we can inline root -UnixFS nodes into their CIDs so we can end up fetching the same number of nodes if +containing directory. In addition, metadata is kept small enough that we can inline root +UnixFS nodes into their CIDs so that we can end up fetching the same number of nodes if we decide to keep file data in a leaf node for deduplication reasons. Downsides to this approach are: 1. Two users adding the same file to IPFS at different times will have different [CID]s due to the `mtime`s being different. If the content is stored in another - node, its [CID] will be constant between the two users but you can't navigate - to it unless you have the parent node which will be less available due to the + node, its [CID] will be constant between the two users, but you can't navigate + to it unless you have the parent node, which will be less available due to the proliferation of [CID]s. 1. Metadata is also impossible to remove without changing the [CID], so metadata becomes part of the content. 2. Performance may be impacted as well as if we don't inline UnixFS root nodes - into [CID]s, additional fetches will be required to load a given UnixFS entry. + into [CID]s, so additional fetches will be required to load a given UnixFS entry. #### Side Trees -With this approach we would maintain a separate data structure outside of the +With this approach, we would maintain a separate data structure outside of the UnixFS tree to hold metadata. This was rejected due to concerns about added complexity, recovery after system @@ -525,7 +519,7 @@ when resolving [CID]s from peers. This scheme would see metadata stored in an external database. The downsides to this are that metadata would not be transferred from one node -to another when syncing as [Bitswap] is not aware of the database, and in-tree +to another when syncing, as [Bitswap] is not aware of the database and in-tree metadata. ### UnixTime Protobuf Datatype Rationale @@ -534,15 +528,14 @@ metadata. The integer portion of UnixTime is represented on the wire using a `varint` encoding. While this is inefficient for negative values, it avoids introducing zig-zag encoding. -Values before the year 1970 will be exceedingly rare, and it would be handy having -such cases stand out, while at the same keeping the "usual" positive values easy -to eyeball. The `varint` representing the time of writing this text is 5 bytes +Values before the year `1970` are exceedingly rare, and it would be handy having +such cases stand out, while ensuring that the "usual" positive values are easily readable. The `varint` representing the time of writing this text is 5 bytes long. It will remain so until October 26, 3058 (34,359,738,367). #### FractionalNanoseconds -Fractional values are effectively a random number in the range 1 ~ 999,999,999. -Such values will exceed 2^28 nanoseconds (268,435,456) in most cases. Therefore, +Fractional values are effectively a random number in the range 1 to 999,999,999. +In most cases, such values will exceed 2^28 (268,435,456) nanoseconds. Therefore, the fractional part is represented as a 4-byte `fixed32`, [as per Google's recommendation](https://developers.google.com/protocol-buffers/docs/proto#scalar). @@ -568,7 +561,7 @@ This section and included subsections are not authoritative. In this example, we will build a `Raw` file with the string `test` as its content. -1. First hash the data: +1. First, hash the data: ```console $ echo -n "test" | sha256sum @@ -586,7 +579,7 @@ f this is the multibase prefix, we need it because we are working with a hex CID 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 the digest we computed earlier ``` -3. Profit: assuming we stored this block in some implementation of our choice which makes it accessible to our client, we can try to decode it. +3. Profit! Assuming we stored this block in some implementation of our choice, which makes it accessible to our client, we can try to decode it. ```console $ ipfs cat f015512209f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 @@ -615,4 +608,4 @@ This will tell you which offset inside this node the children at the correspondi [multicodec]: https://github.com/multiformats/multicodec [multihash]: https://github.com/multiformats/multihash [Bitswap]: https://github.com/ipfs/specs/blob/master/BITSWAP.md -[ipld-dag-pb]: https://ipld.io/specs/codecs/dag-pb/spec/ +[ipld-dag-pb]: https://ipld.io/specs/codecs/dag-pb/spec/ \ No newline at end of file From 1667bd4e6d71b040d2313588d82aa04f89f4ced5 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Fri, 6 Sep 2024 18:41:20 +0200 Subject: [PATCH 07/46] unixfs: Tsize suggestions from review --- src/architecture/unixfs.md | 37 +++++++++++++++++++++++-------------- 1 file changed, 23 insertions(+), 14 deletions(-) diff --git a/src/architecture/unixfs.md b/src/architecture/unixfs.md index a9d2f4663..9c2d5624c 100644 --- a/src/architecture/unixfs.md +++ b/src/architecture/unixfs.md @@ -317,24 +317,33 @@ To resolve the path inside a HAMT: name you were trying to resolve, you have successfully resolved a path component. Everything past the hex encoded prefix is the name of that element, which is useful when listing children of this directory. -### `TSize` / `DagSize` +### `TSize` (child DAG size hint) -This is an optional field in `PBNode.Links[]`. It **does not** represent any -meaningful information of the underlying structure, and there is no known -usage of it to this day, although some implementations omit these. +`Tsize` is an optional field in `PBNode.Links[]` which represents the precomputed size of the specific child DAG. It provides a performance optimization: a hint about the total size of child DAG can be read without having to fetch any child nodes. -To compute the `DagSize` of a node, which is stored in the parents, sum the length of the `dag-pb` outside message binary length and the `blocksizes` of all child files. +To compute the `Tsize` of a child DAG, sum the length of the `dag-pb` outside message binary length and the `blocksizes` of all nodes in the child DAG. -An example of where this could be useful is as a hint to smart download clients. -For example, if you are downloading a file concurrently from two sources that have -radically different speeds, it would probably be more efficient to download bigger -links from the fastest source, and smaller ones from the slowest source. +:::note - -There is no failure mode known for this field, so your implementation should be -able to decode nodes where this field is wrong (not the value you expect), or -partially or completely missing. This also allows smarter encoders to give a -more accurate picture (Don't count duplicate blocks, etc.). +Examples of where `Tsize` is useful: + +- User interfaces, where total size of a DAG needs to be displayed immediately, without having to do the full DAG walk. +- Smart download clients, downloading a file concurrently from two sources that have radically different speeds. It may be more efficient to parallelize and download bigger +links from the fastest source, and smaller ones from the slower sources. + +::: + +:::warning + +An implementation SHOULD NOT assume the `TSize` values are correct. The value is only a hint that provides performance optimization for better UX. + +Following the [Robustness Principle](https://specs.ipfs.tech/architecture/principles/#robustness), implementation SHOULD be +able to decode nodes where the `Tsize` field is wrong (not matching the sizes of sub-DAGs), or +partially or completely missing. + +When total data size is needed for important purposes such as accounting, billing, and cost estimation, the `Tsize` SHOULD NOT be used, and instead a full DAG walk SHOULD to be performed. + +::: ### Metadata From 0559e6585c8ed49915e8df0b6e004b933f7e023b Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Fri, 6 Sep 2024 18:44:25 +0200 Subject: [PATCH 08/46] unxifs: suggestions from code review Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> --- src/architecture/unixfs.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/architecture/unixfs.md b/src/architecture/unixfs.md index 9c2d5624c..21c41c4d5 100644 --- a/src/architecture/unixfs.md +++ b/src/architecture/unixfs.md @@ -67,7 +67,7 @@ be recognized because their CIDs are encoded using the `raw` codec: - The file content is purely the block body. - They never have any children nodes, and thus are also known as single block files. -- Their size (both `dagsize` and `blocksize`) is the length of the block body. +- Their size is the length of the block body (`Tsize` in parent is equal to `blocksize`). ## `dag-pb` Nodes @@ -270,7 +270,7 @@ remaining components and on the CID you popped. A :dfn[Symlink] represents a POSIX [symbolic link](https://pubs.opengroup.org/onlinepubs/9699919799/functions/symlink.html). A symlink MUST NOT have children. -The `PBNode.Data.Data` field is a POSIX path that MAY be appended in front of the +The `PBNode.Data.Data` field is a POSIX path that MAY be inserted in front of the currently remaining path component stack. ##### Path Resolution @@ -279,7 +279,7 @@ There is no current consensus on how pathing over symlinks should behave. Some implementations return symlink objects and fail if a consumer tries to follow them through. -Following the POSIX specification over the current UnixFS path context is probably fine. +Symlink path resolution SHOULD follow the POSIX specification, over the current UnixFS path context, as much as is applicable. #### `HAMTDirectory` From 6c79d5bec4f6c3b704aef90d30abf6dc89c411ff Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Fri, 6 Sep 2024 18:44:39 +0200 Subject: [PATCH 09/46] unixfs: editorial changes --- UNIXFS.md | 2 +- src/data-formats/index.html | 13 ++++++++++ src/index.html | 13 ++++++++++ .../unixfs.md => unixfs-data-format.md} | 25 ++++++++++++------- 4 files changed, 43 insertions(+), 10 deletions(-) create mode 100644 src/data-formats/index.html rename src/{architecture/unixfs.md => unixfs-data-format.md} (97%) diff --git a/UNIXFS.md b/UNIXFS.md index 00444fe49..243f5f030 100644 --- a/UNIXFS.md +++ b/UNIXFS.md @@ -1,3 +1,3 @@ # UnixFS -Moved to https://specs.ipfs.tech/architecture/unixfs/ +Moved to https://specs.ipfs.tech/unixfs-data-format/ diff --git a/src/data-formats/index.html b/src/data-formats/index.html new file mode 100644 index 000000000..c34ea9654 --- /dev/null +++ b/src/data-formats/index.html @@ -0,0 +1,13 @@ +--- +title: Data formats +description: | + IPFS basic primitive is an opaque block of bytes identified by a CID. CID includes codec that informs IPFS System about data format: how to parse the block, and how to link from one block to another. +--- + +{% include 'header.html' %} + +
+ {% include 'list.html', posts: collections.data-formats %} +
+ +{% include 'footer.html' %} diff --git a/src/index.html b/src/index.html index d2d31ad64..96f8b950e 100644 --- a/src/index.html +++ b/src/index.html @@ -108,6 +108,19 @@

HTTP Gateways

{% include 'list.html', posts: collections.webHttpGateways %} +
+

Data Formats

+

+ IPFS basic primitive is an opaque block of bytes identified by a CID. CID includes codec that informs IPFS System about data format: how to parse the block, and how to link from one block to another. +

+

+ The most popular data formats used by IPFS Systems are RAW (opaque block), CAR (archive of opaque blocks), UnixFS (filesystem abstraction built with DAG-PB and RAW codecs), DAG-CBOR/DAG-JSON, however IPFS ecosystem is not limited to them, and IPFS systems are free to choose the level of interoperability, or even implement support for own, additional formats. A complimentary CAR is a codec-agnostic archive format for transporting multiple opaque blocks. +

+

+ Specifications: +

+ {% include 'list.html', posts: collections.data-formats %} +

InterPlanetary Naming System

diff --git a/src/architecture/unixfs.md b/src/unixfs-data-format.md similarity index 97% rename from src/architecture/unixfs.md rename to src/unixfs-data-format.md index 21c41c4d5..ad08a87e9 100644 --- a/src/architecture/unixfs.md +++ b/src/unixfs-data-format.md @@ -2,9 +2,9 @@ title: UnixFS description: > UnixFS is a Protocol Buffers-based format for describing files, directories, - and symlinks as DAGs in IPFS. -date: 2022-10-10 -maturity: reliable + and symlinks as dag-pb and raw DAGs in IPFS. +date: 2024-09-06 +maturity: draft editors: - name: David Dias github: daviddias @@ -31,8 +31,13 @@ editors: affiliation: name: Protocol Labs url: https://protocol.ai/ + - name: Marcin Rataj + github: lidel + affiliation: + name: Interplanetary Shipyard + url: https://ipshipyard.com/ -tags: ['architecture'] +tags: ['data-formats'] order: 1 --- @@ -63,7 +68,7 @@ In UnixFS, a node can be encoded using two different multicodecs, listed below. ## `Raw` Nodes The simplest nodes use `raw` encoding and are implicitly a :ref[File]. They can -be recognized because their CIDs are encoded using the `raw` codec: +be recognized because their CIDs are encoded using the `raw` (`0x55`) codec: - The file content is purely the block body. - They never have any children nodes, and thus are also known as single block files. @@ -71,7 +76,7 @@ be recognized because their CIDs are encoded using the `raw` codec: ## `dag-pb` Nodes -More complex nodes use the `dag-pb` encoding. These nodes require two steps of +More complex nodes use the `dag-pb` (`0x70`) encoding. These nodes require two steps of decoding. The first step is to decode the outer container of the block. This is encoded using the IPLD [`dag-pb`][ipld-dag-pb] specification, which can be summarized as follows: @@ -117,8 +122,8 @@ message Data { repeated uint64 blocksizes = 4; optional uint64 hashType = 5; optional uint64 fanout = 6; - optional uint32 mode = 7; - optional UnixTime mtime = 8; + optional uint32 mode = 7; // opt-in, AKA UnixFS 1.5 + optional UnixTime mtime = 8; // opt-in, AKA UnixFS 1.5 } message Metadata { @@ -176,8 +181,10 @@ size in bytes of the partial file content present in children DAGs. Each index i `PBNode.Links` MUST have a corresponding chunk size stored at the same index in `decode(PBNode.Data).blocksizes`. +:::warning Implementers need to be extra careful to ensure the values in `Data.blocksizes` are calculated by following the definition from [`Blocksize`](#decodepbnodedatablocksize). +::: This allows for fast indexing into the file. For example, if someone is trying to read bytes 25 to 35, we can compute an offset list by summing all previous @@ -617,4 +624,4 @@ This will tell you which offset inside this node the children at the correspondi [multicodec]: https://github.com/multiformats/multicodec [multihash]: https://github.com/multiformats/multihash [Bitswap]: https://github.com/ipfs/specs/blob/master/BITSWAP.md -[ipld-dag-pb]: https://ipld.io/specs/codecs/dag-pb/spec/ \ No newline at end of file +[ipld-dag-pb]: https://ipld.io/specs/codecs/dag-pb/spec/ From 02dec5aedbe5d11ed802446cb835962aecaa36bd Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Fri, 6 Sep 2024 18:58:20 +0200 Subject: [PATCH 10/46] unixfs: editorial, updated implementations links --- src/unixfs-data-format.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/src/unixfs-data-format.md b/src/unixfs-data-format.md index ad08a87e9..a657c92a3 100644 --- a/src/unixfs-data-format.md +++ b/src/unixfs-data-format.md @@ -19,8 +19,8 @@ editors: - name: Alex Potsides github: achingbrain affiliation: - name: Protocol Labs - url: https://protocol.ai/ + name: Interplanetary Shipyard + url: https://ipshipyard.com/ - name: Peter Rabbitson github: ribasushi affiliation: @@ -449,7 +449,7 @@ The following names SHOULD NOT be used: terminations in some systems, such as C-compatible systems. Many unix file systems do not accept this character in path components. -## Design Decision Rationale +## Appendix: Design Decision Rationale ### `mtime` and `mode` Metadata Support in UnixFSv1.5 @@ -555,20 +555,23 @@ In most cases, such values will exceed 2^28 (268,435,456) nanoseconds. Therefore the fractional part is represented as a 4-byte `fixed32`, [as per Google's recommendation](https://developers.google.com/protocol-buffers/docs/proto#scalar). -# Notes for Implementers +# Appendix: Notes for Implementers This section and included subsections are not authoritative. ## Implementations - JavaScript + - [`@helia/unixfs`](https://www.npmjs.com/package/@helia/unixfs) implementation of a filesystem compatible with [Helia SDK](https://github.com/ipfs/helia#readme) - Data Formats - [unixfs](https://github.com/ipfs/js-ipfs-unixfs) - - Importer - [unixfs-importer](https://github.com/ipfs/js-ipfs-unixfs-importer) - - Exporter - [unixfs-exporter](https://github.com/ipfs/js-ipfs-unixfs-exporter) + - Importer - [unixfs-importer](https://github.com/ipfs/js-ipfs-unixfs/tree/main/packages/ipfs-unixfs-importer) + - Exporter - [unixfs-exporter](https://github.com/ipfs/js-ipfs-unixfs/tree/main/packages/ipfs-unixfs-exporter) - Go - - Protocol Buffer Definitions - [`ipfs/go-unixfs/pb`](https://github.com/ipfs/go-unixfs/blob/707110f05dac4309bdcf581450881fb00f5bc578/pb/unixfs.proto) - - [`ipfs/go-unixfs`](https://github.com/ipfs/go-unixfs/) - - `go-ipld-prime` implementation [`ipfs/go-unixfsnode`](https://github.com/ipfs/go-unixfsnode) + - [Boxo SDK](https://github.com/ipfs/boxo#readme) includes implementation of UnixFS filesystem + - Protocol Buffer Definitions - [`ipfs/boxo/../unixfs.proto`](https://github.com/ipfs/boxo/blob/v0.23.0/ipld/unixfs/pb/unixfs.proto) + - [`boxo/files`](https://github.com/ipfs/boxo/tree/main/files) + - [`ipfs/boxo/ipld/unixfs`](https://github.com/ipfs/boxo/tree/main/ipld/unixfs/) + - Alternative `go-ipld-prime` implementation: [`ipfs/go-unixfsnode`](https://github.com/ipfs/go-unixfsnode) - Rust - [`iroh-unixfs`](https://github.com/n0-computer/iroh/tree/b7a4dd2b01dbc665435659951e3e06d900966f5f/iroh-unixfs) - [`unixfs-v1`](https://github.com/ipfs-rust/unixfsv1) From c70c6145b788f09d7879d57c3d53bfb42c4275a8 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Fri, 6 Sep 2024 19:49:52 +0200 Subject: [PATCH 11/46] chore: remove trailing spaces --- src/unixfs-data-format.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/unixfs-data-format.md b/src/unixfs-data-format.md index a657c92a3..0066c6fff 100644 --- a/src/unixfs-data-format.md +++ b/src/unixfs-data-format.md @@ -268,7 +268,7 @@ a child under `PBNode.Links`. If you find a match, you can then remember the CID You MUST continue the search. If you find another match, you MUST error since duplicate names are not allowed. -Assuming no errors were raised, you can continue to the path resolution on the +Assuming no errors were raised, you can continue to the path resolution on the remaining components and on the CID you popped. @@ -345,7 +345,7 @@ links from the fastest source, and smaller ones from the slower sources. An implementation SHOULD NOT assume the `TSize` values are correct. The value is only a hint that provides performance optimization for better UX. Following the [Robustness Principle](https://specs.ipfs.tech/architecture/principles/#robustness), implementation SHOULD be -able to decode nodes where the `Tsize` field is wrong (not matching the sizes of sub-DAGs), or +able to decode nodes where the `Tsize` field is wrong (not matching the sizes of sub-DAGs), or partially or completely missing. When total data size is needed for important purposes such as accounting, billing, and cost estimation, the `Tsize` SHOULD NOT be used, and instead a full DAG walk SHOULD to be performed. @@ -397,7 +397,7 @@ non-IPFS target MUST observe the following: - If the target supports a distinction between `unspecified` and `0`/`1970-01-01T00:00:00Z`, the distinction must be preserved within the target. For example, if no `mtime` structure is available, a web gateway must **not** render a `Last-Modified:` header. -- If the target requires an `mtime` ( e.g. a FUSE interface ) and no `mtime` is +- If the target requires an `mtime` ( e.g. a FUSE interface ) and no `mtime` is supplied OR the supplied `mtime` falls outside of the targets accepted range: - When no `mtime` is specified or the resulting `UnixTime` is negative: implementations must assume `0`/`1970-01-01T00:00:00Z` (note that such values @@ -513,10 +513,10 @@ Downsides to this approach are: 1. Two users adding the same file to IPFS at different times will have different [CID]s due to the `mtime`s being different. If the content is stored in another - node, its [CID] will be constant between the two users, but you can't navigate - to it unless you have the parent node, which will be less available due to the + node, its [CID] will be constant between the two users, but you can't navigate + to it unless you have the parent node, which will be less available due to the proliferation of [CID]s. -1. Metadata is also impossible to remove without changing the [CID], so +1. Metadata is also impossible to remove without changing the [CID], so metadata becomes part of the content. 2. Performance may be impacted as well as if we don't inline UnixFS root nodes into [CID]s, so additional fetches will be required to load a given UnixFS entry. From 556a6c8e5dbc2f3def2d830cca399cd71b327ab2 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Fri, 6 Sep 2024 20:17:35 +0200 Subject: [PATCH 12/46] chore: remove duplicated mention of cars --- src/index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/index.html b/src/index.html index 78cbcd882..5a630da49 100644 --- a/src/index.html +++ b/src/index.html @@ -114,7 +114,7 @@

Data Formats

IPFS basic primitive is an opaque block of bytes identified by a CID. CID includes codec that informs IPFS System about data format: how to parse the block, and how to link from one block to another.

- The most popular data formats used by IPFS Systems are RAW (opaque block), CAR (archive of opaque blocks), UnixFS (filesystem abstraction built with DAG-PB and RAW codecs), DAG-CBOR/DAG-JSON, however IPFS ecosystem is not limited to them, and IPFS systems are free to choose the level of interoperability, or even implement support for own, additional formats. A complimentary CAR is a codec-agnostic archive format for transporting multiple opaque blocks. + The most popular data formats used by IPFS Systems are RAW (an opaque block), UnixFS (filesystem abstraction built with DAG-PB and RAW codecs), DAG-CBOR/DAG-JSON, however IPFS ecosystem is not limited to them, and IPFS systems are free to choose the level of interoperability, or even implement support for own, additional formats. A complimentary CAR is a codec-agnostic archive format for transporting multiple opaque blocks.

Specifications: From 9131a7cfb56bf2ef4fb916feec53836c75ba1a05 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Fri, 6 Sep 2024 20:32:56 +0200 Subject: [PATCH 13/46] chore: markdown lint --- src/unixfs-data-format.md | 47 +++++++++++++++++++++------------------ 1 file changed, 25 insertions(+), 22 deletions(-) diff --git a/src/unixfs-data-format.md b/src/unixfs-data-format.md index 0066c6fff..3de0ae4cd 100644 --- a/src/unixfs-data-format.md +++ b/src/unixfs-data-format.md @@ -271,7 +271,6 @@ duplicate names are not allowed. Assuming no errors were raised, you can continue to the path resolution on the remaining components and on the CID you popped. - #### `Symlink` type A :dfn[Symlink] represents a POSIX [symbolic link](https://pubs.opengroup.org/onlinepubs/9699919799/functions/symlink.html). @@ -490,14 +489,14 @@ both UnixFS v1 and v1.5 nodes. This was rejected for the following reasons: -1. When creating a UnixFS node, there's no way to record metadata without wrapping - it in a directory. -2. If you access any UnixFS node directly by its [CID], there is no way of recreating - the metadata which limits flexibility. -3. In order to list the contents of a directory including entry types and sizes, - you have to fetch the root node of each entry, so the performance benefit - of including some metadata in the containing directory is negligible in this - use case. +1. When creating a UnixFS node, there's no way to record metadata without + wrapping it in a directory. +2. If you access any UnixFS node directly by its [CID], there is no way of + recreating the metadata which limits flexibility. +3. In order to list the contents of a directory including entry types and + sizes, you have to fetch the root node of each entry, so the performance + benefit of including some metadata in the containing directory is negligible + in this use case. #### Metadata in the File @@ -511,15 +510,16 @@ we decide to keep file data in a leaf node for deduplication reasons. Downsides to this approach are: -1. Two users adding the same file to IPFS at different times will have different - [CID]s due to the `mtime`s being different. If the content is stored in another - node, its [CID] will be constant between the two users, but you can't navigate - to it unless you have the parent node, which will be less available due to the - proliferation of [CID]s. -1. Metadata is also impossible to remove without changing the [CID], so - metadata becomes part of the content. -2. Performance may be impacted as well as if we don't inline UnixFS root nodes - into [CID]s, so additional fetches will be required to load a given UnixFS entry. +1. Two users adding the same file to IPFS at different times will have + different [CID]s due to the `mtime`s being different. If the content is + stored in another node, its [CID] will be constant between the two users, + but you can't navigate to it unless you have the parent node, which will be + less available due to the proliferation of [CID]s. +2. Metadata is also impossible to remove without changing the [CID], so + metadata becomes part of the content. +3. Performance may be impacted as well as if we don't inline UnixFS root nodes + into [CID]s, so additional fetches will be required to load a given UnixFS + entry. #### Side Trees @@ -580,25 +580,28 @@ This section and included subsections are not authoritative. In this example, we will build a `Raw` file with the string `test` as its content. -1. First, hash the data: +First, hash the data: ```console $ echo -n "test" | sha256sum 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 - ``` -2. Add the CID prefix: +Add the CID prefix: ``` +f01551220 + 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 + f this is the multibase prefix, we need it because we are working with a hex CID, this is omitted for binary CIDs 01 the CID version, here one 55 the codec, here we MUST use Raw because this is a Raw file 12 the hashing function used, here sha256 20 the digest length 32 bytes - 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 the digest we computed earlier + 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 is the the digest we computed earlier ``` -3. Profit! Assuming we stored this block in some implementation of our choice, which makes it accessible to our client, we can try to decode it. +Done. Assuming we stored this block in some implementation of our choice, which makes it accessible to our client, we can try to decode it. ```console $ ipfs cat f015512209f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 From 5ef46128fd09f54b928eaa186ba190437903f8b8 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sat, 1 Mar 2025 01:22:17 +0100 Subject: [PATCH 14/46] chore: editorial cleanup - fixed headers - added some warnings - updated metadata section to be more clear --- src/unixfs-data-format.md | 181 +++++++++++++++++++++----------------- 1 file changed, 100 insertions(+), 81 deletions(-) diff --git a/src/unixfs-data-format.md b/src/unixfs-data-format.md index 3de0ae4cd..76c73bd26 100644 --- a/src/unixfs-data-format.md +++ b/src/unixfs-data-format.md @@ -3,19 +3,13 @@ title: UnixFS description: > UnixFS is a Protocol Buffers-based format for describing files, directories, and symlinks as dag-pb and raw DAGs in IPFS. -date: 2024-09-06 +date: 2025-03-01 maturity: draft editors: - name: David Dias github: daviddias - affiliation: - name: Protocol Labs - url: https://protocol.ai/ - name: Jeromy Johnson github: whyrusleeping - affiliation: - name: Protocol Labs - url: https://protocol.ai/ - name: Alex Potsides github: achingbrain affiliation: @@ -23,14 +17,8 @@ editors: url: https://ipshipyard.com/ - name: Peter Rabbitson github: ribasushi - affiliation: - name: Protocol Labs - url: https://protocol.ai/ - name: Hugo Valtier github: jorropo - affiliation: - name: Protocol Labs - url: https://protocol.ai/ - name: Marcin Rataj github: lidel affiliation: @@ -41,10 +29,7 @@ tags: ['data-formats'] order: 1 --- -UnixFS is a [protocol-buffers][protobuf]-based format for describing files, -directories and symlinks as Directed Acyclic Graphs (DAGs) in IPFS. - -## Nodes +# Node Types A :dfn[Node] is the smallest unit present in a graph, and it comes from graph theory. In UnixFS, there is a 1-to-1 mapping between nodes and blocks. Therefore, @@ -62,19 +47,19 @@ hash function specified in the multihash, gives us the same multihash value back In UnixFS, a node can be encoded using two different multicodecs, listed below. More details are provided in the following sections: -- `raw` (`0x55`), which are single block :ref[Files]. -- `dag-pb` (`0x70`), which can be of any other type. +- [`raw`](#raw-node) (`0x55`), which are single block files without any metadata. +- [`dag-pb`](#dag-pb-node) (`0x70`), which can be of any other type. -## `Raw` Nodes +# `raw` Node The simplest nodes use `raw` encoding and are implicitly a :ref[File]. They can be recognized because their CIDs are encoded using the `raw` (`0x55`) codec: -- The file content is purely the block body. +- The block is the file data. There is no protobuf envelope or metadata. - They never have any children nodes, and thus are also known as single block files. - Their size is the length of the block body (`Tsize` in parent is equal to `blocksize`). -## `dag-pb` Nodes +# `dag-pb` Node More complex nodes use the `dag-pb` (`0x70`) encoding. These nodes require two steps of decoding. The first step is to decode the outer container of the block. This is encoded using the IPLD [`dag-pb`][ipld-dag-pb] specification, which can be @@ -111,7 +96,7 @@ message Data { Raw = 0; Directory = 1; File = 2; - Metadata = 3; + Metadata = 3; // reserved for future use Symlink = 4; HAMTShard = 5; } @@ -141,17 +126,17 @@ whose `Data` field is a UnixFSV1 Protobuf message. For clarity, the specificatio document may represent these nested Protobufs as one object. In this representation, it is implied that the `PBNode.Data` field is encoded in a protobuf. -### Data Types +## `dag-pb` Types A `dag-pb` UnixFS node supports different types, which are defined in `decode(PBNode.Data).Type`. Every type is handled differently. -#### `File` type +### `dag-pb` `File` A :dfn[File] is a container over an arbitrary sized amount of bytes. Files are either single block or multi-block. A multi-block file is a concatenation of multiple child files. -##### The _sister-lists_ `PBNode.Links` and `decode(PBNode.Data).blocksizes` +#### The _sister-lists_ `PBNode.Links` and `decode(PBNode.Data).blocksizes` The _sister-lists_ are the key point of why IPLD `dag-pb` is important for files. They allow us to concatenate smaller files together. @@ -195,14 +180,14 @@ In the example above, the offset list would be `[0, 20]`. Thus, we know we only UnixFS parser MUST error if `blocksizes` or `Links` are not of the same length. -##### `decode(PBNode.Data).Data` +#### `decode(PBNode.Data).Data` An array of bytes that is the file content and is appended before the links. This must be taken into account when doing offset calculations; that is, the length of `decode(PBNode.Data).Data` defines the value of the zeroth element of the offset list when computing offsets. -##### `PBNode.Links[].Name` +#### `PBNode.Links[].Name` This field makes sense only in :ref[Directories] contexts and MUST be absent when creating a new file. For historical reasons, implementations parsing @@ -211,24 +196,24 @@ third-party data SHOULD accept empty values here. If this field is present and non-empty, the file is invalid and the parser MUST error. -##### `decode(PBNode.Data).Blocksize` +#### `decode(PBNode.Data).Blocksize` This field is not directly present in the block, but rather a computable property of a `dag-pb`, which would be used in the parent node in `decode(PBNode.Data).blocksizes`. It is the sum of the length of `decode(PBNode.Data).Data` field plus the sum of all link's `blocksizes`. -##### `decode(PBNode.Data).filesize` +#### `decode(PBNode.Data).filesize` If present, this field MUST be equal to the `Blocksize` computation above. Otherwise, this file is invalid. -##### Path Resolution +#### `dag-pb` `File` Path Resolution A file terminates a UnixFS content path. Any attempt to resolve a path past a file MUST error. -#### `Directory` Type +### `dag-pb` `Directory` A :dfn[Directory], also known as folder, is a named collection of child :ref[Nodes]: @@ -237,6 +222,8 @@ A :dfn[Directory], also known as folder, is a named collection of child :ref[Nod - Duplicate names are not allowed. Therefore, two elements of `PBNode.Link` CANNOT have the same `Name`. If two identical names are present in a directory, the decoder MUST fail. +- Implementations SHOULD detect when directory becomes too big to fit in a single + `Directory` block and use [`HAMTDirectory`] type instead. The minimum valid `PBNode.Data` field for a directory is as follows: @@ -246,9 +233,7 @@ The minimum valid `PBNode.Data` field for a directory is as follows: } ``` -The remaining relevant values are covered in [Metadata](#metadata). - -##### Link Ordering +#### `dag-pb` `Directory` Link Ordering The canonical sorting order is lexicographical over the names. @@ -261,7 +246,7 @@ it consumed those names. However, when some implementations decode, modify and t re-encode, the original link order loses it's original meaning, given that there is no way to indicate which sorting was used originally. -##### Path Resolution +#### `dag-pb` `Directory` Path Resolution Pop the left-most component of the path, and try to match it to the `Name` of a child under `PBNode.Links`. If you find a match, you can then remember the CID. @@ -271,23 +256,7 @@ duplicate names are not allowed. Assuming no errors were raised, you can continue to the path resolution on the remaining components and on the CID you popped. -#### `Symlink` type - -A :dfn[Symlink] represents a POSIX [symbolic link](https://pubs.opengroup.org/onlinepubs/9699919799/functions/symlink.html). -A symlink MUST NOT have children. - -The `PBNode.Data.Data` field is a POSIX path that MAY be inserted in front of the -currently remaining path component stack. - -##### Path Resolution - -There is no current consensus on how pathing over symlinks should behave. Some -implementations return symlink objects and fail if a consumer tries to follow them -through. - -Symlink path resolution SHOULD follow the POSIX specification, over the current UnixFS path context, as much as is applicable. - -#### `HAMTDirectory` +### `dag-pb` `HAMTDirectory` A :dfn[HAMT Directory] is a [Hashed-Array-Mapped-Trie](https://en.wikipedia.org/wiki/Hash_array_mapped_trie) data structure representing a :ref[Directory]. It is generally used to represent @@ -307,7 +276,7 @@ directories:, since they allow you to split large directories into multiple bloc The field `Name` of an element of `PBNode.Links` for a HAMT starts with an uppercase hex-encoded prefix, which is `log2(fanout)` bits wide. -##### Path Resolution +#### `dag-pb` `HAMTDirectory` Path Resolution To resolve the path inside a HAMT: @@ -323,7 +292,27 @@ To resolve the path inside a HAMT: name you were trying to resolve, you have successfully resolved a path component. Everything past the hex encoded prefix is the name of that element, which is useful when listing children of this directory. -### `TSize` (child DAG size hint) +### `dag-pb` `Symlink` + +A :dfn[Symlink] represents a POSIX [symbolic link](https://pubs.opengroup.org/onlinepubs/9699919799/functions/symlink.html). +A symlink MUST NOT have children. + +The `PBNode.Data.Data` field is a POSIX path that MAY be inserted in front of the +currently remaining path component stack. + +#### `dag-pb` `Symlink` Path Resolution + +Symlink path resolution SHOULD follow the POSIX specification, over the current UnixFS path context, as much as is applicable. + +:::warning + +There is no current consensus on how pathing over symlinks should behave. Some +implementations return symlink objects and fail if a consumer tries to follow them +through. + +::: + +### `dag-pb` `TSize` (child DAG size hint) `Tsize` is an optional field in `PBNode.Links[]` which represents the precomputed size of the specific child DAG. It provides a performance optimization: a hint about the total size of child DAG can be read without having to fetch any child nodes. @@ -347,20 +336,24 @@ Following the [Robustness Principle](https://specs.ipfs.tech/architecture/princi able to decode nodes where the `Tsize` field is wrong (not matching the sizes of sub-DAGs), or partially or completely missing. +::: + +:::warning + When total data size is needed for important purposes such as accounting, billing, and cost estimation, the `Tsize` SHOULD NOT be used, and instead a full DAG walk SHOULD to be performed. ::: -### Metadata +### `dag-pb` Optional Metadata -UnixFS currently supports two optional metadata fields. +UnixFS currently supports below optional metadata fields. -#### `mode` +#### `mode` Field The `mode` is for persisting the file permissions in [numeric notation](https://en.wikipedia.org/wiki/File_system_permissions#Numeric_notation) \[[spec](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html)\]. -- If unspecified, this defaults to +- If unspecified, implementations MAY default to - `0755` for directories/HAMT shards - `0644` for all other types where applicable - The nine least significant bits represent `ugo-rwx` @@ -369,7 +362,7 @@ The `mode` is for persisting the file permissions in [numeric notation](https:// - For future-proofing, the (de)serialization layer must preserve the entire uint32 value during clone/copy operations, modifying only bit values that have a well defined meaning: `clonedValue = ( modifiedBits & 07777 ) | ( originalValue & 0xFFFFF000 )` - Implementations of this spec must proactively mask off bits without a defined meaning in the implemented version of the spec: `interpretedValue = originalValue & 07777` -#### `mtime` +#### `mtime` Field A two-element structure ( `Seconds`, `FractionalNanoseconds` ) representing the modification time in seconds relative to the unix epoch `1970-01-01T00:00:00Z`. @@ -405,7 +398,7 @@ non-IPFS target MUST observe the following: vs 64bit mismatch), implementations must assume the highest possible value in the targets range. In most cases, this would be `2038-01-19T03:14:07Z`. -## Paths +## UnixFS Paths Paths begin with a `/` or `/ipfs//`, where `` is a [multibase] encoded [CID]. The CID encoding MUST NOT use a multibase alphabet that contains @@ -421,21 +414,31 @@ inspired by POSIX paths. - Components SHOULD be UTF8 unicode. - Components are case-sensitive. -### Escaping +### Path Escaping + +:::warning + +Behavior is not defined. -The `\` may be used to trigger an escape sequence. However, it is currently -broken and inconsistent across implementations. Until we agree on a specification -for this, you SHOULD NOT use any escape sequences and/or non-ASCII characters. +Until we agree on a specification for this, implementations SHOULD NOT depend on any escape +sequences and/or non-ASCII characters for mission-critical applications, or limit escaping to specific context. + +- HTTP interfaces such as Gateways have limited support for [percent-encoding](https://developer.mozilla.org/en-US/docs/Glossary/Percent-encoding). +- The `\` may be used to trigger an escape sequence. However, it is currently broken and inconsistent across implementations. + +::: ### Relative Path Components Relative path components MUST be resolved before trying to work on the path: -- `.` points to the current node and MUST be removed. +- `.` points to the current node and MUST be removed. - `..` points to the parent node and MUST be removed left to right. When removing a `..`, the path component on the left MUST also be removed. If there is no path component on the left, you MUST error to avoid out-of-bounds path resolution. +- Implementations MUST error when resolving a relative path that attempts to go + beyond the root CID (example: `/ipfs/cid/../foo`). ### Restricted Names @@ -448,16 +451,29 @@ The following names SHOULD NOT be used: terminations in some systems, such as C-compatible systems. Many unix file systems do not accept this character in path components. -## Appendix: Design Decision Rationale +# Appendix: Historical Design Decisions + +:::warning +Below section explains some of historical decisions. This is not part of specification, +and is provided here only for extra context. +::: -### `mtime` and `mode` Metadata Support in UnixFSv1.5 +## Design Considerations: Extra Metadata Metadata support in UnixFSv1.5 has been expanded to increase the number of possible use cases. These include `rsync` and filesystem-based package managers. Several metadata systems were evaluated, as discussed in the following sections. -#### Separate Metadata Node +:::note + +UnixFS 1.5 stores optional `mode` and `mtime` metadata in the `Data` fields of +the root `dag-pb` node, however below analysis may be useful when additional +metadata is being discussed, or UnixFS 1.5 approach is revisited. + +::: + +### Pros and Cons: Metadata in a Separate Metadata Node In this scheme, the existing `Metadata` message is expanded to include additional metadata types (`mtime`, `mode`, etc). It contains links to the actual file data, @@ -474,7 +490,7 @@ This was ultimately rejected for a number of reasons: UnixFSv2, as mapping between metadata formats potentially requires multiple fetch operations. -#### Metadata in the Directory +### Pros and Cons: Metadata in the Directory Repeated `Metadata` messages are added to UnixFS `Directory` and `HAMTShard` nodes, the index of which indicates which entry they are to be applied to. Where entries are @@ -498,7 +514,7 @@ This was rejected for the following reasons: benefit of including some metadata in the containing directory is negligible in this use case. -#### Metadata in the File +### Pros and Cons: Metadata in the File This adds new fields to the UnixFS `Data` message to represent the various metadata fields. @@ -521,7 +537,7 @@ Downsides to this approach are: into [CID]s, so additional fetches will be required to load a given UnixFS entry. -#### Side Trees +### Pros and Cons: Metadata in Side Trees With this approach, we would maintain a separate data structure outside of the UnixFS tree to hold metadata. @@ -530,7 +546,7 @@ This was rejected due to concerns about added complexity, recovery after system crashes while writing, and having to make extra requests to fetch metadata nodes when resolving [CID]s from peers. -#### Side Database +### Pros and Cons: Metadata in Side Database This scheme would see metadata stored in an external database. @@ -538,9 +554,9 @@ The downsides to this are that metadata would not be transferred from one node to another when syncing, as [Bitswap] is not aware of the database and in-tree metadata. -### UnixTime Protobuf Datatype Rationale +## Design Decision: UnixTime Protobuf Datatype -#### Seconds +### UnixTime Seconds The integer portion of UnixTime is represented on the wire using a `varint` encoding. While this is inefficient for negative values, it avoids introducing zig-zag encoding. @@ -548,7 +564,7 @@ Values before the year `1970` are exceedingly rare, and it would be handy having such cases stand out, while ensuring that the "usual" positive values are easily readable. The `varint` representing the time of writing this text is 5 bytes long. It will remain so until October 26, 3058 (34,359,738,367). -#### FractionalNanoseconds +### UnixTime FractionalNanoseconds Fractional values are effectively a random number in the range 1 to 999,999,999. In most cases, such values will exceed 2^28 (268,435,456) nanoseconds. Therefore, @@ -559,7 +575,7 @@ the fractional part is represented as a 4-byte `fixed32`, This section and included subsections are not authoritative. -## Implementations +## Popular Implementations - JavaScript - [`@helia/unixfs`](https://www.npmjs.com/package/@helia/unixfs) implementation of a filesystem compatible with [Helia SDK](https://github.com/ipfs/helia#readme) @@ -569,16 +585,19 @@ This section and included subsections are not authoritative. - Go - [Boxo SDK](https://github.com/ipfs/boxo#readme) includes implementation of UnixFS filesystem - Protocol Buffer Definitions - [`ipfs/boxo/../unixfs.proto`](https://github.com/ipfs/boxo/blob/v0.23.0/ipld/unixfs/pb/unixfs.proto) - - [`boxo/files`](https://github.com/ipfs/boxo/tree/main/files) + - [`ipfs/boxo/files`](https://github.com/ipfs/boxo/tree/main/files) - [`ipfs/boxo/ipld/unixfs`](https://github.com/ipfs/boxo/tree/main/ipld/unixfs/) - Alternative `go-ipld-prime` implementation: [`ipfs/go-unixfsnode`](https://github.com/ipfs/go-unixfsnode) + + -## Simple `Raw` Example +## Simple `raw` Example -In this example, we will build a `Raw` file with the string `test` as its content. +In this example, we will build a single `raw` block with the string `test` as its content. First, hash the data: @@ -629,5 +648,5 @@ This will tell you which offset inside this node the children at the correspondi [CID]: https://github.com/multiformats/cid/ [multicodec]: https://github.com/multiformats/multicodec [multihash]: https://github.com/multiformats/multihash -[Bitswap]: https://github.com/ipfs/specs/blob/master/BITSWAP.md +[Bitswap]: https://specs.ipfs.tech/bitswap-protocol/ [ipld-dag-pb]: https://ipld.io/specs/codecs/dag-pb/spec/ From 84762b385770bce6d3ecc47dd715b1d3c3e515a8 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Fri, 22 Aug 2025 00:59:08 +0200 Subject: [PATCH 15/46] feat(unixfs): add acknowledgments and thanks includes thanks to contributors who made significant impact on the initial reference implementations over the years --- Makefile | 2 +- src/unixfs-data-format.md | 29 +++++++++++++++++++++++------ 2 files changed, 24 insertions(+), 7 deletions(-) diff --git a/Makefile b/Makefile index 989e549bc..1394a429e 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -SPEC_GENERATOR_VER=1.5.0 +SPEC_GENERATOR_VER=1.6.0 .PHONY: install diff --git a/src/unixfs-data-format.md b/src/unixfs-data-format.md index 76c73bd26..c608f8380 100644 --- a/src/unixfs-data-format.md +++ b/src/unixfs-data-format.md @@ -6,10 +6,29 @@ description: > date: 2025-03-01 maturity: draft editors: - - name: David Dias - github: daviddias + - name: Marcin Rataj + github: lidel + affiliation: + name: Interplanetary Shipyard + url: https://ipshipyard.com/ +contributors: + - name: Hugo Valtier + github: jorropo + affiliation: + name: Interplanetary Shipyard + url: https://ipshipyard.com/ +thanks: - name: Jeromy Johnson github: whyrusleeping + - name: Steven Allen + github: Stebalien + - name: Hector Sanjuan + github: hsanjuan + affiliation: + name: Interplanetary Shipyard + url: https://ipshipyard.com/ + - name: Łukasz Magiera + github: magik6k - name: Alex Potsides github: achingbrain affiliation: @@ -17,10 +36,8 @@ editors: url: https://ipshipyard.com/ - name: Peter Rabbitson github: ribasushi - - name: Hugo Valtier - github: jorropo - - name: Marcin Rataj - github: lidel + - name: Henrique Dias + github: hacdias affiliation: name: Interplanetary Shipyard url: https://ipshipyard.com/ From 9099d6e619e26bc200a3eb3b1b1627e9bc7219f9 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sat, 23 Aug 2025 00:50:15 +0200 Subject: [PATCH 16/46] feat(unixfs): add test vectors appendix add test vectors with block size analysis for files, directories, and special cases to help implementers validate their code --- src/unixfs-data-format.md | 495 ++++++++++++++++++++++++++++++++------ 1 file changed, 415 insertions(+), 80 deletions(-) diff --git a/src/unixfs-data-format.md b/src/unixfs-data-format.md index c608f8380..950a9a504 100644 --- a/src/unixfs-data-format.md +++ b/src/unixfs-data-format.md @@ -69,7 +69,7 @@ In UnixFS, a node can be encoded using two different multicodecs, listed below. # `raw` Node -The simplest nodes use `raw` encoding and are implicitly a :ref[File]. They can +The simplest nodes use `raw` encoding and are implicitly a [File](#dag-pb-file). They can be recognized because their CIDs are encoded using the `raw` (`0x55`) codec: - The block is the file data. There is no protobuf envelope or metadata. @@ -79,7 +79,7 @@ be recognized because their CIDs are encoded using the `raw` (`0x55`) codec: # `dag-pb` Node More complex nodes use the `dag-pb` (`0x70`) encoding. These nodes require two steps of -decoding. The first step is to decode the outer container of the block. This is encoded using the IPLD [`dag-pb`][ipld-dag-pb] specification, which can be +decoding. The first step is to decode the outer container of the block. This is encoded using the [`dag-pb`][ipld-dag-pb] specification, which can be summarized as follows: ```protobuf @@ -138,7 +138,7 @@ message UnixTime { } ``` -Summarizing, a `dag-pb` UnixFS node is an IPLD [`dag-pb`][ipld-dag-pb] protobuf, +Summarizing, a `dag-pb` UnixFS node is a [`dag-pb`][ipld-dag-pb] protobuf, whose `Data` field is a UnixFSV1 Protobuf message. For clarity, the specification document may represent these nested Protobufs as one object. In this representation, it is implied that the `PBNode.Data` field is encoded in a protobuf. @@ -155,13 +155,13 @@ single block or multi-block. A multi-block file is a concatenation of multiple c #### The _sister-lists_ `PBNode.Links` and `decode(PBNode.Data).blocksizes` -The _sister-lists_ are the key point of why IPLD `dag-pb` is important for files. They +The _sister-lists_ are the key point of why `dag-pb` is important for files. They allow us to concatenate smaller files together. Linked files would be loaded recursively with the same process following a DFS (Depth-First-Search) order. -Child nodes must be of type File; either a `dag-pb`:ref[File], or a +Child nodes must be of type File; either a `dag-pb` [File](#dag-pb-file), or a [`raw` block](#raw-blocks). For example, consider this pseudo-json block: @@ -206,7 +206,7 @@ of the offset list when computing offsets. #### `PBNode.Links[].Name` -This field makes sense only in :ref[Directories] contexts and MUST be absent +This field makes sense only in [Directories](#dag-pb-directory) contexts and MUST be absent when creating a new file. For historical reasons, implementations parsing third-party data SHOULD accept empty values here. @@ -232,7 +232,7 @@ file MUST error. ### `dag-pb` `Directory` -A :dfn[Directory], also known as folder, is a named collection of child :ref[Nodes]: +A :dfn[Directory], also known as folder, is a named collection of child [Nodes](#dag-pb-node): - Every link in `PBNode.Links` is an entry (child) of the directory, and `PBNode.Links[].Name` gives you the name of that child. @@ -276,7 +276,7 @@ remaining components and on the CID you popped. ### `dag-pb` `HAMTDirectory` A :dfn[HAMT Directory] is a [Hashed-Array-Mapped-Trie](https://en.wikipedia.org/wiki/Hash_array_mapped_trie) -data structure representing a :ref[Directory]. It is generally used to represent +data structure representing a [Directory](#dag-pb-directory). It is generally used to represent directories that cannot fit inside a single block. These are also known as "sharded directories:, since they allow you to split large directories into multiple blocks, known as "shards". @@ -468,6 +468,413 @@ The following names SHOULD NOT be used: terminations in some systems, such as C-compatible systems. Many unix file systems do not accept this character in path components. +# Appendix: Test Vectors + +:::warning +**Implementations SHOULD validate against these test vectors and reference implementations before production use.** +::: + +This section provides test vectors organized by UnixFS structure type, progressing from simple to complex within each category. + +## File Test Vectors + +Test vectors for UnixFS file structures, progressing from simple single-block files to complex multi-block files. + +### Single `raw` Block File + +- Fixture: [`dir-with-files.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_unixfs/dir-with-files.car) + - CID: `bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4` (hello.txt) + - Type: [`raw` Node](#raw-node) + - Content: "hello world\n" (12 bytes) + - Block Analysis: + - Block size (`ipfs block stat`): 12 bytes + - Data size (`ipfs cat`): 12 bytes + - DAG-PB envelope: N/A (raw blocks have no envelope overhead) + - Purpose: Single block using `raw` codec, no protobuf wrapper + - Validation: Block content IS the file content, no UnixFS metadata + +### Single `dag-pb` Block File + +- Fixture: Well-known test CID from IPFS Gateway Checker +- CID: `bafybeifx7yeb55armcsxwwitkymga5xf53dxiarykms3ygqic223w5sk3m` +- Type: [`dag-pb` File](#dag-pb-file) with data in the same block +- Content: "Hello from IPFS Gateway Checker\n" (32 bytes) +- Block Analysis: + - Block size (`ipfs block stat`): 40 bytes + - Data size (`ipfs cat`): 32 bytes + - DAG-PB envelope: 8 bytes (40 - 32) +- Structure: + ``` + 📄 small-file.txt # bafybeifx7yeb55armcsxwwitkymga5xf53dxiarykms3ygqic223w5sk3m (dag-pb) + └── 📦 Data.Data # "Hello from IPFS Gateway Checker\n" (32 bytes, stored inline in UnixFS protobuf) + ``` +- Purpose: Small file stored within dag-pb Data field +- Validation: File content extracted from UnixFS Data.Data field + +### Multi-block File + +- Fixture: [`dir-with-files.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_unixfs/dir-with-files.car) + - CID: `bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa` (multiblock.txt) + - Type: [`dag-pb` File](#dag-pb-file) with multiple [`raw` Node](#raw-node) leaves + - Content: Lorem ipsum text (1026 bytes total) + - Block Analysis: + - Root block size (`ipfs block stat`): 245 bytes (dag-pb) + - Total data size (`ipfs cat`): 1026 bytes + - Child blocks: + - Block 1: 256 bytes (raw) + - Block 2: 256 bytes (raw) + - Block 3: 256 bytes (raw) + - Block 4: 256 bytes (raw) + - Block 5: 2 bytes (raw) + - DAG-PB envelope: 245 bytes (root block containing metadata + links) + - Structure: + ``` + 📄 multiblock.txt # bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa (dag-pb root) + ├── 📦 [0-255] # bafkreie5noke3mb7hqxukzcy73nl23k6lxszxi5w3dtmuwz62wnvkpsscm (raw, 256 bytes) + ├── 📦 [256-511] # bafkreih4ephajybraj6wnxsbwjwa77fukurtpl7oj7t7pfq545duhot7cq (raw, 256 bytes) + ├── 📦 [512-767] # bafkreigu7buvm3cfunb35766dn7tmqyh2um62zcio63en2btvxuybgcpue (raw, 256 bytes) + ├── 📦 [768-1023] # bafkreicll3huefkc3qnrzeony7zcfo7cr3nbx64hnxrqzsixpceg332fhe (raw, 256 bytes) + └── 📦 [1024-1025] # bafkreifst3pqztuvj57lycamoi7z34b4emf7gawxs74nwrc2c7jncmpaqm (raw, 2 bytes) + ``` + - Purpose: File chunking and reassembly + - Validation: + - Links have no Names (must be absent) + - Blocksizes array matches Links array length + - Reassembled content matches original + +### File with Missing Blocks + +- Fixture: [`bafybeibfhhww5bpsu34qs7nz25wp7ve36mcc5mxd5du26sr45bbnjhpkei.dag-pb`](https://github.com/ipld/codec-fixtures/raw/381e762b85862b2bbdb6ef2ba140b3c505e31a44/fixtures/dagpb_7unnamedlinks%2Bdata/bafybeibfhhww5bpsu34qs7nz25wp7ve36mcc5mxd5du26sr45bbnjhpkei.dag-pb) + - CID: `bafybeibfhhww5bpsu34qs7nz25wp7ve36mcc5mxd5du26sr45bbnjhpkei` + - Type: [`dag-pb` File](#dag-pb-file) with 7 links to child blocks + - Size: 306MB total (from metadata) + - Structure: + ``` + 📄 large-file # bafybeibfhhww5bpsu34qs7nz25wp7ve36mcc5mxd5du26sr45bbnjhpkei (dag-pb root) + ├── ⚠️ block[0] # (missing child block) + ├── ⚠️ block[1] # (missing child block) + ├── ⚠️ block[2] # (missing child block) + ├── ⚠️ block[3] # (missing child block) + ├── ⚠️ block[4] # (missing child block) + ├── ⚠️ block[5] # (missing child block) + └── ⚠️ block[6] # (missing child block) + ``` + - Note: Child blocks are NOT included - they may be unavailable locally or missing entirely + - Purpose: + - Reading UnixFS file metadata should require only the root block + - File size and structure can be determined without fetching children + - Operations should not block waiting for child blocks unless content is actually requested + - Validation: Can extract file size and chunking info from root block alone + +### Range Requests with Missing Blocks + +- Fixture: [`file-3k-and-3-blocks-missing-block.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/trustless_gateway_car/file-3k-and-3-blocks-missing-block.car) + - CID: `QmYhmPjhFjYFyaoiuNzYv8WGavpSRDwdHWe5B4M5du5Rtk` + - Type: [`dag-pb` File](#dag-pb-file) with 3 links but middle block intentionally missing + - Structure: + ``` + 📄 file-3k # QmYhmPjhFjYFyaoiuNzYv8WGavpSRDwdHWe5B4M5du5Rtk (dag-pb root) + ├── 📦 [0-1023] # QmPKt7ptM2ZYSGPUc8PmPT2VBkLDK3iqpG9TBJY7PCE9rF (raw, 1024 bytes) + ├── ⚠️ [1024-2047] # (missing block - intentionally removed) + └── 📦 [2048-3071] # QmWXY482zQdwecnfBsj78poUUuPXvyw2JAFAEMw4tzTavV (raw, 1024 bytes) + ``` + - Critical requirement: Must support seeking without all blocks available + - Purpose: + - Fetch only required blocks for byte range requests (e.g., bytes=0-1023 or bytes=2048-3071) + - Gateway conformance tests verify that first block (`QmPKt7ptM2ZYSGPUc8PmPT2VBkLDK3iqpG9TBJY7PCE9rF`) and third block (`QmWXY482zQdwecnfBsj78poUUuPXvyw2JAFAEMw4tzTavV`) can be fetched independently + - Requests for middle block or byte ranges requiring it should fail gracefully + +## Directory Test Vectors + +Test vectors for UnixFS directory structures, progressing from simple flat directories to complex HAMT-sharded directories. + +### Simple Directory + +- Fixture: [`dir-with-files.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_unixfs/dir-with-files.car) + - CID: `bafybeihchr7vmgjaasntayyatmp5sv6xza57iy2h4xj7g46bpjij6yhrmy` + - Type: [`dag-pb` Directory](#dag-pb-directory) + - Block Analysis: + - Directory block size (`ipfs block stat`): 185 bytes + - Contains UnixFS Type=Directory metadata + 4 links + - Structure: + ``` + 📁 / # bafybeihchr7vmgjaasntayyatmp5sv6xza57iy2h4xj7g46bpjij6yhrmy + ├── 📄 ascii-copy.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" + ├── 📄 ascii.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" + ├── 📄 hello.txt # bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4 (raw, 12 bytes) "hello world\n" + └── 📄 multiblock.txt # bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa (dag-pb, 1026 bytes) Lorem ipsum text + ``` + - Purpose: Directory listing, link sorting, deduplication (ascii.txt and ascii-copy.txt share same CID) + - Validation: Links sorted lexicographically by Name, each has valid Tsize + +### Nested Directories + +- Fixture: [`subdir-with-two-single-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/trustless_gateway_car/subdir-with-two-single-block-files.car) + - CID: `bafybeietjm63oynimmv5yyqay33nui4y4wx6u3peezwetxgiwvfmelutzu` + - Type: [`dag-pb` Directory](#dag-pb-directory) containing another Directory + - Block Analysis: + - Root directory block size: 55 bytes + - Subdirectory block size: 110 bytes + - Structure: + ``` + 📁 / # bafybeietjm63oynimmv5yyqay33nui4y4wx6u3peezwetxgiwvfmelutzu + └── 📁 subdir/ # bafybeiggghzz6dlue3m6nb2dttnbrygxh3lrjl5764f2m4gq7dgzdt55o4 (dag-pb Directory) + ├── 📄 ascii.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" + └── 📄 hello.txt # bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4 (raw, 12 bytes) "hello world\n" + ``` + - Purpose: Path traversal through directory hierarchy + - Validation: Can traverse `/subdir/hello.txt` path correctly + +- Fixture: [`dag-pb.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_dag/dag-pb.car) + - CID: `bafybeiegxwlgmoh2cny7qlolykdf7aq7g6dlommarldrbm7c4hbckhfcke` + - Type: [`dag-pb` Directory](#dag-pb-directory) + - Structure: + ``` + 📁 / # bafybeiegxwlgmoh2cny7qlolykdf7aq7g6dlommarldrbm7c4hbckhfcke + ├── 📁 foo/ # bafybeidryarwh34ygbtyypbu7qjkl4euiwxby6cql6uvosonohkq2kwnkm (dag-pb Directory) + │ └── 📄 bar.txt # bafkreigzafgemjeejks3vqyuo46ww2e22rt7utq5djikdofjtvnjl5zp6u (raw, 14 bytes) "Hello, world!" + └── 📄 foo.txt # bafkreic3ondyhizrzeoufvoodehinugpj3ecruwokaygl7elezhn2khqfa (raw, 13 bytes) "Hello, IPFS!" + ``` + - Purpose: Another example of standard UnixFS directory with raw leaf blocks + +### Special Characters in Filenames + +- Fixture: [`path_gateway_tar/fixtures.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_tar/fixtures.car) + - CID: `bafybeig6ka5mlwkl4subqhaiatalkcleo4jgnr3hqwvpmsqfca27cijp3i` + - Type: [`dag-pb` Directory](#dag-pb-directory) with nested subdirectories + - Structure: + ``` + 📁 / # bafybeig6ka5mlwkl4subqhaiatalkcleo4jgnr3hqwvpmsqfca27cijp3i + └── 📁 ą/ # (dag-pb Directory) + └── 📁 ę/ # (dag-pb Directory) + └── 📄 file-źł.txt # (raw, 34 bytes) "I am a txt file on path with utf8" + ``` + - Path with Polish diacritics: `/ipfs/bafybeig6ka5mlwkl4subqhaiatalkcleo4jgnr3hqwvpmsqfca27cijp3i/ą/ę/file-źł.txt` + - Purpose: UTF-8 characters in directory and file names (ą, ę, ź, ł) + - Validation: Directory traversal works with UTF-8 paths + +- Fixture: [`dir-with-percent-encoded-filename.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_unixfs/dir-with-percent-encoded-filename.car) + - CID: `bafybeig675grnxcmshiuzdaz2xalm6ef4thxxds6o6ypakpghm5kghpc34` + - Type: [`dag-pb` Directory](#dag-pb-directory) + - Structure: + ``` + 📁 / # bafybeig675grnxcmshiuzdaz2xalm6ef4thxxds6o6ypakpghm5kghpc34 + └── 📄 Portugal%2C+España=Peninsula Ibérica.txt # bafkreihfmctcb2kuvoljqeuphqr2fg2r45vz5cxgq5c2yrxnqg5erbitmq (raw, 38 bytes) "hello from a percent encoded filename" + ``` + - Purpose: Filenames with percent-encoding (`%2C`), plus signs, equals, and non-ASCII characters + - Validation: + - Implementations MUST preserve the original filename exactly as stored in UnixFS + - Must not be confused by filenames mixing Unicode characters with percent-encoding + - Gateway example: In gateway-conformance, accessing this file from a web browser requires double-encoding the `%2C` as `%252C` in the URL path (`/ipfs/{{CID}}/Portugal%252C+España=Peninsula%20Ibérica.txt`) + - Browser implementations should preserve `%2C` in the filename to avoid conflicts with URL encoding + +### Directory with Missing Blocks + +- Fixture: [`bafybeigcsevw74ssldzfwhiijzmg7a35lssfmjkuoj2t5qs5u5aztj47tq.dag-pb`](https://github.com/ipld/codec-fixtures/raw/381e762b85862b2bbdb6ef2ba140b3c505e31a44/fixtures/dagpb_4namedlinks%2Bdata/bafybeigcsevw74ssldzfwhiijzmg7a35lssfmjkuoj2t5qs5u5aztj47tq.dag-pb) + - CID: `bafybeigcsevw74ssldzfwhiijzmg7a35lssfmjkuoj2t5qs5u5aztj47tq` + - Type: [`dag-pb` Directory](#dag-pb-directory) + - Structure: + ``` + 📁 / # bafybeigcsevw74ssldzfwhiijzmg7a35lssfmjkuoj2t5qs5u5aztj47tq + ├── ⚠️ audio_only.m4a # (link to missing block, ~24MB) + ├── ⚠️ chat.txt # (link to missing block, ~1KB) + ├── ⚠️ playback.m3u # (link to missing block, ~116 bytes) + └── ⚠️ zoom_0.mp4 # (link to missing block) + ``` + - Note: Child blocks are NOT included - they may be unavailable locally or missing entirely + - Purpose: + - Directory enumeration should require only the root block + - Can list all filenames and their CIDs without fetching child blocks + - Operations should not block waiting for child blocks unless content is actually requested + - Validation: Can enumerate directory contents from root block alone + +### HAMT Sharded Directory + +- Fixture: [`single-layer-hamt-with-multi-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/trustless_gateway_car/single-layer-hamt-with-multi-block-files.car) + - CID: `bafybeidbclfqleg2uojchspzd4bob56dqetqjsj27gy2cq3klkkgxtpn4i` + - Type: [`dag-pb` HAMTDirectory](#dag-pb-hamtdirectory) + - Block Analysis: + - Root HAMT block size (`ipfs block stat`): 12046 bytes + - Contains UnixFS Type=HAMTShard metadata with fanout=256 + - Links use 2-character hex prefixes for hash buckets (00-FF) + - Structure: + ``` + 📂 / # bafybeidbclfqleg2uojchspzd4bob56dqetqjsj27gy2cq3klkkgxtpn4i (HAMT root) + ├── 📄 1.txt # (dag-pb file, multi-block) + ├── 📄 2.txt # (dag-pb file, multi-block) + ├── ... + └── 📄 1000.txt # (dag-pb file, multi-block) + ``` + - Contents: 1000 numbered files (1.txt through 1000.txt), each containing Lorem ipsum text + - Purpose: HAMT sharding for large directories + - Validation: + - Fanout field = 256 + - Link Names in HAMT have 2-character hex prefix (hash buckets) + - Can retrieve any file by name through hash bucket calculation + +## Special Cases and Advanced Features + +Test vectors for special UnixFS features and edge cases. + +### Symbolic Links + +- Fixture: [`symlink.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_unixfs/symlink.car) + - CID: `QmWvY6FaqFMS89YAQ9NAPjVP4WZKA1qbHbicc9HeSKQTgt` + - Types: [`dag-pb` Directory](#dag-pb-directory) containing [`dag-pb` Symlink](#dag-pb-symlink) + - Block Analysis: + - Root directory block: Not measured (V0 CID) + - Symlink block (`QmTB8BaCJdCH5H3k7GrxJsxgDNmNYGGR71C58ERkivXoj5`): 9 bytes + - Target file block (`Qme2y5HA5kvo2jAx13UsnV5bQJVijiAJCPvaW3JGQWhvJZ`): 16 bytes + - Structure: + ``` + 📁 / # QmWvY6FaqFMS89YAQ9NAPjVP4WZKA1qbHbicc9HeSKQTgt + ├── 📄 foo # Qme2y5HA5kvo2jAx13UsnV5bQJVijiAJCPvaW3JGQWhvJZ - file containing "content" + └── 🔗 bar # QmTB8BaCJdCH5H3k7GrxJsxgDNmNYGGR71C58ERkivXoj5 - symlink pointing to "foo" + ``` + - Purpose: UnixFS symlink resolution + - Security note: Critical for preventing path traversal vulnerabilities + +### Mixed Block Sizes + +- Fixture: [`subdir-with-mixed-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/trustless_gateway_car/subdir-with-mixed-block-files.car) + - CID: `bafybeidh6k2vzukelqtrjsmd4p52cpmltd2ufqrdtdg6yigi73in672fwu` + - Type: [`dag-pb` Directory](#dag-pb-directory) with subdirectory + - Structure: + ``` + 📁 / # bafybeidh6k2vzukelqtrjsmd4p52cpmltd2ufqrdtdg6yigi73in672fwu + └── 📁 subdir/ # bafybeicnmple4ehlz3ostv2sbojz3zhh5q7tz5r2qkfdpqfilgggeen7xm + ├── 📄 ascii.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" + ├── 📄 hello.txt # bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4 (raw, 12 bytes) "hello world\n" + └── 📄 multiblock.txt # bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa (dag-pb, 1271 bytes total) + ``` + - Purpose: Directories containing both single-block raw files and multi-block dag-pb files + - Validation: Can handle mixed file types in same directory + +### Deduplication + +- Fixture: [`dir-with-duplicate-files.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/trustless_gateway_car/dir-with-duplicate-files.car) + - CID: `bafybeihchr7vmgjaasntayyatmp5sv6xza57iy2h4xj7g46bpjij6yhrmy` + - Type: [`dag-pb` Directory](#dag-pb-directory) + - Structure: + ``` + 📁 / # bafybeihchr7vmgjaasntayyatmp5sv6xza57iy2h4xj7g46bpjij6yhrmy + ├── 🔗 ascii-copy.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (same CID as ascii.txt) + ├── 📄 ascii.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" + ├── 📄 hello.txt # bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4 (raw, 12 bytes) "hello world\n" + └── 📄 multiblock.txt # bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa (dag-pb, multi-block) + ``` + - Purpose: Multiple directory entries pointing to the same content CID (deduplication) + - Validation: Both ascii.txt and ascii-copy.txt resolve to the same content block + +### Invalid Test Cases + +These fixtures test raw dag-pb codec capabilities and serve as invalid test vectors for UnixFS implementations. Most lack UnixFS metadata - meaning their dag-pb Data field either doesn't exist, is empty, or contains bytes that aren't a valid UnixFS protobuf (which requires at minimum a `Type` field specifying File/Directory/Symlink etc). + +These validate that implementations properly reject malformed or non-UnixFS dag-pb nodes rather than crashing or behaving unpredictably: + +- 💢 [`bafybeihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku.dag-pb`](https://github.com/ipld/codec-fixtures/raw/381e762b85862b2bbdb6ef2ba140b3c505e31a44/fixtures/dagpb_empty/bafybeihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku.dag-pb) - Empty dag-pb node, 0 bytes (no UnixFS metadata) +- 💢 [`bafybeihyivpglm6o6wrafbe36fp5l67abmewk7i2eob5wacdbhz7as5obe.dag-pb`](https://github.com/ipld/codec-fixtures/raw/381e762b85862b2bbdb6ef2ba140b3c505e31a44/fixtures/dagpb_1link/bafybeihyivpglm6o6wrafbe36fp5l67abmewk7i2eob5wacdbhz7as5obe.dag-pb) - Single link without data, bytes: `12240a2212207521fe19c374a97759226dc5c0c8e674e73950e81b211f7dd3b6b30883a08a51` (no UnixFS metadata) +- 💢 [`bafybeibh647pmxyksmdm24uad6b5f7tx4dhvilzbg2fiqgzll4yek7g7y4.dag-pb`](https://github.com/ipld/codec-fixtures/raw/381e762b85862b2bbdb6ef2ba140b3c505e31a44/fixtures/dagpb_2link%2Bdata/bafybeibh647pmxyksmdm24uad6b5f7tx4dhvilzbg2fiqgzll4yek7g7y4.dag-pb) - Two links with data, bytes: `12340a2212208ab7a6c5e74737878ac73863cb76739d15d4666de44e5756bf55a2f9e9ab5f431209736f6d65206c696e6b1880c2d72f12370a2212208ab7a6c5e74737878ac73863cb76739d15d4666de44e5756bf55a2f9e9ab5f44120f736f6d65206f74686572206c696e6b18080a09736f6d652064617461` (invalid UnixFS protobuf) +- 💢 [`bafybeie7xh3zqqmeedkotykfsnj2pi4sacvvsjq6zddvcff4pq7dvyenhu.dag-pb`](https://github.com/ipld/codec-fixtures/raw/381e762b85862b2bbdb6ef2ba140b3c505e31a44/fixtures/dagpb_11unnamedlinks%2Bdata/bafybeie7xh3zqqmeedkotykfsnj2pi4sacvvsjq6zddvcff4pq7dvyenhu.dag-pb) - Eleven unnamed links with data (invalid UnixFS protobuf) +- 💢 [`bafybeibazl2z4vqp2tmwcfag6wirmtpnomxknqcgrauj7m2yisrz3qjbom.dag-pb`](https://github.com/ipld/codec-fixtures/raw/381e762b85862b2bbdb6ef2ba140b3c505e31a44/fixtures/dagpb_Data_some/bafybeibazl2z4vqp2tmwcfag6wirmtpnomxknqcgrauj7m2yisrz3qjbom.dag-pb) - Node with data field populated, bytes: `0a050001020304` (invalid UnixFS protobuf) +- 💢 [`bafybeiaqfni3s5s2k2r6rgpxz4hohdsskh44ka5tk6ztbjerqpvxwfkwaq.dag-pb`](https://github.com/ipld/codec-fixtures/raw/381e762b85862b2bbdb6ef2ba140b3c505e31a44/fixtures/dagpb_Data_zero/bafybeiaqfni3s5s2k2r6rgpxz4hohdsskh44ka5tk6ztbjerqpvxwfkwaq.dag-pb) - Node with empty data field, bytes: `0a00` (no UnixFS metadata) +- 💢 [`bafybeia53f5n75ituvc3yupuf7tdnxf6fqetrmo2alc6g6iljkmk7ys5mm.dag-pb`](https://github.com/ipld/codec-fixtures/raw/381e762b85862b2bbdb6ef2ba140b3c505e31a44/fixtures/dagpb_Links_Hash_some/bafybeia53f5n75ituvc3yupuf7tdnxf6fqetrmo2alc6g6iljkmk7ys5mm.dag-pb) - Links with hash only, bytes: `120b0a09015500050001020304` (no UnixFS metadata) +- 💢 [`bafybeifq4hcxma3kjljrpxtunnljtc6tvbkgsy3vldyfpfbx2lij76niyu.dag-pb`](https://github.com/ipld/codec-fixtures/raw/381e762b85862b2bbdb6ef2ba140b3c505e31a44/fixtures/dagpb_Links_Hash_some_Name_some/bafybeifq4hcxma3kjljrpxtunnljtc6tvbkgsy3vldyfpfbx2lij76niyu.dag-pb) - Links with hash and name, bytes: `12160a090155000500010203041209736f6d65206e616d65` (no UnixFS metadata) +- 💢 [`bafybeie7fstnkm4yshfwnmpp7d3mlh4f4okmk7a54d6c3ffr755q7qzk44.dag-pb`](https://github.com/ipld/codec-fixtures/raw/381e762b85862b2bbdb6ef2ba140b3c505e31a44/fixtures/dagpb_Links_Hash_some_Name_zero/bafybeie7fstnkm4yshfwnmpp7d3mlh4f4okmk7a54d6c3ffr755q7qzk44.dag-pb) - Links with hash but empty name, bytes: `120d0a090155000500010203041200` (no UnixFS metadata) +- 💢 [`bafybeiezymjvhwfuharanxmzxwuomzjjuzqjewjolr4phaiyp6l7qfwo64.dag-pb`](https://github.com/ipld/codec-fixtures/raw/381e762b85862b2bbdb6ef2ba140b3c505e31a44/fixtures/dagpb_Links_Hash_some_Tsize_some/bafybeiezymjvhwfuharanxmzxwuomzjjuzqjewjolr4phaiyp6l7qfwo64.dag-pb) - Links with hash and Tsize, bytes: `12140a0901550005000102030418ffffffffffffff0f` (no UnixFS metadata) +- 💢 [`bafybeichjs5otecmbvwh5azdr4jc45mp2qcofh2fr54wjdxhz4znahod2i.dag-pb`](https://github.com/ipld/codec-fixtures/raw/381e762b85862b2bbdb6ef2ba140b3c505e31a44/fixtures/dagpb_Links_Hash_some_Tsize_zero/bafybeichjs5otecmbvwh5azdr4jc45mp2qcofh2fr54wjdxhz4znahod2i.dag-pb) - Links with hash but zero Tsize, bytes: `120d0a090155000500010203041800` (no UnixFS metadata) +- 💢 [`bafybeia2qk4u55f2qj7zimmtpulejgz7urp7rzs44cvledcaj42gltkk3u.dag-pb`](https://github.com/ipld/codec-fixtures/raw/381e762b85862b2bbdb6ef2ba140b3c505e31a44/fixtures/dagpb_simple_forms_1/bafybeia2qk4u55f2qj7zimmtpulejgz7urp7rzs44cvledcaj42gltkk3u.dag-pb) - Simple form variant 1, bytes: `0a03010203` (invalid UnixFS protobuf) +- 💢 [`bafybeiahfgovhod2uvww72vwdgatl5r6qkoeegg7at2bghiokupfphqcku.dag-pb`](https://github.com/ipld/codec-fixtures/raw/381e762b85862b2bbdb6ef2ba140b3c505e31a44/fixtures/dagpb_simple_forms_2/bafybeiahfgovhod2uvww72vwdgatl5r6qkoeegg7at2bghiokupfphqcku.dag-pb) - Simple form variant 2, bytes: `120b0a0901550005000102030412100a09015500050001020304120362617212100a090155000500010203041203666f6f` (no UnixFS metadata) +- 💢 [`bafybeidrg2f6slbv4yzydqtgmsi2vzojajnt7iufcreynfpxndca4z5twm.dag-pb`](https://github.com/ipld/codec-fixtures/raw/381e762b85862b2bbdb6ef2ba140b3c505e31a44/fixtures/dagpb_simple_forms_3/bafybeidrg2f6slbv4yzydqtgmsi2vzojajnt7iufcreynfpxndca4z5twm.dag-pb) - Simple form variant 3, bytes: `120b0a09015500050001020304120e0a09015500050001020304120161120e0a09015500050001020304120161` (no UnixFS metadata) +- 💢 [`bafybeieube7zxmzoc5bgttub2aqofi6xdzimv5munkjseeqccn36a6v6j4.dag-pb`](https://github.com/ipld/codec-fixtures/raw/381e762b85862b2bbdb6ef2ba140b3c505e31a44/fixtures/dagpb_simple_forms_4/bafybeieube7zxmzoc5bgttub2aqofi6xdzimv5munkjseeqccn36a6v6j4.dag-pb) - Simple form variant 4, bytes: `120e0a09015500050001020304120161120e0a09015500050001020304120161` (no UnixFS metadata) + +## Additional Testing Resources + +- Gateway Conformance Suite: [ipfs/gateway-conformance](https://github.com/ipfs/gateway-conformance) + - Real-world test suite with UnixFS fixtures + - Tests gateway behaviors with various UnixFS structures + - Includes edge cases and performance scenarios + +- Test fixture generator: [go-fixtureplate](https://github.com/ipld/go-fixtureplate) + - Tool for generating custom test fixtures + - Includes UnixFS files and directories of arbitrary shapes + +Report specification issues or submit corrections via [ipfs/specs](https://github.com/ipfs/specs/issues). + +# Appendix: Notes for Implementers + +This section and included subsections are not authoritative. + +## Popular Implementations + +- JavaScript + - [`@helia/unixfs`](https://www.npmjs.com/package/@helia/unixfs) implementation of a filesystem compatible with [Helia SDK](https://github.com/ipfs/helia#readme) + - Data Formats - [unixfs](https://github.com/ipfs/js-ipfs-unixfs) + - Importer - [unixfs-importer](https://github.com/ipfs/js-ipfs-unixfs/tree/main/packages/ipfs-unixfs-importer) + - Exporter - [unixfs-exporter](https://github.com/ipfs/js-ipfs-unixfs/tree/main/packages/ipfs-unixfs-exporter) +- Go + - [Boxo SDK](https://github.com/ipfs/boxo#readme) includes implementation of UnixFS filesystem + - Protocol Buffer Definitions - [`ipfs/boxo/../unixfs.proto`](https://github.com/ipfs/boxo/blob/v0.23.0/ipld/unixfs/pb/unixfs.proto) + - [`ipfs/boxo/files`](https://github.com/ipfs/boxo/tree/main/files) + - [`ipfs/boxo/ipld/unixfs`](https://github.com/ipfs/boxo/tree/main/ipld/unixfs/) + - Alternative `go-ipld-prime` implementation: [`ipfs/go-unixfsnode`](https://github.com/ipfs/go-unixfsnode) + + + +## Simple `raw` Example + +In this example, we will build a single `raw` block with the string `test` as its content. + +First, hash the data: + +```console +$ echo -n "test" | sha256sum +9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 - +``` + +Add the CID prefix: + +``` +f01551220 + 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 + +f this is the multibase prefix, we need it because we are working with a hex CID, this is omitted for binary CIDs + 01 the CID version, here one + 55 the codec, here we MUST use Raw because this is a Raw file + 12 the hashing function used, here sha256 + 20 the digest length 32 bytes + 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 is the the digest we computed earlier +``` + +Done. Assuming we stored this block in some implementation of our choice, which makes it accessible to our client, we can try to decode it. + +```console +$ ipfs cat f015512209f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 +test +``` + +## Offset List + +The offset list isn't the only way to use blocksizes and reach a correct implementation, +it is a simple canonical one, python pseudo code to compute it looks like this: + +```python +def offsetlist(node): + unixfs = decodeDataField(node.Data) + if len(node.Links) != len(unixfs.Blocksizes): + raise "unmatched sister-lists" # error messages are implementation details + + cursor = len(unixfs.Data) if unixfs.Data else 0 + return [cursor] + [cursor := cursor + size for size in unixfs.Blocksizes[:-1]] +``` + +This will tell you which offset inside this node the children at the corresponding index starts to cover. (using `[x,y)` ranging) + + # Appendix: Historical Design Decisions :::warning @@ -588,78 +995,6 @@ In most cases, such values will exceed 2^28 (268,435,456) nanoseconds. Therefore the fractional part is represented as a 4-byte `fixed32`, [as per Google's recommendation](https://developers.google.com/protocol-buffers/docs/proto#scalar). -# Appendix: Notes for Implementers - -This section and included subsections are not authoritative. - -## Popular Implementations - -- JavaScript - - [`@helia/unixfs`](https://www.npmjs.com/package/@helia/unixfs) implementation of a filesystem compatible with [Helia SDK](https://github.com/ipfs/helia#readme) - - Data Formats - [unixfs](https://github.com/ipfs/js-ipfs-unixfs) - - Importer - [unixfs-importer](https://github.com/ipfs/js-ipfs-unixfs/tree/main/packages/ipfs-unixfs-importer) - - Exporter - [unixfs-exporter](https://github.com/ipfs/js-ipfs-unixfs/tree/main/packages/ipfs-unixfs-exporter) -- Go - - [Boxo SDK](https://github.com/ipfs/boxo#readme) includes implementation of UnixFS filesystem - - Protocol Buffer Definitions - [`ipfs/boxo/../unixfs.proto`](https://github.com/ipfs/boxo/blob/v0.23.0/ipld/unixfs/pb/unixfs.proto) - - [`ipfs/boxo/files`](https://github.com/ipfs/boxo/tree/main/files) - - [`ipfs/boxo/ipld/unixfs`](https://github.com/ipfs/boxo/tree/main/ipld/unixfs/) - - Alternative `go-ipld-prime` implementation: [`ipfs/go-unixfsnode`](https://github.com/ipfs/go-unixfsnode) - - - -## Simple `raw` Example - -In this example, we will build a single `raw` block with the string `test` as its content. - -First, hash the data: - -```console -$ echo -n "test" | sha256sum -9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 - -``` - -Add the CID prefix: - -``` -f01551220 - 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 - -f this is the multibase prefix, we need it because we are working with a hex CID, this is omitted for binary CIDs - 01 the CID version, here one - 55 the codec, here we MUST use Raw because this is a Raw file - 12 the hashing function used, here sha256 - 20 the digest length 32 bytes - 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 is the the digest we computed earlier -``` - -Done. Assuming we stored this block in some implementation of our choice, which makes it accessible to our client, we can try to decode it. - -```console -$ ipfs cat f015512209f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 -test -``` - -## Offset List - -The offset list isn't the only way to use blocksizes and reach a correct implementation, -it is a simple canonical one, python pseudo code to compute it looks like this: - -```python -def offsetlist(node): - unixfs = decodeDataField(node.Data) - if len(node.Links) != len(unixfs.Blocksizes): - raise "unmatched sister-lists" # error messages are implementation details - - cursor = len(unixfs.Data) if unixfs.Data else 0 - return [cursor] + [cursor := cursor + size for size in unixfs.Blocksizes[:-1]] -``` - -This will tell you which offset inside this node the children at the corresponding index starts to cover. (using `[x,y)` ranging) [protobuf]: https://developers.google.com/protocol-buffers/ [CID]: https://github.com/multiformats/cid/ From 6fee57914f3c6dbbbdcfbb9720f30110b36b42b3 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sat, 23 Aug 2025 02:17:49 +0200 Subject: [PATCH 17/46] feat(unixfs): clarify HAMT specification - add concrete example using test vector with 1000 files - document structure parameters (type, hash, fanout, bitmap) - clarify path resolution with step-by-step example - specify uppercase hex prefixes and little endian bit order - note sharding thresholds and implementation flexibility --- src/unixfs-data-format.md | 134 +++++++++++++++++++++++++++++++------- 1 file changed, 109 insertions(+), 25 deletions(-) diff --git a/src/unixfs-data-format.md b/src/unixfs-data-format.md index 950a9a504..da4e4b527 100644 --- a/src/unixfs-data-format.md +++ b/src/unixfs-data-format.md @@ -278,36 +278,120 @@ remaining components and on the CID you popped. A :dfn[HAMT Directory] is a [Hashed-Array-Mapped-Trie](https://en.wikipedia.org/wiki/Hash_array_mapped_trie) data structure representing a [Directory](#dag-pb-directory). It is generally used to represent directories that cannot fit inside a single block. These are also known as "sharded -directories:, since they allow you to split large directories into multiple blocks, known as "shards". +directories", since they allow you to split large directories into multiple blocks, known as "shards". +#### HAMT Structure and Parameters + +The HAMT directory is configured through the UnixFS metadata in `PBNode.Data`: + +- `decode(PBNode.Data).Type` MUST be `HAMTShard` (value `5`) - `decode(PBNode.Data).hashType` indicates the [multihash] function to use to digest - the path components used for sharding. It MUST be `murmur3-x64-64` (`0x22`). -- `decode(PBNode.Data).Data.Data` is a bit field, which indicates whether or not - links are part of this HAMT, or its leaves. The usage of this field is unknown, given - that you can deduce the same information from the link names. -- `decode(PBNode.Data).Data.fanout` MUST be a power of two. This encodes the number - of hash permutations that will be used on each resolution step. The log base 2 - of the `fanout` indicate how wide the bitmask will be on the hash at for that step. - `fanout` MUST be between 8 and probably 65536. . - -The field `Name` of an element of `PBNode.Links` for a HAMT starts with an -uppercase hex-encoded prefix, which is `log2(fanout)` bits wide. + the path components for sharding. Currently, all HAMT implementations use `murmur3-x64-64` (`0x22`), + and this value MUST be consistent across all shards within the same HAMT structure +- `decode(PBNode.Data).fanout` MUST be a power of two. This determines the number + of possible bucket indices (permutations) at each level of the trie. For example, + fanout=256 provides 256 possible buckets (0x00 to 0xFF), requiring 8 bits from the hash. + The hex prefix length is `log2(fanout)/4` characters (since each hex character represents 4 bits). + The same fanout value is used throughout all levels of a single HAMT structure. + Implementations choose fanout based on their specific trade-offs between tree depth and node size +- `decode(PBNode.Data).Data` is a bitmap field indicating which buckets contain entries. + Each bit represents one bucket. While included in the protobuf, implementations + typically derive bucket occupancy from the link names directly + +The field `Name` of an element of `PBNode.Links` for a HAMT uses a +hex-encoded prefix corresponding to the bucket index, zero-padded to a width +of `log2(fanout)/4` characters. + +Implementations choose when to convert a regular directory to HAMT based on various criteria +such as estimated block size (commonly around 256KiB-1MiB to produce blocks that can be +transported over Bitswap) or number of directory entries. + +To illustrate the HAMT structure with a concrete example: + +```protobuf +// Root HAMT shard (bafybeidbclfqleg2uojchspzd4bob56dqetqjsj27gy2cq3klkkgxtpn4i) +// This shard contains 1000 files distributed across buckets +message PBNode { + // UnixFS metadata in Data field + Data = { + Type = HAMTShard // Type = 5 + Data = 0xffffff... // Bitmap: bits set for populated buckets + hashType = 0x22 // murmur3-x64-64 + fanout = 256 // 256 buckets (8-bit width) + } + + // Links to sub-shards or entries + Links = [ + { + Hash = bafybeiaebmuestgbpqhkkbrwl2qtjtvs3whkmp2trkbkimuod4yv7oygni + Name = "00" // Bucket 0x00 + Tsize = 2693 // Cumulative size of this subtree + }, + { + Hash = bafybeia322onepwqofne3l3ptwltzns52fgapeauhmyynvoojmcvchxptu + Name = "01" // Bucket 0x01 + Tsize = 7977 + }, + // ... more buckets as needed up to "FF" + ] +} + +// Sub-shard for bucket "00" (multiple files hash to 00 at first level) +message PBNode { + Data = { + Type = HAMTShard // Still a HAMT at second level + Data = 0x800000... // Bitmap for this sub-level + hashType = 0x22 // murmur3-x64-64 + fanout = 256 // Same fanout throughout + } + + Links = [ + { + Hash = bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa + Name = "6E470.txt" // Bucket 0x6E + filename + Tsize = 1271 + }, + { + Hash = bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa + Name = "FF742.txt" // Bucket 0xFF + filename + Tsize = 1271 + } + ] +} +``` #### `dag-pb` `HAMTDirectory` Path Resolution -To resolve the path inside a HAMT: - -1. Take the current path component, then hash it using the [multihash] represented - by the value of `decode(PBNode.Data).hashType`. -2. Pop the `log2(fanout)` lowest bits from the path component hash digest, then - hex encode (using 0-F) those bits using little endian. Find the link that starts - with this hex encoded path. -3. If the link `Name` is exactly as long as the hex encoded representation, follow - the link and repeat step 2 with the child node and the remaining bit stack. - The child node MUST be a HAMT directory, or else the directory is invalid. Otherwise, continue. -4. Compare the remaining part of the last name you found. If it matches the original - name you were trying to resolve, you have successfully resolved a path component. - Everything past the hex encoded prefix is the name of that element, which is useful when listing children of this directory. +To resolve a path inside a HAMT: + +1. Hash the filename using the hash function specified in `decode(PBNode.Data).hashType` +2. Pop `log2(fanout)` bits from the hash digest (lowest/least significant bits first), + then hex encode those bits using little endian to form the bucket prefix. The prefix MUST use uppercase hex characters (00-FF, not 00-ff) +3. Find the link whose `Name` starts with this hex prefix: + - If `Name` equals the prefix exactly → this is a sub-shard, follow the link and repeat from step 2 + - If `Name` equals prefix + filename → target found + - If no matching prefix → file not in directory +4. When following to a sub-shard, continue consuming bits from the same hash + +Note: Empty intermediate shards are typically collapsed during deletion operations to maintain consistency +and avoid having HAMT structures that differ based on insertion/deletion history. + +:::note +**Example: Finding "470.txt" in a HAMT with fanout=256** (see [HAMT Sharded Directory test vector](#hamt-sharded-directory)) + +Given a HAMT-sharded directory containing 1000 files: + +1. Hash the filename "470.txt" using murmur3-x64-64 (multihash `0x22`) +2. With fanout=256, we consume 8 bits at a time from the hash: + - First 8 bits determine root bucket → `0x00` → link name "00" + - Follow link "00" to sub-shard (`bafybeiaebmuestgbpqhkkbrwl2qtjtvs3whkmp2trkbkimuod4yv7oygni`) +3. The sub-shard is also a HAMT (has Type=HAMTShard): + - Next 8 bits from hash → `0x6E` + - Find entry with name "6E470.txt" (prefix + original filename) +4. Link name format at leaf level: `[hex_prefix][original_filename]` + - "6E470.txt" means: file "470.txt" that hashed to bucket 6E at this level + - "FF742.txt" means: file "742.txt" that hashed to bucket FF at this level +::: ### `dag-pb` `Symlink` From 91729bf068abd5384cef408e19b34e280d75bfec Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sat, 23 Aug 2025 02:38:11 +0200 Subject: [PATCH 18/46] docs(unixfs): clarify raw codec vs Raw DataType - add deprecation comment to Raw enum value - add warning distinguishing raw codec from Raw DataType - specify MUST NOT produce, MAY read Raw DataType --- src/unixfs-data-format.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/unixfs-data-format.md b/src/unixfs-data-format.md index da4e4b527..eb1599fa8 100644 --- a/src/unixfs-data-format.md +++ b/src/unixfs-data-format.md @@ -76,6 +76,12 @@ be recognized because their CIDs are encoded using the `raw` (`0x55`) codec: - They never have any children nodes, and thus are also known as single block files. - Their size is the length of the block body (`Tsize` in parent is equal to `blocksize`). +:::warning +**Important**: Do not confuse `raw` codec blocks (`0x55`) with the deprecated `Raw` DataType (enum value `0`): +- **`raw` codec** - Modern way to store data without protobuf wrapper (used for small files and leaves) +- **`Raw` DataType** - Legacy UnixFS type that wrapped raw data in dag-pb protobuf (implementations MUST NOT produce, MAY read for compatibility) +::: + # `dag-pb` Node More complex nodes use the `dag-pb` (`0x70`) encoding. These nodes require two steps of @@ -110,7 +116,7 @@ a protobuf message specified in the UnixFSV1 format: ```protobuf message Data { enum DataType { - Raw = 0; + Raw = 0; // deprecated, use raw codec blocks without dag-pb instead Directory = 1; File = 2; Metadata = 3; // reserved for future use From 04df2676d828f0d8f7832957250b7308cb0f8d98 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sat, 23 Aug 2025 03:36:53 +0200 Subject: [PATCH 19/46] fix(unixfs): clarify blocksize vs Tsize distinction - blocksize: raw file data only (no protobuf overhead) - tsize: cumulative DAG size including all serialized blocks - add concrete examples using Simple Directory fixture - clarify PBNode.Links[].Name handling for file chunks - note decoder should preserve original file order --- src/unixfs-data-format.md | 71 +++++++++++++++++++++++++++++---------- 1 file changed, 53 insertions(+), 18 deletions(-) diff --git a/src/unixfs-data-format.md b/src/unixfs-data-format.md index eb1599fa8..89daefc18 100644 --- a/src/unixfs-data-format.md +++ b/src/unixfs-data-format.md @@ -212,19 +212,31 @@ of the offset list when computing offsets. #### `PBNode.Links[].Name` -This field makes sense only in [Directories](#dag-pb-directory) contexts and MUST be absent -when creating a new file. For historical reasons, implementations parsing -third-party data SHOULD accept empty values here. +The `Name` field is primarily used in directories to identify child entries. -If this field is present and non-empty, the file is invalid and the parser MUST -error. +**For internal file chunks:** +- Implementations SHOULD NOT produce `Name` fields (the field should be absent in the protobuf, not an empty string) +- For compatibility with historical data, implementations SHOULD treat empty string values ("") the same as absent when parsing +- If a non-empty `Name` is present in an internal file chunk, the parser MUST error as this indicates an invalid file structure #### `decode(PBNode.Data).Blocksize` This field is not directly present in the block, but rather a computable property of a `dag-pb`, which would be used in the parent node in `decode(PBNode.Data).blocksizes`. -It is the sum of the length of `decode(PBNode.Data).Data` field plus the sum -of all link's `blocksizes`. + +**Important:** `blocksize` represents only the raw file data size, NOT including the protobuf envelope overhead. + +It is calculated as: +- For `dag-pb` blocks: the length of `decode(PBNode.Data).Data` field plus the sum of all child `blocksizes` +- For `raw` blocks (small files, raw leaves): the length of the entire raw block + +:::note + +Examples of where `blocksize` is useful: + +- Seeking and range requests (e.g., HTTP Range headers for video streaming). The `blocksizes` array allows calculating byte offsets (see [Offset List](#offset-list)) to determine which blocks contain the requested range without downloading unnecessary blocks. + +::: #### `decode(PBNode.Data).filesize` @@ -258,16 +270,16 @@ The minimum valid `PBNode.Data` field for a directory is as follows: #### `dag-pb` `Directory` Link Ordering -The canonical sorting order is lexicographical over the names. +Directory links SHOULD be sorted lexicographically by the `Name` field when creating +new directories. This ensures consistent, deterministic directory structures across +implementations. -In theory, there is no reason an encoder couldn't use an other ordering. However, -this loses some of its meaning when mapped into most file systems today, as most file -systems consider directories to be unordered key-value objects. +While decoders MUST accept directories with any link ordering, encoders SHOULD use +lexicographic sorting for better interoperability and deterministic CIDs. A decoder +SHOULD, if it can, preserve the order of the original files. -A decoder SHOULD, if it can, preserve the order of the original files in the same way -it consumed those names. However, when some implementations decode, modify and then -re-encode, the original link order loses it's original meaning, given that there -is no way to indicate which sorting was used originally. +Note: The sorting requirement helps with deduplication detection and enables more +efficient directory traversal algorithms in some implementations. #### `dag-pb` `Directory` Path Resolution @@ -419,11 +431,34 @@ through. ::: -### `dag-pb` `TSize` (child DAG size hint) +### `dag-pb` `TSize` (cumulative DAG size) + +`Tsize` is an optional field in `PBNode.Links[]` which represents the cumulative size of the entire DAG rooted at that link, including all protobuf encoding overhead. -`Tsize` is an optional field in `PBNode.Links[]` which represents the precomputed size of the specific child DAG. It provides a performance optimization: a hint about the total size of child DAG can be read without having to fetch any child nodes. +**Key distinction from blocksize:** +- **`blocksize`**: Only the raw file data (no protobuf overhead) +- **`Tsize`**: Total size of all serialized blocks in the DAG (includes protobuf overhead) -To compute the `Tsize` of a child DAG, sum the length of the `dag-pb` outside message binary length and the `blocksizes` of all nodes in the child DAG. +To compute `Tsize`: sum the serialized size of the current dag-pb block and the Tsize values of all child links. + +:::note + +**Example: Directory with multi-block file** + +Consider the [Simple Directory fixture](#simple-directory) (`bafybeihchr7vmgjaasntayyatmp5sv6xza57iy2h4xj7g46bpjij6yhrmy`): + +The directory has a total `Tsize` of 1572 bytes: +- Directory block itself: 227 bytes when serialized +- Child entries with Tsizes: 31 + 31 + 12 + 1271 = 1345 bytes + +The `multiblock.txt` file within this directory demonstrates how `Tsize` accumulates: +- Raw file content: 1026 bytes (blocksizes: [256, 256, 256, 256, 2]) +- Root dag-pb block: 245 bytes when serialized +- Total `Tsize`: 245 + 1026 = 1271 bytes + +This shows how `Tsize` includes both the protobuf overhead and all child data, while `blocksize` only counts the raw file data. + +::: :::note From 523cb6ae181b8813040a499f8025d518ba963ea6 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sat, 23 Aug 2025 03:53:18 +0200 Subject: [PATCH 20/46] fix(unixfs): clarify dag stability and sorting behavior - explain 'sort on write, not on read' principle for dag stability - clarify that sorting happens when links list is modified --- src/unixfs-data-format.md | 854 +++++++++++++++++++------------------- 1 file changed, 429 insertions(+), 425 deletions(-) diff --git a/src/unixfs-data-format.md b/src/unixfs-data-format.md index 89daefc18..c870f4bbb 100644 --- a/src/unixfs-data-format.md +++ b/src/unixfs-data-format.md @@ -1,46 +1,46 @@ --- title: UnixFS description: > - UnixFS is a Protocol Buffers-based format for describing files, directories, - and symlinks as dag-pb and raw DAGs in IPFS. +UnixFS is a Protocol Buffers-based format for describing files, directories, +and symlinks as dag-pb and raw DAGs in IPFS. date: 2025-03-01 maturity: draft editors: - - name: Marcin Rataj - github: lidel - affiliation: - name: Interplanetary Shipyard - url: https://ipshipyard.com/ +- name: Marcin Rataj +github: lidel +affiliation: +name: Interplanetary Shipyard +url: https://ipshipyard.com/ contributors: - - name: Hugo Valtier - github: jorropo - affiliation: - name: Interplanetary Shipyard - url: https://ipshipyard.com/ +- name: Hugo Valtier +github: jorropo +affiliation: +name: Interplanetary Shipyard +url: https://ipshipyard.com/ thanks: - - name: Jeromy Johnson - github: whyrusleeping - - name: Steven Allen - github: Stebalien - - name: Hector Sanjuan - github: hsanjuan - affiliation: - name: Interplanetary Shipyard - url: https://ipshipyard.com/ - - name: Łukasz Magiera - github: magik6k - - name: Alex Potsides - github: achingbrain - affiliation: - name: Interplanetary Shipyard - url: https://ipshipyard.com/ - - name: Peter Rabbitson - github: ribasushi - - name: Henrique Dias - github: hacdias - affiliation: - name: Interplanetary Shipyard - url: https://ipshipyard.com/ +- name: Jeromy Johnson +github: whyrusleeping +- name: Steven Allen +github: Stebalien +- name: Hector Sanjuan +github: hsanjuan +affiliation: +name: Interplanetary Shipyard +url: https://ipshipyard.com/ +- name: Łukasz Magiera +github: magik6k +- name: Alex Potsides +github: achingbrain +affiliation: +name: Interplanetary Shipyard +url: https://ipshipyard.com/ +- name: Peter Rabbitson +github: ribasushi +- name: Henrique Dias +github: hacdias +affiliation: +name: Interplanetary Shipyard +url: https://ipshipyard.com/ tags: ['data-formats'] order: 1 @@ -57,7 +57,7 @@ required. A [CID] includes two important pieces of information: 1. A [multicodec], simply known as a codec. 2. A [multihash] used to specify the hashing algorithm, the hash parameters and - the hash digest. +the hash digest. Thus, the block must be retrieved; that is, the bytes which ,when hashed using the hash function specified in the multihash, gives us the same multihash value back. @@ -90,22 +90,22 @@ summarized as follows: ```protobuf message PBLink { - // binary CID (with no multibase prefix) of the target object - optional bytes Hash = 1; +// binary CID (with no multibase prefix) of the target object +optional bytes Hash = 1; - // UTF-8 string name - optional string Name = 2; +// UTF-8 string name +optional string Name = 2; - // cumulative size of target object - optional uint64 Tsize = 3; +// cumulative size of target object +optional uint64 Tsize = 3; } message PBNode { - // refs to other objects - repeated PBLink Links = 2; +// refs to other objects +repeated PBLink Links = 2; - // opaque user data - optional bytes Data = 1; +// opaque user data +optional bytes Data = 1; } ``` @@ -115,32 +115,32 @@ a protobuf message specified in the UnixFSV1 format: ```protobuf message Data { - enum DataType { - Raw = 0; // deprecated, use raw codec blocks without dag-pb instead - Directory = 1; - File = 2; - Metadata = 3; // reserved for future use - Symlink = 4; - HAMTShard = 5; - } - - required DataType Type = 1; - optional bytes Data = 2; - optional uint64 filesize = 3; - repeated uint64 blocksizes = 4; - optional uint64 hashType = 5; - optional uint64 fanout = 6; - optional uint32 mode = 7; // opt-in, AKA UnixFS 1.5 - optional UnixTime mtime = 8; // opt-in, AKA UnixFS 1.5 +enum DataType { +Raw = 0; // deprecated, use raw codec blocks without dag-pb instead +Directory = 1; +File = 2; +Metadata = 3; // reserved for future use +Symlink = 4; +HAMTShard = 5; +} + +required DataType Type = 1; +optional bytes Data = 2; +optional uint64 filesize = 3; +repeated uint64 blocksizes = 4; +optional uint64 hashType = 5; +optional uint64 fanout = 6; +optional uint32 mode = 7; // opt-in, AKA UnixFS 1.5 +optional UnixTime mtime = 8; // opt-in, AKA UnixFS 1.5 } message Metadata { - optional string MimeType = 1; +optional string MimeType = 1; } message UnixTime { - required int64 Seconds = 1; - optional fixed32 FractionalNanoseconds = 2; +required int64 Seconds = 1; +optional fixed32 FractionalNanoseconds = 2; } ``` @@ -174,11 +174,11 @@ For example, consider this pseudo-json block: ```json { - "Links": [{"Hash":"Qmfoo"}, {"Hash":"Qmbar"}], - "Data": { - "Type": "File", - "blocksizes": [20, 30] - } +"Links": [{"Hash":"Qmfoo"}, {"Hash":"Qmbar"}], +"Data": { +"Type": "File", +"blocksizes": [20, 30] +} } ``` @@ -253,18 +253,18 @@ file MUST error. A :dfn[Directory], also known as folder, is a named collection of child [Nodes](#dag-pb-node): - Every link in `PBNode.Links` is an entry (child) of the directory, and - `PBNode.Links[].Name` gives you the name of that child. +`PBNode.Links[].Name` gives you the name of that child. - Duplicate names are not allowed. Therefore, two elements of `PBNode.Link` CANNOT - have the same `Name`. If two identical names are present in a directory, the - decoder MUST fail. +have the same `Name`. If two identical names are present in a directory, the +decoder MUST fail. - Implementations SHOULD detect when directory becomes too big to fit in a single - `Directory` block and use [`HAMTDirectory`] type instead. +`Directory` block and use [`HAMTDirectory`] type instead. The minimum valid `PBNode.Data` field for a directory is as follows: ```json { - "Type": "Directory" +"Type": "Directory" } ``` @@ -275,10 +275,14 @@ new directories. This ensures consistent, deterministic directory structures acr implementations. While decoders MUST accept directories with any link ordering, encoders SHOULD use -lexicographic sorting for better interoperability and deterministic CIDs. A decoder -SHOULD, if it can, preserve the order of the original files. +lexicographic sorting for better interoperability and deterministic CIDs. + +A decoder SHOULD, if it can, preserve the order of the original files. This "sort on write, +not on read" approach maintains DAG stability - existing unsorted directories remain unchanged +when accessed or traversed, preventing unintentional mutations of intermediate nodes that could +alter their CIDs. -Note: The sorting requirement helps with deduplication detection and enables more +Note: Sorting on write (when the Links list is modified) helps with deduplication detection and enables more efficient directory traversal algorithms in some implementations. #### `dag-pb` `Directory` Path Resolution @@ -304,17 +308,17 @@ The HAMT directory is configured through the UnixFS metadata in `PBNode.Data`: - `decode(PBNode.Data).Type` MUST be `HAMTShard` (value `5`) - `decode(PBNode.Data).hashType` indicates the [multihash] function to use to digest - the path components for sharding. Currently, all HAMT implementations use `murmur3-x64-64` (`0x22`), - and this value MUST be consistent across all shards within the same HAMT structure +the path components for sharding. Currently, all HAMT implementations use `murmur3-x64-64` (`0x22`), +and this value MUST be consistent across all shards within the same HAMT structure - `decode(PBNode.Data).fanout` MUST be a power of two. This determines the number - of possible bucket indices (permutations) at each level of the trie. For example, - fanout=256 provides 256 possible buckets (0x00 to 0xFF), requiring 8 bits from the hash. - The hex prefix length is `log2(fanout)/4` characters (since each hex character represents 4 bits). - The same fanout value is used throughout all levels of a single HAMT structure. - Implementations choose fanout based on their specific trade-offs between tree depth and node size +of possible bucket indices (permutations) at each level of the trie. For example, +fanout=256 provides 256 possible buckets (0x00 to 0xFF), requiring 8 bits from the hash. +The hex prefix length is `log2(fanout)/4` characters (since each hex character represents 4 bits). +The same fanout value is used throughout all levels of a single HAMT structure. +Implementations choose fanout based on their specific trade-offs between tree depth and node size - `decode(PBNode.Data).Data` is a bitmap field indicating which buckets contain entries. - Each bit represents one bucket. While included in the protobuf, implementations - typically derive bucket occupancy from the link names directly +Each bit represents one bucket. While included in the protobuf, implementations +typically derive bucket occupancy from the link names directly The field `Name` of an element of `PBNode.Links` for a HAMT uses a hex-encoded prefix corresponding to the bucket index, zero-padded to a width @@ -330,51 +334,51 @@ To illustrate the HAMT structure with a concrete example: // Root HAMT shard (bafybeidbclfqleg2uojchspzd4bob56dqetqjsj27gy2cq3klkkgxtpn4i) // This shard contains 1000 files distributed across buckets message PBNode { - // UnixFS metadata in Data field - Data = { - Type = HAMTShard // Type = 5 - Data = 0xffffff... // Bitmap: bits set for populated buckets - hashType = 0x22 // murmur3-x64-64 - fanout = 256 // 256 buckets (8-bit width) - } - - // Links to sub-shards or entries - Links = [ - { - Hash = bafybeiaebmuestgbpqhkkbrwl2qtjtvs3whkmp2trkbkimuod4yv7oygni - Name = "00" // Bucket 0x00 - Tsize = 2693 // Cumulative size of this subtree - }, - { - Hash = bafybeia322onepwqofne3l3ptwltzns52fgapeauhmyynvoojmcvchxptu - Name = "01" // Bucket 0x01 - Tsize = 7977 - }, - // ... more buckets as needed up to "FF" - ] +// UnixFS metadata in Data field +Data = { +Type = HAMTShard // Type = 5 +Data = 0xffffff... // Bitmap: bits set for populated buckets +hashType = 0x22 // murmur3-x64-64 +fanout = 256 // 256 buckets (8-bit width) +} + +// Links to sub-shards or entries +Links = [ +{ +Hash = bafybeiaebmuestgbpqhkkbrwl2qtjtvs3whkmp2trkbkimuod4yv7oygni +Name = "00" // Bucket 0x00 +Tsize = 2693 // Cumulative size of this subtree +}, +{ +Hash = bafybeia322onepwqofne3l3ptwltzns52fgapeauhmyynvoojmcvchxptu +Name = "01" // Bucket 0x01 +Tsize = 7977 +}, +// ... more buckets as needed up to "FF" +] } // Sub-shard for bucket "00" (multiple files hash to 00 at first level) message PBNode { - Data = { - Type = HAMTShard // Still a HAMT at second level - Data = 0x800000... // Bitmap for this sub-level - hashType = 0x22 // murmur3-x64-64 - fanout = 256 // Same fanout throughout - } - - Links = [ - { - Hash = bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa - Name = "6E470.txt" // Bucket 0x6E + filename - Tsize = 1271 - }, - { - Hash = bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa - Name = "FF742.txt" // Bucket 0xFF + filename - Tsize = 1271 - } - ] +Data = { +Type = HAMTShard // Still a HAMT at second level +Data = 0x800000... // Bitmap for this sub-level +hashType = 0x22 // murmur3-x64-64 +fanout = 256 // Same fanout throughout +} + +Links = [ +{ +Hash = bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa +Name = "6E470.txt" // Bucket 0x6E + filename +Tsize = 1271 +}, +{ +Hash = bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa +Name = "FF742.txt" // Bucket 0xFF + filename +Tsize = 1271 +} +] } ``` @@ -384,11 +388,11 @@ To resolve a path inside a HAMT: 1. Hash the filename using the hash function specified in `decode(PBNode.Data).hashType` 2. Pop `log2(fanout)` bits from the hash digest (lowest/least significant bits first), - then hex encode those bits using little endian to form the bucket prefix. The prefix MUST use uppercase hex characters (00-FF, not 00-ff) +then hex encode those bits using little endian to form the bucket prefix. The prefix MUST use uppercase hex characters (00-FF, not 00-ff) 3. Find the link whose `Name` starts with this hex prefix: - - If `Name` equals the prefix exactly → this is a sub-shard, follow the link and repeat from step 2 - - If `Name` equals prefix + filename → target found - - If no matching prefix → file not in directory +- If `Name` equals the prefix exactly → this is a sub-shard, follow the link and repeat from step 2 +- If `Name` equals prefix + filename → target found +- If no matching prefix → file not in directory 4. When following to a sub-shard, continue consuming bits from the same hash Note: Empty intermediate shards are typically collapsed during deletion operations to maintain consistency @@ -401,14 +405,14 @@ Given a HAMT-sharded directory containing 1000 files: 1. Hash the filename "470.txt" using murmur3-x64-64 (multihash `0x22`) 2. With fanout=256, we consume 8 bits at a time from the hash: - - First 8 bits determine root bucket → `0x00` → link name "00" - - Follow link "00" to sub-shard (`bafybeiaebmuestgbpqhkkbrwl2qtjtvs3whkmp2trkbkimuod4yv7oygni`) +- First 8 bits determine root bucket → `0x00` → link name "00" +- Follow link "00" to sub-shard (`bafybeiaebmuestgbpqhkkbrwl2qtjtvs3whkmp2trkbkimuod4yv7oygni`) 3. The sub-shard is also a HAMT (has Type=HAMTShard): - - Next 8 bits from hash → `0x6E` - - Find entry with name "6E470.txt" (prefix + original filename) +- Next 8 bits from hash → `0x6E` +- Find entry with name "6E470.txt" (prefix + original filename) 4. Link name format at leaf level: `[hex_prefix][original_filename]` - - "6E470.txt" means: file "470.txt" that hashed to bucket 6E at this level - - "FF742.txt" means: file "742.txt" that hashed to bucket FF at this level +- "6E470.txt" means: file "470.txt" that hashed to bucket 6E at this level +- "FF742.txt" means: file "742.txt" that hashed to bucket FF at this level ::: ### `dag-pb` `Symlink` @@ -496,13 +500,13 @@ The `mode` is for persisting the file permissions in [numeric notation](https:// \[[spec](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html)\]. - If unspecified, implementations MAY default to - - `0755` for directories/HAMT shards - - `0644` for all other types where applicable +- `0755` for directories/HAMT shards +- `0644` for all other types where applicable - The nine least significant bits represent `ugo-rwx` - The next three least significant bits represent `setuid`, `setgid` and the `sticky bit` - The remaining 20 bits are reserved for future use, and are subject to change. Spec implementations **MUST** handle bits they do not expect as follows: - - For future-proofing, the (de)serialization layer must preserve the entire uint32 value during clone/copy operations, modifying only bit values that have a well defined meaning: `clonedValue = ( modifiedBits & 07777 ) | ( originalValue & 0xFFFFF000 )` - - Implementations of this spec must proactively mask off bits without a defined meaning in the implemented version of the spec: `interpretedValue = originalValue & 07777` +- For future-proofing, the (de)serialization layer must preserve the entire uint32 value during clone/copy operations, modifying only bit values that have a well defined meaning: `clonedValue = ( modifiedBits & 07777 ) | ( originalValue & 0xFFFFF000 )` +- Implementations of this spec must proactively mask off bits without a defined meaning in the implemented version of the spec: `interpretedValue = originalValue & 07777` #### `mtime` Field @@ -516,29 +520,29 @@ The two fields are: Implementations encoding or decoding wire-representations MUST observe the following: - An `mtime` structure with `FractionalNanoseconds` outside of the on-wire range - `[1, 999999999]` is **not** valid. This includes a fractional value of `0`. - Implementations encountering such values should consider the entire enclosing - metadata block malformed and abort the processing of the corresponding DAG. +`[1, 999999999]` is **not** valid. This includes a fractional value of `0`. +Implementations encountering such values should consider the entire enclosing +metadata block malformed and abort the processing of the corresponding DAG. - The `mtime` structure is optional. Its absence implies `unspecified` rather - than `0`. +than `0`. - For ergonomic reasons, a surface API of an encoder MUST allow fractional `0` as - input, while at the same time MUST ensure it is stripped from the final structure - before encoding, satisfying the above constraints. +input, while at the same time MUST ensure it is stripped from the final structure +before encoding, satisfying the above constraints. Implementations interpreting the `mtime` metadata in order to apply it within a non-IPFS target MUST observe the following: - If the target supports a distinction between `unspecified` and `0`/`1970-01-01T00:00:00Z`, - the distinction must be preserved within the target. For example, if no `mtime` structure - is available, a web gateway must **not** render a `Last-Modified:` header. +the distinction must be preserved within the target. For example, if no `mtime` structure +is available, a web gateway must **not** render a `Last-Modified:` header. - If the target requires an `mtime` ( e.g. a FUSE interface ) and no `mtime` is - supplied OR the supplied `mtime` falls outside of the targets accepted range: - - When no `mtime` is specified or the resulting `UnixTime` is negative: - implementations must assume `0`/`1970-01-01T00:00:00Z` (note that such values - are not merely academic: e.g. the OpenVMS epoch is `1858-11-17T00:00:00Z`) - - When the resulting `UnixTime` is larger than the targets range ( e.g. 32bit - vs 64bit mismatch), implementations must assume the highest possible value - in the targets range. In most cases, this would be `2038-01-19T03:14:07Z`. +supplied OR the supplied `mtime` falls outside of the targets accepted range: +- When no `mtime` is specified or the resulting `UnixTime` is negative: +implementations must assume `0`/`1970-01-01T00:00:00Z` (note that such values +are not merely academic: e.g. the OpenVMS epoch is `1858-11-17T00:00:00Z`) +- When the resulting `UnixTime` is larger than the targets range ( e.g. 32bit +vs 64bit mismatch), implementations must assume the highest possible value +in the targets range. In most cases, this would be `2038-01-19T03:14:07Z`. ## UnixFS Paths @@ -552,7 +556,7 @@ separated by `/` (`0x2F`). UnixFS paths read from left to right, and are inspired by POSIX paths. - Components MUST NOT contain `/` unicode codepoints because it would break - the path into two components. +the path into two components. - Components SHOULD be UTF8 unicode. - Components are case-sensitive. @@ -576,11 +580,11 @@ Relative path components MUST be resolved before trying to work on the path: - `.` points to the current node and MUST be removed. - `..` points to the parent node and MUST be removed left to right. When removing - a `..`, the path component on the left MUST also be removed. If there is no path - component on the left, you MUST error to avoid out-of-bounds - path resolution. +a `..`, the path component on the left MUST also be removed. If there is no path +component on the left, you MUST error to avoid out-of-bounds +path resolution. - Implementations MUST error when resolving a relative path that attempts to go - beyond the root CID (example: `/ipfs/cid/../foo`). +beyond the root CID (example: `/ipfs/cid/../foo`). ### Restricted Names @@ -590,8 +594,8 @@ The following names SHOULD NOT be used: - The `..` string, as it represents the parent node in POSIX pathing. - The empty string. - Any string containing a `NULL` (`0x00`) byte, as this is often used to signify string - terminations in some systems, such as C-compatible systems. Many unix - file systems do not accept this character in path components. +terminations in some systems, such as C-compatible systems. Many unix +file systems do not accept this character in path components. # Appendix: Test Vectors @@ -608,15 +612,15 @@ Test vectors for UnixFS file structures, progressing from simple single-block fi ### Single `raw` Block File - Fixture: [`dir-with-files.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_unixfs/dir-with-files.car) - - CID: `bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4` (hello.txt) - - Type: [`raw` Node](#raw-node) - - Content: "hello world\n" (12 bytes) - - Block Analysis: - - Block size (`ipfs block stat`): 12 bytes - - Data size (`ipfs cat`): 12 bytes - - DAG-PB envelope: N/A (raw blocks have no envelope overhead) - - Purpose: Single block using `raw` codec, no protobuf wrapper - - Validation: Block content IS the file content, no UnixFS metadata +- CID: `bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4` (hello.txt) +- Type: [`raw` Node](#raw-node) +- Content: "hello world\n" (12 bytes) +- Block Analysis: +- Block size (`ipfs block stat`): 12 bytes +- Data size (`ipfs cat`): 12 bytes +- DAG-PB envelope: N/A (raw blocks have no envelope overhead) +- Purpose: Single block using `raw` codec, no protobuf wrapper +- Validation: Block content IS the file content, no UnixFS metadata ### Single `dag-pb` Block File @@ -625,89 +629,89 @@ Test vectors for UnixFS file structures, progressing from simple single-block fi - Type: [`dag-pb` File](#dag-pb-file) with data in the same block - Content: "Hello from IPFS Gateway Checker\n" (32 bytes) - Block Analysis: - - Block size (`ipfs block stat`): 40 bytes - - Data size (`ipfs cat`): 32 bytes - - DAG-PB envelope: 8 bytes (40 - 32) +- Block size (`ipfs block stat`): 40 bytes +- Data size (`ipfs cat`): 32 bytes +- DAG-PB envelope: 8 bytes (40 - 32) - Structure: - ``` - 📄 small-file.txt # bafybeifx7yeb55armcsxwwitkymga5xf53dxiarykms3ygqic223w5sk3m (dag-pb) - └── 📦 Data.Data # "Hello from IPFS Gateway Checker\n" (32 bytes, stored inline in UnixFS protobuf) - ``` +``` +📄 small-file.txt # bafybeifx7yeb55armcsxwwitkymga5xf53dxiarykms3ygqic223w5sk3m (dag-pb) +└── 📦 Data.Data # "Hello from IPFS Gateway Checker\n" (32 bytes, stored inline in UnixFS protobuf) +``` - Purpose: Small file stored within dag-pb Data field - Validation: File content extracted from UnixFS Data.Data field ### Multi-block File - Fixture: [`dir-with-files.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_unixfs/dir-with-files.car) - - CID: `bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa` (multiblock.txt) - - Type: [`dag-pb` File](#dag-pb-file) with multiple [`raw` Node](#raw-node) leaves - - Content: Lorem ipsum text (1026 bytes total) - - Block Analysis: - - Root block size (`ipfs block stat`): 245 bytes (dag-pb) - - Total data size (`ipfs cat`): 1026 bytes - - Child blocks: - - Block 1: 256 bytes (raw) - - Block 2: 256 bytes (raw) - - Block 3: 256 bytes (raw) - - Block 4: 256 bytes (raw) - - Block 5: 2 bytes (raw) - - DAG-PB envelope: 245 bytes (root block containing metadata + links) - - Structure: - ``` - 📄 multiblock.txt # bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa (dag-pb root) - ├── 📦 [0-255] # bafkreie5noke3mb7hqxukzcy73nl23k6lxszxi5w3dtmuwz62wnvkpsscm (raw, 256 bytes) - ├── 📦 [256-511] # bafkreih4ephajybraj6wnxsbwjwa77fukurtpl7oj7t7pfq545duhot7cq (raw, 256 bytes) - ├── 📦 [512-767] # bafkreigu7buvm3cfunb35766dn7tmqyh2um62zcio63en2btvxuybgcpue (raw, 256 bytes) - ├── 📦 [768-1023] # bafkreicll3huefkc3qnrzeony7zcfo7cr3nbx64hnxrqzsixpceg332fhe (raw, 256 bytes) - └── 📦 [1024-1025] # bafkreifst3pqztuvj57lycamoi7z34b4emf7gawxs74nwrc2c7jncmpaqm (raw, 2 bytes) - ``` - - Purpose: File chunking and reassembly - - Validation: - - Links have no Names (must be absent) - - Blocksizes array matches Links array length - - Reassembled content matches original +- CID: `bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa` (multiblock.txt) +- Type: [`dag-pb` File](#dag-pb-file) with multiple [`raw` Node](#raw-node) leaves +- Content: Lorem ipsum text (1026 bytes total) +- Block Analysis: +- Root block size (`ipfs block stat`): 245 bytes (dag-pb) +- Total data size (`ipfs cat`): 1026 bytes +- Child blocks: +- Block 1: 256 bytes (raw) +- Block 2: 256 bytes (raw) +- Block 3: 256 bytes (raw) +- Block 4: 256 bytes (raw) +- Block 5: 2 bytes (raw) +- DAG-PB envelope: 245 bytes (root block containing metadata + links) +- Structure: +``` +📄 multiblock.txt # bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa (dag-pb root) +├── 📦 [0-255] # bafkreie5noke3mb7hqxukzcy73nl23k6lxszxi5w3dtmuwz62wnvkpsscm (raw, 256 bytes) +├── 📦 [256-511] # bafkreih4ephajybraj6wnxsbwjwa77fukurtpl7oj7t7pfq545duhot7cq (raw, 256 bytes) +├── 📦 [512-767] # bafkreigu7buvm3cfunb35766dn7tmqyh2um62zcio63en2btvxuybgcpue (raw, 256 bytes) +├── 📦 [768-1023] # bafkreicll3huefkc3qnrzeony7zcfo7cr3nbx64hnxrqzsixpceg332fhe (raw, 256 bytes) +└── 📦 [1024-1025] # bafkreifst3pqztuvj57lycamoi7z34b4emf7gawxs74nwrc2c7jncmpaqm (raw, 2 bytes) +``` +- Purpose: File chunking and reassembly +- Validation: +- Links have no Names (must be absent) +- Blocksizes array matches Links array length +- Reassembled content matches original ### File with Missing Blocks - Fixture: [`bafybeibfhhww5bpsu34qs7nz25wp7ve36mcc5mxd5du26sr45bbnjhpkei.dag-pb`](https://github.com/ipld/codec-fixtures/raw/381e762b85862b2bbdb6ef2ba140b3c505e31a44/fixtures/dagpb_7unnamedlinks%2Bdata/bafybeibfhhww5bpsu34qs7nz25wp7ve36mcc5mxd5du26sr45bbnjhpkei.dag-pb) - - CID: `bafybeibfhhww5bpsu34qs7nz25wp7ve36mcc5mxd5du26sr45bbnjhpkei` - - Type: [`dag-pb` File](#dag-pb-file) with 7 links to child blocks - - Size: 306MB total (from metadata) - - Structure: - ``` - 📄 large-file # bafybeibfhhww5bpsu34qs7nz25wp7ve36mcc5mxd5du26sr45bbnjhpkei (dag-pb root) - ├── ⚠️ block[0] # (missing child block) - ├── ⚠️ block[1] # (missing child block) - ├── ⚠️ block[2] # (missing child block) - ├── ⚠️ block[3] # (missing child block) - ├── ⚠️ block[4] # (missing child block) - ├── ⚠️ block[5] # (missing child block) - └── ⚠️ block[6] # (missing child block) - ``` - - Note: Child blocks are NOT included - they may be unavailable locally or missing entirely - - Purpose: - - Reading UnixFS file metadata should require only the root block - - File size and structure can be determined without fetching children - - Operations should not block waiting for child blocks unless content is actually requested - - Validation: Can extract file size and chunking info from root block alone +- CID: `bafybeibfhhww5bpsu34qs7nz25wp7ve36mcc5mxd5du26sr45bbnjhpkei` +- Type: [`dag-pb` File](#dag-pb-file) with 7 links to child blocks +- Size: 306MB total (from metadata) +- Structure: +``` +📄 large-file # bafybeibfhhww5bpsu34qs7nz25wp7ve36mcc5mxd5du26sr45bbnjhpkei (dag-pb root) +├── ⚠️ block[0] # (missing child block) +├── ⚠️ block[1] # (missing child block) +├── ⚠️ block[2] # (missing child block) +├── ⚠️ block[3] # (missing child block) +├── ⚠️ block[4] # (missing child block) +├── ⚠️ block[5] # (missing child block) +└── ⚠️ block[6] # (missing child block) +``` +- Note: Child blocks are NOT included - they may be unavailable locally or missing entirely +- Purpose: +- Reading UnixFS file metadata should require only the root block +- File size and structure can be determined without fetching children +- Operations should not block waiting for child blocks unless content is actually requested +- Validation: Can extract file size and chunking info from root block alone ### Range Requests with Missing Blocks - Fixture: [`file-3k-and-3-blocks-missing-block.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/trustless_gateway_car/file-3k-and-3-blocks-missing-block.car) - - CID: `QmYhmPjhFjYFyaoiuNzYv8WGavpSRDwdHWe5B4M5du5Rtk` - - Type: [`dag-pb` File](#dag-pb-file) with 3 links but middle block intentionally missing - - Structure: - ``` - 📄 file-3k # QmYhmPjhFjYFyaoiuNzYv8WGavpSRDwdHWe5B4M5du5Rtk (dag-pb root) - ├── 📦 [0-1023] # QmPKt7ptM2ZYSGPUc8PmPT2VBkLDK3iqpG9TBJY7PCE9rF (raw, 1024 bytes) - ├── ⚠️ [1024-2047] # (missing block - intentionally removed) - └── 📦 [2048-3071] # QmWXY482zQdwecnfBsj78poUUuPXvyw2JAFAEMw4tzTavV (raw, 1024 bytes) - ``` - - Critical requirement: Must support seeking without all blocks available - - Purpose: - - Fetch only required blocks for byte range requests (e.g., bytes=0-1023 or bytes=2048-3071) - - Gateway conformance tests verify that first block (`QmPKt7ptM2ZYSGPUc8PmPT2VBkLDK3iqpG9TBJY7PCE9rF`) and third block (`QmWXY482zQdwecnfBsj78poUUuPXvyw2JAFAEMw4tzTavV`) can be fetched independently - - Requests for middle block or byte ranges requiring it should fail gracefully +- CID: `QmYhmPjhFjYFyaoiuNzYv8WGavpSRDwdHWe5B4M5du5Rtk` +- Type: [`dag-pb` File](#dag-pb-file) with 3 links but middle block intentionally missing +- Structure: +``` +📄 file-3k # QmYhmPjhFjYFyaoiuNzYv8WGavpSRDwdHWe5B4M5du5Rtk (dag-pb root) +├── 📦 [0-1023] # QmPKt7ptM2ZYSGPUc8PmPT2VBkLDK3iqpG9TBJY7PCE9rF (raw, 1024 bytes) +├── ⚠️ [1024-2047] # (missing block - intentionally removed) +└── 📦 [2048-3071] # QmWXY482zQdwecnfBsj78poUUuPXvyw2JAFAEMw4tzTavV (raw, 1024 bytes) +``` +- Critical requirement: Must support seeking without all blocks available +- Purpose: +- Fetch only required blocks for byte range requests (e.g., bytes=0-1023 or bytes=2048-3071) +- Gateway conformance tests verify that first block (`QmPKt7ptM2ZYSGPUc8PmPT2VBkLDK3iqpG9TBJY7PCE9rF`) and third block (`QmWXY482zQdwecnfBsj78poUUuPXvyw2JAFAEMw4tzTavV`) can be fetched independently +- Requests for middle block or byte ranges requiring it should fail gracefully ## Directory Test Vectors @@ -716,126 +720,126 @@ Test vectors for UnixFS directory structures, progressing from simple flat direc ### Simple Directory - Fixture: [`dir-with-files.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_unixfs/dir-with-files.car) - - CID: `bafybeihchr7vmgjaasntayyatmp5sv6xza57iy2h4xj7g46bpjij6yhrmy` - - Type: [`dag-pb` Directory](#dag-pb-directory) - - Block Analysis: - - Directory block size (`ipfs block stat`): 185 bytes - - Contains UnixFS Type=Directory metadata + 4 links - - Structure: - ``` - 📁 / # bafybeihchr7vmgjaasntayyatmp5sv6xza57iy2h4xj7g46bpjij6yhrmy - ├── 📄 ascii-copy.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" - ├── 📄 ascii.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" - ├── 📄 hello.txt # bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4 (raw, 12 bytes) "hello world\n" - └── 📄 multiblock.txt # bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa (dag-pb, 1026 bytes) Lorem ipsum text - ``` - - Purpose: Directory listing, link sorting, deduplication (ascii.txt and ascii-copy.txt share same CID) - - Validation: Links sorted lexicographically by Name, each has valid Tsize +- CID: `bafybeihchr7vmgjaasntayyatmp5sv6xza57iy2h4xj7g46bpjij6yhrmy` +- Type: [`dag-pb` Directory](#dag-pb-directory) +- Block Analysis: +- Directory block size (`ipfs block stat`): 185 bytes +- Contains UnixFS Type=Directory metadata + 4 links +- Structure: +``` +📁 / # bafybeihchr7vmgjaasntayyatmp5sv6xza57iy2h4xj7g46bpjij6yhrmy +├── 📄 ascii-copy.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" +├── 📄 ascii.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" +├── 📄 hello.txt # bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4 (raw, 12 bytes) "hello world\n" +└── 📄 multiblock.txt # bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa (dag-pb, 1026 bytes) Lorem ipsum text +``` +- Purpose: Directory listing, link sorting, deduplication (ascii.txt and ascii-copy.txt share same CID) +- Validation: Links sorted lexicographically by Name, each has valid Tsize ### Nested Directories - Fixture: [`subdir-with-two-single-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/trustless_gateway_car/subdir-with-two-single-block-files.car) - - CID: `bafybeietjm63oynimmv5yyqay33nui4y4wx6u3peezwetxgiwvfmelutzu` - - Type: [`dag-pb` Directory](#dag-pb-directory) containing another Directory - - Block Analysis: - - Root directory block size: 55 bytes - - Subdirectory block size: 110 bytes - - Structure: - ``` - 📁 / # bafybeietjm63oynimmv5yyqay33nui4y4wx6u3peezwetxgiwvfmelutzu - └── 📁 subdir/ # bafybeiggghzz6dlue3m6nb2dttnbrygxh3lrjl5764f2m4gq7dgzdt55o4 (dag-pb Directory) - ├── 📄 ascii.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" - └── 📄 hello.txt # bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4 (raw, 12 bytes) "hello world\n" - ``` - - Purpose: Path traversal through directory hierarchy - - Validation: Can traverse `/subdir/hello.txt` path correctly +- CID: `bafybeietjm63oynimmv5yyqay33nui4y4wx6u3peezwetxgiwvfmelutzu` +- Type: [`dag-pb` Directory](#dag-pb-directory) containing another Directory +- Block Analysis: +- Root directory block size: 55 bytes +- Subdirectory block size: 110 bytes +- Structure: +``` +📁 / # bafybeietjm63oynimmv5yyqay33nui4y4wx6u3peezwetxgiwvfmelutzu +└── 📁 subdir/ # bafybeiggghzz6dlue3m6nb2dttnbrygxh3lrjl5764f2m4gq7dgzdt55o4 (dag-pb Directory) +├── 📄 ascii.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" +└── 📄 hello.txt # bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4 (raw, 12 bytes) "hello world\n" +``` +- Purpose: Path traversal through directory hierarchy +- Validation: Can traverse `/subdir/hello.txt` path correctly - Fixture: [`dag-pb.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_dag/dag-pb.car) - - CID: `bafybeiegxwlgmoh2cny7qlolykdf7aq7g6dlommarldrbm7c4hbckhfcke` - - Type: [`dag-pb` Directory](#dag-pb-directory) - - Structure: - ``` - 📁 / # bafybeiegxwlgmoh2cny7qlolykdf7aq7g6dlommarldrbm7c4hbckhfcke - ├── 📁 foo/ # bafybeidryarwh34ygbtyypbu7qjkl4euiwxby6cql6uvosonohkq2kwnkm (dag-pb Directory) - │ └── 📄 bar.txt # bafkreigzafgemjeejks3vqyuo46ww2e22rt7utq5djikdofjtvnjl5zp6u (raw, 14 bytes) "Hello, world!" - └── 📄 foo.txt # bafkreic3ondyhizrzeoufvoodehinugpj3ecruwokaygl7elezhn2khqfa (raw, 13 bytes) "Hello, IPFS!" - ``` - - Purpose: Another example of standard UnixFS directory with raw leaf blocks +- CID: `bafybeiegxwlgmoh2cny7qlolykdf7aq7g6dlommarldrbm7c4hbckhfcke` +- Type: [`dag-pb` Directory](#dag-pb-directory) +- Structure: +``` +📁 / # bafybeiegxwlgmoh2cny7qlolykdf7aq7g6dlommarldrbm7c4hbckhfcke +├── 📁 foo/ # bafybeidryarwh34ygbtyypbu7qjkl4euiwxby6cql6uvosonohkq2kwnkm (dag-pb Directory) +│ └── 📄 bar.txt # bafkreigzafgemjeejks3vqyuo46ww2e22rt7utq5djikdofjtvnjl5zp6u (raw, 14 bytes) "Hello, world!" +└── 📄 foo.txt # bafkreic3ondyhizrzeoufvoodehinugpj3ecruwokaygl7elezhn2khqfa (raw, 13 bytes) "Hello, IPFS!" +``` +- Purpose: Another example of standard UnixFS directory with raw leaf blocks ### Special Characters in Filenames - Fixture: [`path_gateway_tar/fixtures.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_tar/fixtures.car) - - CID: `bafybeig6ka5mlwkl4subqhaiatalkcleo4jgnr3hqwvpmsqfca27cijp3i` - - Type: [`dag-pb` Directory](#dag-pb-directory) with nested subdirectories - - Structure: - ``` - 📁 / # bafybeig6ka5mlwkl4subqhaiatalkcleo4jgnr3hqwvpmsqfca27cijp3i - └── 📁 ą/ # (dag-pb Directory) - └── 📁 ę/ # (dag-pb Directory) - └── 📄 file-źł.txt # (raw, 34 bytes) "I am a txt file on path with utf8" - ``` - - Path with Polish diacritics: `/ipfs/bafybeig6ka5mlwkl4subqhaiatalkcleo4jgnr3hqwvpmsqfca27cijp3i/ą/ę/file-źł.txt` - - Purpose: UTF-8 characters in directory and file names (ą, ę, ź, ł) - - Validation: Directory traversal works with UTF-8 paths +- CID: `bafybeig6ka5mlwkl4subqhaiatalkcleo4jgnr3hqwvpmsqfca27cijp3i` +- Type: [`dag-pb` Directory](#dag-pb-directory) with nested subdirectories +- Structure: +``` +📁 / # bafybeig6ka5mlwkl4subqhaiatalkcleo4jgnr3hqwvpmsqfca27cijp3i +└── 📁 ą/ # (dag-pb Directory) +└── 📁 ę/ # (dag-pb Directory) +└── 📄 file-źł.txt # (raw, 34 bytes) "I am a txt file on path with utf8" +``` +- Path with Polish diacritics: `/ipfs/bafybeig6ka5mlwkl4subqhaiatalkcleo4jgnr3hqwvpmsqfca27cijp3i/ą/ę/file-źł.txt` +- Purpose: UTF-8 characters in directory and file names (ą, ę, ź, ł) +- Validation: Directory traversal works with UTF-8 paths - Fixture: [`dir-with-percent-encoded-filename.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_unixfs/dir-with-percent-encoded-filename.car) - - CID: `bafybeig675grnxcmshiuzdaz2xalm6ef4thxxds6o6ypakpghm5kghpc34` - - Type: [`dag-pb` Directory](#dag-pb-directory) - - Structure: - ``` - 📁 / # bafybeig675grnxcmshiuzdaz2xalm6ef4thxxds6o6ypakpghm5kghpc34 - └── 📄 Portugal%2C+España=Peninsula Ibérica.txt # bafkreihfmctcb2kuvoljqeuphqr2fg2r45vz5cxgq5c2yrxnqg5erbitmq (raw, 38 bytes) "hello from a percent encoded filename" - ``` - - Purpose: Filenames with percent-encoding (`%2C`), plus signs, equals, and non-ASCII characters - - Validation: - - Implementations MUST preserve the original filename exactly as stored in UnixFS - - Must not be confused by filenames mixing Unicode characters with percent-encoding - - Gateway example: In gateway-conformance, accessing this file from a web browser requires double-encoding the `%2C` as `%252C` in the URL path (`/ipfs/{{CID}}/Portugal%252C+España=Peninsula%20Ibérica.txt`) - - Browser implementations should preserve `%2C` in the filename to avoid conflicts with URL encoding +- CID: `bafybeig675grnxcmshiuzdaz2xalm6ef4thxxds6o6ypakpghm5kghpc34` +- Type: [`dag-pb` Directory](#dag-pb-directory) +- Structure: +``` +📁 / # bafybeig675grnxcmshiuzdaz2xalm6ef4thxxds6o6ypakpghm5kghpc34 +└── 📄 Portugal%2C+España=Peninsula Ibérica.txt # bafkreihfmctcb2kuvoljqeuphqr2fg2r45vz5cxgq5c2yrxnqg5erbitmq (raw, 38 bytes) "hello from a percent encoded filename" +``` +- Purpose: Filenames with percent-encoding (`%2C`), plus signs, equals, and non-ASCII characters +- Validation: +- Implementations MUST preserve the original filename exactly as stored in UnixFS +- Must not be confused by filenames mixing Unicode characters with percent-encoding +- Gateway example: In gateway-conformance, accessing this file from a web browser requires double-encoding the `%2C` as `%252C` in the URL path (`/ipfs/{{CID}}/Portugal%252C+España=Peninsula%20Ibérica.txt`) +- Browser implementations should preserve `%2C` in the filename to avoid conflicts with URL encoding ### Directory with Missing Blocks - Fixture: [`bafybeigcsevw74ssldzfwhiijzmg7a35lssfmjkuoj2t5qs5u5aztj47tq.dag-pb`](https://github.com/ipld/codec-fixtures/raw/381e762b85862b2bbdb6ef2ba140b3c505e31a44/fixtures/dagpb_4namedlinks%2Bdata/bafybeigcsevw74ssldzfwhiijzmg7a35lssfmjkuoj2t5qs5u5aztj47tq.dag-pb) - - CID: `bafybeigcsevw74ssldzfwhiijzmg7a35lssfmjkuoj2t5qs5u5aztj47tq` - - Type: [`dag-pb` Directory](#dag-pb-directory) - - Structure: - ``` - 📁 / # bafybeigcsevw74ssldzfwhiijzmg7a35lssfmjkuoj2t5qs5u5aztj47tq - ├── ⚠️ audio_only.m4a # (link to missing block, ~24MB) - ├── ⚠️ chat.txt # (link to missing block, ~1KB) - ├── ⚠️ playback.m3u # (link to missing block, ~116 bytes) - └── ⚠️ zoom_0.mp4 # (link to missing block) - ``` - - Note: Child blocks are NOT included - they may be unavailable locally or missing entirely - - Purpose: - - Directory enumeration should require only the root block - - Can list all filenames and their CIDs without fetching child blocks - - Operations should not block waiting for child blocks unless content is actually requested - - Validation: Can enumerate directory contents from root block alone +- CID: `bafybeigcsevw74ssldzfwhiijzmg7a35lssfmjkuoj2t5qs5u5aztj47tq` +- Type: [`dag-pb` Directory](#dag-pb-directory) +- Structure: +``` +📁 / # bafybeigcsevw74ssldzfwhiijzmg7a35lssfmjkuoj2t5qs5u5aztj47tq +├── ⚠️ audio_only.m4a # (link to missing block, ~24MB) +├── ⚠️ chat.txt # (link to missing block, ~1KB) +├── ⚠️ playback.m3u # (link to missing block, ~116 bytes) +└── ⚠️ zoom_0.mp4 # (link to missing block) +``` +- Note: Child blocks are NOT included - they may be unavailable locally or missing entirely +- Purpose: +- Directory enumeration should require only the root block +- Can list all filenames and their CIDs without fetching child blocks +- Operations should not block waiting for child blocks unless content is actually requested +- Validation: Can enumerate directory contents from root block alone ### HAMT Sharded Directory - Fixture: [`single-layer-hamt-with-multi-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/trustless_gateway_car/single-layer-hamt-with-multi-block-files.car) - - CID: `bafybeidbclfqleg2uojchspzd4bob56dqetqjsj27gy2cq3klkkgxtpn4i` - - Type: [`dag-pb` HAMTDirectory](#dag-pb-hamtdirectory) - - Block Analysis: - - Root HAMT block size (`ipfs block stat`): 12046 bytes - - Contains UnixFS Type=HAMTShard metadata with fanout=256 - - Links use 2-character hex prefixes for hash buckets (00-FF) - - Structure: - ``` - 📂 / # bafybeidbclfqleg2uojchspzd4bob56dqetqjsj27gy2cq3klkkgxtpn4i (HAMT root) - ├── 📄 1.txt # (dag-pb file, multi-block) - ├── 📄 2.txt # (dag-pb file, multi-block) - ├── ... - └── 📄 1000.txt # (dag-pb file, multi-block) - ``` - - Contents: 1000 numbered files (1.txt through 1000.txt), each containing Lorem ipsum text - - Purpose: HAMT sharding for large directories - - Validation: - - Fanout field = 256 - - Link Names in HAMT have 2-character hex prefix (hash buckets) - - Can retrieve any file by name through hash bucket calculation +- CID: `bafybeidbclfqleg2uojchspzd4bob56dqetqjsj27gy2cq3klkkgxtpn4i` +- Type: [`dag-pb` HAMTDirectory](#dag-pb-hamtdirectory) +- Block Analysis: +- Root HAMT block size (`ipfs block stat`): 12046 bytes +- Contains UnixFS Type=HAMTShard metadata with fanout=256 +- Links use 2-character hex prefixes for hash buckets (00-FF) +- Structure: +``` +📂 / # bafybeidbclfqleg2uojchspzd4bob56dqetqjsj27gy2cq3klkkgxtpn4i (HAMT root) +├── 📄 1.txt # (dag-pb file, multi-block) +├── 📄 2.txt # (dag-pb file, multi-block) +├── ... +└── 📄 1000.txt # (dag-pb file, multi-block) +``` +- Contents: 1000 numbered files (1.txt through 1000.txt), each containing Lorem ipsum text +- Purpose: HAMT sharding for large directories +- Validation: +- Fanout field = 256 +- Link Names in HAMT have 2-character hex prefix (hash buckets) +- Can retrieve any file by name through hash bucket calculation ## Special Cases and Advanced Features @@ -844,52 +848,52 @@ Test vectors for special UnixFS features and edge cases. ### Symbolic Links - Fixture: [`symlink.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_unixfs/symlink.car) - - CID: `QmWvY6FaqFMS89YAQ9NAPjVP4WZKA1qbHbicc9HeSKQTgt` - - Types: [`dag-pb` Directory](#dag-pb-directory) containing [`dag-pb` Symlink](#dag-pb-symlink) - - Block Analysis: - - Root directory block: Not measured (V0 CID) - - Symlink block (`QmTB8BaCJdCH5H3k7GrxJsxgDNmNYGGR71C58ERkivXoj5`): 9 bytes - - Target file block (`Qme2y5HA5kvo2jAx13UsnV5bQJVijiAJCPvaW3JGQWhvJZ`): 16 bytes - - Structure: - ``` - 📁 / # QmWvY6FaqFMS89YAQ9NAPjVP4WZKA1qbHbicc9HeSKQTgt - ├── 📄 foo # Qme2y5HA5kvo2jAx13UsnV5bQJVijiAJCPvaW3JGQWhvJZ - file containing "content" - └── 🔗 bar # QmTB8BaCJdCH5H3k7GrxJsxgDNmNYGGR71C58ERkivXoj5 - symlink pointing to "foo" - ``` - - Purpose: UnixFS symlink resolution - - Security note: Critical for preventing path traversal vulnerabilities +- CID: `QmWvY6FaqFMS89YAQ9NAPjVP4WZKA1qbHbicc9HeSKQTgt` +- Types: [`dag-pb` Directory](#dag-pb-directory) containing [`dag-pb` Symlink](#dag-pb-symlink) +- Block Analysis: +- Root directory block: Not measured (V0 CID) +- Symlink block (`QmTB8BaCJdCH5H3k7GrxJsxgDNmNYGGR71C58ERkivXoj5`): 9 bytes +- Target file block (`Qme2y5HA5kvo2jAx13UsnV5bQJVijiAJCPvaW3JGQWhvJZ`): 16 bytes +- Structure: +``` +📁 / # QmWvY6FaqFMS89YAQ9NAPjVP4WZKA1qbHbicc9HeSKQTgt +├── 📄 foo # Qme2y5HA5kvo2jAx13UsnV5bQJVijiAJCPvaW3JGQWhvJZ - file containing "content" +└── 🔗 bar # QmTB8BaCJdCH5H3k7GrxJsxgDNmNYGGR71C58ERkivXoj5 - symlink pointing to "foo" +``` +- Purpose: UnixFS symlink resolution +- Security note: Critical for preventing path traversal vulnerabilities ### Mixed Block Sizes - Fixture: [`subdir-with-mixed-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/trustless_gateway_car/subdir-with-mixed-block-files.car) - - CID: `bafybeidh6k2vzukelqtrjsmd4p52cpmltd2ufqrdtdg6yigi73in672fwu` - - Type: [`dag-pb` Directory](#dag-pb-directory) with subdirectory - - Structure: - ``` - 📁 / # bafybeidh6k2vzukelqtrjsmd4p52cpmltd2ufqrdtdg6yigi73in672fwu - └── 📁 subdir/ # bafybeicnmple4ehlz3ostv2sbojz3zhh5q7tz5r2qkfdpqfilgggeen7xm - ├── 📄 ascii.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" - ├── 📄 hello.txt # bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4 (raw, 12 bytes) "hello world\n" - └── 📄 multiblock.txt # bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa (dag-pb, 1271 bytes total) - ``` - - Purpose: Directories containing both single-block raw files and multi-block dag-pb files - - Validation: Can handle mixed file types in same directory +- CID: `bafybeidh6k2vzukelqtrjsmd4p52cpmltd2ufqrdtdg6yigi73in672fwu` +- Type: [`dag-pb` Directory](#dag-pb-directory) with subdirectory +- Structure: +``` +📁 / # bafybeidh6k2vzukelqtrjsmd4p52cpmltd2ufqrdtdg6yigi73in672fwu +└── 📁 subdir/ # bafybeicnmple4ehlz3ostv2sbojz3zhh5q7tz5r2qkfdpqfilgggeen7xm +├── 📄 ascii.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" +├── 📄 hello.txt # bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4 (raw, 12 bytes) "hello world\n" +└── 📄 multiblock.txt # bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa (dag-pb, 1271 bytes total) +``` +- Purpose: Directories containing both single-block raw files and multi-block dag-pb files +- Validation: Can handle mixed file types in same directory ### Deduplication - Fixture: [`dir-with-duplicate-files.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/trustless_gateway_car/dir-with-duplicate-files.car) - - CID: `bafybeihchr7vmgjaasntayyatmp5sv6xza57iy2h4xj7g46bpjij6yhrmy` - - Type: [`dag-pb` Directory](#dag-pb-directory) - - Structure: - ``` - 📁 / # bafybeihchr7vmgjaasntayyatmp5sv6xza57iy2h4xj7g46bpjij6yhrmy - ├── 🔗 ascii-copy.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (same CID as ascii.txt) - ├── 📄 ascii.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" - ├── 📄 hello.txt # bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4 (raw, 12 bytes) "hello world\n" - └── 📄 multiblock.txt # bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa (dag-pb, multi-block) - ``` - - Purpose: Multiple directory entries pointing to the same content CID (deduplication) - - Validation: Both ascii.txt and ascii-copy.txt resolve to the same content block +- CID: `bafybeihchr7vmgjaasntayyatmp5sv6xza57iy2h4xj7g46bpjij6yhrmy` +- Type: [`dag-pb` Directory](#dag-pb-directory) +- Structure: +``` +📁 / # bafybeihchr7vmgjaasntayyatmp5sv6xza57iy2h4xj7g46bpjij6yhrmy +├── 🔗 ascii-copy.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (same CID as ascii.txt) +├── 📄 ascii.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" +├── 📄 hello.txt # bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4 (raw, 12 bytes) "hello world\n" +└── 📄 multiblock.txt # bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa (dag-pb, multi-block) +``` +- Purpose: Multiple directory entries pointing to the same content CID (deduplication) +- Validation: Both ascii.txt and ascii-copy.txt resolve to the same content block ### Invalid Test Cases @@ -916,13 +920,13 @@ These validate that implementations properly reject malformed or non-UnixFS dag- ## Additional Testing Resources - Gateway Conformance Suite: [ipfs/gateway-conformance](https://github.com/ipfs/gateway-conformance) - - Real-world test suite with UnixFS fixtures - - Tests gateway behaviors with various UnixFS structures - - Includes edge cases and performance scenarios +- Real-world test suite with UnixFS fixtures +- Tests gateway behaviors with various UnixFS structures +- Includes edge cases and performance scenarios - Test fixture generator: [go-fixtureplate](https://github.com/ipld/go-fixtureplate) - - Tool for generating custom test fixtures - - Includes UnixFS files and directories of arbitrary shapes +- Tool for generating custom test fixtures +- Includes UnixFS files and directories of arbitrary shapes Report specification issues or submit corrections via [ipfs/specs](https://github.com/ipfs/specs/issues). @@ -933,21 +937,21 @@ This section and included subsections are not authoritative. ## Popular Implementations - JavaScript - - [`@helia/unixfs`](https://www.npmjs.com/package/@helia/unixfs) implementation of a filesystem compatible with [Helia SDK](https://github.com/ipfs/helia#readme) - - Data Formats - [unixfs](https://github.com/ipfs/js-ipfs-unixfs) - - Importer - [unixfs-importer](https://github.com/ipfs/js-ipfs-unixfs/tree/main/packages/ipfs-unixfs-importer) - - Exporter - [unixfs-exporter](https://github.com/ipfs/js-ipfs-unixfs/tree/main/packages/ipfs-unixfs-exporter) +- [`@helia/unixfs`](https://www.npmjs.com/package/@helia/unixfs) implementation of a filesystem compatible with [Helia SDK](https://github.com/ipfs/helia#readme) +- Data Formats - [unixfs](https://github.com/ipfs/js-ipfs-unixfs) +- Importer - [unixfs-importer](https://github.com/ipfs/js-ipfs-unixfs/tree/main/packages/ipfs-unixfs-importer) +- Exporter - [unixfs-exporter](https://github.com/ipfs/js-ipfs-unixfs/tree/main/packages/ipfs-unixfs-exporter) - Go - - [Boxo SDK](https://github.com/ipfs/boxo#readme) includes implementation of UnixFS filesystem - - Protocol Buffer Definitions - [`ipfs/boxo/../unixfs.proto`](https://github.com/ipfs/boxo/blob/v0.23.0/ipld/unixfs/pb/unixfs.proto) - - [`ipfs/boxo/files`](https://github.com/ipfs/boxo/tree/main/files) - - [`ipfs/boxo/ipld/unixfs`](https://github.com/ipfs/boxo/tree/main/ipld/unixfs/) - - Alternative `go-ipld-prime` implementation: [`ipfs/go-unixfsnode`](https://github.com/ipfs/go-unixfsnode) +- [Boxo SDK](https://github.com/ipfs/boxo#readme) includes implementation of UnixFS filesystem +- Protocol Buffer Definitions - [`ipfs/boxo/../unixfs.proto`](https://github.com/ipfs/boxo/blob/v0.23.0/ipld/unixfs/pb/unixfs.proto) +- [`ipfs/boxo/files`](https://github.com/ipfs/boxo/tree/main/files) +- [`ipfs/boxo/ipld/unixfs`](https://github.com/ipfs/boxo/tree/main/ipld/unixfs/) +- Alternative `go-ipld-prime` implementation: [`ipfs/go-unixfsnode`](https://github.com/ipfs/go-unixfsnode) ## Simple `raw` Example @@ -965,14 +969,14 @@ Add the CID prefix: ``` f01551220 - 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 +9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 f this is the multibase prefix, we need it because we are working with a hex CID, this is omitted for binary CIDs - 01 the CID version, here one - 55 the codec, here we MUST use Raw because this is a Raw file - 12 the hashing function used, here sha256 - 20 the digest length 32 bytes - 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 is the the digest we computed earlier +01 the CID version, here one +55 the codec, here we MUST use Raw because this is a Raw file +12 the hashing function used, here sha256 +20 the digest length 32 bytes +9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 is the the digest we computed earlier ``` Done. Assuming we stored this block in some implementation of our choice, which makes it accessible to our client, we can try to decode it. @@ -989,12 +993,12 @@ it is a simple canonical one, python pseudo code to compute it looks like this: ```python def offsetlist(node): - unixfs = decodeDataField(node.Data) - if len(node.Links) != len(unixfs.Blocksizes): - raise "unmatched sister-lists" # error messages are implementation details +unixfs = decodeDataField(node.Data) +if len(node.Links) != len(unixfs.Blocksizes): +raise "unmatched sister-lists" # error messages are implementation details - cursor = len(unixfs.Data) if unixfs.Data else 0 - return [cursor] + [cursor := cursor + size for size in unixfs.Blocksizes[:-1]] +cursor = len(unixfs.Data) if unixfs.Data else 0 +return [cursor] + [cursor := cursor + size for size in unixfs.Blocksizes[:-1]] ``` This will tell you which offset inside this node the children at the corresponding index starts to cover. (using `[x,y)` ranging) @@ -1031,13 +1035,13 @@ but never the file data itself. This was ultimately rejected for a number of reasons: 1. You would always need to retrieve an additional node to access file data, which - limits the kind of optimizations that are possible. For example, many files are - under the 256 KiB block size limit, so we tend to inline them into the describing - UnixFS `File` node. This would not be possible with an intermediate `Metadata` node. +limits the kind of optimizations that are possible. For example, many files are +under the 256 KiB block size limit, so we tend to inline them into the describing +UnixFS `File` node. This would not be possible with an intermediate `Metadata` node. 2. The `File` node already contains some metadata (e.g. the file size), so metadata - would be stored in multiple places. This complicates forwards compatibility with - UnixFSv2, as mapping between metadata formats potentially requires multiple fetch - operations. +would be stored in multiple places. This complicates forwards compatibility with +UnixFSv2, as mapping between metadata formats potentially requires multiple fetch +operations. ### Pros and Cons: Metadata in the Directory @@ -1055,13 +1059,13 @@ both UnixFS v1 and v1.5 nodes. This was rejected for the following reasons: 1. When creating a UnixFS node, there's no way to record metadata without - wrapping it in a directory. +wrapping it in a directory. 2. If you access any UnixFS node directly by its [CID], there is no way of - recreating the metadata which limits flexibility. +recreating the metadata which limits flexibility. 3. In order to list the contents of a directory including entry types and - sizes, you have to fetch the root node of each entry, so the performance - benefit of including some metadata in the containing directory is negligible - in this use case. +sizes, you have to fetch the root node of each entry, so the performance +benefit of including some metadata in the containing directory is negligible +in this use case. ### Pros and Cons: Metadata in the File @@ -1076,15 +1080,15 @@ we decide to keep file data in a leaf node for deduplication reasons. Downsides to this approach are: 1. Two users adding the same file to IPFS at different times will have - different [CID]s due to the `mtime`s being different. If the content is - stored in another node, its [CID] will be constant between the two users, - but you can't navigate to it unless you have the parent node, which will be - less available due to the proliferation of [CID]s. +different [CID]s due to the `mtime`s being different. If the content is +stored in another node, its [CID] will be constant between the two users, +but you can't navigate to it unless you have the parent node, which will be +less available due to the proliferation of [CID]s. 2. Metadata is also impossible to remove without changing the [CID], so - metadata becomes part of the content. +metadata becomes part of the content. 3. Performance may be impacted as well as if we don't inline UnixFS root nodes - into [CID]s, so additional fetches will be required to load a given UnixFS - entry. +into [CID]s, so additional fetches will be required to load a given UnixFS +entry. ### Pros and Cons: Metadata in Side Trees From 53979a8914ebaf5bd46ea8ab63061f81ae2aa1ea Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sat, 23 Aug 2025 23:33:27 +0200 Subject: [PATCH 21/46] chore: update date to 2025-08-23 --- src/unixfs-data-format.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/unixfs-data-format.md b/src/unixfs-data-format.md index c870f4bbb..8119bec70 100644 --- a/src/unixfs-data-format.md +++ b/src/unixfs-data-format.md @@ -3,7 +3,7 @@ title: UnixFS description: > UnixFS is a Protocol Buffers-based format for describing files, directories, and symlinks as dag-pb and raw DAGs in IPFS. -date: 2025-03-01 +date: 2025-08-23 maturity: draft editors: - name: Marcin Rataj From e4cb33f95eb50d738496bfe0708d2096ec4fd68d Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sun, 24 Aug 2025 00:37:39 +0200 Subject: [PATCH 22/46] fix: restore indentation removed by commit 523cb6a MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - fix yaml frontmatter indentation - fix protobuf and json block indentation - fix nested list indentation - fix list numbering (4→1, 3→1) - fix multiple consecutive blank lines - add protobuf reference in dag-pb section - update protobuf link to protobuf.dev - add blanks-around-fences exception to markdownlint config --- .markdownlint.json | 3 +- src/unixfs-data-format.md | 850 +++++++++++++++++++------------------- 2 files changed, 426 insertions(+), 427 deletions(-) diff --git a/.markdownlint.json b/.markdownlint.json index 4197e44b0..fb66d6e5c 100644 --- a/.markdownlint.json +++ b/.markdownlint.json @@ -8,5 +8,6 @@ "blanks-around-lists": false, "single-trailing-newline": false, "link-fragments": false, - "line-length": false + "line-length": false, + "blanks-around-fences": false } diff --git a/src/unixfs-data-format.md b/src/unixfs-data-format.md index 8119bec70..b31159e31 100644 --- a/src/unixfs-data-format.md +++ b/src/unixfs-data-format.md @@ -1,46 +1,46 @@ --- title: UnixFS description: > -UnixFS is a Protocol Buffers-based format for describing files, directories, -and symlinks as dag-pb and raw DAGs in IPFS. + UnixFS is a Protocol Buffers-based format for describing files, directories, + and symlinks as dag-pb and raw DAGs in IPFS. date: 2025-08-23 maturity: draft editors: -- name: Marcin Rataj -github: lidel -affiliation: -name: Interplanetary Shipyard -url: https://ipshipyard.com/ + - name: Marcin Rataj + github: lidel + affiliation: + name: Interplanetary Shipyard + url: https://ipshipyard.com/ contributors: -- name: Hugo Valtier -github: jorropo -affiliation: -name: Interplanetary Shipyard -url: https://ipshipyard.com/ + - name: Hugo Valtier + github: jorropo + affiliation: + name: Interplanetary Shipyard + url: https://ipshipyard.com/ thanks: -- name: Jeromy Johnson -github: whyrusleeping -- name: Steven Allen -github: Stebalien -- name: Hector Sanjuan -github: hsanjuan -affiliation: -name: Interplanetary Shipyard -url: https://ipshipyard.com/ -- name: Łukasz Magiera -github: magik6k -- name: Alex Potsides -github: achingbrain -affiliation: -name: Interplanetary Shipyard -url: https://ipshipyard.com/ -- name: Peter Rabbitson -github: ribasushi -- name: Henrique Dias -github: hacdias -affiliation: -name: Interplanetary Shipyard -url: https://ipshipyard.com/ + - name: Jeromy Johnson + github: whyrusleeping + - name: Steven Allen + github: Stebalien + - name: Hector Sanjuan + github: hsanjuan + affiliation: + name: Interplanetary Shipyard + url: https://ipshipyard.com/ + - name: Łukasz Magiera + github: magik6k + - name: Alex Potsides + github: achingbrain + affiliation: + name: Interplanetary Shipyard + url: https://ipshipyard.com/ + - name: Peter Rabbitson + github: ribasushi + - name: Henrique Dias + github: hacdias + affiliation: + name: Interplanetary Shipyard + url: https://ipshipyard.com/ tags: ['data-formats'] order: 1 @@ -57,7 +57,7 @@ required. A [CID] includes two important pieces of information: 1. A [multicodec], simply known as a codec. 2. A [multihash] used to specify the hashing algorithm, the hash parameters and -the hash digest. + the hash digest. Thus, the block must be retrieved; that is, the bytes which ,when hashed using the hash function specified in the multihash, gives us the same multihash value back. @@ -85,27 +85,27 @@ be recognized because their CIDs are encoded using the `raw` (`0x55`) codec: # `dag-pb` Node More complex nodes use the `dag-pb` (`0x70`) encoding. These nodes require two steps of -decoding. The first step is to decode the outer container of the block. This is encoded using the [`dag-pb`][ipld-dag-pb] specification, which can be +decoding. The first step is to decode the outer container of the block. This is encoded using the [`dag-pb`][ipld-dag-pb] specification, which uses [Protocol Buffers][protobuf] and can be summarized as follows: ```protobuf message PBLink { -// binary CID (with no multibase prefix) of the target object -optional bytes Hash = 1; + // binary CID (with no multibase prefix) of the target object + optional bytes Hash = 1; -// UTF-8 string name -optional string Name = 2; + // UTF-8 string name + optional string Name = 2; -// cumulative size of target object -optional uint64 Tsize = 3; + // cumulative size of target object + optional uint64 Tsize = 3; } message PBNode { -// refs to other objects -repeated PBLink Links = 2; + // refs to other objects + repeated PBLink Links = 2; -// opaque user data -optional bytes Data = 1; + // opaque user data + optional bytes Data = 1; } ``` @@ -115,32 +115,32 @@ a protobuf message specified in the UnixFSV1 format: ```protobuf message Data { -enum DataType { -Raw = 0; // deprecated, use raw codec blocks without dag-pb instead -Directory = 1; -File = 2; -Metadata = 3; // reserved for future use -Symlink = 4; -HAMTShard = 5; -} - -required DataType Type = 1; -optional bytes Data = 2; -optional uint64 filesize = 3; -repeated uint64 blocksizes = 4; -optional uint64 hashType = 5; -optional uint64 fanout = 6; -optional uint32 mode = 7; // opt-in, AKA UnixFS 1.5 -optional UnixTime mtime = 8; // opt-in, AKA UnixFS 1.5 + enum DataType { + Raw = 0; // deprecated, use raw codec blocks without dag-pb instead + Directory = 1; + File = 2; + Metadata = 3; // reserved for future use + Symlink = 4; + HAMTShard = 5; + } + + required DataType Type = 1; + optional bytes Data = 2; + optional uint64 filesize = 3; + repeated uint64 blocksizes = 4; + optional uint64 hashType = 5; + optional uint64 fanout = 6; + optional uint32 mode = 7; // opt-in, AKA UnixFS 1.5 + optional UnixTime mtime = 8; // opt-in, AKA UnixFS 1.5 } message Metadata { -optional string MimeType = 1; + optional string MimeType = 1; } message UnixTime { -required int64 Seconds = 1; -optional fixed32 FractionalNanoseconds = 2; + required int64 Seconds = 1; + optional fixed32 FractionalNanoseconds = 2; } ``` @@ -174,11 +174,11 @@ For example, consider this pseudo-json block: ```json { -"Links": [{"Hash":"Qmfoo"}, {"Hash":"Qmbar"}], -"Data": { -"Type": "File", -"blocksizes": [20, 30] -} + "Links": [{"Hash":"Qmfoo"}, {"Hash":"Qmbar"}], + "Data": { + "Type": "File", + "blocksizes": [20, 30] + } } ``` @@ -253,18 +253,18 @@ file MUST error. A :dfn[Directory], also known as folder, is a named collection of child [Nodes](#dag-pb-node): - Every link in `PBNode.Links` is an entry (child) of the directory, and -`PBNode.Links[].Name` gives you the name of that child. + `PBNode.Links[].Name` gives you the name of that child. - Duplicate names are not allowed. Therefore, two elements of `PBNode.Link` CANNOT -have the same `Name`. If two identical names are present in a directory, the -decoder MUST fail. + have the same `Name`. If two identical names are present in a directory, the + decoder MUST fail. - Implementations SHOULD detect when directory becomes too big to fit in a single -`Directory` block and use [`HAMTDirectory`] type instead. + `Directory` block and use [`HAMTDirectory`] type instead. The minimum valid `PBNode.Data` field for a directory is as follows: ```json { -"Type": "Directory" + "Type": "Directory" } ``` @@ -308,17 +308,17 @@ The HAMT directory is configured through the UnixFS metadata in `PBNode.Data`: - `decode(PBNode.Data).Type` MUST be `HAMTShard` (value `5`) - `decode(PBNode.Data).hashType` indicates the [multihash] function to use to digest -the path components for sharding. Currently, all HAMT implementations use `murmur3-x64-64` (`0x22`), -and this value MUST be consistent across all shards within the same HAMT structure + the path components for sharding. Currently, all HAMT implementations use `murmur3-x64-64` (`0x22`), + and this value MUST be consistent across all shards within the same HAMT structure - `decode(PBNode.Data).fanout` MUST be a power of two. This determines the number -of possible bucket indices (permutations) at each level of the trie. For example, -fanout=256 provides 256 possible buckets (0x00 to 0xFF), requiring 8 bits from the hash. -The hex prefix length is `log2(fanout)/4` characters (since each hex character represents 4 bits). -The same fanout value is used throughout all levels of a single HAMT structure. -Implementations choose fanout based on their specific trade-offs between tree depth and node size + of possible bucket indices (permutations) at each level of the trie. For example, + fanout=256 provides 256 possible buckets (0x00 to 0xFF), requiring 8 bits from the hash. + The hex prefix length is `log2(fanout)/4` characters (since each hex character represents 4 bits). + The same fanout value is used throughout all levels of a single HAMT structure. + Implementations choose fanout based on their specific trade-offs between tree depth and node size - `decode(PBNode.Data).Data` is a bitmap field indicating which buckets contain entries. -Each bit represents one bucket. While included in the protobuf, implementations -typically derive bucket occupancy from the link names directly + Each bit represents one bucket. While included in the protobuf, implementations + typically derive bucket occupancy from the link names directly The field `Name` of an element of `PBNode.Links` for a HAMT uses a hex-encoded prefix corresponding to the bucket index, zero-padded to a width @@ -334,51 +334,51 @@ To illustrate the HAMT structure with a concrete example: // Root HAMT shard (bafybeidbclfqleg2uojchspzd4bob56dqetqjsj27gy2cq3klkkgxtpn4i) // This shard contains 1000 files distributed across buckets message PBNode { -// UnixFS metadata in Data field -Data = { -Type = HAMTShard // Type = 5 -Data = 0xffffff... // Bitmap: bits set for populated buckets -hashType = 0x22 // murmur3-x64-64 -fanout = 256 // 256 buckets (8-bit width) -} - -// Links to sub-shards or entries -Links = [ -{ -Hash = bafybeiaebmuestgbpqhkkbrwl2qtjtvs3whkmp2trkbkimuod4yv7oygni -Name = "00" // Bucket 0x00 -Tsize = 2693 // Cumulative size of this subtree -}, -{ -Hash = bafybeia322onepwqofne3l3ptwltzns52fgapeauhmyynvoojmcvchxptu -Name = "01" // Bucket 0x01 -Tsize = 7977 -}, -// ... more buckets as needed up to "FF" -] + // UnixFS metadata in Data field + Data = { + Type = HAMTShard // Type = 5 + Data = 0xffffff... // Bitmap: bits set for populated buckets + hashType = 0x22 // murmur3-x64-64 + fanout = 256 // 256 buckets (8-bit width) + } + + // Links to sub-shards or entries + Links = [ + { + Hash = bafybeiaebmuestgbpqhkkbrwl2qtjtvs3whkmp2trkbkimuod4yv7oygni + Name = "00" // Bucket 0x00 + Tsize = 2693 // Cumulative size of this subtree + }, + { + Hash = bafybeia322onepwqofne3l3ptwltzns52fgapeauhmyynvoojmcvchxptu + Name = "01" // Bucket 0x01 + Tsize = 7977 + }, + // ... more buckets as needed up to "FF" + ] } // Sub-shard for bucket "00" (multiple files hash to 00 at first level) message PBNode { -Data = { -Type = HAMTShard // Still a HAMT at second level -Data = 0x800000... // Bitmap for this sub-level -hashType = 0x22 // murmur3-x64-64 -fanout = 256 // Same fanout throughout -} - -Links = [ -{ -Hash = bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa -Name = "6E470.txt" // Bucket 0x6E + filename -Tsize = 1271 -}, -{ -Hash = bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa -Name = "FF742.txt" // Bucket 0xFF + filename -Tsize = 1271 -} -] + Data = { + Type = HAMTShard // Still a HAMT at second level + Data = 0x800000... // Bitmap for this sub-level + hashType = 0x22 // murmur3-x64-64 + fanout = 256 // Same fanout throughout + } + + Links = [ + { + Hash = bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa + Name = "6E470.txt" // Bucket 0x6E + filename + Tsize = 1271 + }, + { + Hash = bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa + Name = "FF742.txt" // Bucket 0xFF + filename + Tsize = 1271 + } + ] } ``` @@ -388,11 +388,11 @@ To resolve a path inside a HAMT: 1. Hash the filename using the hash function specified in `decode(PBNode.Data).hashType` 2. Pop `log2(fanout)` bits from the hash digest (lowest/least significant bits first), -then hex encode those bits using little endian to form the bucket prefix. The prefix MUST use uppercase hex characters (00-FF, not 00-ff) + then hex encode those bits using little endian to form the bucket prefix. The prefix MUST use uppercase hex characters (00-FF, not 00-ff) 3. Find the link whose `Name` starts with this hex prefix: -- If `Name` equals the prefix exactly → this is a sub-shard, follow the link and repeat from step 2 -- If `Name` equals prefix + filename → target found -- If no matching prefix → file not in directory + - If `Name` equals the prefix exactly → this is a sub-shard, follow the link and repeat from step 2 + - If `Name` equals prefix + filename → target found + - If no matching prefix → file not in directory 4. When following to a sub-shard, continue consuming bits from the same hash Note: Empty intermediate shards are typically collapsed during deletion operations to maintain consistency @@ -405,14 +405,14 @@ Given a HAMT-sharded directory containing 1000 files: 1. Hash the filename "470.txt" using murmur3-x64-64 (multihash `0x22`) 2. With fanout=256, we consume 8 bits at a time from the hash: -- First 8 bits determine root bucket → `0x00` → link name "00" -- Follow link "00" to sub-shard (`bafybeiaebmuestgbpqhkkbrwl2qtjtvs3whkmp2trkbkimuod4yv7oygni`) + - First 8 bits determine root bucket → `0x00` → link name "00" + - Follow link "00" to sub-shard (`bafybeiaebmuestgbpqhkkbrwl2qtjtvs3whkmp2trkbkimuod4yv7oygni`) 3. The sub-shard is also a HAMT (has Type=HAMTShard): -- Next 8 bits from hash → `0x6E` -- Find entry with name "6E470.txt" (prefix + original filename) + - Next 8 bits from hash → `0x6E` + - Find entry with name "6E470.txt" (prefix + original filename) 4. Link name format at leaf level: `[hex_prefix][original_filename]` -- "6E470.txt" means: file "470.txt" that hashed to bucket 6E at this level -- "FF742.txt" means: file "742.txt" that hashed to bucket FF at this level + - "6E470.txt" means: file "470.txt" that hashed to bucket 6E at this level + - "FF742.txt" means: file "742.txt" that hashed to bucket FF at this level ::: ### `dag-pb` `Symlink` @@ -500,13 +500,13 @@ The `mode` is for persisting the file permissions in [numeric notation](https:// \[[spec](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html)\]. - If unspecified, implementations MAY default to -- `0755` for directories/HAMT shards -- `0644` for all other types where applicable + - `0755` for directories/HAMT shards + - `0644` for all other types where applicable - The nine least significant bits represent `ugo-rwx` - The next three least significant bits represent `setuid`, `setgid` and the `sticky bit` - The remaining 20 bits are reserved for future use, and are subject to change. Spec implementations **MUST** handle bits they do not expect as follows: -- For future-proofing, the (de)serialization layer must preserve the entire uint32 value during clone/copy operations, modifying only bit values that have a well defined meaning: `clonedValue = ( modifiedBits & 07777 ) | ( originalValue & 0xFFFFF000 )` -- Implementations of this spec must proactively mask off bits without a defined meaning in the implemented version of the spec: `interpretedValue = originalValue & 07777` + - For future-proofing, the (de)serialization layer must preserve the entire uint32 value during clone/copy operations, modifying only bit values that have a well defined meaning: `clonedValue = ( modifiedBits & 07777 ) | ( originalValue & 0xFFFFF000 )` + - Implementations of this spec must proactively mask off bits without a defined meaning in the implemented version of the spec: `interpretedValue = originalValue & 07777` #### `mtime` Field @@ -520,29 +520,29 @@ The two fields are: Implementations encoding or decoding wire-representations MUST observe the following: - An `mtime` structure with `FractionalNanoseconds` outside of the on-wire range -`[1, 999999999]` is **not** valid. This includes a fractional value of `0`. -Implementations encountering such values should consider the entire enclosing -metadata block malformed and abort the processing of the corresponding DAG. + `[1, 999999999]` is **not** valid. This includes a fractional value of `0`. + Implementations encountering such values should consider the entire enclosing + metadata block malformed and abort the processing of the corresponding DAG. - The `mtime` structure is optional. Its absence implies `unspecified` rather -than `0`. + than `0`. - For ergonomic reasons, a surface API of an encoder MUST allow fractional `0` as -input, while at the same time MUST ensure it is stripped from the final structure -before encoding, satisfying the above constraints. + input, while at the same time MUST ensure it is stripped from the final structure + before encoding, satisfying the above constraints. Implementations interpreting the `mtime` metadata in order to apply it within a non-IPFS target MUST observe the following: - If the target supports a distinction between `unspecified` and `0`/`1970-01-01T00:00:00Z`, -the distinction must be preserved within the target. For example, if no `mtime` structure -is available, a web gateway must **not** render a `Last-Modified:` header. + the distinction must be preserved within the target. For example, if no `mtime` structure + is available, a web gateway must **not** render a `Last-Modified:` header. - If the target requires an `mtime` ( e.g. a FUSE interface ) and no `mtime` is -supplied OR the supplied `mtime` falls outside of the targets accepted range: -- When no `mtime` is specified or the resulting `UnixTime` is negative: -implementations must assume `0`/`1970-01-01T00:00:00Z` (note that such values -are not merely academic: e.g. the OpenVMS epoch is `1858-11-17T00:00:00Z`) -- When the resulting `UnixTime` is larger than the targets range ( e.g. 32bit -vs 64bit mismatch), implementations must assume the highest possible value -in the targets range. In most cases, this would be `2038-01-19T03:14:07Z`. + supplied OR the supplied `mtime` falls outside of the targets accepted range: + - When no `mtime` is specified or the resulting `UnixTime` is negative: + implementations must assume `0`/`1970-01-01T00:00:00Z` (note that such values + are not merely academic: e.g. the OpenVMS epoch is `1858-11-17T00:00:00Z`) + - When the resulting `UnixTime` is larger than the targets range ( e.g. 32bit + vs 64bit mismatch), implementations must assume the highest possible value + in the targets range. In most cases, this would be `2038-01-19T03:14:07Z`. ## UnixFS Paths @@ -556,7 +556,7 @@ separated by `/` (`0x2F`). UnixFS paths read from left to right, and are inspired by POSIX paths. - Components MUST NOT contain `/` unicode codepoints because it would break -the path into two components. + the path into two components. - Components SHOULD be UTF8 unicode. - Components are case-sensitive. @@ -580,11 +580,11 @@ Relative path components MUST be resolved before trying to work on the path: - `.` points to the current node and MUST be removed. - `..` points to the parent node and MUST be removed left to right. When removing -a `..`, the path component on the left MUST also be removed. If there is no path -component on the left, you MUST error to avoid out-of-bounds -path resolution. + a `..`, the path component on the left MUST also be removed. If there is no path + component on the left, you MUST error to avoid out-of-bounds + path resolution. - Implementations MUST error when resolving a relative path that attempts to go -beyond the root CID (example: `/ipfs/cid/../foo`). + beyond the root CID (example: `/ipfs/cid/../foo`). ### Restricted Names @@ -594,8 +594,8 @@ The following names SHOULD NOT be used: - The `..` string, as it represents the parent node in POSIX pathing. - The empty string. - Any string containing a `NULL` (`0x00`) byte, as this is often used to signify string -terminations in some systems, such as C-compatible systems. Many unix -file systems do not accept this character in path components. + terminations in some systems, such as C-compatible systems. Many unix + file systems do not accept this character in path components. # Appendix: Test Vectors @@ -612,15 +612,15 @@ Test vectors for UnixFS file structures, progressing from simple single-block fi ### Single `raw` Block File - Fixture: [`dir-with-files.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_unixfs/dir-with-files.car) -- CID: `bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4` (hello.txt) -- Type: [`raw` Node](#raw-node) -- Content: "hello world\n" (12 bytes) -- Block Analysis: -- Block size (`ipfs block stat`): 12 bytes -- Data size (`ipfs cat`): 12 bytes -- DAG-PB envelope: N/A (raw blocks have no envelope overhead) -- Purpose: Single block using `raw` codec, no protobuf wrapper -- Validation: Block content IS the file content, no UnixFS metadata + - CID: `bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4` (hello.txt) + - Type: [`raw` Node](#raw-node) + - Content: "hello world\n" (12 bytes) + - Block Analysis: + - Block size (`ipfs block stat`): 12 bytes + - Data size (`ipfs cat`): 12 bytes + - DAG-PB envelope: N/A (raw blocks have no envelope overhead) + - Purpose: Single block using `raw` codec, no protobuf wrapper + - Validation: Block content IS the file content, no UnixFS metadata ### Single `dag-pb` Block File @@ -629,89 +629,89 @@ Test vectors for UnixFS file structures, progressing from simple single-block fi - Type: [`dag-pb` File](#dag-pb-file) with data in the same block - Content: "Hello from IPFS Gateway Checker\n" (32 bytes) - Block Analysis: -- Block size (`ipfs block stat`): 40 bytes -- Data size (`ipfs cat`): 32 bytes -- DAG-PB envelope: 8 bytes (40 - 32) + - Block size (`ipfs block stat`): 40 bytes + - Data size (`ipfs cat`): 32 bytes + - DAG-PB envelope: 8 bytes (40 - 32) - Structure: -``` -📄 small-file.txt # bafybeifx7yeb55armcsxwwitkymga5xf53dxiarykms3ygqic223w5sk3m (dag-pb) -└── 📦 Data.Data # "Hello from IPFS Gateway Checker\n" (32 bytes, stored inline in UnixFS protobuf) -``` + ``` + 📄 small-file.txt # bafybeifx7yeb55armcsxwwitkymga5xf53dxiarykms3ygqic223w5sk3m (dag-pb) + └── 📦 Data.Data # "Hello from IPFS Gateway Checker\n" (32 bytes, stored inline in UnixFS protobuf) + ``` - Purpose: Small file stored within dag-pb Data field - Validation: File content extracted from UnixFS Data.Data field ### Multi-block File - Fixture: [`dir-with-files.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_unixfs/dir-with-files.car) -- CID: `bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa` (multiblock.txt) -- Type: [`dag-pb` File](#dag-pb-file) with multiple [`raw` Node](#raw-node) leaves -- Content: Lorem ipsum text (1026 bytes total) -- Block Analysis: -- Root block size (`ipfs block stat`): 245 bytes (dag-pb) -- Total data size (`ipfs cat`): 1026 bytes -- Child blocks: -- Block 1: 256 bytes (raw) -- Block 2: 256 bytes (raw) -- Block 3: 256 bytes (raw) -- Block 4: 256 bytes (raw) -- Block 5: 2 bytes (raw) -- DAG-PB envelope: 245 bytes (root block containing metadata + links) -- Structure: -``` -📄 multiblock.txt # bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa (dag-pb root) -├── 📦 [0-255] # bafkreie5noke3mb7hqxukzcy73nl23k6lxszxi5w3dtmuwz62wnvkpsscm (raw, 256 bytes) -├── 📦 [256-511] # bafkreih4ephajybraj6wnxsbwjwa77fukurtpl7oj7t7pfq545duhot7cq (raw, 256 bytes) -├── 📦 [512-767] # bafkreigu7buvm3cfunb35766dn7tmqyh2um62zcio63en2btvxuybgcpue (raw, 256 bytes) -├── 📦 [768-1023] # bafkreicll3huefkc3qnrzeony7zcfo7cr3nbx64hnxrqzsixpceg332fhe (raw, 256 bytes) -└── 📦 [1024-1025] # bafkreifst3pqztuvj57lycamoi7z34b4emf7gawxs74nwrc2c7jncmpaqm (raw, 2 bytes) -``` -- Purpose: File chunking and reassembly -- Validation: -- Links have no Names (must be absent) -- Blocksizes array matches Links array length -- Reassembled content matches original + - CID: `bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa` (multiblock.txt) + - Type: [`dag-pb` File](#dag-pb-file) with multiple [`raw` Node](#raw-node) leaves + - Content: Lorem ipsum text (1026 bytes total) + - Block Analysis: + - Root block size (`ipfs block stat`): 245 bytes (dag-pb) + - Total data size (`ipfs cat`): 1026 bytes + - Child blocks: + - Block 1: 256 bytes (raw) + - Block 2: 256 bytes (raw) + - Block 3: 256 bytes (raw) + - Block 4: 256 bytes (raw) + - Block 5: 2 bytes (raw) + - DAG-PB envelope: 245 bytes (root block containing metadata + links) + - Structure: + ``` + 📄 multiblock.txt # bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa (dag-pb root) + ├── 📦 [0-255] # bafkreie5noke3mb7hqxukzcy73nl23k6lxszxi5w3dtmuwz62wnvkpsscm (raw, 256 bytes) + ├── 📦 [256-511] # bafkreih4ephajybraj6wnxsbwjwa77fukurtpl7oj7t7pfq545duhot7cq (raw, 256 bytes) + ├── 📦 [512-767] # bafkreigu7buvm3cfunb35766dn7tmqyh2um62zcio63en2btvxuybgcpue (raw, 256 bytes) + ├── 📦 [768-1023] # bafkreicll3huefkc3qnrzeony7zcfo7cr3nbx64hnxrqzsixpceg332fhe (raw, 256 bytes) + └── 📦 [1024-1025] # bafkreifst3pqztuvj57lycamoi7z34b4emf7gawxs74nwrc2c7jncmpaqm (raw, 2 bytes) + ``` + - Purpose: File chunking and reassembly + - Validation: + - Links have no Names (must be absent) + - Blocksizes array matches Links array length + - Reassembled content matches original ### File with Missing Blocks - Fixture: [`bafybeibfhhww5bpsu34qs7nz25wp7ve36mcc5mxd5du26sr45bbnjhpkei.dag-pb`](https://github.com/ipld/codec-fixtures/raw/381e762b85862b2bbdb6ef2ba140b3c505e31a44/fixtures/dagpb_7unnamedlinks%2Bdata/bafybeibfhhww5bpsu34qs7nz25wp7ve36mcc5mxd5du26sr45bbnjhpkei.dag-pb) -- CID: `bafybeibfhhww5bpsu34qs7nz25wp7ve36mcc5mxd5du26sr45bbnjhpkei` -- Type: [`dag-pb` File](#dag-pb-file) with 7 links to child blocks -- Size: 306MB total (from metadata) -- Structure: -``` -📄 large-file # bafybeibfhhww5bpsu34qs7nz25wp7ve36mcc5mxd5du26sr45bbnjhpkei (dag-pb root) -├── ⚠️ block[0] # (missing child block) -├── ⚠️ block[1] # (missing child block) -├── ⚠️ block[2] # (missing child block) -├── ⚠️ block[3] # (missing child block) -├── ⚠️ block[4] # (missing child block) -├── ⚠️ block[5] # (missing child block) -└── ⚠️ block[6] # (missing child block) -``` -- Note: Child blocks are NOT included - they may be unavailable locally or missing entirely -- Purpose: -- Reading UnixFS file metadata should require only the root block -- File size and structure can be determined without fetching children -- Operations should not block waiting for child blocks unless content is actually requested -- Validation: Can extract file size and chunking info from root block alone + - CID: `bafybeibfhhww5bpsu34qs7nz25wp7ve36mcc5mxd5du26sr45bbnjhpkei` + - Type: [`dag-pb` File](#dag-pb-file) with 7 links to child blocks + - Size: 306MB total (from metadata) + - Structure: + ``` + 📄 large-file # bafybeibfhhww5bpsu34qs7nz25wp7ve36mcc5mxd5du26sr45bbnjhpkei (dag-pb root) + ├── ⚠️ block[0] # (missing child block) + ├── ⚠️ block[1] # (missing child block) + ├── ⚠️ block[2] # (missing child block) + ├── ⚠️ block[3] # (missing child block) + ├── ⚠️ block[4] # (missing child block) + ├── ⚠️ block[5] # (missing child block) + └── ⚠️ block[6] # (missing child block) + ``` + - Note: Child blocks are NOT included - they may be unavailable locally or missing entirely + - Purpose: + - Reading UnixFS file metadata should require only the root block + - File size and structure can be determined without fetching children + - Operations should not block waiting for child blocks unless content is actually requested + - Validation: Can extract file size and chunking info from root block alone ### Range Requests with Missing Blocks - Fixture: [`file-3k-and-3-blocks-missing-block.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/trustless_gateway_car/file-3k-and-3-blocks-missing-block.car) -- CID: `QmYhmPjhFjYFyaoiuNzYv8WGavpSRDwdHWe5B4M5du5Rtk` -- Type: [`dag-pb` File](#dag-pb-file) with 3 links but middle block intentionally missing -- Structure: -``` -📄 file-3k # QmYhmPjhFjYFyaoiuNzYv8WGavpSRDwdHWe5B4M5du5Rtk (dag-pb root) -├── 📦 [0-1023] # QmPKt7ptM2ZYSGPUc8PmPT2VBkLDK3iqpG9TBJY7PCE9rF (raw, 1024 bytes) -├── ⚠️ [1024-2047] # (missing block - intentionally removed) -└── 📦 [2048-3071] # QmWXY482zQdwecnfBsj78poUUuPXvyw2JAFAEMw4tzTavV (raw, 1024 bytes) -``` -- Critical requirement: Must support seeking without all blocks available -- Purpose: -- Fetch only required blocks for byte range requests (e.g., bytes=0-1023 or bytes=2048-3071) -- Gateway conformance tests verify that first block (`QmPKt7ptM2ZYSGPUc8PmPT2VBkLDK3iqpG9TBJY7PCE9rF`) and third block (`QmWXY482zQdwecnfBsj78poUUuPXvyw2JAFAEMw4tzTavV`) can be fetched independently -- Requests for middle block or byte ranges requiring it should fail gracefully + - CID: `QmYhmPjhFjYFyaoiuNzYv8WGavpSRDwdHWe5B4M5du5Rtk` + - Type: [`dag-pb` File](#dag-pb-file) with 3 links but middle block intentionally missing + - Structure: + ``` + 📄 file-3k # QmYhmPjhFjYFyaoiuNzYv8WGavpSRDwdHWe5B4M5du5Rtk (dag-pb root) + ├── 📦 [0-1023] # QmPKt7ptM2ZYSGPUc8PmPT2VBkLDK3iqpG9TBJY7PCE9rF (raw, 1024 bytes) + ├── ⚠️ [1024-2047] # (missing block - intentionally removed) + └── 📦 [2048-3071] # QmWXY482zQdwecnfBsj78poUUuPXvyw2JAFAEMw4tzTavV (raw, 1024 bytes) + ``` + - Critical requirement: Must support seeking without all blocks available + - Purpose: + - Fetch only required blocks for byte range requests (e.g., bytes=0-1023 or bytes=2048-3071) + - Gateway conformance tests verify that first block (`QmPKt7ptM2ZYSGPUc8PmPT2VBkLDK3iqpG9TBJY7PCE9rF`) and third block (`QmWXY482zQdwecnfBsj78poUUuPXvyw2JAFAEMw4tzTavV`) can be fetched independently + - Requests for middle block or byte ranges requiring it should fail gracefully ## Directory Test Vectors @@ -720,126 +720,126 @@ Test vectors for UnixFS directory structures, progressing from simple flat direc ### Simple Directory - Fixture: [`dir-with-files.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_unixfs/dir-with-files.car) -- CID: `bafybeihchr7vmgjaasntayyatmp5sv6xza57iy2h4xj7g46bpjij6yhrmy` -- Type: [`dag-pb` Directory](#dag-pb-directory) -- Block Analysis: -- Directory block size (`ipfs block stat`): 185 bytes -- Contains UnixFS Type=Directory metadata + 4 links -- Structure: -``` -📁 / # bafybeihchr7vmgjaasntayyatmp5sv6xza57iy2h4xj7g46bpjij6yhrmy -├── 📄 ascii-copy.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" -├── 📄 ascii.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" -├── 📄 hello.txt # bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4 (raw, 12 bytes) "hello world\n" -└── 📄 multiblock.txt # bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa (dag-pb, 1026 bytes) Lorem ipsum text -``` -- Purpose: Directory listing, link sorting, deduplication (ascii.txt and ascii-copy.txt share same CID) -- Validation: Links sorted lexicographically by Name, each has valid Tsize + - CID: `bafybeihchr7vmgjaasntayyatmp5sv6xza57iy2h4xj7g46bpjij6yhrmy` + - Type: [`dag-pb` Directory](#dag-pb-directory) + - Block Analysis: + - Directory block size (`ipfs block stat`): 185 bytes + - Contains UnixFS Type=Directory metadata + 4 links + - Structure: + ``` + 📁 / # bafybeihchr7vmgjaasntayyatmp5sv6xza57iy2h4xj7g46bpjij6yhrmy + ├── 📄 ascii-copy.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" + ├── 📄 ascii.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" + ├── 📄 hello.txt # bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4 (raw, 12 bytes) "hello world\n" + └── 📄 multiblock.txt # bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa (dag-pb, 1026 bytes) Lorem ipsum text + ``` + - Purpose: Directory listing, link sorting, deduplication (ascii.txt and ascii-copy.txt share same CID) + - Validation: Links sorted lexicographically by Name, each has valid Tsize ### Nested Directories - Fixture: [`subdir-with-two-single-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/trustless_gateway_car/subdir-with-two-single-block-files.car) -- CID: `bafybeietjm63oynimmv5yyqay33nui4y4wx6u3peezwetxgiwvfmelutzu` -- Type: [`dag-pb` Directory](#dag-pb-directory) containing another Directory -- Block Analysis: -- Root directory block size: 55 bytes -- Subdirectory block size: 110 bytes -- Structure: -``` -📁 / # bafybeietjm63oynimmv5yyqay33nui4y4wx6u3peezwetxgiwvfmelutzu -└── 📁 subdir/ # bafybeiggghzz6dlue3m6nb2dttnbrygxh3lrjl5764f2m4gq7dgzdt55o4 (dag-pb Directory) -├── 📄 ascii.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" -└── 📄 hello.txt # bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4 (raw, 12 bytes) "hello world\n" -``` -- Purpose: Path traversal through directory hierarchy -- Validation: Can traverse `/subdir/hello.txt` path correctly + - CID: `bafybeietjm63oynimmv5yyqay33nui4y4wx6u3peezwetxgiwvfmelutzu` + - Type: [`dag-pb` Directory](#dag-pb-directory) containing another Directory + - Block Analysis: + - Root directory block size: 55 bytes + - Subdirectory block size: 110 bytes + - Structure: + ``` + 📁 / # bafybeietjm63oynimmv5yyqay33nui4y4wx6u3peezwetxgiwvfmelutzu + └── 📁 subdir/ # bafybeiggghzz6dlue3m6nb2dttnbrygxh3lrjl5764f2m4gq7dgzdt55o4 (dag-pb Directory) + ├── 📄 ascii.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" + └── 📄 hello.txt # bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4 (raw, 12 bytes) "hello world\n" + ``` + - Purpose: Path traversal through directory hierarchy + - Validation: Can traverse `/subdir/hello.txt` path correctly - Fixture: [`dag-pb.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_dag/dag-pb.car) -- CID: `bafybeiegxwlgmoh2cny7qlolykdf7aq7g6dlommarldrbm7c4hbckhfcke` -- Type: [`dag-pb` Directory](#dag-pb-directory) -- Structure: -``` -📁 / # bafybeiegxwlgmoh2cny7qlolykdf7aq7g6dlommarldrbm7c4hbckhfcke -├── 📁 foo/ # bafybeidryarwh34ygbtyypbu7qjkl4euiwxby6cql6uvosonohkq2kwnkm (dag-pb Directory) -│ └── 📄 bar.txt # bafkreigzafgemjeejks3vqyuo46ww2e22rt7utq5djikdofjtvnjl5zp6u (raw, 14 bytes) "Hello, world!" -└── 📄 foo.txt # bafkreic3ondyhizrzeoufvoodehinugpj3ecruwokaygl7elezhn2khqfa (raw, 13 bytes) "Hello, IPFS!" -``` -- Purpose: Another example of standard UnixFS directory with raw leaf blocks + - CID: `bafybeiegxwlgmoh2cny7qlolykdf7aq7g6dlommarldrbm7c4hbckhfcke` + - Type: [`dag-pb` Directory](#dag-pb-directory) + - Structure: + ``` + 📁 / # bafybeiegxwlgmoh2cny7qlolykdf7aq7g6dlommarldrbm7c4hbckhfcke + ├── 📁 foo/ # bafybeidryarwh34ygbtyypbu7qjkl4euiwxby6cql6uvosonohkq2kwnkm (dag-pb Directory) + │ └── 📄 bar.txt # bafkreigzafgemjeejks3vqyuo46ww2e22rt7utq5djikdofjtvnjl5zp6u (raw, 14 bytes) "Hello, world!" + └── 📄 foo.txt # bafkreic3ondyhizrzeoufvoodehinugpj3ecruwokaygl7elezhn2khqfa (raw, 13 bytes) "Hello, IPFS!" + ``` + - Purpose: Another example of standard UnixFS directory with raw leaf blocks ### Special Characters in Filenames - Fixture: [`path_gateway_tar/fixtures.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_tar/fixtures.car) -- CID: `bafybeig6ka5mlwkl4subqhaiatalkcleo4jgnr3hqwvpmsqfca27cijp3i` -- Type: [`dag-pb` Directory](#dag-pb-directory) with nested subdirectories -- Structure: -``` -📁 / # bafybeig6ka5mlwkl4subqhaiatalkcleo4jgnr3hqwvpmsqfca27cijp3i -└── 📁 ą/ # (dag-pb Directory) -└── 📁 ę/ # (dag-pb Directory) -└── 📄 file-źł.txt # (raw, 34 bytes) "I am a txt file on path with utf8" -``` -- Path with Polish diacritics: `/ipfs/bafybeig6ka5mlwkl4subqhaiatalkcleo4jgnr3hqwvpmsqfca27cijp3i/ą/ę/file-źł.txt` -- Purpose: UTF-8 characters in directory and file names (ą, ę, ź, ł) -- Validation: Directory traversal works with UTF-8 paths + - CID: `bafybeig6ka5mlwkl4subqhaiatalkcleo4jgnr3hqwvpmsqfca27cijp3i` + - Type: [`dag-pb` Directory](#dag-pb-directory) with nested subdirectories + - Structure: + ``` + 📁 / # bafybeig6ka5mlwkl4subqhaiatalkcleo4jgnr3hqwvpmsqfca27cijp3i + └── 📁 ą/ # (dag-pb Directory) + └── 📁 ę/ # (dag-pb Directory) + └── 📄 file-źł.txt # (raw, 34 bytes) "I am a txt file on path with utf8" + ``` + - Path with Polish diacritics: `/ipfs/bafybeig6ka5mlwkl4subqhaiatalkcleo4jgnr3hqwvpmsqfca27cijp3i/ą/ę/file-źł.txt` + - Purpose: UTF-8 characters in directory and file names (ą, ę, ź, ł) + - Validation: Directory traversal works with UTF-8 paths - Fixture: [`dir-with-percent-encoded-filename.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_unixfs/dir-with-percent-encoded-filename.car) -- CID: `bafybeig675grnxcmshiuzdaz2xalm6ef4thxxds6o6ypakpghm5kghpc34` -- Type: [`dag-pb` Directory](#dag-pb-directory) -- Structure: -``` -📁 / # bafybeig675grnxcmshiuzdaz2xalm6ef4thxxds6o6ypakpghm5kghpc34 -└── 📄 Portugal%2C+España=Peninsula Ibérica.txt # bafkreihfmctcb2kuvoljqeuphqr2fg2r45vz5cxgq5c2yrxnqg5erbitmq (raw, 38 bytes) "hello from a percent encoded filename" -``` -- Purpose: Filenames with percent-encoding (`%2C`), plus signs, equals, and non-ASCII characters -- Validation: -- Implementations MUST preserve the original filename exactly as stored in UnixFS -- Must not be confused by filenames mixing Unicode characters with percent-encoding -- Gateway example: In gateway-conformance, accessing this file from a web browser requires double-encoding the `%2C` as `%252C` in the URL path (`/ipfs/{{CID}}/Portugal%252C+España=Peninsula%20Ibérica.txt`) -- Browser implementations should preserve `%2C` in the filename to avoid conflicts with URL encoding + - CID: `bafybeig675grnxcmshiuzdaz2xalm6ef4thxxds6o6ypakpghm5kghpc34` + - Type: [`dag-pb` Directory](#dag-pb-directory) + - Structure: + ``` + 📁 / # bafybeig675grnxcmshiuzdaz2xalm6ef4thxxds6o6ypakpghm5kghpc34 + └── 📄 Portugal%2C+España=Peninsula Ibérica.txt # bafkreihfmctcb2kuvoljqeuphqr2fg2r45vz5cxgq5c2yrxnqg5erbitmq (raw, 38 bytes) "hello from a percent encoded filename" + ``` + - Purpose: Filenames with percent-encoding (`%2C`), plus signs, equals, and non-ASCII characters + - Validation: + - Implementations MUST preserve the original filename exactly as stored in UnixFS + - Must not be confused by filenames mixing Unicode characters with percent-encoding + - Gateway example: In gateway-conformance, accessing this file from a web browser requires double-encoding the `%2C` as `%252C` in the URL path (`/ipfs/{{CID}}/Portugal%252C+España=Peninsula%20Ibérica.txt`) + - Browser implementations should preserve `%2C` in the filename to avoid conflicts with URL encoding ### Directory with Missing Blocks - Fixture: [`bafybeigcsevw74ssldzfwhiijzmg7a35lssfmjkuoj2t5qs5u5aztj47tq.dag-pb`](https://github.com/ipld/codec-fixtures/raw/381e762b85862b2bbdb6ef2ba140b3c505e31a44/fixtures/dagpb_4namedlinks%2Bdata/bafybeigcsevw74ssldzfwhiijzmg7a35lssfmjkuoj2t5qs5u5aztj47tq.dag-pb) -- CID: `bafybeigcsevw74ssldzfwhiijzmg7a35lssfmjkuoj2t5qs5u5aztj47tq` -- Type: [`dag-pb` Directory](#dag-pb-directory) -- Structure: -``` -📁 / # bafybeigcsevw74ssldzfwhiijzmg7a35lssfmjkuoj2t5qs5u5aztj47tq -├── ⚠️ audio_only.m4a # (link to missing block, ~24MB) -├── ⚠️ chat.txt # (link to missing block, ~1KB) -├── ⚠️ playback.m3u # (link to missing block, ~116 bytes) -└── ⚠️ zoom_0.mp4 # (link to missing block) -``` -- Note: Child blocks are NOT included - they may be unavailable locally or missing entirely -- Purpose: -- Directory enumeration should require only the root block -- Can list all filenames and their CIDs without fetching child blocks -- Operations should not block waiting for child blocks unless content is actually requested -- Validation: Can enumerate directory contents from root block alone + - CID: `bafybeigcsevw74ssldzfwhiijzmg7a35lssfmjkuoj2t5qs5u5aztj47tq` + - Type: [`dag-pb` Directory](#dag-pb-directory) + - Structure: + ``` + 📁 / # bafybeigcsevw74ssldzfwhiijzmg7a35lssfmjkuoj2t5qs5u5aztj47tq + ├── ⚠️ audio_only.m4a # (link to missing block, ~24MB) + ├── ⚠️ chat.txt # (link to missing block, ~1KB) + ├── ⚠️ playback.m3u # (link to missing block, ~116 bytes) + └── ⚠️ zoom_0.mp4 # (link to missing block) + ``` + - Note: Child blocks are NOT included - they may be unavailable locally or missing entirely + - Purpose: + - Directory enumeration should require only the root block + - Can list all filenames and their CIDs without fetching child blocks + - Operations should not block waiting for child blocks unless content is actually requested + - Validation: Can enumerate directory contents from root block alone ### HAMT Sharded Directory - Fixture: [`single-layer-hamt-with-multi-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/trustless_gateway_car/single-layer-hamt-with-multi-block-files.car) -- CID: `bafybeidbclfqleg2uojchspzd4bob56dqetqjsj27gy2cq3klkkgxtpn4i` -- Type: [`dag-pb` HAMTDirectory](#dag-pb-hamtdirectory) -- Block Analysis: -- Root HAMT block size (`ipfs block stat`): 12046 bytes -- Contains UnixFS Type=HAMTShard metadata with fanout=256 -- Links use 2-character hex prefixes for hash buckets (00-FF) -- Structure: -``` -📂 / # bafybeidbclfqleg2uojchspzd4bob56dqetqjsj27gy2cq3klkkgxtpn4i (HAMT root) -├── 📄 1.txt # (dag-pb file, multi-block) -├── 📄 2.txt # (dag-pb file, multi-block) -├── ... -└── 📄 1000.txt # (dag-pb file, multi-block) -``` -- Contents: 1000 numbered files (1.txt through 1000.txt), each containing Lorem ipsum text -- Purpose: HAMT sharding for large directories -- Validation: -- Fanout field = 256 -- Link Names in HAMT have 2-character hex prefix (hash buckets) -- Can retrieve any file by name through hash bucket calculation + - CID: `bafybeidbclfqleg2uojchspzd4bob56dqetqjsj27gy2cq3klkkgxtpn4i` + - Type: [`dag-pb` HAMTDirectory](#dag-pb-hamtdirectory) + - Block Analysis: + - Root HAMT block size (`ipfs block stat`): 12046 bytes + - Contains UnixFS Type=HAMTShard metadata with fanout=256 + - Links use 2-character hex prefixes for hash buckets (00-FF) + - Structure: + ``` + 📂 / # bafybeidbclfqleg2uojchspzd4bob56dqetqjsj27gy2cq3klkkgxtpn4i (HAMT root) + ├── 📄 1.txt # (dag-pb file, multi-block) + ├── 📄 2.txt # (dag-pb file, multi-block) + ├── ... + └── 📄 1000.txt # (dag-pb file, multi-block) + ``` + - Contents: 1000 numbered files (1.txt through 1000.txt), each containing Lorem ipsum text + - Purpose: HAMT sharding for large directories + - Validation: + - Fanout field = 256 + - Link Names in HAMT have 2-character hex prefix (hash buckets) + - Can retrieve any file by name through hash bucket calculation ## Special Cases and Advanced Features @@ -848,52 +848,52 @@ Test vectors for special UnixFS features and edge cases. ### Symbolic Links - Fixture: [`symlink.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_unixfs/symlink.car) -- CID: `QmWvY6FaqFMS89YAQ9NAPjVP4WZKA1qbHbicc9HeSKQTgt` -- Types: [`dag-pb` Directory](#dag-pb-directory) containing [`dag-pb` Symlink](#dag-pb-symlink) -- Block Analysis: -- Root directory block: Not measured (V0 CID) -- Symlink block (`QmTB8BaCJdCH5H3k7GrxJsxgDNmNYGGR71C58ERkivXoj5`): 9 bytes -- Target file block (`Qme2y5HA5kvo2jAx13UsnV5bQJVijiAJCPvaW3JGQWhvJZ`): 16 bytes -- Structure: -``` -📁 / # QmWvY6FaqFMS89YAQ9NAPjVP4WZKA1qbHbicc9HeSKQTgt -├── 📄 foo # Qme2y5HA5kvo2jAx13UsnV5bQJVijiAJCPvaW3JGQWhvJZ - file containing "content" -└── 🔗 bar # QmTB8BaCJdCH5H3k7GrxJsxgDNmNYGGR71C58ERkivXoj5 - symlink pointing to "foo" -``` -- Purpose: UnixFS symlink resolution -- Security note: Critical for preventing path traversal vulnerabilities + - CID: `QmWvY6FaqFMS89YAQ9NAPjVP4WZKA1qbHbicc9HeSKQTgt` + - Types: [`dag-pb` Directory](#dag-pb-directory) containing [`dag-pb` Symlink](#dag-pb-symlink) + - Block Analysis: + - Root directory block: Not measured (V0 CID) + - Symlink block (`QmTB8BaCJdCH5H3k7GrxJsxgDNmNYGGR71C58ERkivXoj5`): 9 bytes + - Target file block (`Qme2y5HA5kvo2jAx13UsnV5bQJVijiAJCPvaW3JGQWhvJZ`): 16 bytes + - Structure: + ``` + 📁 / # QmWvY6FaqFMS89YAQ9NAPjVP4WZKA1qbHbicc9HeSKQTgt + ├── 📄 foo # Qme2y5HA5kvo2jAx13UsnV5bQJVijiAJCPvaW3JGQWhvJZ - file containing "content" + └── 🔗 bar # QmTB8BaCJdCH5H3k7GrxJsxgDNmNYGGR71C58ERkivXoj5 - symlink pointing to "foo" + ``` + - Purpose: UnixFS symlink resolution + - Security note: Critical for preventing path traversal vulnerabilities ### Mixed Block Sizes - Fixture: [`subdir-with-mixed-block-files.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/trustless_gateway_car/subdir-with-mixed-block-files.car) -- CID: `bafybeidh6k2vzukelqtrjsmd4p52cpmltd2ufqrdtdg6yigi73in672fwu` -- Type: [`dag-pb` Directory](#dag-pb-directory) with subdirectory -- Structure: -``` -📁 / # bafybeidh6k2vzukelqtrjsmd4p52cpmltd2ufqrdtdg6yigi73in672fwu -└── 📁 subdir/ # bafybeicnmple4ehlz3ostv2sbojz3zhh5q7tz5r2qkfdpqfilgggeen7xm -├── 📄 ascii.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" -├── 📄 hello.txt # bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4 (raw, 12 bytes) "hello world\n" -└── 📄 multiblock.txt # bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa (dag-pb, 1271 bytes total) -``` -- Purpose: Directories containing both single-block raw files and multi-block dag-pb files -- Validation: Can handle mixed file types in same directory + - CID: `bafybeidh6k2vzukelqtrjsmd4p52cpmltd2ufqrdtdg6yigi73in672fwu` + - Type: [`dag-pb` Directory](#dag-pb-directory) with subdirectory + - Structure: + ``` + 📁 / # bafybeidh6k2vzukelqtrjsmd4p52cpmltd2ufqrdtdg6yigi73in672fwu + └── 📁 subdir/ # bafybeicnmple4ehlz3ostv2sbojz3zhh5q7tz5r2qkfdpqfilgggeen7xm + ├── 📄 ascii.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" + ├── 📄 hello.txt # bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4 (raw, 12 bytes) "hello world\n" + └── 📄 multiblock.txt # bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa (dag-pb, 1271 bytes total) + ``` + - Purpose: Directories containing both single-block raw files and multi-block dag-pb files + - Validation: Can handle mixed file types in same directory ### Deduplication - Fixture: [`dir-with-duplicate-files.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/trustless_gateway_car/dir-with-duplicate-files.car) -- CID: `bafybeihchr7vmgjaasntayyatmp5sv6xza57iy2h4xj7g46bpjij6yhrmy` -- Type: [`dag-pb` Directory](#dag-pb-directory) -- Structure: -``` -📁 / # bafybeihchr7vmgjaasntayyatmp5sv6xza57iy2h4xj7g46bpjij6yhrmy -├── 🔗 ascii-copy.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (same CID as ascii.txt) -├── 📄 ascii.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" -├── 📄 hello.txt # bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4 (raw, 12 bytes) "hello world\n" -└── 📄 multiblock.txt # bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa (dag-pb, multi-block) -``` -- Purpose: Multiple directory entries pointing to the same content CID (deduplication) -- Validation: Both ascii.txt and ascii-copy.txt resolve to the same content block + - CID: `bafybeihchr7vmgjaasntayyatmp5sv6xza57iy2h4xj7g46bpjij6yhrmy` + - Type: [`dag-pb` Directory](#dag-pb-directory) + - Structure: + ``` + 📁 / # bafybeihchr7vmgjaasntayyatmp5sv6xza57iy2h4xj7g46bpjij6yhrmy + ├── 🔗 ascii-copy.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (same CID as ascii.txt) + ├── 📄 ascii.txt # bafkreifkam6ns4aoolg3wedr4uzrs3kvq66p4pecirz6y2vlrngla62mxm (raw, 31 bytes) "hello application/vnd.ipld.car" + ├── 📄 hello.txt # bafkreifjjcie6lypi6ny7amxnfftagclbuxndqonfipmb64f2km2devei4 (raw, 12 bytes) "hello world\n" + └── 📄 multiblock.txt # bafybeigcisqd7m5nf3qmuvjdbakl5bdnh4ocrmacaqkpuh77qjvggmt2sa (dag-pb, multi-block) + ``` + - Purpose: Multiple directory entries pointing to the same content CID (deduplication) + - Validation: Both ascii.txt and ascii-copy.txt resolve to the same content block ### Invalid Test Cases @@ -920,13 +920,13 @@ These validate that implementations properly reject malformed or non-UnixFS dag- ## Additional Testing Resources - Gateway Conformance Suite: [ipfs/gateway-conformance](https://github.com/ipfs/gateway-conformance) -- Real-world test suite with UnixFS fixtures -- Tests gateway behaviors with various UnixFS structures -- Includes edge cases and performance scenarios + - Real-world test suite with UnixFS fixtures + - Tests gateway behaviors with various UnixFS structures + - Includes edge cases and performance scenarios - Test fixture generator: [go-fixtureplate](https://github.com/ipld/go-fixtureplate) -- Tool for generating custom test fixtures -- Includes UnixFS files and directories of arbitrary shapes + - Tool for generating custom test fixtures + - Includes UnixFS files and directories of arbitrary shapes Report specification issues or submit corrections via [ipfs/specs](https://github.com/ipfs/specs/issues). @@ -937,21 +937,21 @@ This section and included subsections are not authoritative. ## Popular Implementations - JavaScript -- [`@helia/unixfs`](https://www.npmjs.com/package/@helia/unixfs) implementation of a filesystem compatible with [Helia SDK](https://github.com/ipfs/helia#readme) -- Data Formats - [unixfs](https://github.com/ipfs/js-ipfs-unixfs) -- Importer - [unixfs-importer](https://github.com/ipfs/js-ipfs-unixfs/tree/main/packages/ipfs-unixfs-importer) -- Exporter - [unixfs-exporter](https://github.com/ipfs/js-ipfs-unixfs/tree/main/packages/ipfs-unixfs-exporter) + - [`@helia/unixfs`](https://www.npmjs.com/package/@helia/unixfs) implementation of a filesystem compatible with [Helia SDK](https://github.com/ipfs/helia#readme) + - Data Formats - [unixfs](https://github.com/ipfs/js-ipfs-unixfs) + - Importer - [unixfs-importer](https://github.com/ipfs/js-ipfs-unixfs/tree/main/packages/ipfs-unixfs-importer) + - Exporter - [unixfs-exporter](https://github.com/ipfs/js-ipfs-unixfs/tree/main/packages/ipfs-unixfs-exporter) - Go -- [Boxo SDK](https://github.com/ipfs/boxo#readme) includes implementation of UnixFS filesystem -- Protocol Buffer Definitions - [`ipfs/boxo/../unixfs.proto`](https://github.com/ipfs/boxo/blob/v0.23.0/ipld/unixfs/pb/unixfs.proto) -- [`ipfs/boxo/files`](https://github.com/ipfs/boxo/tree/main/files) -- [`ipfs/boxo/ipld/unixfs`](https://github.com/ipfs/boxo/tree/main/ipld/unixfs/) -- Alternative `go-ipld-prime` implementation: [`ipfs/go-unixfsnode`](https://github.com/ipfs/go-unixfsnode) + - [Boxo SDK](https://github.com/ipfs/boxo#readme) includes implementation of UnixFS filesystem + - Protocol Buffer Definitions - [`ipfs/boxo/../unixfs.proto`](https://github.com/ipfs/boxo/blob/v0.23.0/ipld/unixfs/pb/unixfs.proto) + - [`ipfs/boxo/files`](https://github.com/ipfs/boxo/tree/main/files) + - [`ipfs/boxo/ipld/unixfs`](https://github.com/ipfs/boxo/tree/main/ipld/unixfs/) + - Alternative `go-ipld-prime` implementation: [`ipfs/go-unixfsnode`](https://github.com/ipfs/go-unixfsnode) ## Simple `raw` Example @@ -969,14 +969,14 @@ Add the CID prefix: ``` f01551220 -9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 + 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 f this is the multibase prefix, we need it because we are working with a hex CID, this is omitted for binary CIDs -01 the CID version, here one -55 the codec, here we MUST use Raw because this is a Raw file -12 the hashing function used, here sha256 -20 the digest length 32 bytes -9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 is the the digest we computed earlier + 01 the CID version, here one + 55 the codec, here we MUST use Raw because this is a Raw file + 12 the hashing function used, here sha256 + 20 the digest length 32 bytes + 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08 is the the digest we computed earlier ``` Done. Assuming we stored this block in some implementation of our choice, which makes it accessible to our client, we can try to decode it. @@ -993,17 +993,16 @@ it is a simple canonical one, python pseudo code to compute it looks like this: ```python def offsetlist(node): -unixfs = decodeDataField(node.Data) -if len(node.Links) != len(unixfs.Blocksizes): -raise "unmatched sister-lists" # error messages are implementation details + unixfs = decodeDataField(node.Data) + if len(node.Links) != len(unixfs.Blocksizes): + raise "unmatched sister-lists" # error messages are implementation details -cursor = len(unixfs.Data) if unixfs.Data else 0 -return [cursor] + [cursor := cursor + size for size in unixfs.Blocksizes[:-1]] + cursor = len(unixfs.Data) if unixfs.Data else 0 + return [cursor] + [cursor := cursor + size for size in unixfs.Blocksizes[:-1]] ``` This will tell you which offset inside this node the children at the corresponding index starts to cover. (using `[x,y)` ranging) - # Appendix: Historical Design Decisions :::warning @@ -1035,13 +1034,13 @@ but never the file data itself. This was ultimately rejected for a number of reasons: 1. You would always need to retrieve an additional node to access file data, which -limits the kind of optimizations that are possible. For example, many files are -under the 256 KiB block size limit, so we tend to inline them into the describing -UnixFS `File` node. This would not be possible with an intermediate `Metadata` node. + limits the kind of optimizations that are possible. For example, many files are + under the 256 KiB block size limit, so we tend to inline them into the describing + UnixFS `File` node. This would not be possible with an intermediate `Metadata` node. 2. The `File` node already contains some metadata (e.g. the file size), so metadata -would be stored in multiple places. This complicates forwards compatibility with -UnixFSv2, as mapping between metadata formats potentially requires multiple fetch -operations. + would be stored in multiple places. This complicates forwards compatibility with + UnixFSv2, as mapping between metadata formats potentially requires multiple fetch + operations. ### Pros and Cons: Metadata in the Directory @@ -1059,13 +1058,13 @@ both UnixFS v1 and v1.5 nodes. This was rejected for the following reasons: 1. When creating a UnixFS node, there's no way to record metadata without -wrapping it in a directory. + wrapping it in a directory. 2. If you access any UnixFS node directly by its [CID], there is no way of -recreating the metadata which limits flexibility. + recreating the metadata which limits flexibility. 3. In order to list the contents of a directory including entry types and -sizes, you have to fetch the root node of each entry, so the performance -benefit of including some metadata in the containing directory is negligible -in this use case. + sizes, you have to fetch the root node of each entry, so the performance + benefit of including some metadata in the containing directory is negligible + in this use case. ### Pros and Cons: Metadata in the File @@ -1080,15 +1079,15 @@ we decide to keep file data in a leaf node for deduplication reasons. Downsides to this approach are: 1. Two users adding the same file to IPFS at different times will have -different [CID]s due to the `mtime`s being different. If the content is -stored in another node, its [CID] will be constant between the two users, -but you can't navigate to it unless you have the parent node, which will be -less available due to the proliferation of [CID]s. + different [CID]s due to the `mtime`s being different. If the content is + stored in another node, its [CID] will be constant between the two users, + but you can't navigate to it unless you have the parent node, which will be + less available due to the proliferation of [CID]s. 2. Metadata is also impossible to remove without changing the [CID], so -metadata becomes part of the content. + metadata becomes part of the content. 3. Performance may be impacted as well as if we don't inline UnixFS root nodes -into [CID]s, so additional fetches will be required to load a given UnixFS -entry. + into [CID]s, so additional fetches will be required to load a given UnixFS + entry. ### Pros and Cons: Metadata in Side Trees @@ -1124,8 +1123,7 @@ In most cases, such values will exceed 2^28 (268,435,456) nanoseconds. Therefore the fractional part is represented as a 4-byte `fixed32`, [as per Google's recommendation](https://developers.google.com/protocol-buffers/docs/proto#scalar). - -[protobuf]: https://developers.google.com/protocol-buffers/ +[protobuf]: https://protobuf.dev/ [CID]: https://github.com/multiformats/cid/ [multicodec]: https://github.com/multiformats/multicodec [multihash]: https://github.com/multiformats/multihash From 28b6a142921bd89b08e3a97b812976c1525613f7 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sun, 24 Aug 2025 01:11:56 +0200 Subject: [PATCH 23/46] docs: add HAMT fanout security limit incorporate CVE-2023-23625 vulnerability details from #507 - enforce maximum fanout of 1024 to prevent DoS attacks - add security warning with links to CVE and GHSA advisory --- src/unixfs-data-format.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/src/unixfs-data-format.md b/src/unixfs-data-format.md index b31159e31..79421286d 100644 --- a/src/unixfs-data-format.md +++ b/src/unixfs-data-format.md @@ -310,12 +310,19 @@ The HAMT directory is configured through the UnixFS metadata in `PBNode.Data`: - `decode(PBNode.Data).hashType` indicates the [multihash] function to use to digest the path components for sharding. Currently, all HAMT implementations use `murmur3-x64-64` (`0x22`), and this value MUST be consistent across all shards within the same HAMT structure -- `decode(PBNode.Data).fanout` MUST be a power of two. This determines the number +- `decode(PBNode.Data).fanout` MUST be a power of two and at most 1024. This determines the number of possible bucket indices (permutations) at each level of the trie. For example, fanout=256 provides 256 possible buckets (0x00 to 0xFF), requiring 8 bits from the hash. The hex prefix length is `log2(fanout)/4` characters (since each hex character represents 4 bits). The same fanout value is used throughout all levels of a single HAMT structure. Implementations choose fanout based on their specific trade-offs between tree depth and node size + :::warning + Implementations MUST limit the `fanout` parameter to a maximum of 1024 to prevent + denial-of-service attacks. Excessively large fanout values can cause memory exhaustion + when allocating bucket arrays. See [CVE-2023-23625](https://nvd.nist.gov/vuln/detail/CVE-2023-23625) and + [GHSA-q264-w97q-q778](https://github.com/advisories/GHSA-q264-w97q-q778) for details + on this vulnerability. + ::: - `decode(PBNode.Data).Data` is a bitmap field indicating which buckets contain entries. Each bit represents one bucket. While included in the protobuf, implementations typically derive bucket occupancy from the link names directly From 9b48ce7cd6b1738aa21efd29b058a103634a5dd2 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sun, 24 Aug 2025 01:32:00 +0200 Subject: [PATCH 24/46] docs: rename unixfs-data-format to unixfs update file name and all references for shorter, cleaner URL --- UNIXFS.md | 2 +- src/index.html | 2 +- src/{unixfs-data-format.md => unixfs.md} | 0 3 files changed, 2 insertions(+), 2 deletions(-) rename src/{unixfs-data-format.md => unixfs.md} (100%) diff --git a/UNIXFS.md b/UNIXFS.md index 243f5f030..95e346ca4 100644 --- a/UNIXFS.md +++ b/UNIXFS.md @@ -1,3 +1,3 @@ # UnixFS -Moved to https://specs.ipfs.tech/unixfs-data-format/ +Moved to https://specs.ipfs.tech/unixfs/ diff --git a/src/index.html b/src/index.html index 5a630da49..d5b3e1b3b 100644 --- a/src/index.html +++ b/src/index.html @@ -114,7 +114,7 @@

Data Formats

IPFS basic primitive is an opaque block of bytes identified by a CID. CID includes codec that informs IPFS System about data format: how to parse the block, and how to link from one block to another.

- The most popular data formats used by IPFS Systems are RAW (an opaque block), UnixFS (filesystem abstraction built with DAG-PB and RAW codecs), DAG-CBOR/DAG-JSON, however IPFS ecosystem is not limited to them, and IPFS systems are free to choose the level of interoperability, or even implement support for own, additional formats. A complimentary CAR is a codec-agnostic archive format for transporting multiple opaque blocks. + The most popular data formats used by IPFS Systems are RAW (an opaque block), UnixFS (filesystem abstraction built with DAG-PB and RAW codecs), DAG-CBOR/DAG-JSON, however IPFS ecosystem is not limited to them, and IPFS systems are free to choose the level of interoperability, or even implement support for own, additional formats. A complimentary CAR is a codec-agnostic archive format for transporting multiple opaque blocks.

Specifications: diff --git a/src/unixfs-data-format.md b/src/unixfs.md similarity index 100% rename from src/unixfs-data-format.md rename to src/unixfs.md From d2c28a30c559508952fb44be10769d601a461b82 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sun, 24 Aug 2025 02:24:54 +0200 Subject: [PATCH 25/46] docs: clarify CID representation in dag-pb add link to CID spec and clarify that Hash field contains raw CID bytes with no multibase prefix addresses review comments: https://github.com/ipfs/specs/pull/331#discussion_r2018652563 https://github.com/ipfs/specs/pull/331#discussion_r2018698487 --- src/unixfs.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/unixfs.md b/src/unixfs.md index 79421286d..208fdf333 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -70,7 +70,7 @@ In UnixFS, a node can be encoded using two different multicodecs, listed below. # `raw` Node The simplest nodes use `raw` encoding and are implicitly a [File](#dag-pb-file). They can -be recognized because their CIDs are encoded using the `raw` (`0x55`) codec: +be recognized because their [CIDs](https://github.com/multiformats/cid) are encoded using the `raw` (`0x55`) codec: - The block is the file data. There is no protobuf envelope or metadata. - They never have any children nodes, and thus are also known as single block files. @@ -90,7 +90,10 @@ summarized as follows: ```protobuf message PBLink { - // binary CID (with no multibase prefix) of the target object + // Binary representation of CID (https://github.com/multiformats/cid) of the target object. + // This contains raw CID bytes (either CIDv0 or CIDv1) with no multibase prefix. + // CIDv1 is a binary format composed of unsigned varints, while CIDv0 is a raw multihash. + // In both cases, the bytes are stored directly without any additional prefix. optional bytes Hash = 1; // UTF-8 string name From ade1d6f9aeb88458b4aff75b064f899931f76deb Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sun, 24 Aug 2025 02:46:43 +0200 Subject: [PATCH 26/46] docs: define concrete error handling behavior clarify that parsers must reject nodes and halt processing when encountering errors, with descriptive error messages addresses review comment: https://github.com/ipfs/specs/pull/331#discussion_r2018723186 --- src/unixfs.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/src/unixfs.md b/src/unixfs.md index 208fdf333..97060e2c1 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -204,7 +204,10 @@ range we are interested in. In the example above, the offset list would be `[0, 20]`. Thus, we know we only need to download `Qmbar` to get the range we are interested in. -UnixFS parser MUST error if `blocksizes` or `Links` are not of the same length. +A UnixFS parser MUST reject the node and halt processing if the `blocksizes` array and +`Links` array contain different numbers of elements. Implementations SHOULD return a +descriptive error indicating the array length mismatch rather than silently failing or +attempting to process partial data. #### `decode(PBNode.Data).Data` @@ -220,7 +223,7 @@ The `Name` field is primarily used in directories to identify child entries. **For internal file chunks:** - Implementations SHOULD NOT produce `Name` fields (the field should be absent in the protobuf, not an empty string) - For compatibility with historical data, implementations SHOULD treat empty string values ("") the same as absent when parsing -- If a non-empty `Name` is present in an internal file chunk, the parser MUST error as this indicates an invalid file structure +- If a non-empty `Name` is present in an internal file chunk, the parser MUST reject the file and halt processing as this indicates an invalid file structure #### `decode(PBNode.Data).Blocksize` @@ -249,7 +252,7 @@ Otherwise, this file is invalid. #### `dag-pb` `File` Path Resolution A file terminates a UnixFS content path. Any attempt to resolve a path past a -file MUST error. +file MUST be rejected with an error indicating that UnixFS files cannot have children. ### `dag-pb` `Directory` @@ -292,8 +295,8 @@ efficient directory traversal algorithms in some implementations. Pop the left-most component of the path, and try to match it to the `Name` of a child under `PBNode.Links`. If you find a match, you can then remember the CID. -You MUST continue the search. If you find another match, you MUST error since -duplicate names are not allowed. +You MUST continue the search. If you find another match, you MUST reject the directory +and halt path resolution since duplicate names are not allowed. Assuming no errors were raised, you can continue to the path resolution on the remaining components and on the CID you popped. @@ -591,10 +594,10 @@ Relative path components MUST be resolved before trying to work on the path: - `.` points to the current node and MUST be removed. - `..` points to the parent node and MUST be removed left to right. When removing a `..`, the path component on the left MUST also be removed. If there is no path - component on the left, you MUST error to avoid out-of-bounds - path resolution. -- Implementations MUST error when resolving a relative path that attempts to go - beyond the root CID (example: `/ipfs/cid/../foo`). + component on the left, implementations MUST reject the path with an error to avoid + out-of-bounds path resolution. +- Implementations MUST reject paths that attempt to traverse beyond the root CID + (example: `/ipfs/cid/../foo`) with an error indicating invalid path traversal. ### Restricted Names From e95b7aabf53eb3f85005712b0bad6bb05f251b5c Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sun, 24 Aug 2025 03:04:10 +0200 Subject: [PATCH 27/46] docs: clarify field requirements and duplicate names - define "identical" names as byte-for-byte equal - clarify Data field requirements for directories - note historical compatibility with empty Data fields addresses review comments: https://github.com/ipfs/specs/pull/331#discussion_r2018751577 https://github.com/ipfs/specs/pull/331#discussion_r2018782817 --- src/unixfs.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/src/unixfs.md b/src/unixfs.md index 97060e2c1..262acc825 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -261,12 +261,14 @@ A :dfn[Directory], also known as folder, is a named collection of child [Nodes]( - Every link in `PBNode.Links` is an entry (child) of the directory, and `PBNode.Links[].Name` gives you the name of that child. - Duplicate names are not allowed. Therefore, two elements of `PBNode.Link` CANNOT - have the same `Name`. If two identical names are present in a directory, the - decoder MUST fail. + have the same `Name`. Names are considered identical if they are byte-for-byte + equal (not just semantically equivalent). If two identical names are present in + a directory, the decoder MUST fail. - Implementations SHOULD detect when directory becomes too big to fit in a single `Directory` block and use [`HAMTDirectory`] type instead. -The minimum valid `PBNode.Data` field for a directory is as follows: +The `PBNode.Data` field MUST contain valid UnixFS protobuf data for all UnixFS nodes. +For directories (DataType==1), the minimum valid `PBNode.Data` field is as follows: ```json { @@ -274,6 +276,9 @@ The minimum valid `PBNode.Data` field for a directory is as follows: } ``` +For historical compatibility, implementations MAY encounter dag-pb nodes with empty or +missing Data fields from older IPFS versions, but MUST NOT produce such nodes. + #### `dag-pb` `Directory` Link Ordering Directory links SHOULD be sorted lexicographically by the `Name` field when creating From 4d30808885b61427b6d542d6b2ea6140a721c109 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sun, 24 Aug 2025 03:19:49 +0200 Subject: [PATCH 28/46] docs: clarify field requirements in protobuf schema - annotate which fields are required for specific DataTypes - clarify fanout is required for HAMTShard despite optional in schema - specify when implementations should include Tsize values - document Data field usage for each type - mark MimeType as reserved for future use addresses review comments: https://github.com/ipfs/specs/pull/331#discussion_r1039771631 https://github.com/ipfs/specs/pull/331#discussion_r1345748774 --- src/unixfs.md | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/src/unixfs.md b/src/unixfs.md index 262acc825..b06997769 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -128,17 +128,17 @@ message Data { } required DataType Type = 1; - optional bytes Data = 2; + optional bytes Data = 2; // file content (File), symlink target (Symlink), bitmap (HAMTShard), unused (Directory) optional uint64 filesize = 3; - repeated uint64 blocksizes = 4; - optional uint64 hashType = 5; - optional uint64 fanout = 6; - optional uint32 mode = 7; // opt-in, AKA UnixFS 1.5 - optional UnixTime mtime = 8; // opt-in, AKA UnixFS 1.5 + repeated uint64 blocksizes = 4; // required for multi-block files (Type=File) with Links + optional uint64 hashType = 5; // required for Type=HAMTShard (currently always murmur3-x64-64) + optional uint64 fanout = 6; // required for Type=HAMTShard (power of 2, max 1024) + optional uint32 mode = 7; // opt-in, AKA UnixFS 1.5 + optional UnixTime mtime = 8; // opt-in, AKA UnixFS 1.5 } message Metadata { - optional string MimeType = 1; + optional string MimeType = 1; // reserved for future use } message UnixTime { @@ -321,7 +321,8 @@ The HAMT directory is configured through the UnixFS metadata in `PBNode.Data`: - `decode(PBNode.Data).hashType` indicates the [multihash] function to use to digest the path components for sharding. Currently, all HAMT implementations use `murmur3-x64-64` (`0x22`), and this value MUST be consistent across all shards within the same HAMT structure -- `decode(PBNode.Data).fanout` MUST be a power of two and at most 1024. This determines the number +- `decode(PBNode.Data).fanout` is REQUIRED for HAMTShard nodes (though marked optional in the + protobuf schema). The value MUST be a power of two and at most 1024. This determines the number of possible bucket indices (permutations) at each level of the trie. For example, fanout=256 provides 256 possible buckets (0x00 to 0xFF), requiring 8 bits from the hash. The hex prefix length is `log2(fanout)/4` characters (since each hex character represents 4 bits). @@ -457,6 +458,11 @@ through. `Tsize` is an optional field in `PBNode.Links[]` which represents the cumulative size of the entire DAG rooted at that link, including all protobuf encoding overhead. +While optional in the protobuf schema, implementations SHOULD include `Tsize` for: +- All directory entries (enables fast directory size display) +- Multi-block files (enables parallel downloading and progress tracking) +- HAMT shard links (enables efficient traversal decisions) + **Key distinction from blocksize:** - **`blocksize`**: Only the raw file data (no protobuf overhead) - **`Tsize`**: Total size of all serialized blocks in the DAG (includes protobuf overhead) From ac9188fa3bda7e36dd84d19471fe96d676afc8df Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sun, 24 Aug 2025 04:05:46 +0200 Subject: [PATCH 29/46] docs(unixfs): move block size discussion to implementation notes Moves implementation-specific block size guidance to appendix section. Keeps spec focused on format while providing practical guidance separately. Related to PR #331 review feedback --- src/unixfs.md | 35 ++++++++++++++++++++++++++++++----- 1 file changed, 30 insertions(+), 5 deletions(-) diff --git a/src/unixfs.md b/src/unixfs.md index b06997769..6a7e926a8 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -344,8 +344,8 @@ hex-encoded prefix corresponding to the bucket index, zero-padded to a width of `log2(fanout)/4` characters. Implementations choose when to convert a regular directory to HAMT based on various criteria -such as estimated block size (commonly around 256KiB-1MiB to produce blocks that can be -transported over Bitswap) or number of directory entries. +such as estimated block size or number of directory entries. See [Block Size Considerations](#block-size-considerations) +for typical thresholds. To illustrate the HAMT structure with a concrete example: @@ -978,6 +978,30 @@ This section and included subsections are not authoritative. - [`unixfs-v1`](https://github.com/ipfs-rust/unixfsv1) --> +## Block Size Considerations + +While UnixFS itself does not mandate specific block size limits, implementations typically +enforce practical constraints for operational efficiency: + +- **Safe conventions for producing new blocks**: Implementations SHOULD use 256 KiB (popular + legacy size) or 1 MiB (modern maximum recommended) for newly created blocks +- **Decoding requirement**: Implementations MUST be able to decode blocks up to 2 MiB + as it is effectively the maximum message size in Bitswap, which acts as ecosystem-wide + common denominator of what is the max block size at the time of writing this note (2025Q3) + +These limits affect several UnixFS behaviors: +- Small files that fit in a single chunk (most common: 256 KiB, 1 MiB) are typically + stored as single `raw` blocks or within the `Data` field of a single `dag-pb` node +- Directories automatically convert to HAMT sharding when approaching the block size + limit (commonly triggered around 256 KiB-1 MiB) +- File chunking algorithms target block sizes that stay within these limits while + maximizing deduplication opportunities + +Note that specific block size policies are implementation-dependent and may be +configurable. If you want to maximize the interoperability of your data, make sure +to keep chunk sizes no bigger than 1 MiB. Consult your implementation's documentation +for exact limits and configuration options. + ## Simple `raw` Example In this example, we will build a single `raw` block with the string `test` as its content. @@ -1058,9 +1082,10 @@ but never the file data itself. This was ultimately rejected for a number of reasons: 1. You would always need to retrieve an additional node to access file data, which - limits the kind of optimizations that are possible. For example, many files are - under the 256 KiB block size limit, so we tend to inline them into the describing - UnixFS `File` node. This would not be possible with an intermediate `Metadata` node. + limits the kind of optimizations that are possible. For example, many files fit + within a single block (see [Block Size Considerations](#block-size-considerations)), + so we tend to inline them into the describing UnixFS `File` node. This would not be + possible with an intermediate `Metadata` node. 2. The `File` node already contains some metadata (e.g. the file size), so metadata would be stored in multiple places. This complicates forwards compatibility with UnixFSv2, as mapping between metadata formats potentially requires multiple fetch From 08d895d99f11f811769c85e8bd8065fe0b242522 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sun, 24 Aug 2025 04:30:57 +0200 Subject: [PATCH 30/46] docs(unixfs): clarify filesize field requirement for File and Raw types Filesize is mandatory for Type=File and Type=Raw, defaults to 0 if omitted. While marked optional in protobuf for compatibility with other types. Addresses review feedback about filesize field requirements --- src/unixfs.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/src/unixfs.md b/src/unixfs.md index 6a7e926a8..88dce8132 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -129,7 +129,7 @@ message Data { required DataType Type = 1; optional bytes Data = 2; // file content (File), symlink target (Symlink), bitmap (HAMTShard), unused (Directory) - optional uint64 filesize = 3; + optional uint64 filesize = 3; // mandatory for Type=File and Type=Raw, defaults to 0 if omitted repeated uint64 blocksizes = 4; // required for multi-block files (Type=File) with Links optional uint64 hashType = 5; // required for Type=HAMTShard (currently always murmur3-x64-64) optional uint64 fanout = 6; // required for Type=HAMTShard (power of 2, max 1024) @@ -246,8 +246,11 @@ Examples of where `blocksize` is useful: #### `decode(PBNode.Data).filesize` -If present, this field MUST be equal to the `Blocksize` computation above. -Otherwise, this file is invalid. +For `Type=File` (0) and `Type=Raw` (2), this field is mandatory. While marked as "optional" +in the protobuf schema (for compatibility with other types like Directory), implementations: +- MUST include this field when creating File or Raw nodes +- When reading, if this field is absent, MUST interpret it as 0 (zero-length file) +- If present, this field MUST be equal to the `Blocksize` computation above, otherwise the file is invalid #### `dag-pb` `File` Path Resolution From 36aa713b43af120e91870ea8c56a1213cf938172 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sun, 24 Aug 2025 04:45:20 +0200 Subject: [PATCH 31/46] docs(unixfs): clarify mode/mtime metadata expectations Mode and mtime SHOULD NOT be included by default to preserve deduplication. Must be explicitly requested by user. https://github.com/ipfs/specs/pull/331#discussion_r1357154429 --- src/unixfs.md | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/src/unixfs.md b/src/unixfs.md index 88dce8132..85bcfb657 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -519,11 +519,11 @@ When total data size is needed for important purposes such as accounting, billin ### `dag-pb` Optional Metadata -UnixFS currently supports below optional metadata fields. +UnixFS defines the following optional metadata fields. #### `mode` Field -The `mode` is for persisting the file permissions in [numeric notation](https://en.wikipedia.org/wiki/File_system_permissions#Numeric_notation) +The `mode` (introduced in UnixFS v1.5) is for persisting the file permissions in [numeric notation](https://en.wikipedia.org/wiki/File_system_permissions#Numeric_notation) \[[spec](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_stat.h.html)\]. - If unspecified, implementations MAY default to @@ -535,10 +535,17 @@ The `mode` is for persisting the file permissions in [numeric notation](https:// - For future-proofing, the (de)serialization layer must preserve the entire uint32 value during clone/copy operations, modifying only bit values that have a well defined meaning: `clonedValue = ( modifiedBits & 07777 ) | ( originalValue & 0xFFFFF000 )` - Implementations of this spec must proactively mask off bits without a defined meaning in the implemented version of the spec: `interpretedValue = originalValue & 07777` +**Implementation guidance:** +- When importing new data, implementations SHOULD NOT include the mode field unless the user explicitly requests preserving permissions + - Including mode changes the root CID, causing unnecessary deduplication failures when permission differences are irrelevant +- Implementations MUST be able to parse UnixFS nodes both with and without this field +- When present during operations like copying, implementations SHOULD preserve this field + #### `mtime` Field -A two-element structure ( `Seconds`, `FractionalNanoseconds` ) representing the +The `mtime` (introduced in UnixFS v1.5) is a two-element structure ( `Seconds`, `FractionalNanoseconds` ) representing the modification time in seconds relative to the unix epoch `1970-01-01T00:00:00Z`. + The two fields are: 1. `Seconds` ( always present, signed 64bit integer ): represents the amount of seconds after **or before** the epoch. @@ -571,6 +578,12 @@ non-IPFS target MUST observe the following: vs 64bit mismatch), implementations must assume the highest possible value in the targets range. In most cases, this would be `2038-01-19T03:14:07Z`. +**Implementation guidance:** +- When importing new data, implementations SHOULD NOT include the mtime field unless the user explicitly requests preserving timestamps + - Including mtime changes the root CID, causing unnecessary deduplication failures when timestamp differences are irrelevant +- Implementations MUST be able to parse UnixFS nodes both with and without this field +- When present during operations like copying, implementations SHOULD preserve this field + ## UnixFS Paths Paths begin with a `/` or `/ipfs//`, where `` is a [multibase] From 4326adfc4597c66d68a62713f9731dce8735fe4c Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sun, 24 Aug 2025 05:25:05 +0200 Subject: [PATCH 32/46] docs(unixfs): clarify HAMT fanout and bitfield - Fanout SHOULD be multiple of 8 for byte-aligned bitfield - Bitfield MUST be written, SHOULD be read for efficiency - Document common sharding threshold (256 KiB - 1 MiB) - Clarify bitfield is little-endian, size is fanout/8 bytes Based on analysis of boxo, go-unixfsnode, and JS implementations https://github.com/ipfs/specs/pull/331#discussion_r1491800817 --- src/unixfs.md | 39 +++++++++++++++++++++++++++------------ 1 file changed, 27 insertions(+), 12 deletions(-) diff --git a/src/unixfs.md b/src/unixfs.md index 85bcfb657..955a4f2b8 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -325,12 +325,15 @@ The HAMT directory is configured through the UnixFS metadata in `PBNode.Data`: the path components for sharding. Currently, all HAMT implementations use `murmur3-x64-64` (`0x22`), and this value MUST be consistent across all shards within the same HAMT structure - `decode(PBNode.Data).fanout` is REQUIRED for HAMTShard nodes (though marked optional in the - protobuf schema). The value MUST be a power of two and at most 1024. This determines the number - of possible bucket indices (permutations) at each level of the trie. For example, - fanout=256 provides 256 possible buckets (0x00 to 0xFF), requiring 8 bits from the hash. + protobuf schema). The value MUST be a power of two and at most 1024. Implementations SHOULD + require fanout to be a multiple of 8 to ensure the bitfield aligns to byte boundaries. + Popular values are: 256, 512, 1024. The most common fanout is 256 (8-bit buckets), + providing a good balance between tree depth and node size. + + This determines the number of possible bucket indices (permutations) at each level of the trie. + For example, fanout=256 provides 256 possible buckets (0x00 to 0xFF), requiring 8 bits from the hash. The hex prefix length is `log2(fanout)/4` characters (since each hex character represents 4 bits). - The same fanout value is used throughout all levels of a single HAMT structure. - Implementations choose fanout based on their specific trade-offs between tree depth and node size + The same fanout value is used throughout all levels of a single HAMT structure :::warning Implementations MUST limit the `fanout` parameter to a maximum of 1024 to prevent denial-of-service attacks. Excessively large fanout values can cause memory exhaustion @@ -338,18 +341,20 @@ The HAMT directory is configured through the UnixFS metadata in `PBNode.Data`: [GHSA-q264-w97q-q778](https://github.com/advisories/GHSA-q264-w97q-q778) for details on this vulnerability. ::: -- `decode(PBNode.Data).Data` is a bitmap field indicating which buckets contain entries. - Each bit represents one bucket. While included in the protobuf, implementations - typically derive bucket occupancy from the link names directly +- `decode(PBNode.Data).Data` contains a bitfield indicating which buckets contain entries. + Each bit corresponds to one bucket (0 to fanout-1), with bit value 1 indicating the bucket + is occupied. The bitfield is stored in little-endian byte order. The bitfield size in bytes + is `fanout/8`, which is why fanout SHOULD be a multiple of 8. + - Implementations MUST write this bitfield when creating HAMT nodes + - Implementations SHOULD use this bitfield for efficient traversal (checking which buckets + exist without examining all links) + - Note: Some implementations derive bucket occupancy from link names instead of reading + the bitfield, but this is less efficient The field `Name` of an element of `PBNode.Links` for a HAMT uses a hex-encoded prefix corresponding to the bucket index, zero-padded to a width of `log2(fanout)/4` characters. -Implementations choose when to convert a regular directory to HAMT based on various criteria -such as estimated block size or number of directory entries. See [Block Size Considerations](#block-size-considerations) -for typical thresholds. - To illustrate the HAMT structure with a concrete example: ```protobuf @@ -437,6 +442,16 @@ Given a HAMT-sharded directory containing 1000 files: - "FF742.txt" means: file "742.txt" that hashed to bucket FF at this level ::: +#### When to Use HAMT Sharding + +Implementations typically convert regular directories to HAMT when the serialized directory +node exceeds a size threshold between 256 KiB and 1 MiB. This threshold: +- Prevents directories from exceeding block size limits +- Is implementation-specific and may be configurable +- Common values range from 256 KiB (conservative) to 1 MiB (modern) + +See [Block Size Considerations](#block-size-considerations) for details on block size limits and conventions. + ### `dag-pb` `Symlink` A :dfn[Symlink] represents a POSIX [symbolic link](https://pubs.opengroup.org/onlinepubs/9699919799/functions/symlink.html). From fac6386642b1308d754f2dad162808867c59fe3d Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sun, 24 Aug 2025 05:28:11 +0200 Subject: [PATCH 33/46] docs(unixfs): improve CID verification clarity https://github.com/ipfs/specs/pull/331#discussion_r1674333701 --- src/unixfs.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/unixfs.md b/src/unixfs.md index 955a4f2b8..7e6fb4743 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -59,8 +59,8 @@ required. A [CID] includes two important pieces of information: 2. A [multihash] used to specify the hashing algorithm, the hash parameters and the hash digest. -Thus, the block must be retrieved; that is, the bytes which ,when hashed using the -hash function specified in the multihash, gives us the same multihash value back. +Thus, when a block is retrieved and its bytes are hashed using the +hash function specified in the multihash, this gives the same multihash value contained in the CID. In UnixFS, a node can be encoded using two different multicodecs, listed below. More details are provided in the following sections: From 50f6343e401003217a804c1d52b6f84b421d54a7 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Sun, 24 Aug 2025 05:42:07 +0200 Subject: [PATCH 34/46] docs(unixfs): convert protobuf definitions to proto3 syntax Remove optional/required keywords for proto3 compatibility. Add comments for semantically required fields that must be validated at the application layer since proto3 doesn't support required fields. https://github.com/ipfs/specs/pull/331#discussion_r1674347918 --- src/unixfs.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/src/unixfs.md b/src/unixfs.md index 7e6fb4743..8bdd98283 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -94,13 +94,13 @@ message PBLink { // This contains raw CID bytes (either CIDv0 or CIDv1) with no multibase prefix. // CIDv1 is a binary format composed of unsigned varints, while CIDv0 is a raw multihash. // In both cases, the bytes are stored directly without any additional prefix. - optional bytes Hash = 1; + bytes Hash = 1; // UTF-8 string name - optional string Name = 2; + string Name = 2; // cumulative size of target object - optional uint64 Tsize = 3; + uint64 Tsize = 3; } message PBNode { @@ -108,7 +108,7 @@ message PBNode { repeated PBLink Links = 2; // opaque user data - optional bytes Data = 1; + bytes Data = 1; } ``` @@ -127,23 +127,23 @@ message Data { HAMTShard = 5; } - required DataType Type = 1; - optional bytes Data = 2; // file content (File), symlink target (Symlink), bitmap (HAMTShard), unused (Directory) - optional uint64 filesize = 3; // mandatory for Type=File and Type=Raw, defaults to 0 if omitted + DataType Type = 1; // MUST be present - validate at application layer + bytes Data = 2; // file content (File), symlink target (Symlink), bitmap (HAMTShard), unused (Directory) + uint64 filesize = 3; // mandatory for Type=File and Type=Raw, defaults to 0 if omitted repeated uint64 blocksizes = 4; // required for multi-block files (Type=File) with Links - optional uint64 hashType = 5; // required for Type=HAMTShard (currently always murmur3-x64-64) - optional uint64 fanout = 6; // required for Type=HAMTShard (power of 2, max 1024) - optional uint32 mode = 7; // opt-in, AKA UnixFS 1.5 - optional UnixTime mtime = 8; // opt-in, AKA UnixFS 1.5 + uint64 hashType = 5; // required for Type=HAMTShard (currently always murmur3-x64-64) + uint64 fanout = 6; // required for Type=HAMTShard (power of 2, max 1024) + uint32 mode = 7; // opt-in, AKA UnixFS 1.5 + UnixTime mtime = 8; // opt-in, AKA UnixFS 1.5 } message Metadata { - optional string MimeType = 1; // reserved for future use + string MimeType = 1; // reserved for future use } message UnixTime { - required int64 Seconds = 1; - optional fixed32 FractionalNanoseconds = 2; + int64 Seconds = 1; // MUST be present when UnixTime is used + fixed32 FractionalNanoseconds = 2; } ``` From 7a060724c3d6b6384f6e7be172da9376817bf960 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Mon, 25 Aug 2025 19:47:44 +0200 Subject: [PATCH 35/46] docs: apply text/grammar fixes from willscott's review Applied text suggestions from PR #331 review comments: - https://github.com/ipfs/specs/pull/331#discussion_r2298237926 - https://github.com/ipfs/specs/pull/331#discussion_r2298243882 - https://github.com/ipfs/specs/pull/331#discussion_r2298246458 - https://github.com/ipfs/specs/pull/331#discussion_r2298256851 Note: Suggestions r1039081771, r1039082139, r1039082250 were for the old UNIXFSv1.md file and don't apply to the current unixfs.md which has already been revised. --- src/unixfs.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/unixfs.md b/src/unixfs.md index 8bdd98283..9d9e5dc60 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -2,7 +2,7 @@ title: UnixFS description: > UnixFS is a Protocol Buffers-based format for describing files, directories, - and symlinks as dag-pb and raw DAGs in IPFS. + and symlinks as dag-pb DAGs and raw blocks in IPFS. date: 2025-08-23 maturity: draft editors: @@ -113,7 +113,7 @@ message PBNode { ``` After decoding the node, we obtain a `PBNode`. This `PBNode` contains a field -`Data` that contains the bytes that require the second decoding. These are also +`Data` that contains the bytes that require the second decoding. This will also be a protobuf message specified in the UnixFSV1 format: ```protobuf @@ -150,7 +150,7 @@ message UnixTime { Summarizing, a `dag-pb` UnixFS node is a [`dag-pb`][ipld-dag-pb] protobuf, whose `Data` field is a UnixFSV1 Protobuf message. For clarity, the specification document may represent these nested Protobufs as one object. In this representation, -it is implied that the `PBNode.Data` field is encoded in a protobuf. +it is implied that the `PBNode.Data` field is protobuf-encoded. ## `dag-pb` Types @@ -267,7 +267,7 @@ A :dfn[Directory], also known as folder, is a named collection of child [Nodes]( have the same `Name`. Names are considered identical if they are byte-for-byte equal (not just semantically equivalent). If two identical names are present in a directory, the decoder MUST fail. -- Implementations SHOULD detect when directory becomes too big to fit in a single +- Implementations SHOULD detect when a directory becomes too big to fit in a single `Directory` block and use [`HAMTDirectory`] type instead. The `PBNode.Data` field MUST contain valid UnixFS protobuf data for all UnixFS nodes. From 06df67466dcd5d56e79117032700ecef99dc0110 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Mon, 25 Aug 2025 20:08:45 +0200 Subject: [PATCH 36/46] docs: remove TODO and clarify symlinks cannot have children Verified through code inspection that symlinks MUST NOT have children in PBNode.Links. Removed the TODO comment and made the requirement explicit. Addresses: https://github.com/ipfs/specs/pull/331#discussion_r1039083293 --- src/unixfs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/unixfs.md b/src/unixfs.md index 9d9e5dc60..9ebaa6ed0 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -455,7 +455,7 @@ See [Block Size Considerations](#block-size-considerations) for details on block ### `dag-pb` `Symlink` A :dfn[Symlink] represents a POSIX [symbolic link](https://pubs.opengroup.org/onlinepubs/9699919799/functions/symlink.html). -A symlink MUST NOT have children. +A symlink MUST NOT have children in `PBNode.Links`. The `PBNode.Data.Data` field is a POSIX path that MAY be inserted in front of the currently remaining path component stack. From a479f22160fec81b0110574cf24064ee04f3d5b5 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Mon, 25 Aug 2025 20:11:54 +0200 Subject: [PATCH 37/46] docs: clarify raw codec preference for single-block files Added note that single-block files SHOULD prefer raw codec (0x55) over dag-pb for canonical CID to avoid protobuf overhead. Addresses: https://github.com/ipfs/specs/pull/331#discussion_r2298249871 --- src/unixfs.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/unixfs.md b/src/unixfs.md index 9ebaa6ed0..1f9e1939d 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -162,6 +162,12 @@ A `dag-pb` UnixFS node supports different types, which are defined in A :dfn[File] is a container over an arbitrary sized amount of bytes. Files are either single block or multi-block. A multi-block file is a concatenation of multiple child files. +:::note +Single-block files SHOULD prefer the `raw` codec (0x55) over `dag-pb` for the canonical CID, +as it's more efficient and avoids the protobuf overhead. The `raw` encoding is described +in the [`raw` Node](#raw-node) section. +::: + #### The _sister-lists_ `PBNode.Links` and `decode(PBNode.Data).blocksizes` The _sister-lists_ are the key point of why `dag-pb` is important for files. They From 775af4ca3670d83864e623cd836759889af4a82b Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Mon, 25 Aug 2025 20:18:57 +0200 Subject: [PATCH 38/46] docs: clarify path resolution context in UnixFS Paths section Added note explaining that path resolution is mostly IPFS semantics over UnixFS, except for types like HAMTDirectory where the resolution algorithm is part of the data structure itself. Addresses: https://github.com/ipfs/specs/pull/331#discussion_r2298255011 --- src/unixfs.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/unixfs.md b/src/unixfs.md index 1f9e1939d..c1e4a44a7 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -607,6 +607,14 @@ non-IPFS target MUST observe the following: ## UnixFS Paths +:::note +Path resolution describes how IPFS systems traverse UnixFS DAGs. While path resolution +behavior is mostly IPFS semantics layered over UnixFS data structures, certain UnixFS +types (notably HAMTDirectory) define specific resolution algorithms as part of their +data structure specification. Each UnixFS type includes a "Path Resolution" subsection +documenting its specific requirements. +::: + Paths begin with a `/` or `/ipfs//`, where `` is a [multibase] encoded [CID]. The CID encoding MUST NOT use a multibase alphabet that contains `/` (`0x2f`) unicode codepoints. However, CIDs may use a multibase encoding with From ce2c71e76cbb73abd6f8c911e9f995b2cadf4578 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Mon, 25 Aug 2025 20:37:54 +0200 Subject: [PATCH 39/46] docs: improve HAMT fanout value guidance Simplified MUST requirements, clarified fanout MUST be multiple of 8, and added clearer guidance on choosing fanout values (256 vs 1024) with their trade-offs. Addresses: https://github.com/ipfs/specs/pull/331#discussion_r2298263710 --- src/unixfs.md | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/src/unixfs.md b/src/unixfs.md index c1e4a44a7..6b22a201d 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -331,15 +331,23 @@ The HAMT directory is configured through the UnixFS metadata in `PBNode.Data`: the path components for sharding. Currently, all HAMT implementations use `murmur3-x64-64` (`0x22`), and this value MUST be consistent across all shards within the same HAMT structure - `decode(PBNode.Data).fanout` is REQUIRED for HAMTShard nodes (though marked optional in the - protobuf schema). The value MUST be a power of two and at most 1024. Implementations SHOULD - require fanout to be a multiple of 8 to ensure the bitfield aligns to byte boundaries. - Popular values are: 256, 512, 1024. The most common fanout is 256 (8-bit buckets), - providing a good balance between tree depth and node size. + protobuf schema). The value MUST be a power of two, a multiple of 8 (for byte-aligned + bitfields), and at most 1024. This determines the number of possible bucket indices (permutations) at each level of the trie. For example, fanout=256 provides 256 possible buckets (0x00 to 0xFF), requiring 8 bits from the hash. The hex prefix length is `log2(fanout)/4` characters (since each hex character represents 4 bits). The same fanout value is used throughout all levels of a single HAMT structure + + :::note + Implementations that onboard user data to create new HAMTDirectory structures are free to choose a `fanout` value or allow users to configure it based on their use case: + - **256**: Balanced tree depth and node size, suitable for most use cases + - **1024**: Creates wider, shallower DAGs with fewer levels + - Advantages: Minimizes tree depth for faster lookups, reduces number of intermediate nodes to traverse + - Trade-offs: Larger blocks mean higher latency on cold cache reads and more data + rewritten when modifying directories (each change affects a larger block) + ::: + :::warning Implementations MUST limit the `fanout` parameter to a maximum of 1024 to prevent denial-of-service attacks. Excessively large fanout values can cause memory exhaustion @@ -350,7 +358,7 @@ The HAMT directory is configured through the UnixFS metadata in `PBNode.Data`: - `decode(PBNode.Data).Data` contains a bitfield indicating which buckets contain entries. Each bit corresponds to one bucket (0 to fanout-1), with bit value 1 indicating the bucket is occupied. The bitfield is stored in little-endian byte order. The bitfield size in bytes - is `fanout/8`, which is why fanout SHOULD be a multiple of 8. + is `fanout/8`, which is why fanout MUST be a multiple of 8. - Implementations MUST write this bitfield when creating HAMT nodes - Implementations SHOULD use this bitfield for efficient traversal (checking which buckets exist without examining all links) From 237b67fa4272d81b1f9648fd89a03aea4f30490d Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Mon, 25 Aug 2025 21:49:21 +0200 Subject: [PATCH 40/46] clarify why lexicographic sorting was chosen for directories adds rationale for the specific sorting choice - universal and locale-independent ordering across implementations https://github.com/ipfs/specs/pull/331#discussion_r2027073454 --- src/unixfs.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/unixfs.md b/src/unixfs.md index 6b22a201d..ff6e69a5b 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -198,6 +198,10 @@ size in bytes of the partial file content present in children DAGs. Each index i `PBNode.Links` MUST have a corresponding chunk size stored at the same index in `decode(PBNode.Data).blocksizes`. +The child blocks containing the partial file data can be either: +- `raw` blocks (0x55): Direct file data without protobuf wrapper +- `dag-pb` blocks (0x70): File data wrapped in protobuf, potentially with further children + :::warning Implementers need to be extra careful to ensure the values in `Data.blocksizes` are calculated by following the definition from [`Blocksize`](#decodepbnodedatablocksize). @@ -302,7 +306,9 @@ not on read" approach maintains DAG stability - existing unsorted directories re when accessed or traversed, preventing unintentional mutations of intermediate nodes that could alter their CIDs. -Note: Sorting on write (when the Links list is modified) helps with deduplication detection and enables more +Note: Lexicographic sorting was chosen as the standard because it provides a universal, +locale-independent ordering that works consistently across all implementations and languages. +Sorting on write (when the Links list is modified) helps with deduplication detection and enables more efficient directory traversal algorithms in some implementations. #### `dag-pb` `Directory` Path Resolution From 719d0bac0d2f9ab0d7ae790aedfc1790c0c15a31 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Mon, 25 Aug 2025 22:15:37 +0200 Subject: [PATCH 41/46] clarify duplicate name handling in directories specifies that duplicates are not allowed but readers must handle them gracefully by returning first match, following robustness principle. writers must clean up duplicates when mutating. resolves inline TODO at line 322 --- src/unixfs.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/src/unixfs.md b/src/unixfs.md index ff6e69a5b..7180a60dc 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -313,10 +313,16 @@ efficient directory traversal algorithms in some implementations. #### `dag-pb` `Directory` Path Resolution -Pop the left-most component of the path, and try to match it to the `Name` of -a child under `PBNode.Links`. If you find a match, you can then remember the CID. -You MUST continue the search. If you find another match, you MUST reject the directory -and halt path resolution since duplicate names are not allowed. +Pop the left-most component of the path, and match it to the `Name` of +a child under `PBNode.Links`. + +Duplicate names are not allowed in UnixFS directories. However, when reading +third-party data that contains duplicates, implementations MUST always return +the first matching entry and ignore subsequent ones (following the +[Robustness Principle](https://specs.ipfs.tech/architecture/principles/#robustness)). +Similarly, when writers mutate a UnixFS directory that has duplicate +names, they MUST drop the redundant entries and only keep the first occurrence +of each name. Assuming no errors were raised, you can continue to the path resolution on the remaining components and on the CID you popped. From 396bc4d488aa3b2f66dd6082aaaf9288ebd9606e Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Mon, 25 Aug 2025 22:42:25 +0200 Subject: [PATCH 42/46] clarify empty string restriction in directories removes TODO and documents that POSIX explicitly prohibits zero-length filenames resolves inline TODO at line 684 --- src/unixfs.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/unixfs.md b/src/unixfs.md index 7180a60dc..9d9fde047 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -677,11 +677,11 @@ Relative path components MUST be resolved before trying to work on the path: ### Restricted Names -The following names SHOULD NOT be used: +The following names SHOULD NOT be used in UnixFS directories: - The `.` string, as it represents the self node in POSIX pathing. - The `..` string, as it represents the parent node in POSIX pathing. -- The empty string. +- The empty string, as POSIX explicitly prohibits zero-length filenames - Any string containing a `NULL` (`0x00`) byte, as this is often used to signify string terminations in some systems, such as C-compatible systems. Many unix file systems do not accept this character in path components. From db12679e3362a50013510af1382efb54351aedee Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Mon, 25 Aug 2025 22:46:15 +0200 Subject: [PATCH 43/46] remove TODO about rust libraries rust-ipfs was archived in 2022 and unixfsv1 is not actively maintained, confirming they should remain hidden resolves inline TODO at line 1040 --- src/unixfs.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/src/unixfs.md b/src/unixfs.md index 9d9fde047..6a2ab90da 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -1037,11 +1037,6 @@ This section and included subsections are not authoritative. - [`ipfs/boxo/ipld/unixfs`](https://github.com/ipfs/boxo/tree/main/ipld/unixfs/) - Alternative `go-ipld-prime` implementation: [`ipfs/go-unixfsnode`](https://github.com/ipfs/go-unixfsnode) - ## Block Size Considerations From 0a111862bbf32f25a0e3382e96376f6fab058772 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Tue, 26 Aug 2025 01:42:45 +0200 Subject: [PATCH 44/46] fix: remove trailing spaces and extra blank line fixes markdown linting issues found by superlinter: - removed trailing spaces from lines 259, 319-324 - removed extra blank line at line 1040 --- src/unixfs.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/src/unixfs.md b/src/unixfs.md index 6a2ab90da..701da60a0 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -256,7 +256,7 @@ Examples of where `blocksize` is useful: #### `decode(PBNode.Data).filesize` -For `Type=File` (0) and `Type=Raw` (2), this field is mandatory. While marked as "optional" +For `Type=File` (0) and `Type=Raw` (2), this field is mandatory. While marked as "optional" in the protobuf schema (for compatibility with other types like Directory), implementations: - MUST include this field when creating File or Raw nodes - When reading, if this field is absent, MUST interpret it as 0 (zero-length file) @@ -316,12 +316,12 @@ efficient directory traversal algorithms in some implementations. Pop the left-most component of the path, and match it to the `Name` of a child under `PBNode.Links`. -Duplicate names are not allowed in UnixFS directories. However, when reading -third-party data that contains duplicates, implementations MUST always return -the first matching entry and ignore subsequent ones (following the -[Robustness Principle](https://specs.ipfs.tech/architecture/principles/#robustness)). -Similarly, when writers mutate a UnixFS directory that has duplicate -names, they MUST drop the redundant entries and only keep the first occurrence +Duplicate names are not allowed in UnixFS directories. However, when reading +third-party data that contains duplicates, implementations MUST always return +the first matching entry and ignore subsequent ones (following the +[Robustness Principle](https://specs.ipfs.tech/architecture/principles/#robustness)). +Similarly, when writers mutate a UnixFS directory that has duplicate +names, they MUST drop the redundant entries and only keep the first occurrence of each name. Assuming no errors were raised, you can continue to the path resolution on the @@ -1037,7 +1037,6 @@ This section and included subsections are not authoritative. - [`ipfs/boxo/ipld/unixfs`](https://github.com/ipfs/boxo/tree/main/ipld/unixfs/) - Alternative `go-ipld-prime` implementation: [`ipfs/go-unixfsnode`](https://github.com/ipfs/go-unixfsnode) - ## Block Size Considerations While UnixFS itself does not mandate specific block size limits, implementations typically From 8911276f68caadb89b34014ee13d38672533b153 Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Tue, 26 Aug 2025 03:04:51 +0200 Subject: [PATCH 45/46] feat: add well-known UnixFS CIDs section documents commonly encountered empty structures including: - empty dag-pb directories (CIDv0, CIDv1, inlined) - empty dag-pb files (CIDv0, CIDv1) - empty raw blocks (CIDv1, inlined) these CIDs are frequently hardcoded in implementations for performance optimization --- src/unixfs.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/src/unixfs.md b/src/unixfs.md index 701da60a0..00d8c3859 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -934,6 +934,25 @@ Test vectors for UnixFS directory structures, progressing from simple flat direc Test vectors for special UnixFS features and edge cases. +### Well-Known UnixFS CIDs + +Common empty structures that implementations frequently encounter: + +* **Empty dag-pb directory** + - **CIDv0**: `QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn` + - **CIDv1**: `bafybeiczsscdsbs7ffqz55asqdf3smv6klcw3gofszvwlyarci47bgf354` + - **Inlined**: `bafyaabakaieac` (identity multihash) + +* **Empty dag-pb file** + - **CIDv0**: `QmbFMke1KXqnYyBBWxB74N4c5SBnJMVAiMNRcGu6x1AwQH` + - **CIDv1**: `bafybeif7ztnhq65lumvvtr4ekcwd2ifwgm3awq4zfr3srh462rwyinlb4y` + +* **Empty raw block** + - **CIDv1**: `bafkreihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku` + - **Inlined**: `bafkqaaa` (identity multihash) + +These CIDs appear frequently in UnixFS implementations and are often hardcoded for performance optimization. + ### Symbolic Links - Fixture: [`symlink.car`](https://github.com/ipfs/gateway-conformance/raw/refs/tags/v0.8.1/fixtures/path_gateway_unixfs/symlink.car) From 0a107a079148c8e2df4f2e0babc89dd0db90213d Mon Sep 17 00:00:00 2001 From: Marcin Rataj Date: Thu, 28 Aug 2025 02:26:32 +0200 Subject: [PATCH 46/46] chore: update UnixFS spec contributors moved Jorropo to former_editors as he authored the comprehensive UnixFS spec rewrite in PR #331 added contributors from PR #331 review process to thanks section, ordered chronologically by first contribution to ipfs ecosystem touching UnixFS, including useful code comments that helped shaping this spec: - earliest contributions to ipfs/specs repo (2015-2021) - UnixFS implementation work in kubo/boxo (2019-2020) - PR #331 reviews and feedback (2022-2025) --- src/unixfs.md | 43 +++++++++++++++++++++++++++++++++---------- 1 file changed, 33 insertions(+), 10 deletions(-) diff --git a/src/unixfs.md b/src/unixfs.md index 00d8c3859..ede3e5524 100644 --- a/src/unixfs.md +++ b/src/unixfs.md @@ -11,15 +11,27 @@ editors: affiliation: name: Interplanetary Shipyard url: https://ipshipyard.com/ -contributors: +former_editors: - name: Hugo Valtier - github: jorropo + github: Jorropo affiliation: name: Interplanetary Shipyard url: https://ipshipyard.com/ +contributors: thanks: + - name: David Dias + github: daviddias + - name: Łukasz Magiera + github: magik6k - name: Jeromy Johnson github: whyrusleeping + - name: Alex Potsides + github: achingbrain + affiliation: + name: Interplanetary Shipyard + url: https://ipshipyard.com/ + - name: Peter Rabbitson + github: ribasushi - name: Steven Allen github: Stebalien - name: Hector Sanjuan @@ -27,20 +39,31 @@ thanks: affiliation: name: Interplanetary Shipyard url: https://ipshipyard.com/ - - name: Łukasz Magiera - github: magik6k - - name: Alex Potsides - github: achingbrain + - name: Henrique Dias + github: hacdias affiliation: name: Interplanetary Shipyard url: https://ipshipyard.com/ - - name: Peter Rabbitson - github: ribasushi - - name: Henrique Dias - github: hacdias + - name: Marten Seemann + github: marten-seemann + - name: Adin Schmahmann + github: aschmahmann + affiliation: + name: Interplanetary Shipyard + url: https://ipshipyard.com/ + - name: Will Scott + github: willscott + - name: John Turpish + github: John-LittleBearLabs + - name: Alan Shaw + github: alanshaw + - name: Andrew Gillis + github: gammazero affiliation: name: Interplanetary Shipyard url: https://ipshipyard.com/ + - name: bumblefudge + github: bumblefudge tags: ['data-formats'] order: 1