Specification: Content Addressable aRchives (CAR / .car) v1

Status: Final

Summary

The CAR format (Content Addressable aRchives) can be used to store content addressable objects in the form of IPLD block data as a sequence of bytes; typically in a file with a .car filename extension.

The CAR format is intended as a serialized representation of any IPLD DAG (graph) as the concatenation of its blocks, plus a header that describes the graphs in the file (via root CIDs). The requirement for the blocks in a CAR to form coherent DAGs is not strict, so the CAR format may also be used to store arbitrary IPLD blocks.

In addition to the binary block data, storage overhead for the CAR format consists of:

This diagram shows how IPLD blocks, their root CID, and a header combine to form a CAR.

Content Addressable aRchive Diagram

The name Certified ARchive has also previously been used to refer to the CAR format.

Format Description

The CAR format comprises a sequence of length-prefixed IPLD block data, where the first block in the CAR is the Header encoded as CBOR, and the remaining blocks form the Data component of the CAR and are each additionally prefixed with their CIDs. The length prefix of each block in a CAR is encoded as a "varint"—an unsigned LEB128 integer. This integer specifies the number of remaining bytes for that block entry—excluding the bytes used to encode the integer, but including the CID for non-header blocks.

|--------- Header --------| |---------------------------------- Data -----------------------------------|

[ varint | DAG-CBOR block ] [ varint | CID | block ] [ varint | CID | block ] [ varint | CID | block ] …

The first bytes of the CAR format hold a varint, this unsigned integer specifies the number of bytes beyond the varint itself that contain the Header block. This Header block is a byte array DAG-CBOR (CBOR with tag 42 for CIDs) encoded object holding the version number and array of roots. As an IPLD Schema:

type CarHeader struct {
  version Int
  roots [&Any]
}

Constraints

(Caveats: see Number of roots and Root CID block existence under Unresolved Issues.)

Data

Immediately following the Header block, one or more IPLD blocks are concatenated to form the Data section of the CAR format. (Caveat: see Zero blocks under Unresolved Issues.) Each block is encoded into a Section by the concatenation of the following values:

  1. Length in bytes of the combined CID and data in this Section, encoded as a varint
  2. CID of the block in this Section, encoded in the raw byte form of the CID
  3. Binary data of the block

Length

Each Section begins with a varint representation of an unsigned integer indicating the number of bytes containing the remainder of the section.

CID

Following the Length, the CID of the block is included in raw byte form. A decoder reading a Section must decode the CID according to CID byte encoding rules, which don't provide a stable length. See https://github.com/multiformats/cid for details on the encoding of a CID. CIDv0 and CIDv1 are currently supported. (Caveat: see CID version under Unresolved Issues.)

CID byte decoding summary

See the CID specification for full details.

A CIDv0 is indicated by a first byte of 0x12 followed by 0x20 which specifies a 32-byte (0x20) length SHA2-256 (0x12) digest.

Failure to find 0x12, 0x20 indicates a CIDv1 which is decoded by reading:

  1. Version as an unsigned varint (should be 1)
  2. Codec as an unsigned varint (valid according to the multicodec table)
  3. The raw bytes of a multihash

Reading the multihash requires a partial decode in order to determine the length:

| hash function code (varint) | digest size (varint) | digest |

The first two bytes of a multihash are varints, where the second varint is an unsigned integer indicating the length of the remaining portion of the multihash. Therefore, a manual decode requires two varint reads and then copying the bytes of those varints in addition to the number of bytes indicated by the second varint into a byte array.

Data

The remainder of a Section, after length-prefix and CID, comprises the raw byte data of the IPLD block. The encoded block may be any IPLD block format as specified by the codec in the CID. Typical codecs will be DAG-PB, DAG-CBOR or RAW.

Additional Considerations

Determinism

Deterministic CAR creation is not covered by this specification. However, deterministic generation of a CAR from a given graph is possible and is relied upon by certain uses of the format, most notably, Filecoin. Specifically a filecoin-deterministic car-file is currently implementation-defined as containing all DAG-forming blocks in first-seen order, as a result of a depth-first DAG traversal starting from a single root.

Additional rules for the generation of the CAR format may be applied in order to ensure that the same CAR is always generated from the same data. The burden of this determinism is primarily placed on selectors whereby a given selector applied to a given graph will always yield blocks in the same order regardless of implementation.

Care regarding the ordering of the roots array in the Header, as well as consideration for CID version (see below) and avoidance of duplicate blocks (see below) may also be required for strict determinism.

All such considerations are deferred to the user of the CAR format and should be documented there as this specification does not inherently support determinism.

Performance

Some considerations regarding performance:

Security and Verifiability

The roots specified by the Header of a CAR must appear somewhere in its Data section, however there is no requirement that the roots define entire DAGs, nor that all blocks in a CAR must be part of DAGs described by the root CIDs in the Header. Therefore, the roots must not be used alone to determine or differentiate the contents of a CAR.

The CAR format contains no internal means, beyond the IPLD block formats and their CIDs, to verify or differentiate contents. Where such a requirement exists, this must be performed externally, such as creating a digest of the entire CAR.

Indexing and Seeking Reads

The CAR format contains no internal indexing, any indexing must be stored externally to a CAR. However, such indexing is possible and makes seeking reads practical. An index storing byte offset (of Section start or block data start) and length (of Section or block data), keyed by CID, will enable a single block read by seeking to the offset and reading the block data. The format of any such index is not specified here and is left up to CAR format parsing implementations.

Padding

The CAR format contains no specified means of padding to achieve specific total archive sizes or internal byte offset alignment of block data. Because it is not a requirement that each block be part of a coherent DAG under one of the roots of the CAR, dummy block entries may be used to achieve padding. Such padding should also account for the size of the length-prefix varint and the CID for a section. All sections must be valid and dummy entries should still decode to valid IPLD blocks.

Implementations

Go

https://github.com/ipld/go-car

As used in Filecoin for genesis block sharing. Supports creation via a DAG walk from a datastore:

WriteCar(ctx context.Context, ds format.DAGService, roots []cid.Cid, w io.Writer) (error)

And writing to a data store via Put(block) operations:

LoadCar(s Store, r io.Reader) (*CarHeader, error)

JavaScript

https://github.com/ipld/js-car

@ipld/car is consumed through factory methods on its different classes. Each class represents a discrete set of functionality, such as writing or reading .car files:

import fs from 'fs'
import { Readable } from 'stream'
import { CarReader, CarWriter } from '@ipld/car'
import * as raw from 'multiformats/codecs/raw'
import { CID } from 'multiformats/cid'
import { sha256 } from 'multiformats/hashes/sha2'

async function example () {
  const bytes = new TextEncoder().encode('random meaningless bytes')
  const hash = await sha256.digest(raw.encode(bytes))
  const cid = CID.create(1, raw.code, hash)
  const { writer, out } = await CarWriter.create([cid])
  Readable.from(out).pipe(fs.createWriteStream('example.car'))

  await writer.put({ cid, bytes })
  await writer.close()

  const inStream = fs.createReadStream('example.car')
  const reader = await CarReader.fromIterable(inStream)
  const roots = await reader.getRoots()
  const got = await reader.get(roots[0])

  console.log('Retrieved [%s] from example.car with CID [%s]',
    new TextDecoder().decode(got.bytes),
    roots[0].toString())
}

Unresolved Items

Number of roots

Regarding the roots property of the Header block:

It is unresolved how the roots array should be constrained. It is recommended that only a single root CID be used in this version of the CAR format.

A work-around for use-cases where the inclusion of a root CID is difficult but needing to be safely within the "at least one" recommendation is to use an empty CID: \x01\x55\x00\x00 (zero-length "identity" multihash with "raw" codec). Since current implementations for this version of the CAR specification don't check for the existence of root CIDs (see Root CID block existence), this will be safe as far as CAR implementations are concerned. However, there is no guarantee that applications that use CAR files will correctly consume (ignore) this empty root CID.

Zero blocks

It is unresolved whether a valid CAR must contain at least one block or whether the empty CAR is a valid format and should be accepted by encoders and decoders.

Root CID block existence

It is unresolved whether an implementation must verify that a CID present in the roots array of the Header also appears as a block in the archive. While it is expected that this would be the case, it is unresolved whether encoders and decoders must validate the existence of root blocks in the archive.

Current implementations of this version of the CAR specification do not check for root block existence in the CAR body.

CID version

It is unresolved whether both CID versions 0 and 1 format are valid in the roots array and at the start of each block Section. Current implementations do not check CID version in the roots array, and both CID versions are also acceptable in each block Section. Discussions on this specification have suggested limiting CIDs used throughout the format (not within blocks) to CIDv1—requiring conversion if an encoder is provided with a CIDv0 and requiring readers of a CAR to ensure CIDv1 is the only available block key.

Duplicate Blocks

The possibility of duplicate blocks in a single CAR (such as for padding) is currently not specified.

Test Fixtures

To assist implementations in confirming compliance to this specification, the following test fixtures are available: