LayerFS

Ephemeral Workspaces. Durable Shared History.

LayerFS gives every agent an isolated, disposable filesystem fork without copying the shared base. Useful states become durable, deduplicated checkpoints with workspace-scoped tool history—ready to branch, rewind, or reuse across parallel development, environment experiments, and MCTS-style rollouts.

01 · LayerStack storage model

Core storage mechanisms

CAS, CDC, and COW make LayerStack history storage-efficient by reusing unchanged objects, file regions, and filesystem structure.

01 · Identity

Content-addressed storage

Names immutable objects from their canonical bytes, verifies reads, and reuses exact duplicates across files, layer stacks, and agents.

02 · Byte locality

Content-defined chunking

Keeps chunk boundaries stable around localized edits, so changing a small region does not require storing an entirely new large file.

03 · Structural locality

Copy-on-write

Publishes a change in a new layer by rebuilding only the changed file and directory path while preserving every unchanged subtree from its parent.

Check out an ephemeral filesystem from any layer

A LayerStack records complete filesystem checkpoints. Agent A can start from L1 while Agent B independently starts from L3. Each gets a private place to work while the selected history remains shared.

03 · Reading path

Build it from first principles

The book follows the same dependency order as the storage model: establish immutable content identity, localize file edits, rebuild only changed filesystem paths, and then organize those states into agent histories.

04 · Ecosystem

Collaborating projects

LayerFS supplies storage mechanics. The surrounding projects own execution environments and version-control workflows.

↗ AgentsGit Version control for agent work in motion: checkpoint, branch, compare, recover, and promote. ↗ Ephemeral Sandbox Isolated execution environments for parallel agents.

Powered by Ephemeral AI Lab

Chapter 1 · Foundations

Foundations: CAS + CDC + COW

A filesystem state should be complete logically but incremental physically. That is a requirement—not a later optimization—when many agents explore from the same environment.

Why storage efficiency is a design constraint

LayerFS is designed for one environment to support many isolated agent workspaces. Each workspace must behave like a complete filesystem: tools can modify it, checkpoint it, fork it, resume it, compare it, or roll it back.

The straightforward implementation is to copy the current filesystem whenever an agent forks or records a checkpoint. That provides isolation, but it gives these operations the wrong cost model: a small edit to one workspace can require storing another copy of the entire workspace.

Multi-agent development makes that mismatch fundamental rather than incidental:

• Shared origin. Agents usually begin from the same repository, dependencies, and generated environment.
• Sparse change. A tool call normally changes a small set of files relative to that shared state.
• High fan-out and history. Parallel attempts, environment experiments, and MCTS-style rollouts create many related states, while checkpoints retain earlier ones.

If every logical state owns its bytes, storage grows with the total size of every workspace. LayerFS instead chooses a different invariant: the physical cost of a new state should follow what changed, not the size of the filesystem that state exposes.

This choice shapes the filesystem model. A workspace starts from an immutable root. Reads reuse objects reachable from that root; writes create new objects without modifying the shared base; a checkpoint records a new root that still references everything unchanged. Isolation comes from giving each agent its own evolving root—not from duplicating all of its bytes.

How CAS + CDC + COW implement the choice

Structural sharing can be lost at the object, file, or tree level. LayerFS therefore applies reuse at all three levels, and Chapter 1 builds them in that order:

Content-Addressed Storage

Most storage systems name data by where it is stored: a pathname, an object key, a URL, or a database row. Content-addressed storage (CAS) uses a different rule. It derives an object’s name from the object’s bytes, then uses that name to store, retrieve, reuse, and verify the object.

That rule is small, but it changes what a storage reference means. A path such as /docs/report.txt can point to different bytes tomorrow. A content address continues to identify the same bytes; changing the bytes produces a new address.

This section develops a vanilla CAS from first principles. It covers the architecture, core algorithm, exact-content deduplication, real-world applications, and why file-scale objects need a chunked representation. It does not yet cover the content-defined boundary algorithm, LayerFS object formats, typed identities, object graphs, admission, packs, or production publication. Those mechanisms make sense only after the basic CAS contract is clear.

1. The Address Is the Content

Consider a conventional filesystem path:

/docs/report.txt

The path tells the filesystem where to look. It does not permanently identify the bytes found there. A process can overwrite the file while keeping the same path, or move the same file to a different path. The relationship between name and content is mutable.

A content-addressed store starts from the opposite direction:

object address = hash(object bytes)

The address is derived from the content rather than chosen independently. If the content remains unchanged, so does the address. If the content changes, its address changes too.

The distinction is not that one system has locations and the other does not. Every physical store eventually places bytes somewhere. The distinction is what the public identity means:

This gives CAS a useful vocabulary for immutable data. Two callers referring to the same content address mean the same expected bytes, even if the storage system later changes the object’s physical placement.

It also changes lookup. CAS does not search the store for bytes that resemble a request. The caller already has an object identifier. The store uses that identifier to locate the object directly, just as a key-value store uses a key. The difference is that the CAS key is derived from the value.

2. Identity from Bytes

Let x be an object’s byte sequence and H a cryptographic hash function. A minimal content identity is:

id(x) = H(x)

The hash function processes an input of arbitrary length and produces a fixed-length digest. A CAS depends on several properties:

• Determinism: hashing the same bytes with the same algorithm produces the same digest.
• Sensitivity to change: changing the input should produce an unrelated digest.
• Collision resistance: finding two different inputs with the same digest should be computationally infeasible for the chosen algorithm.
• Streaming evaluation: the digest can be computed incrementally without loading the whole object into memory.

A digest is not a mathematical proof that collisions are impossible. It is a security assumption: the digest is large enough and the algorithm strong enough that accidental or deliberate collisions are infeasible for the system’s threat model. Long-lived systems also need a way to evolve when a hash algorithm ages; Git’s hash-function transition design is one example of why algorithm choice cannot be assumed permanent.

Identity enables verification

Suppose a caller requests an object by expected_id, and the store returns content. The caller or store recomputes the digest and compares it with the requested ID:

actual_id = H(content)

if actual_id != expected_id:
    return integrity_error

If the values differ, the returned bytes are not the requested object. This can detect corruption, truncation, a faulty locator, or storage returning the wrong object.

This is an integrity check, not a complete security system. It does not prove who created the bytes, whether the bytes are trustworthy, whether they are encrypted, or whether another copy is available for repair.

Identity enables exact deduplication

If an object with address d is already stored, another insertion of the same object can reuse it instead of writing another payload copy. This is exact-content deduplication at the CAS object’s boundary.

Object boundary matters. A CAS storing whole files recognizes two identical files, but a one-byte change makes the edited file a different whole object. Reusing unchanged regions inside the file requires the system to divide the file into smaller objects. Chunking is an additional design choice, not part of the basic CAS definition.

The distinction among CAS, deduplication, and incremental backup is worth making explicit:

These techniques often appear together, especially in backup products, but they are not synonyms. The BlinkDisk CAS overview describes a chunked backup system; the chunking is what allows unchanged regions inside an edited file to be reused. Its separate introductions to deduplication and incremental backup show the related storage and backup concepts. A vanilla CAS can deduplicate exact objects without implementing either chunking or an incremental-backup chain.

3. Architecture: A Minimal Content-Addressed Store

A minimal CAS needs a narrow set of responsibilities:

1. accept a sequence of bytes;
2. derive its object identifier;
3. retain one immutable object for that identifier;
4. locate an object from its identifier; and
5. optionally rehash returned bytes to verify them.

The locator does not have to be a database. A simple store can derive a pathname directly from the digest. A larger store may use an index because objects are packed, remote, replicated, or moved between storage tiers. Both designs preserve the same separation:

logical identity:  which exact bytes?
physical locator:  where are those bytes currently stored?

Above the CAS, an application usually maintains names and relationships that people care about. A backup system maps snapshots and pathnames to stored objects. Git maps trees and commits to objects, then uses mutable branch names to select commits. A container registry maps tags to manifests whose descriptors identify blobs by digest. These upper layers supply meaning; the vanilla CAS stores opaque bytes.

That boundary is deliberate. CAS is easier to reason about when its core contract does not also try to be a filesystem, version-control system, backup catalog, or distributed availability service.

4. Core Algorithm: Put, Get, and Verify

At the conceptual level, a vanilla CAS has two operations:

put(bytes) -> object_id
get(object_id) -> bytes

The core algorithm can be written as:

put(bytes):
    id = hash(bytes)

    if id is not already stored:
        publish bytes as the immutable object named by id

    return id

get(id):
    bytes = locate and read the object named by id

    if hash(bytes) != id:
        return integrity error

    return bytes

The phrase publish ... as immutable hides real engineering work. A production implementation must handle crashes, concurrent writers, a destination that already exists, partial writes, and storage errors without replacing trusted data. Chapter 2 returns to those mechanics. For now, the logical rule is enough: after successful publication, the bytes associated with an object ID never change.

A put operation

Both paths return the same object ID; only the publication work differs.

The same input produces the same candidate ID, so exact duplicates converge on one name. A trustworthy implementation must not blindly trust an existing occupant merely because it has the expected pathname; Chapter 2 will strengthen this reuse path with authenticated admission.

A get operation

A caller supplies the ID of the bytes it expects. The store resolves the ID, reads the bytes, and recomputes the digest. A matching digest authenticates the byte sequence against the requested content identity. A mismatch must be reported rather than returning bytes as though the read succeeded.

Verification can occur at different boundaries. A local store may verify every read, verify during admission and rely on trusted immutable media, or let a client verify a remote response. The placement changes cost and trust assumptions, but not the equation H(bytes) = requested ID.

A worked example

Assume strings are stored as their UTF-8 bytes:

The application submitted 16 logical bytes, but the CAS retained 11 unique payload bytes. The second hello still had to be identified—normally by reading and hashing its five bytes—but it did not require another payload copy.

This small example captures both the strength and the boundary of vanilla CAS:

• exact repeats are cheap to retain because they share an ID;
• any changed byte creates a new whole-object identity; and
• higher-level references are needed to explain why an application cares about either object.

5. Content Addressing in Real Systems

CAS is rarely the whole product. Mature systems use it as a stable object layer and add structures that give objects meaning, reachability, policy, and efficient physical representation.

Git is the most familiar teaching example. Its object database stores content-addressed blobs, trees, commits, and tags, as described in Git Objects. A branch such as main is not itself an immutable content address; it is a mutable reference that selects a commit.

This separation lets Git retain stable historical objects while allowing a branch name to advance. Git also demonstrates that logical identity and physical encoding can evolve separately: packfiles can store objects compactly using deltas without changing the identities used by commits and trees.

Venti is an early, influential example of a network storage system built around content-addressed immutable blocks. Its clients build archival data structures above the block store, reinforcing the same boundary: CAS provides stable blocks, while higher layers provide roots and interpretation.

Container and build systems use the idea for distribution and reuse. The OCI image specification’s descriptors carry digests used to identify and verify referenced content. Bazel remote caching separates a content-addressable store for files from an action cache that maps actions to results. In both cases, CAS is one component in a larger protocol.

Distributed CAS makes another boundary visible. An IPFS content identifier identifies content independently of a particular host, but knowing a CID does not guarantee that a reachable peer currently provides the data. Content identity and content availability are different properties.

Backup systems commonly combine all three concepts introduced earlier: chunking chooses deduplication units, content addressing names them, and snapshot or incremental metadata records recoverable states. Storage savings and restore behavior therefore belong to the complete backup design, not to hashing alone.

6. Why CAS Needs a Chunked Representation

The unit of deduplication in a CAS is the object. If an entire file is one object, two identical files share one identity, but changing one byte changes the identity of the entire file. Without another encoding such as delta compression, the edited version contributes another whole-file payload.

A chunked representation changes the deduplication unit from the whole file to smaller byte regions. Each chunk receives its own content identity, and a small manifest records their order. After a localized edit, unchanged chunks retain their identities and can be reused; only the affected boundary region and the manifest need new objects.

For example, if a 1 GiB file is stored as one CAS object, a one-byte edit can add another 1 GiB payload. With chunking, the added payload is instead proportional to the affected chunking region plus a small manifest. The exact amount depends on chunk size and boundary behavior; chunking improves the reuse opportunity but does not promise that every edit changes exactly one chunk.

CAS supplies identity and exact reuse; chunking chooses the smaller regions that CAS can reuse.

This creates the next design question: where should the boundaries fall? Fixed-size boundaries are simple, but an insertion can shift every later chunk. The next section introduces content-defined chunking, which chooses boundaries from the content so the stream can resynchronize after a localized edit.

Chunking and Content-Defined Boundaries

The previous section established the basic CAS rule: an object’s identity comes from its bytes. This section asks how to represent a large file as smaller, reusable objects without losing the original byte sequence.

CAS can reuse only the object it receives. If an entire file is one object, a one-byte edit gives the edited file a new identity, even though almost all of its bytes may still be unchanged.

The unchanged regions are still present, but whole-file CAS cannot see them. We need a smaller unit of reuse. That smaller unit is a chunk.

1. What Is a Chunk?

Chunking means splitting a byte stream into contiguous pieces. The pieces preserve the original bytes and their order; the representation simply gives us smaller units to store, compare, transfer, and reuse.

Here || means concatenate: read C₀, then C₁, then C₂:

B = C₀ || C₁ || C₂

A chunk is only a range of bytes. It is not a file-format record, a paragraph, or a cryptographic identity. The order of the chunks is part of the representation: the same pieces in a different order describe different bytes.

The file therefore needs a small ordered recipe that says which chunks to read. The chunk data can live in one shared pool:

The second file is not stored as a second full copy. Its index changes from [C1, C2, C3] to [C1, C4, C3]: the unchanged outer chunks are reused, and only the changed middle chunk is new.

A chunk is easy to define. The difficult question is where its end should fall.

2. Why Positional Boundaries Fail

Suppose abcdefghij is split into ranges of three bytes. If X is inserted after c, every later range starts one byte later:

Before:  abcdefghij       → [abc] [def] [ghi] [j]
After:   abcXdefghij      → [abc] [Xde] [fgh] [ij]

The first range can still be reused, but the later ranges no longer contain the same bytes. Their content addresses change too. This is the boundary-shift problem: a local edit has caused every later fixed-position boundary to move.

Fixed-size boundaries are simple and fast, but they follow offsets rather than content. We need a rule that can recognize a suitable boundary from the bytes near the current position.

3. Content-Defined Chunking (CDC)

Content-defined chunking makes the boundary a function of the content. The scanner does not cut only because it has reached a predetermined offset. Instead, the bytes near the current position influence whether the current position is a good boundary.

The same bytes, scanned with the same profile, produce the same boundary decision. A local edit may disturb one decision, but the scan can later recognize unchanged content again instead of permanently losing alignment.

At a high level, CDC repeats one small loop:

read bytes → produce a content signal → decide cut or continue

The next section explains the signal. The section after that explains how FastCDC uses it to control chunk sizes.

4. Rolling Fingerprint and GearHash

Generic rolling hash: the moving-window intuition

The scanner advances one byte at a time while reusing its previous state. The following animation gives the generic rolling-hash intuition:

The window moves from ABCD to BCDE: the outgoing A is removed, the incoming E is added, and the next state reuses work from the previous state. The point is not the particular arithmetic; it is that the scanner does not start from scratch at every position.

GearHash: the fast fingerprint

GearHash is a particularly simple rolling fingerprint for CDC. For each input byte bᵢ, it updates the previous fingerprint with a shift and a lookup in a fixed table:

fpᵢ = ((fpᵢ₋₁ << 1) + G[bᵢ]) mod 2ʷ

w is the fingerprint width, normally 64 bits. G is a fixed table of 256 precomputed w-bit constants, typically chosen to look random. The table and the width are part of the chunking profile and must be identical across implementations. In code, unsigned wrapping arithmetic is equivalent to reducing the result modulo 2ʷ.

The recurrence and its three basic operations—one shift, one addition, and one table lookup—are described in the FastCDC paper and Josh Lee’s GearHash explanation.

GearHash does not retain an explicit byte window or subtract an outgoing byte. It retains only the current w-bit fingerprint. Each shift moves older table contributions toward the discarded end of the state; after roughly w updates, contributions from older bytes no longer affect the fingerprint. This is why GearHash behaves like a rolling window while remaining only a shift, an addition, and an array lookup.

The animation shows the conceptual one-byte recurrence with toy 16-bit values. The current LayerFS scanner unrolls two consecutive updates in its hot loop; algebraically, that is:

fpᵢ₊₂ = ((fpᵢ << 2) + (G[bᵢ] << 1) + G[bᵢ₊₁]) mod 2ʷ

It is the same recurrence applied twice, while allowing the implementation to test a boundary after either byte.

That distinction matters: the generic sliding-window animation is intuition, while GearHash’s precise state transition is the recurrence above. GearHash is fast and useful for finding boundaries, but it is not a cryptographic hash and must not be used as the chunk’s immutable identity.

Hash judgment: cut or continue

Once the scanner has a fingerprint, it needs a separate test for whether to end the current chunk. The zero-mask form used by FastCDC and the current LayerFS profile is:

(fp & mask) == 0

If the selected fingerprint bits are all zero, the scanner cuts; otherwise it continues. Earlier Gear-based descriptions commonly express the judgment as fp mod D == r; FastCDC uses a padded, spread-out mask and the convenient zero test above. The exact predicate is part of the chunking profile.

If the selected bits behave approximately uniformly and independently, an N-bit zero test matches with probability roughly 1 / 2ᴺ at each tested position. That probability controls a tendency toward a target chunk scale, not the exact size of every chunk; minimum and maximum limits and FastCDC normalization also shape the observed distribution.

This completes the basic CDC loop: update the fingerprint, judge it, then emit the chunk or continue scanning. Minimum and maximum range limits keep that loop practical. A later CAS layer can hash the completed bytes to name and verify the chunk; the boundary fingerprint is only a fast signal.

5. FastCDC

GearHash makes the fingerprint cheap, but a cheap fingerprint does not automatically produce a useful chunk-size distribution. Josh Lee’s illustrative 8 KiB GearHash run on the Shakespeare corpus reported 38.49% of chunks below 4 KiB, 13.76% above 16 KiB, and two chunks above 80 KiB. Those numbers are from that example, not a universal GearHash guarantee, but they make the problem concrete: the average can be near the target while individual chunks are still poorly distributed. (GearHash article)

The FastCDC paper identifies two related problems with plain Gear-based CDC: the boundary judgment sees only a short span of recent history, and once GearHash makes fingerprinting cheap, judging every position becomes a noticeable part of the work. In its evaluation, FastCDC was reported as roughly 10× faster than the paper’s best open-source Rabin baseline and roughly 3× faster than Gear- and AE-based CDC, while achieving nearly the same deduplication ratio as Rabin. Those are measurements from the paper’s workloads and hardware, not guarantees for every implementation. (FastCDC paper)

FastCDC keeps GearHash’s fast fingerprint update and changes the boundary policy around it. Think of it as three guardrails and one safety rail:

• Before the minimum: copy bytes only. Do not hash or judge; a cut is not allowed yet.
• From minimum to target: update GearHash and use a strict MaskS, so a match is rare and the chunk is encouraged to grow.
• From target to maximum: update GearHash and use a relaxed MaskL, so a match is more likely and the chunk is encouraged to finish.
• At the maximum: emit if no earlier match appeared. This is a normal size bound, not a fourth FastCDC technique.

The animation uses a toy profile—minimum 4, target 10, maximum 18—so the policy fits on one screen. The exact LayerFS construction profile is an implementation contract, not a universal CDC setting.

The key idea: copy first, then make early matches rare, later matches likely, and progress unconditional at the maximum.

The three named improvements explain why those guardrails work together. Read them as one sequence rather than as separate algorithms. (FastCDC article)

1. Broaden the boundary test — enhanced hash judgment. FastCDC pads the mask with zero bits and distributes its effective bits across a wider range of the fingerprint. The GearHash update itself does not change: the same number of effective bits can keep roughly the same match probability, while the decision reflects a longer span of recent history. FastCDC also simplifies the conventional remainder/threshold test to the zero-mask form fp & mask == 0. The mask must be generated deterministically; a different mask can produce different boundaries and destroy reuse.

2. Skip impossible early cuts — sub-minimum cut-point skipping. Before the minimum length, LayerFS copies bytes into the candidate chunk but does not update or judge the Gear state. Once the candidate reaches the minimum, GearHash scanning begins. This avoids tiny chunks and avoids work where a cut is forbidden. The skipped positions might have been useful content-defined boundaries, so this improves speed at a possible deduplication cost.

3. Pull the result toward the target — normalized chunking. Before the target, MaskS has more effective bits, so matches are rarer and the chunk is encouraged to grow. After the target, MaskL has fewer effective bits, so matches are more likely and the chunk is encouraged to finish. Conceptually, mask = if len < target { MaskS } else { MaskL }. At the maximum, the scanner cuts to guarantee progress. The goal is not an exact target-sized chunk; it is a useful size range with good deduplication behavior.

At end of input, any remaining range is emitted as the tail, even when it is shorter than the minimum. This is normal tail behavior, not a boundary-detection error. If input ends with one byte of an incomplete pair, that byte is appended to the tail; end of input does not create another boundary test.

FastCDC therefore gives LayerFS a repeatable sequence of byte ranges. The next two sections place those ranges in the larger LayerFS design and preview the file representation built from them.

6. Why LayerFS Chooses CAS + CDC + COW

CAS, CDC, and COW solve three different forms of storage amplification. Keeping their contracts separate makes the complete LayerFS design easier to reason about:

This separation is the reason LayerFS does not put version information into a chunk. A chunk is just immutable bytes with a content-derived identity. FastCDC chooses its range; the CAS stores and verifies it; an ordered file representation records how the ranges reconstruct the file.

Structural COW operates one level higher than the file representation. When one file changes, it creates a new file manifest and new directory records only along that file’s path. Every unrelated file and subtree remains shared with the previous immutable root.

The resulting division of responsibility is:

1. FastCDC finds useful, reproducible boundaries without scanning the whole file for every candidate.
2. CAS gives each completed chunk an immutable identity and reuses equal bytes.
3. The file representation preserves chunk order, lengths, and exact reconstruction.
4. Structural COW rebuilds only the affected path and produces a new complete immutable root.

FastCDC is therefore a boundary policy, not the storage format. Changing the physical CAS layout or the way layers are persisted should not change the logical chunk ranges, chunk identities, or reconstructed bytes.

7. What Comes Next

At this point we can explain the complete chunking decision: start with a byte stream, update a content-derived fingerprint, apply the FastCDC policy, and emit bounded ranges. We also know why those ranges are useful: unchanged ranges can keep their CAS identities after a local edit.

One piece still sits above the chunker: a complete file needs an ordered, durable recipe for its chunks. A storage layer can build that representation by:

• hashing each completed chunk to its ChunkId;
• recording chunk IDs and lengths in order;
• reconstructing and verifying the original file bytes; and
• measuring how a small edit reuses existing chunks.

The planned Section 1.3 will take that file representation into the filesystem tree: one edited file, one rebuilt path, and one new immutable root. LayerStacks and agent-workspace history come later, after the root model is established.