PatternBuffer.NET: beam up the bytes that matter

I've used Borg Backup for years, and I've admired its design for just as long. It splits files into content-defined chunks, stores each unique chunk once across every snapshot, and compresses everything. It's elegant. I also mostly live on Windows and .NET, where Borg is a bit awkward. So I did the thing every programmer eventually does with a tool they love: I wrote my own.

PatternBuffer.NET is an independent, cross-platform backup CLI for .NET 10. The command is beam, and the verbs are what you'd expect from a transporter:

beam init  ssh://eric@nas/var/lib/beam/repo --encryption=passphrase
beam up    ~/Projects --repository ssh://eric@nas/var/lib/beam/repo -sdb
beam list  --repository ssh://eric@nas/var/lib/beam/repo
beam down  stardate-2026.67 ~/restore --repository ssh://eric@nas/var/lib/beam/repo

(Yes, the Star Trek Borg theme is intentional. Every archive joins the repository's collective, and unchanged data is assimilated only once.)

To be clear, it's not Borg. It doesn't read Borg repositories, isn't compatible with them, and isn't affiliated with the project. It borrows ideas, not formats. It even got a logo, because a backup tool with a transporter theme deserves one:

Chunks that survive an insert

Fixed-size chunking breaks down the moment you insert a byte near the start of a file: every block after it shifts, and deduplication falls apart. Content-defined chunking puts boundaries where the data says to, so an insert only disturbs the chunks around it.

beam uses FastCDC with a seeded Gear table. The heart of it is a rolling hash that's one shift and one add per byte:

hash = (hash << 1) + gear[value];
if (length >= minimum && ((hash & mask) == 0 || length >= maximum))
{
    // cut here: this byte ends the chunk
}

The Gear table is generated from a per-repository seed with a SplitMix-style mixer. Chunk boundaries are stable within one repository but differ between repositories, which avoids leaking chunk-size patterns across them.

The rules I wrote before the code

The architecture doc opens with a list of principles, and the first is the one everything else hangs on: a backup program must never present a partially committed archive as complete. Some of the others:

• Stored data is immutable. Packs, index fragments, and archive records are written once and replaced, never edited in place.
• Every on-disk format is versioned, with a defined upgrade or rejection path.
• Memory is bounded. Parallel compression is limited by a memory budget, not just the CPU count.
• No silent metadata loss. Metadata the platform can't store produces a warning or a failure, according to an explicit policy.

Compression is always zstd level 3. Encryption is opt-in AES-256-GCM with a passphrase. Every chunk is compressed independently, so the pipeline can do that work in parallel.

Remotes over plain SSH

The first remote backend needs nothing on the server except SSH. beam scans and chunks locally, takes a lock over SSH, uploads new packs and index fragments with SCP, and publishes the archive record last. That record is the commit point, so a dropped connection leaves orphaned packs that compact can clean up, never a half-archive that list would show you. A remote down fetches only the packs the selected archive references.

A future version will run a persistent protocol through SSH for batching, range reads, and resumable uploads. The design leaves room for it without needing it yet.

Stardates

Archives you don't name get one automatically:

var fraction = Math.Floor(local.DayOfYear / (double)daysInYear * 100) / 100;
return (local.Year + fraction).ToString("F2", CultureInfo.InvariantCulture);

It's the “Kelvin timeline” style of stardate: the year plus the fraction of it that's gone by. So stardate-2026.67 is early September. It sorts correctly and reads well in a list. I'll admit it's also just fun.

Next up: a command line is great for building a backup tool, but it's not what makes people run backups. That's the next post.