What the Pool Remembers

A Mac on the desk asks, as it does every few hours, where the Time Machine volume has gone. The answer is sixteen feet away, through a wall, on a FreeBSD jail running on a TrueNAS SCALE box that iXsystems has, as of last year, quietly put on life support. Samba is doing the actual work. The jail is managed by iocage. The pool is a pair of mirrored spinning disks that predate my interest in the question of who owns the operator console. The Mac doesn’t care. The Mac wants the volume back. The human with the Mac has started, dimly, to wonder what it would take to replace the several million lines of Python and TypeScript between its backup request and the spinning disks with something smaller. Something he could read in a weekend and understand by Monday.

This is the account of that wondering. It ends with a design document marked probably not. Before it does, it passes through a threat analysis of credentials in private git repos, a three-tier storage model for secrets that lives inside ZFS user properties, a QR pairing flow borrowed intact from a BEAM observability tool, a proposal to implement Shamir Secret Sharing from scratch in Erlang because the usual injunction against rolling one’s own cryptography does not apply to well-specified finite-field arithmetic, and a list of seven ways to lose one’s keys for good. None of it will be built. All of it was worth thinking through.

The Temptation

FreeNAS began in 2005 as a PHP port of m0n0wall. By 2011 it was FreeNAS 8, written in Django against an ExtJS frontend. By 2020 it had been renamed TrueNAS CORE, the Django was gone, and a single Python asyncio daemon called middlewared had absorbed essentially every decision the system made. By 2024 iXsystems had de-emphasised the FreeBSD line in favour of TrueNAS SCALE, which runs on Debian, uses KVM instead of bhyve, and treats Docker as a first-class citizen. FreeNAS is now a product that exists in the past tense.

The middleware is the product. The current TrueNAS repo holds something on the order of fourteen megabytes of Python, which grepping by file type suggests is roughly four hundred thousand lines. The Angular UI weighs another twelve megabytes of TypeScript — call it two hundred and eighty thousand lines. A hundred plugin namespaces, each of which maps a REST endpoint to a Mako template that renders a chunk of smb4.conf or exports or ctl.conf and then restarts the daemon that consumes it. The Angular is a skin. The Python is the shape of a decade of tickets about SMB ACL mapping, NFSv4 Kerberos, replication retry, iSCSI VAAI, CARP failover, and the forty-seven different ways an operator can break their own pool.

Anyone who has run a TrueNAS box for more than a year has, at some point, thought: I could do this smaller. The thought is always wrong in the specific and right in the general. It is wrong because parity with TrueNAS CORE 13 is a three-to-four person-year project and parity with SCALE on FreeBSD is the battle iXsystems already lost. It is right because you do not want parity. You want the tenth of it that you actually use, implemented with the tools you know, in a shape you can read.

The honest number for a Phoenix LiveView rebuild of the middleware layer, aimed at feature parity, assuming a small team, is four person-years. The honest number for a rebuild of the subset that a Mac-first home or small-office operator actually uses — SMB shares, Time Machine, and S3-compatible object storage, nothing else — is fourteen person-months. The gap between those two numbers is the entire story.

Three Services, Not One Hundred

The narrowed scope looks, from the outside, cowardly. No NFS. No iSCSI. No AFP. No FTP. No WebDAV. No Active Directory. No LDAP. No Kerberos. No bhyve. No jails as a managed product. No Docker. No app store. No HA. No cloud sync. No GPU transcoding. The first reaction of any TrueNAS user reading that list is that this is not a NAS — it is a Samba server with pretensions.

It is a Samba server with pretensions. That is the point. Here is what falls inside the fence:

The zed in the first line is not the FreeBSD ZFS Event Daemon. It is a small Elixir tool I wrote to do declarative BEAM deployments against ZFS-backed hosts. It uses ZFS user properties in the com.zed:* namespace as its only state store — no etcd, no consul, no SQLite alongside. The state travels with zfs send. Snapshots are pre-commit. Rollbacks are zfs rollback, which is constant-time regardless of dataset size. I mention this because the NAS design reuses it, and because the storage model that follows is an extrapolation of a choice I made six months ago and have not yet regretted.

Credentials in the Basement

Every declarative deploy tool eventually faces the secrets question. The pattern: the IR repo describes what should exist, a human commits it, an agent on the target applies it. Everything is fine until the operator realises their IR repo describes an SMB share whose authenticated user has a password, and now they have a file somewhere with the password in it.

The first instinct, and it is widespread, is: the repo is private, the server is in my basement, nobody can reach it, therefore I can commit the password in plaintext. This is worth taking seriously, because if it were true the entire problem would evaporate, and the remainder of this post would not exist.

It is not true.

Mel Brooks, announcing the long-awaited sequel to Spaceballs this month at CinemaCon, retired the working title Spaceballs 2: The Search for More Money on the grounds that he had, after four decades, located the money. It was, he confided, in his basement. The NAS operator is in structurally the same position and is also wrong, because the basement is not actually a hiding place — it is simply the nearest room with a door. Physical security of the git server closes one surface only: unauthenticated network access to the daemon, and people walking off with the disks. Every other surface is open:

1. Every developer with git clone has a full copy of every secret, forever. One stolen laptop, one forgotten coffee-shop open terminal, one badly configured iCloud sync, and the perimeter extends to wherever that clone ended up.
2. Git history is immutable. Rotating a secret does not remove the old one from commit forty-seven. git filter-repo rewrites require coordinating force-push across every clone you have ever created, including ones you have forgotten. Treat any commit as permanent, because it is.
3. Backups escape the perimeter silently. zfs send offsite, Time Machine of a developer laptop, S3 cold storage. The physical basement forks itself into three other locations whenever a backup runs.
4. Read is not different from use. If a developer can read the repo to review an SMB share definition, they can read its password. There is no separation of duties when the review surface includes the secret.
5. The real authentication boundary is authorized_keys on the git server, not the lock on the basement door. One compromised SSH key — one developer whose laptop ran a browser with a weak extension — and full read access is a network-reachable thing.

The answer is not to commit the plaintext. The answer is that the IR carries non-secret declarative config only, and secrets live somewhere else with an explicit storage model.

What the Pool Remembers

The observation that makes the storage model work is that ZFS already is a small distributed database. It has user properties namespaced by domain. Properties set on a dataset travel with zfs send to any replica of that dataset. They are queryable with zfs get, settable with zfs set, and inheritable down the dataset hierarchy. The middlewared authors did not use them for configuration — they used SQLite backed by Django models, for reasons that make sense in Python — but there is no law that says configuration has to live in SQLite.

The zed tool puts its entire deploy state in properties under com.zed:*. The NAS design extends that pattern to secret metadata, while keeping secret values out of it. The split is three tiers:

The split earns three things. First, zfs get output cannot leak a secret value, because the value is never in a property — only its SHA-256 fingerprint is. Second, zfs send -w of the encrypted secrets dataset moves ciphertext to a replica, which needs the passphrase before it can mount and read. Third, the convergence engine can verify integrity by comparing file hashes against the fingerprint property, without ever reading the secret in memory.

The dataset layout is minimal:

<pool>/zed                        # carries com.zed:* metadata
<pool>/zed/secrets   encryption=aes-256-gcm, keyformat=passphrase,
                     mountpoint=/var/db/zed/secrets, canmount=noauto

The canmount=noauto is deliberate. A reboot without the passphrase does not silently serve without secrets; it fails to serve at all. Closed-by-default, in the design parlance that engineers use to sound more careful than they actually are.

The slot catalog for the SMB+TM+S3 MVP is six items:

Per-SMB-user NT hashes are not in this catalog. They are generated per user in the share-creation flow, not at install. Install-time secrets are a small, bounded set. User-scoped secrets are a different lifecycle and belong under share management.

zed bootstrap init --pool jeff walks the catalog, generates any slot that is unset, stamps its fingerprint and metadata into the com.zed:secret.* properties on jeff/zed, and snapshots jeff/zed@bootstrap-<ts>. The first output the operator sees is a banner that prints the admin plaintext, the MinIO keys, and the SSH host pubkey, once, with the header: save this now, it will not be shown again. Any subsequent run of bootstrap init walks the catalog again, finds each slot already stamped, and skips. Rotation is its own subcommand.

The Worst Minute

The worst minute of any NAS install is the first login. TrueNAS handles it by shipping a default password, telling you to change it, and serving the admin UI over a cert that every modern browser refuses to trust. The user is expected to click through a warning page designed by security engineers who explicitly did not want them to click through, then type a password into a form they have not yet verified. The typical NAS operator, having done this on every box they have ever installed, no longer reads the warning. This is the intended outcome of every other piece of the security industry, and at the NAS-install moment it is precisely inverted.

The flow should not require clicking past cert warnings. The flow should not require typing a password. The flow is, ideally, a scan.

There is a small BEAM library called probnik_qr that exists to solve the equivalent problem for remote BEAM node introspection. A probnik_qr:show() call from an Erlang shell renders, in ANSI block characters, a QR code whose payload is an Erlang term: {probnik_pair, Node, Cookie, [opts]}. A mobile scanner app parses the term, writes a config file, and pairs. The ergonomics, once practised, drop below the threshold of deliberate attention. The author describes it as a motor skill, and he is not exaggerating.

The same primitive fits the NAS admin login, with one change of payload:

{zed_admin,
  'zed@super-io',                  % node name, for recognition
  {192, 168, 0, 33}, 4000,         % host IP + port
  "sha256:3f:8a:1b:cc:...",         % cert fingerprint to pin
  "oEzKJ9v...",                     % one-time token
  1713542400                        % expires_at unix seconds
}

The server, at the end of bootstrap init, issues a one-time token that lives in an ETS table with a ten-minute TTL, prints the QR to the installation shell, and waits. The mobile app scans, verifies the expiry locally, prompts the operator for biometric confirmation, opens a WebView whose HTTPS client is pinned to exactly the fingerprint in the QR, and POSTs the token to /admin/qr-login. The server validates the token atomically, flips its used flag, returns a session cookie, and the operator is inside the admin UI with no cert-warning page and no typed password.

The intellectual content of the flow is in two places. The first is cert pinning: the fingerprint is delivered out-of-band, through the QR, which is physically on the installation shell’s screen. The only vector by which the pinning can be subverted is compromise of the installation shell itself, which is the assumed security boundary. There is no TLS interception that gets through.

The second is the one-time token. Its purpose is not to be secret — it appears on the screen as QR bits, visible to anyone in camera range — but to be single-use and short-lived. Two hundred and fifty-six bits of entropy makes brute-force infeasible in the TTL window; rate limiting on the redeem endpoint makes even the attempt unappealing; the used flag, atomically flipped on consume, forecloses replay. Attackers need a camera on the console in the sixty seconds after the operator runs the bootstrap. This is a physical threat that matches the physical assumption. It is the rare case in which the security model admits exactly what it is.

Romans Implement Their Own :crypto

The QR flow is enough for first login. It is not enough for the harder cases: what holds the passphrase for the encrypted secrets dataset across reboots, what authorises the destruction of a dataset that contains four years of Time Machine backups, what survives the theft of the NAS itself.

The answer, in the design, is to treat the paired mobile device as a small vault. Probnik was already a companion app for BEAM observability. It has a stable pairing protocol, an ML Kit QR scanner, a platform keychain, biometric gating. The extension is to let the BEAM node request secrets from it over a second, encrypted sub-channel — not the Scenic rendering channel, which is low-value and can be observed, but a new ECDH-per-session channel with independent audit. The vault supports three operational modes.

Mode one: unlock-at-boot. Server boots, requires the secrets-dataset passphrase, emits {zed_vault_request, :secrets_ds_passphrase, Nonce} to the paired phone. The phone, which has the passphrase in its Keychain behind a Face ID prompt, releases it over the sub-channel. Server unlocks, zeros the passphrase in memory, proceeds. The operator has typed nothing. The passphrase has never been in RAM on any computer the operator does not physically hold.

Mode two: approval. LiveView admin wants to destroy a dataset or rotate a root key. The server’s admin session alone is insufficient — the operation requires out-of-band confirmation. A push arrives on the phone: Destroy tank/archive? Hold to authorise. The phone signs the challenge with the device key. The server verifies, executes. An attacker with a hijacked admin session can browse the UI but cannot destroy data without a thing the attacker does not have.

Mode three: Shamir. The highest-value secrets — the pool encryption key, the shared cross-host replication key — are split across N paired phones with a reconstruction threshold of K. One phone lost is survivable; one phone compromised does not leak the secret; K phones must agree to reconstruct. For the home operator with two phones this collapses to 2-of-2 and becomes either-or with a paper recovery share. For the small-business operator with five paired devices across three locations, 3-of-5 is the canonical configuration.

Shamir Secret Sharing requires arithmetic in GF(2⁸) — a finite field with 256 elements, a multiplication that is a specific lookup table, and a Lagrange interpolation to reconstruct a polynomial from K points. The standard engineering wisdom is never roll your own cryptography, and it is correct in almost every case. This case is one of the exceptions. The randomness comes from :crypto.strong_rand_bytes/1, which is not custom. The field arithmetic is a hundred lines of Elixir that anyone can check against a reference implementation and a test vector. The polynomial evaluation is high-school algebra. There is no novel primitive to get wrong. A hex-published library would add a supply-chain surface for no compensating security gain; keeping the code in-tree means the audit log can record every split and every reconstruct next to the arithmetic that implemented them.

The working title of the chapter in which I will one day write this up is When in Rome, Do It Yourself. The Romans, when they wrote their own :crypto, were not violating the rule against rolling one’s own; they were applying the rule at the right level of abstraction. The rule is against inventing primitives. It is not against implementing specified ones with due care in a small field where the specification is four pages long and every correct implementation agrees with every other on the test vectors.

Seven Ways To Lose Your Keys

Every storage model is a promise about what can and cannot be recovered, and every promise has a fine print. The fine print on this one is a list of ways to lose data beyond recovery. It is worth enumerating, because the design choices downstream — which slots live in which tier, which storage modes are legal for which categories of secret — fall out of this list rather than out of any abstract threat model.

1. Forgotten passphrase, interactive unlock mode. The secrets dataset cannot be mounted. All slots on it are unreachable. The data on other datasets is fine — the pool is not encrypted at the pool level — but the configuration that told the services how to serve that data is gone.
2. Lost keyfile, keyfile unlock mode. The same, minus the operator’s memory as a possible rescue.
3. More than N−K Shamir shares lost. Below the reconstruction threshold. The secret is gone. For a 3-of-5 configuration this requires three simultaneous losses, which sounds unlikely until someone restores three phones from the same iCloud backup and the backup is then deleted.
4. Sole paired vault device destroyed. A single-device probnik_vault slot with no recovery device is a data-loss-level failure. The design forbids this configuration for slots whose loss destroys data: legal values for such slots are :probnik_vault_pair (two devices, either sufficient) or :shamir_k_of_n with k < n.
5. Keychain wipe after an OS restore. iOS has historically, and Android occasionally, lost Keychain items on restore-from-backup-without-keychain. The platform treats this as an authentication boundary enforcement. The secret treats it as fatal.
6. Pool encryption key lost. Not a Plan I or Plan II failure — a ZFS-level failure — but worth listing because operators conflate pool-level encryption with secrets-level encryption. Losing the pool key destroys every byte on the pool, not just the secrets dataset. The blast radius is the thing.
7. Silent fatal: shares on devices from the same failure domain. Three phones, one iCloud account. Two Shamir shares on a laptop and its replicated backup. The fingerprint property stamped on a dataset while the actual secret file was written to a dataset the converge engine never reads again. These look fine during normal operation. They fail only when recovery is attempted. They are the ones that matter.

The operational corollary is a small table of legal storage modes by slot class:

A zed bootstrap verify run is expected to catch every silent fatal before it becomes loud. Fingerprint matches file contents. File mode unchanged. Archive directory retention within policy. Paired Probnik devices respond to a ping. Each Shamir share independently retrievable. If any of these fail, the verify run prints the operator’s next action, not a stack trace.

Coda

The Mac on the desk is still asking where Time Machine has gone. The TrueNAS jail is still serving it. The design described above will not, in any foreseeable near term, be written. The fourteen person-months are not in the budget, and the market for opinionated FreeBSD-first Mac-focused NAS with first-class BEAM operator UX is, at the most generous estimate, six people. The memo is marked probably not, and it will stay marked.

Probably not is not the same as no. The design document exists. Its ideas propagate backward into the tools I actually am writing. zed already uses the ZFS-properties-as-database pattern; the secret-metadata extension is a few more property namespaces. The QR admin-login flow, once you have probnik_qr as a dependency, is a couple of days of integration work regardless of whether it goes in a NAS or in anything else. The Shamir module, written for this imaginary NAS, is a hundred lines of Elixir that would be useful in any multi-device BEAM operator console that ever gets built.

The value of designing a thing you will not build is that it produces a set of answers to questions you would otherwise have to answer later, under pressure, with other constraints in your mind. ZFS user properties as a metadata backbone is now a decision I have made, rather than a thing I might do. The QR pairing flow is a shape I now reach for when I think about first-contact with a new BEAM service. The seven ways to lose one’s keys is a list I will consult the next time I design any system that holds credentials. The thing I never built has told me how to build the things I actually will.

The Mac, for its part, does not care. It asks where the Time Machine volume has gone, I click the icon, Samba returns the volume, the backup proceeds, and several hundred thousand lines of Python between the request and the disks continue to do their work. The disks, Hitchens-like, remain unimpressed by any of this. They turn, as they have always turned, without opinion on the matter.

This post is drawn from two internal design memos — Secret Plan I (install-time bootstrap and three-tier ZFS storage) and Secret Plan II (Probnik Vault extension, three modes, Shamir in-tree) — both shelved under “probably not.” The zed deploy tool is at github.com/borodark/zed and is real. probnik_qr is at github.com/borodark/probnik_qr and is also real. The NAS that sits between them exists only on paper, and on the shelf marked “probably not,” which is where I have learned to keep the good ideas that I will not be permitted to build this year.