Oxide Security Advisory 20240118-1: Unencrypted Control Plane datastores in Oxide software

Oxide software releases version 5 and earlier.

An attacker who has physical access to the Oxide Rack and the knowledge about the Control Plane databases may be able to access the data on the physical storage devices. They may be able to view and modify system data such as user device tokens, rack configurations, VM instance and disk metadata. They may also be able to examine the disk data by decrypting it with the application-level encryption keys stored in the CockroachDB. The issue can manifest as unauthorized use of VM instances and sensitive data stored in the disks.

Update the Oxide software to version 6.

There are no mitigations available besides enforcing controls against unauthorized rack physical access.

Oxide provides encryption at rest for both customer data, and Oxide control plane data. The goal of such encryption is to prevent offline attacks, such that a casual attacker with physical access to the rack cannot steal a subset of disks or sleds and recover sensitive information off of them. Unique encryption keys per U.2 drive are derived from a shared rack secret which is computed at each sled via an implementation of Shamir secret sharing. In order to reconstruct the rack secret and derive encryption keys, at least a threshold, t, of sleds must participate in an online distributed algorithm. During installation, t = N/2 + 1, where N is the number of sleds in a rack. Concretely, for a 16 sled rack, t=9, where for a 32 sled rack, t = 17. If fewer than t sleds can participate, then no data can be learned about the rack secret and no encryption keys may be derived. The rack secret is split into distinct key shares which are stored on M.2 drives of each of the sleds, and are not immediately available to steal without taking a sled out of the rack. At least t M.2 drives would have to be stolen in order for an attacker to reconstruct the rack secret, derive encryption keys, and then access data on stolen U.2 drives. This provides sufficient mitigation against casual theft of few sleds or drives from being useful to an attacker.

Inside an Oxide rack, customer and Oxide control plane data are stored inside ZFS datasets. Each ZFS dataset has a path, which dictates its location in a hierarchy, and this path corresponds to the fileysystem mount path, but is distinct from it. There are up to 10 U.2 disks on each sled in the rack, and each of these has an encrypted root dataset at path oxp_<UUID>/crypt, where <UUID> is distinct per U.2 disk. ZFS allows child datasets to inherit encryption, such that if the child dataset resides under the path of an encrypted filesystem, the child dataset is also encrypted using the same key. By default, encryption is inherited, and so a dataset such as oxp_<UUID>/crypt/zone/cockroachdb would be encrypted. However, since we only encrypt under our crypt root, a dataset not under that hierarchy, such as oxp_<UUID>/cockroachdb will not be encrypted via ZFS dataset encryption. Keys used for ZFS encryption are derived from the Oxide rack secret as discussed above.

We have two primary types of datasets, ephemeral "zone" datasets which act as the root filesystems of illumos zones and may be recreated across reboots, and persistent "data" datasets that store data associated with the zone, including database state. On launch of a zone, the "data" datasets get mounted into the zone and are used to store customer and Oxide control plane specific data. While the ephemeral zone datasets had paths under our crypt root (oxp_<UUID>/crypt) and therefore inherited the encryption property, our persistent zones were created outside of the crypt hierarchy, and therefore were not encrypted. Unfortunately, persistent datasets contain the most critical data of the Oxide rack, such as cockroachdb data, debug data, clickhouse data, and dns data. None of this data was actually encrypted because all of these datasets resided outside the crypt hierarchy. It should be noted that while crucible datasets live outside the crypt hierarchy, they are encrypted with a different application level mechanism and remained encrypted. Unfortunately the encryption keys were stored in cockroachdb datasets which were unencrypted.

By upgrading to at least Oxide release version 6, existing deployments will have their unencrypted datasets migrated into encrypted datasets at boot time, before usage. New deployments will only create encrypted datasets. As an additional safety measure, any time a dataset is mounted into a zone, the zone will check to see if it is encrypted and error if it is not. This allows fail-fast identification of problems during both development and customer usage.

There is no additional information at this time.