A server-side request forgery (SSRF) issue exists in version 4 and older of the Oxide software. This issue allows authenticated users with the collaborator role on a project or silo to send HTTP HEAD and GET requests to arbitrary URLs on the underlay network, the primary physical rack-internal IPv6 network where services internal to the system communicate.
This issue allows such users to send arbitrary queries to the metrics service if the internal underlay network address for the ClickHouse service is known. The responses to those queries cannot be read, but such users can still tamper with metrics data and prevent metrics APIs from working.
This issue also potentially allows users to read from a limited set of debugging endpoints served by CockroachDB if the internal underlay network address is known and the endpoint meets certain requirements. The set of requirements necessary for information disclosure rules out nearly all HTTP endpoints served by CockroachDB, but Oxide is unable to rule out all endpoints at this time.
This issue is fixed in version 5 of the Oxide software; we recommend customers upgrade as soon as possible.
| Revision | Date (YYYYMMDD) | Changes |
|---|---|---|
1.0 | 20231215 | Initial Release |
Impacted Products
Oxide software releases version 4 and earlier.
Impact
An authenticated user with the collaborator role on a project or silo can trigger HTTP HEAD and GET requests to arbitrary URLs on the underlay network. This can manifest as a loss of integrity and availability of metrics data, and can possibly manifest as an information disclosure of debugging information for the primary database.
Oxide calculates a CVSS v3.1 base score of 4.9 (Medium). Vector: CVSS:3.1/AV:N/AC:H/PR:L/UI:N/S:C/C:N/I:L/A:L
Action Required
Update the Oxide software to version 5.
Mitigations
There are no mitigations available.
Technical Background
In version 4 and older of the Oxide software, a user with the
collaborator role on a project or silo could create an image with the
url source via the image_create endpoint, or import blocks from
a URL into a disk via the disk_import_blocks_from_url endpoint.
These were used during development prior to the ability to create
an image from a snapshot and to bulk-write disk blocks via the
disk_bulk_write_import endpoint. Version 5 of the Oxide software
removes the url image source and the disk_import_blocks_from_url
endpoint.
When a user creates an image with the url source, Nexus (the Oxide
API server) makes an HTTP HEAD request to the user-provided URL to
query the Content-Length of the response to determine the image size.
If the response is successful (not HTTP 4xx or 5xx), and contains a
Content-Length header that is divisible by the block size (any of
512, 2048, or 4096), Nexus considers the URL a valid image source, and
the image is created. A user can then create a disk from the image and
issue reads to it from an instance, which cause Propolis (the hypervisor
userspace) to make HTTP GET requests with the Range header set; if
the response is successful and has the requested Content-Length based
on the Range header, that data is sent to the instance to complete the
read operation.
The above process is similar for the disk_import_blocks_from_url
endpoint, except that the Crucible pantry (part of the storage subsystem
that allows modification of disks while no instance is running) makes
the HTTP HEAD request, and that the Crucible pantry immediately
makes HTTP GET requests with the Range header set until all data is
read or an error condition occurs. The same requirements are necessary
for the operation to succeed; if successful, the user can attach the
disk to an instance and issue reads, which do not create additional
HTTP requests to the URL. If the operation was unsuccessful, any
partially-read data is available on the disk.
Because Nexus, Propolis, and the Crucible pantry are on the underlay network, they can make HTTP requests to any service on that network. This allows a user to send HTTP HEAD requests to arbitrary URLs on the underlay network; if the response to a HEAD request is successful and contains a Content-Length header divisible by the block size, and responses to GET requests with the Range header have a body with the requested length divisible by the block size, the response data can be viewed by the user.
Additionally, Nexus has access to networks outside the system, to allow users to access it and allow it to communicate with SAML providers. This allows a user to send HTTP HEAD requests to arbitrary URLs to networks routable from the system. These requests are performed without DNS resolution, so a user must specify an IP address. GET requests routed outside the system via this issue are not possible, as services making GET requests only run on the underlay network.
HTTP services on the underlay network are described in the following sections.
ClickHouse
ClickHouse is used by the Oxide software to store metrics. It offers a SQL-like query interface over HTTP, allowing queries to be passed in via URL query parameters or a POST request body.
The query interface responds to HEAD requests, but with no Content-Length header, so reading from the query interface is not possible. HEAD requests to this interface can run arbitrary SQL queries; thus it is possible for a user with knowledge of the underlay network address of the HTTP interface to tamper with data stored in ClickHouse, either by modifying or deleting metric data or by preventing the storage and querying of metrics altogether.
The rest of the system’s availability is not affected if either ClickHouse or the metrics system are unavailable.
CockroachDB
CockroachDB is used by the Oxide software to store all data except metrics and data stored on disks. Its primary interface is a PostgreSQL-compatible protocol, but by default it starts an HTTP console on the same IP address. Prior to version 5 of the Oxide software, this console was accessible via the underlay network.
Most endpoints served by the CockroachDB console respond 406 Method Not Allowed to HEAD requests, or require a valid session cookie. The endpoints that successfully respond to HEAD requests provide a very limited subset of database metrics and health information, assets for the console UI, and debug endpoints that we do not expect can provide database contents via GET requests alone. For data to be read from any of these endpoints, the Content-Length must be divisible by a valid user-selected block size (512, 2048, or 4096), and the endpoint must support Range requests.
Oxide has evaluated the HTTP endpoints served by CockroachDB and could not identify any which meet these requirements, and we are not confident that this issue could successfully be used for information disclosure; however, we cannot completely rule out the possibility at this time.
Oxide-authored HTTP services
The Oxide software consists of many HTTP services authored by Oxide. These services use a common HTTP library named Dropshot. Dropshot does not automatically handle HTTP HEAD requests, and no endpoints provided by any Oxide-authored HTTP services handle HEAD requests. There is no impact to these services.
Additional Information
There is no additional information at this time.