Key Concepts

User Resource Management

Silos

Silos are the top-level containers for grouping users, projects, and resources. As the name suggests, silos are isolated from each other, providing strict tenancy separation between groups of users. Each silo has its own resource limits and access policies as well as its own subdomain for the web console and API. Resources in a silo cannot be shared with any other silos.

Every Oxide rack includes a Recovery silo which is reserved for built-in users and resources. Rack administrators can create one or more silos based on their usage model. Silos can be backed by an external or local identity provider, as defined by the identity_mode property of the silo:

  • SamlJit: Users are authenticated via SAML using an external identity provider (IdP). The IdP is the source of truth about users and groups. The control plane’s information about users and groups gets imported and updated only during user authentication (i.e., on a "just-in-time" basis). User discovery is not supported at this time for pre-provisioning users prior to their first-time login to the rack.

  • LocalOnly: Users are authenticated by username and password. The identity data is managed through Oxide API and resides only in the rack. IAM roles can only be granted to local users on a per-user basis as groups are not supported for this type of silos.

The identity_mode property is set when the silo is created and cannot be changed afterwards.

While each silo is linked to a single realm of users, the same provider can be configured to back multiple silos. In other words, it is possible to have the same user onboarded to different silos that are set up for different purposes, e.g.

Silos and Identity Providers

Silo DNS Endpoints

As part of the on-premises network integration, a DNS domain is delegated to the Oxide rack at the time of initialization. This allows the rack external endpoints to be accessible using DNS names and secured with TLS certificates.

Each silo has a unique console and API endpoint with the naming convention $silo_dns_name.sys.$delegated_domain. The silo_dns_name is set to the silo’s name by default but is customizable. For this reason, silo names must comply with the domain name standard RFC 1035, i.e. they may contain only the English-language letters a-z, the digits 0-9, and hyphens; hyphens cannot be at the beginning and at the end.

Projects

Projects provide another level of resource and access isolation within each silo. They serve as containers for managing compute, storage, and network resources. Resources such as VM instances and disks are always scoped under specific projects and cannot be shared with other projects. Internal network traffic among VM instances in the same project is bound within the virtual private network (VPC) for that project.

Silo users can be members of one or more projects. Their access permissions are governed by both the silo-level and project-level access policies.

Like silos, project names are also governed by RFC 1035 to enable the use of DNS names for VM instances in the future versions of Oxide Rack.

Identity & Access Management (IAM) Policy

Access to projects and resources may be granted on a per-group or per-user basis. There are three built-in roles with different levels of permissions:

RoleDescription

admin

has all permissions on the target object

collaborator

has read, list_children, and create_child, plus the admin role for all children of the target object, e.g. a project collaborator has full control over the resources within the project, but cannot modify the project and its IAM policy

viewer

has read and list_children on the target object

Here is an example of IAM policies with different access permissions for different members of a project:

Access and IAM

IAM roles granted at the silo level provide cascading permissions to all child objects scoped within the silo. For example, when a group is assigned the "viewer" role in a silo, group members can view all the projects of the silo and their resources. If a particular user within the group is also given the "collaborator" role, the person gains additional permissions to act on the projects.

Virtual Resources

Instances

Instances are virtual machines that run on the Oxide platform. Each VM can be attached to up to eight virtual disks, one private network interface, and multiple external network interfaces.

Instances that have one or more network interfaces get outbound internet access through NAT service on the rack. They can be optionally attached to temporary or permanent external IP addresses to allow them to communicate with external hosts.

Disks, Snapshots, and Images

Disks are persistent volumes that can be managed independently from VM instances. A snapshot is a lightweight point-in-time copy of a disk that can be used to create a reusable image shared within the project or with other projects.

Images and snapshots let you create a new disk based on an existing one, providing the mechanisms for disk cloning and templating. See the Images and Disks and Snapshots guides for more information.

IP Pools and External IP Addresses

IP pools are collections of external IPs you can assign to silos. When a pool is linked to a silo, users in that silo can allocate temporary or permanent external IPs from the pool for their instances.

Temporary external IPs are known as ephemeral IPs. They are allocated and deallocated during instance creation and termination respectively. Permanent external IPs are known as floating IPs. They exist independently of instances as project-scoped resources and can be attached to and detached from instances within the project as needed.

See the Networking and Floating IPs guides to understand more about VM networking.

Virtual Private Networks

VPCs are private RFC1918 networks that isolate sets of instances from each other. Instances within a VPC subnet can talk to each other using private IP addresses (if firewall rules allow it) but traffic between VPCs must go through external IPs.

Rack Network Management

Integrating an Oxide rack into the on-premises network involves the allocation of IP addresses and configuration of rack switches. There are five basic elements on a rack switch that may be configured:

  1. Ports carry physical layer (L1).

  2. Links carry link layer (L2) information.

  3. Interfaces carry internet layer (L3) information and typically belong to a port.

  4. Addresses are assigned to interfaces.

  5. Routes determine forwarding behavior across the switch.

Basic Switch Elements

Address Lots

Usable addresses are communicated to the Oxide Rack through address lots. An address lot contains an array of address ranges. Address lots are named elements with a resource id. Address lots allow operators to associate an intended use to a collection of addresses. To this end address lots have types. Currently there are two types, infra and pool. The infra type indicates that an address lot is to be used for the network infrastructure connecting the rack to upstream networks. The pool type indicates that an address lot is to be bound to an IP pool. For example, for a rack on the public Internet with the address prefix 203.0.113.0/24, an administrator may choose to break this prefix up as follows into lots.

nametypeblock

network infrastructure

infra

203.0.113.2 - 203.0.113.3

control plane

pool

203.0.113.4 - 203.0.113.10

instances

pool

203.0.113.11 - 203.0.113.100
203.0.113.200 - 203.0.113.220

Once these address blocks are created. Addresses within these blocks may be used for things like rack switch addresses and ip-pools. The Oxide control plane tracks the use of these addresses to prevent inconsistencies such as assigning the same address twice or trying to use an address that is not in an address block.

Switch Ports

Port objects are associated with physical switch ports. There are different types of ports, e.g. faceplate data ports, backplane ports. Ports provide a configuration interface for physical-layer properties such as phy lane geometry and power levels supplied to transceivers.

Naming:

  • qsfp0-31: data ports on the faceplate of the rack switch

  • tech0-1: RJ-45 technician ports

  • rear0-31: backplane ports

  • int0: internal 0 for the CPU port

  • mgmt0-34: vsc7448 ports

    • management ports 0-31: service processors.

    • management ports 32-33: power controllers 0 and 1

    • management port 34: adjacent sidecar

Switch Links

Links provide a configuration interface for link-layer properties such as MAC addresses. The number of links per port depends on the phy geometry that has been configured for the port. Ports that support modular interfaces such as QSFP have multiple data lanes that connect the switch ASIC to the internal side of a transceiver. Some transceivers support multiple independent physical cables on the external side of the transceiver. This is called a breakout cable. The phy geometry configuration of a port defines how the internal transceiver data lanes are grouped into layer-2 link elements. In the case of QSFP, the Q stands for quad. Typical breakout configurations include 1xQSFP (no breakout), 2xQSFP (2 lanes per QSFP link, delivering half the available bandwidth to each), 4xSFP (1 lane per SFP link, delivering a quarter of the available bandwidth to each). Thus in the case of QSFP, there may be 1, 2 or 4 links per port with 4, 2 and 1 lanes per link respectively.

Naming:

If the phy geometry only specifies one link, the link has the same name as the port. If the phy geometry specifies multiple links, the link is named as follows "<port-name>:<index>" where index ranges from 0 to one less than the number of links specified in the phy geometry. For example the following enumerates all possible link configurations for the port qsfp7.

Phy GeometryLinks

1xQSFP

qsfp7

2xQSFP

qsfp7:1
qsfp7:2

4xQSFP

qsfp7:1
qsfp7:2
qsfp7:3
qsfp7:4

Switch Interfaces

Interfaces carry internet layer information and typically belong to a port. They come in three flavors, loopback (lo), link and vlan. Loopback interfaces are not associated with any link and are typically created to hold router IPs. There is one loopback interface per switch that cannot be manually created or destroyed. A link interface is automatically created for every link object and there is only one per link. Link interfaces cannot be manually created or destroyed, they simply exist as a companion to a link. Vlan interfaces are associated with a link, and are configured with a VLAN id (VID) to only send and receive traffic on that VID. There can be an arbitrary number of vlan interfaces per link, but they all must have distinct VIDs. These interfaces exist primarily to support routing protocols over VLAN segmented networks. Traffic delivered to routing daemons from the vlan interface has the VLAN tag removed. Traffic sent from routing daemons through a vlan interface has the VLAN tag added. As the product evolves, more interface types such as vxlan, Geneve, gre etc. may be added.

Naming:

  • Loopback interface is named lo as there is only one per switch currently. When there is a need to create multiple loopback interfaces in the future, users can supply a suffix to uniquely identify them.

  • Vlan interfaces always start with the prefix "vlan". Users may also supply a suffix for a vlan interface.

Switch Addresses and Routes

An arbitrary number of IPv4 or IPv6 addresses may be added to any type of interface. Adding an address to a non-loopback interface automatically creates a route whose destination is the subnet prefix of the address. This route is called an on-link route, meaning that it does not have a nexthop gateway. Layer-2 frame information for packets egressing along this type of route is acquired from ARP or NDP neighbor tables for IPv4 and IPv6 respectively. On-link routes may not be manually created or destroyed, they belong to an address and share a lifetime with that address.

Interface routes map a destination prefix to an interface. They depend on IPv6 router advertisements to work. The target interface must have an IPv6 link-local address, and the switch must have received a router advertisement on that address. The switch uses the mac address of the router it received an advertisement from, to form the L2 frame of packets egressing along an interface route. Note that interface routes can have either an IPv4 or IPv6 destination prefix.

Nexthop routes associate a destination prefix with a nexthop address. The nexthop address must be reachable through an existing chain of routes.

Last updated