Network Preparations

When planning your network integration, you must determine:

The following table shows supported QSFP transceiver types.

You must configure upstream switch ports connected to the Oxide rack with a Maximum Transmission Unit (MTU) of 1500 or above.

Technician port addresses are auto-configured via SLAAC or DHCPv6:

In order to access the services provided over the management network, SSH port 22 must be accessible. Both the wicket program and the support shell are accessed through TCP port 22.

The rack switches operate as Layer 3 routers, not Layer 2 switches. There is no common broadcast domain shared between ports. Key implications for your network design:

Egress traffic: Each active port requires an egress route assigned to forward packets to the broader network.

Ingress traffic: The rack will not respond to ARP or NDP requests for IP pool addresses. Configure upstream routers to direct off-subnet traffic to the rack’s gateway address. The switches respond to ARP and NDP requests only for their assigned gateway addresses.

Routing integration: Plan for the rack switches to integrate with your existing routing infrastructure using static routes or BGP.

Configure initial communication paths between your network and the rack using a configuration file (rack.toml). The wicket setup program handles this configuration during the Initial Rack Setup phase.

The sections below detail the network-specific configuration parameters. A complete configuration example appears at the end of this guide.

The following is all of the above configuration in one place.

This guide has primarily focused on network considerations for getting the rack up and running. Once the rack is set up, there are additional considerations for transiting traffic to and from VM instances. The Oxide API provides a set of endpoints for managing IP pools. These IP pools are the same basic abstraction as the internal services IP pool covered above. The only difference is the IP pools that are managed through the Oxide API are used to hand out IP addresses to VM instances. The addresses in these pools need to be routed to the rack, and the rack needs to have egress routes set up pointing at appropriate gateways for the address space covered by the IP pool.

There are no restrictions on what IP ranges can be used in IP pools.

The configuration provided during initial rack setup may be changed later through the Oxide API once the rack is up and running. You may also make other network topology changes, such as:

The following ports are used by the rack and should be made available on the broader network segment the rack is a part of. The direction in identifies traffic to the rack from the broader network, out identifies traffic from the rack to the broader network, and both indicates bidirectional traffic.

Summary: If the Oxide rack and its upstream gateway share a common L3 interface, cross-VPC traffic may trigger ICMP Redirect exception processing on the upstream router, degrading performance. Disable ICMP Redirects on upstream routers connected to the Oxide rack to avoid this issue.

ICMP Redirect messages are a mechanism for an IP router to inform a sender of a better path to their destination which doesn’t require the use of that router. Because of the security implications of allowing a network path to be influenced by an unauthenticated third party device, ICMP Redirect messages are typically ignored when received. However, some network vendors enable the generation of ICMP Redirect messages on their platforms by default.

On most modern router or switch platforms, packets are handled by specialized silicon called Network Processors, which are designed to provide high throughput and consistent low latency. However, there are some packets which require more specific handling than the Network Processor is capable of; these packets are called "exceptions" and are sent to the platform’s CPU for the Operating System to perform the required handling. Exception handling imparts negative performance characteristics on affected packets, since the bandwidth of the link connecting the Network Processor to the platform’s CPU is limited (and often exacerbated by traffic policers intended to protect the platform’s control plane) and latency may be non-deterministic as a result of the OS' process scheduling. One exception condition that is commonly present in Network Processors enables the platform to generate ICMP Redirect messages. A packet satisfies this condition when the Network Processor determines that the route lookup for the packet’s Destination IP is resolved via the same Layer 3 interface that the packet was received on.

The processing sequence for packets matching the ICMP Redirect exception generally looks like this:

In a network where ICMP Redirects are both generated and trusted, the sender can update its forwarding path to avoid the hairpinning router, and packets should only traverse the exception path temporarily. However, in the majority of cases, ICMP Redirects are not trusted by hosts and packets would continue to traverse the exception path indefinitely. In this scenario, all such packets would be negatively impacted by the CPU-based exception handling.

The network isolation that Oxide provides for VPCs does not allow traffic to directly flow from one VPC to another without first exiting the rack. This allows for an intermediate device to apply its own security policies on cross-VPC traffic. For example, some operators may want to ensure traffic inspection is conducted for all network traffic that flows between two different tenants. Because of this strict network isolation and the need to traverse an intermediate device, normal cross-VPC packets may be considered exceptions by an upstream router. To avoid performance issues with cross-VPC traffic due to exception processing, ICMP Redirects may need to be disabled on the upstream routers connecting to an Oxide system.