Jumbo Frames

This guide shows how to enable the fleet-wide opt-in that allows end users to turn on jumbo frames on the primary network interface of their instances.

Note
This guide describes configuring jumbo frames at the fleet level. The end-user guide for enabling jumbo frames on instances after the fleet-wide setting is configured can be found in Networking.

Background

Traffic between instances within an Oxide rack already benefits from a 9000 byte underlay MTU via virtio TCP segmentation offload and tunneled TSO at the sled NIC. The MTU advertised to a guest’s primary network interface — the interface used for traffic leaving the rack — defaults to 1500 bytes because the upstream MTU is a property of the network the rack is attached to, not of the rack itself.

When the upstream network supports jumbo frames end-to-end, raising the primary interface MTU to 8500 bytes can substantially improve throughput for flows that send large messages. Jumbo frames have little or no benefit for workloads whose typical message size already fits within 1500 bytes.

The opt-in is split into two settings:

  • A fleet-wide setting that an operator enables only when the upstream network actually supports jumbo frames.

  • A per-instance setting that end users set on their own instances.

While the fleet-wide setting is disabled, any attempt to set the per-instance setting to true (on instance create or update) is rejected with HTTP 400.

The 8500 byte effective MTU leaves 500 bytes of headroom under the 9000 byte underlay MTU for VPC encapsulation and future header extensions.

Before enabling

Confirm the following before enabling the fleet-wide opt-in:

  • The upstream switches connected to the rack uplinks have an MTU of at least 9000 bytes on the relevant ports.

  • Every device on the paths your users care about — upstream switches, routers, firewalls, peering networks — passes 9000 byte frames without fragmenting or dropping them.

  • ICMPv4 fragmentation-needed and ICMPv6 packet-too-big messages are not filtered along those paths. Guests rely on path MTU discovery (PMTUD) when they send to destinations behind a smaller MTU; PMTUD black holes will cause connections to stall after the handshake.

Important
Enabling the fleet-wide opt-in does not raise the MTU on any existing instance. The primary interface MTU of an instance changes only when its owner sets enable_jumbo_frames to true and restarts the instance. The end-user interface for jumbo frames is described in the Networking guide.

Viewing the current setting

oxide system networking settings view
{
  "external_jumbo_frames_opt_in_enabled": false
}

Enabling the fleet-wide opt-in

oxide system networking settings update \
    --external-jumbo-frames-opt-in-enabled true
{
  "external_jumbo_frames_opt_in_enabled": true
}

Once this is enabled, users in any silo on the rack can set enable_jumbo_frames on their instances. See the Jumbo frames section of the Networking guide for the end-user workflow.

Disabling the fleet-wide opt-in

oxide system networking settings update \
    --external-jumbo-frames-opt-in-enabled false

The fleet-wide opt-in may be disabled at any time, including while instances have per-instance jumbo frames enabled. The effect is:

  • Running instances are not reconfigured. Their primary interface keeps the 8500 byte MTU it was started with.

  • On the next restart of an instance, the effective MTU is recomputed. Because the fleet-wide opt-in is now off, the primary interface comes back with the default 1500 byte MTU even if enable_jumbo_frames is still true on the instance.

  • New attempts to set enable_jumbo_frames=true (on instance create or update) are rejected with HTTP 400 while the fleet-wide opt-in is disabled. Setting it to false is always allowed.

  • Per-instance values are preserved. Re-enabling the fleet-wide opt-in restores jumbo frames on those instances at their next restart, without any per-instance change.

Recommendations for end users

When jumbo frames are enabled, communicating with hosts behind a smaller MTU depends on PMTUD working correctly. Recommend that users running Linux guests enable packetization-layer PMTUD as a fallback:

sysctl -w net.ipv4.tcp_mtu_probing=1

This lets TCP shrink its effective MTU based on retransmission signals when ICMP messages are dropped along the path.

Scope

  • Only the primary OPTE interface of an instance is affected. Secondary interfaces continue to use the default MTU.

  • Switch port MTUs on the rack are not user-configurable. The Tofino switch ASIC natively passes frames larger than 9000 bytes, so the rack switches do not artificially constrain frame sizes on their uplinks.

  • The MTU on tfport data links in the switch zone (used by BGP and DDM) is unaffected and remains at 1500 bytes.

Related API references