External IP pools and subnet pools are two distinct classes of IP resources for different network connectivity requirements of user workloads.
IP pools are named collections of external IP ranges defined by operators and assigned to silos for instance ephemeral or floating IP assignment. These external addresses do not appear within their attached instances. The primary internal NAT’d address within the instance is used for egress traffic. VPC routing rules combined with internet gateways are used to determine which external address is used as the source address for packets egressing the rack when an instance has multiple external addresses assigned.
Similar to IP pools, subnet pools carry named collections of external addresses but are subnets instead of IP ranges. They can be specified in terms of prefix length and attached to instances by project users. Address allocation from the attached subnet to instance-managed resources (e.g., containers) is controlled by the application running on the instance. The subnet IPs are used as the source addresses for egress without translation.
Key concepts
Pool type and IP version
IP pools are categorized by pool type and IP version. Accordingly, there are four kinds of IP pools: unicast IPv4, unicast IPv6, multicast IPv4, and multicast IPv6. Unicast pools are used for assigning ephemeral, floating, and Source NAT (SNAT) IP addresses to instances for one-to-one outbound and inbound traffic. Multicast pools are used for allocating addresses to multicast groups. Instances across different projects or silos may join a group to receive multicast traffic.
Subnet pools are categorized by IP version: IPv4 and IPv6.
Pool assignment
External IP addresses are similar to physical resources like compute and storage in that they are limited in quantity and subject to allocation among different tenants. Operators make IP address assignments by linking one or more pools to silos. They can adjust the size of pools based on demand and other constraints while users can reference the same logical pool in provisioning requests without having to understand the physical setup. An IP or subnet pool can be linked to any number of silos, including zero.
Silo default pools
Default pools are used automatically for allocation when an address or subnet is requested without explicitly specifying a pool. For IP pools, a silo may have a default pool for each combination of pool type (unicast or multicast) and IP version (IPv4 or IPv6). For subnet pools, a silo may have a default pool for each IP version.
External IP allocation can happen on instance create as well as on ephemeral IP or floating IP create. In these requests, the user can name a pool explicitly or ask for automatic allocation, in which case the system uses the silo’s default unicast pool. Pool type is fixed by the operation — these paths never draw from multicast pools — so the only ambiguity is IP version: if both default IPv4 and IPv6 unicast pools are configured, the request must specify an IP version. The same applies to multicast group creation, where the operation is fixed to multicast pools and IP version is the only remaining axis.
Subnet allocation works the same way: an external subnet create request can name a pool explicitly or ask for automatic allocation, in which case the system uses the silo’s default subnet pool for the requested IP version.
Note that every silo must have a default IPv4 pool configured for SNAT IP address assignment. This restriction will be removed in a future release.
Setting up IP pools
Decide allocation strategy
IP pools are designed to allow granular assignment and maximize the use of available addresses. An IP range does not need to be a complete subnet and none of the addresses within it is reserved. For example, you may have an IP range from 172.21.252.0 to 172.21.252.19 to give the pool twenty IP addresses. IP ranges within a pool also do not need to be contiguous. It is possible to add another address range from 172.23.152.131 to 172.23.152.140 to the above pool, for instance. With these flexibilities, you may consider chunking a large subnet into smaller blocks and allocate them as needed instead of committing a large address space to a pool upfront.
You may define IP pools that are shared by different silos or used by specific silos. If you plan to have a common pool of external IP addresses for all silos, you can create one IP pool, add all available IP ranges to it, and make it the default pool for every silo.
Create IP pools
Both IPv4 and IPv6 addresses are supported for unicast communications but the different types of addresses cannot be placed in the same IP pool. When creating an IP pool, ensure to set the appropriate IP version and pool type.
The following policies are enforced by the system when specifying IP address ranges:
The first and last IP addresses must be a valid IPv4 or IPv6 address range and match the IP version of the pool.
No overlapping IP ranges are allowed within the same pool or across different pools.
oxide system networking ip-pool create --description "Demo pool" --name "demopoolv4-uni" --ip-version v4 --pool-type unicastoxide system networking ip-pool range add --pool demopoolv4-uni --first 172.23.152.131 --last 172.23.152.140Link IP pools to silos
Each silo can be linked to any number and type of IP pools depending on the networking requirements of user workloads. Setting a default IP pool for a silo allows IP addresses to be allocated automatically without users having to specify a pool when creating an instance.
IP pools can be linked and unlinked dynamically, though you cannot unlink a pool from a silo if there are IP addresses from that pool in use.
oxide system networking ip-pool silo link --pool demopoolv4-uni --silo my-silo --is-default falseoxide system networking ip-pool silo unlink --pool demopoolv4-uni --silo my-siloSetting up subnet pools
Each subnet pool may include any number of subnets that match the pool’s IP version. Operators can optionally specify minimum/maximum allocatable prefix lengths for each subnet. A larger prefix means smaller allocations are allowed (e.g., a /24 prefix yields smaller subnet allocations than a /16 prefix).
Link subnet pools to silos
Each silo can be linked to any number and type of subnet pools depending on the networking requirements of user workloads. Up to two default pools can be configured for each silo, one for each IP version. When an attached subnet is requested with only the version specified, the corresponding default pool will be used.
Similar to IP pools, subnet pools can be linked and unlinked dynamically when the subnets in them are not attached to any instances.
oxide system networking subnet-pool silo link --pool demo-subnet-pool --silo my-silo --is-default falseoxide system networking subnet-pool silo unlink --pool demo-subnet-pool --silo my-siloView subnet pool utilization
Operators can view subnet pool utilization with the subnet pool utilization API.
Using IP pools
Address types
Up to three different types of external unicast address may be allocated to an instance, as described in the Networking guide. Here, we will focus on the assignment method and how it may affect IP pool utilization and sizing considerations.
Ephemeral IP: A temporary IP address allocated to a new instance. The address is used exclusively by the instance for inbound traffic and is mapped to its primary NIC on the VPC. The address is returned to the IP pool upon instance termination.
Floating IP: A permanent IP address managed within the scope of a project. Project users can create and delete these addresses and choose when to associate them with different instances. Once a floating IP is attached to an instance, it works in the same way as an ephemeral IP for inbound traffic. When the instance is terminated, the floating IP remains in the project and can be attached to another instance rather than being returned to the pool.
SNAT IP: IPv4 Source NAT addresses are allocated automatically to every instance with network interfaces to enable outbound communications with external hosts. Each SNAT IP uses only a quarter of the port range on a given address, allowing up to four instances to share the same SNAT IP over different port ranges. IP addresses currently in use for SNAT purposes will not be allocated as ephemeral or floating IPs.
Address assignment from the default pool
When creating a VM instance through the API, an ephemeral IP is allocated from a
default pool when no pool_selector is specified, e.g.,
"external_ips": [
{
"type": "ephemeral"
}
]If the silo has both an IPv4 and IPv6 default unicast pool configured, the request must specify an IP version:
"external_ips": [
{
"type": "ephemeral",
"pool_selector": { "type": "auto", "ip_version": "v4" }
}
]Address assignment from the non-default pools
To allocate an ephemeral IP from a non-default pool during instance create, the user can specify the pool name, e.g.,
"external_ips": [
{
"type": "ephemeral",
"pool_selector": {
"type": "explicit",
"pool": "inter-dc"
}
}
]To allocate a floating IP from a non-default pool, specify the pool in the
floating IP create request’s address_allocator.pool_selector. The floating IP
can then be attached to an instance by name or ID.
The IP pool used must be linked to the silo. A HTTP 404 (Not found) error will be returned if the IP pool is not linked to the user’s silo.
IP pool utilization and capacity checks
The current IP utilization metrics for individual IP pools are available on the web console under System and through the CLI.
oxide system networking ip-pool utilization --pool env-gmWhen the system is unable to allocate external IP addresses due to capacity issues, users will get a HTTP 507 error ("No external IP addresses available") when attempting to create more instances. In the case of unicast address allocation, the system does not fall back to using another IP pool automatically. The user has several options to resolve the error:
Free up in-use addresses by deleting existing instances
Ask the operator to add more addresses to the relevant IP pool(s)
Request IP addresses from another available pool, if multiple pools are linked to the silo
Using subnet pools
Subnet allocation is supported via API only at this time. Similar to floating IPs, subnets are owned by projects and can be attached to any instance in the project.
Project users may request for an external subnet using the
external_subnet_create API and attach the subnet to their
instance with external_subnet_attach. Other related subnet
operations available to end users are covered in the
Networking guide.
Here are some examples of subnet allocation and attachment requests.
Subnet allocation from the default pool
When a pool name is not specified in the subnet request, the silo default pool is used for automatic assignment, e.g.,
oxide external-subnet create \
--project oxdoc \
--json-body subnet-default-pool.json{
"name": "pod-network-1",
"description": "subnet assigned from default pool",
"allocator": {
"pool_selector": {
"type": "auto",
"ip_version": "v4"
},
"prefix_length": 24,
"type": "auto"
}
}Subnet assignment from the non-default pools
To get an auto-assigned subnet from the non-default pools, user can specify the pool name in the create requests, e.g.,
{
"name": "pod-network-2",
"description": "subnet assigned from non-default pool",
"allocator": {
"pool_selector": {
"type": "explicit",
"pool": "alternate-pool"
},
"prefix_length": 26,
"type": "auto"
}
}Subnet assignment for a specific CIDR
To create a subnet for a specific CIDR, one of the silo’s linked pools must cover the CIDR and the pool name should be included in the request, e.g.,
{
"name": "pod-network-3",
"description": "172.20.37.0/24 subnet slice",
"allocator": {
"type": "explicit",
"subnet": "172.20.37.128/26",
"pool": "alternate-pool"
}
}Attaching a subnet to instance
Once a subnet is allocated, it can be attached to any instance in the
project with the attach subcommand, e.g.,
oxide external-subnet attach \
--project oxdoc \
--external-subnet pod-network-1 \
--instance k8s-controller-1A subnet may only be attached to one instance at a time.
Service IP pool
The service pool is created during rack setup and provides IP addresses exclusively for system services such as DNS, API, and NAT gateway functionality. The pool is managed separately from tenant IP pools via different API endpoints. It cannot be linked to silos and does not appear in the main IP pool list.
The service pool and tenant IP pools are independent. Both must be accessible from client networks, but they can use different subnets or address ranges. For example, the service pool might use 192.168.1.0/24 while tenant pools use 10.0.0.0/8 addresses.