External IP pools and subnet pools are two distinct classes of IP resources for different network connectivity requirements of user workloads.
IP pools are named collections of external IP ranges defined by operators and assigned to silos for instance ephemeral or floating IP assignment. These external addresses do not appear within their attached instances. The primary internal NAT’d address within the instance is used for egress traffic. VPC routing rules combined with internet gateways are used to determine which external address is used as the source address for packets egressing the rack when an instance has multiple external addresses assigned.
Similar to IP pools, subnet pools carry named collections of external addresses but are subnets instead of IP ranges. They can be specified in terms of prefix length and attached to instances by project users. Address allocation from the attached subnet to instance-managed resources (e.g., containers) is controlled by the application running on the instance. The subnet IPs are used as the source addresses for egress without translation.
Key concepts
Pool types
IP pools are categorized by address class and IP version. There are four types of IP pools: unicast IPv4 or IPv6, multicast IPv4 or IPv6. Unicast pools are used for assigning ephemeral, floating, and Source NAT (SNAT) IP addresses to instances for one-to-one outbound and inbound traffic. Multicast pools are used for allocating addresses to multicast groups which instances across different projects or silos may join dynamically to receive multicast traffic.
Subnet pools are categorized by IP version, IPv4 and IPv6.
Pool assignment
External IP addresses are similar to physical resources like compute and storage in that they are limited in quantity and subject to allocation among different tenants. Operators make IP address assignments by linking one or more pools to silos. They can adjust the size of pools based on demand and other constraints while users can reference the same logical pool in provisioning requests without having to understand the physical setup. An IP or subnet pool can be linked to any number of silos, including zero.
Silo default pools
Default pools are used automatically for external IP allocation in that silo when a pool is not specified. A silo may have a default pool for each distinct pool type. Note that every silo must have a default IPv4 pool configured for SNAT IP address assignment. This restriction will be removed in a future release.
Setting up IP pools
Decide allocation strategy
IP pools are designed to allow granular assignment and maximize the use of available addresses. An IP range does not need to be a complete subnet and none of the addresses within it is reserved. For example, you may have an IP range from 172.21.252.0 to 172.21.252.19 to give the pool twenty IP addresses. IP ranges within a pool also do not need to be contiguous. It is possible to add another address range from 172.23.152.131 to 172.23.152.140 to the above pool, for instance. With these flexibilities, you may consider chunking a large subnet into smaller blocks and allocate them as needed instead of committing a large address space to a pool upfront.
You may define IP pools that are shared by different silos or used by specific silos. If you plan to have a common pool of external IP addresses for all silos, you can create one IP pool, add all available IP ranges to it, and make it the default pool for every silo.
Create IP pools
Both IPv4 and IPv6 addresses are supported for unicast communications but the different types of addresses cannot be placed in the same IP pool. When creating an IP pool, ensure to set the appropriate IP version and pool type.
The following policies are enforced by the system when specifying IP address ranges:
The first and last IP addresses must be a valid IPv4 or IPv6 address range and match the IP version of the pool.
No overlapping IP ranges are allowed within the same pool or across different pools.
Link IP pools with silos
Each silo can be linked to any number and type of IP pools depending on the networking requirements of user workloads. The default pool setting allows IP addresses to be allocated automatically without users having to specify a pool when creating an instance.
IP pools can be linked and unlinked dynamically, though you cannot unlink a pool from a silo if there are IP addresses from that pool in use.
Setting up subnet pools
Each subnet pool may include any number of subnets that match the pool’s IP version. Operators can optionally specify minimum/maximum allocatable prefix lengths for each subnet. A larger prefix means smaller allocations are allowed (e.g., a /24 prefix yields smaller subnet allocations than a /16 prefix).
Link subnet pools with silos
Each silo can be linked to any number and type of subnet pools depending on the networking requirements of user workloads. Up to two default pools can be configured for each silo, one for each IP version. When an attached subnet is requested with only the version specified, the corresponding default pool will be used.
Similar to IP pools, subnet pools can be linked and unlinked dynamically when the subnets in them are not attached to any instances.
View subnet pool utilization
Operators can view subnet pool utilization with the subnet pool utilization API.
Using IP pools
Address types
Up to three different types of external unicast address may be allocated to an instance, as described in the Networking guide. Here, we will focus on the assignment method and how it may affect IP pool utilization and sizing considerations.
Ephemeral IP: A temporary IP address allocated to a new instance. The address is used exclusively by the instance for inbound traffic and is mapped to its primary NIC on the VPC. The address is returned to the IP pool upon instance termination.
Floating IP: A permanent IP address managed within the scope of a project. Project users can create and delete these addresses and choose when to associate them with different instances. Once a floating IP is attached to an instance, it works in the same way as an ephemeral IP for inbound traffic. When the instance is terminated, the floating IP remains in the project and can be attached to another instance rather than being returned to the pool.
SNAT IP: IPv4 Source NAT addresses are allocated automatically to every instance with network interfaces to enable outbound communications with external hosts. Each SNAT IP uses only a quarter of the port range on a given address, allowing up to four instances to share the same SNAT IP over different port ranges. IP addresses currently in use for SNAT purposes will not be allocated as ephemeral or floating IPs.
Address assignment from the default pool
When creating a VM instance in the web console, if the user selects the "default" option for networking configuration, an external IP address will be assigned automatically from the silo’s default IP pool. Similarly, the silo default pool is used when the pool name attribute is not specified in an instance or floating IP create request, e.g.
"external_ips": [
{
"type": "ephemeral"
}
]Address assignment from the non-default pools
To get an external IP address from the non-default pools, user can specify the pool name in the instance or floating IP create requests, e.g.
"external_ips": [
{
"type": "ephemeral",
"pool_selector": {
"type": "explicit",
"pool": "inter-dc"
}
}
]The IP pool used must be linked to the silo. A HTTP 404 (Not found) error will be returned if the IP pool is not associated with the silo to which the user belongs.
IP pool utilization and capacity checks
The current IP utilization metrics for individual IP pools are available on the web console under System and through the CLI.
When the system is unable to allocate external IP addresses due to capacity issues, users will get a HTTP 507 error ("No external IP addresses available") when attempting to create more instances. In the case of unicast address allocation, the system does not fall back to using another IP pool automatically. The user has several options to resolve the error:
Free up in-use addresses by deleting existing instances
Ask the operator to add more addresses to the relevant IP pool(s)
Request IP addresses from another available pool, if multiple pools are linked to the silo
Using subnet pools
Subnet allocation is supported via API only at this time. Similar to floating IPs, subnets are owned by projects and can be attached to any instance in the project.
Project users may request for an external subnet using the
external_subnet_create API and attach the subnet to their
instance with external_subnet_attach. Other related subnet
operations available to end users are covered in the
Networking guide.
Here are some examples of subnet allocation and attachment requests.
Subnet allocation from the default pool
When a pool name is not specified in the subnet request, the silo default pool is used for automatic assignment, e.g.,
oxide external-subnet create \
--project oxdoc \
--json-body subnet-default-pool.json{
"name": "pod-network-1",
"description": "subnet assigned from default pool",
"allocator": {
"pool_selector": {
"type": "auto",
"ip_version": "v4"
},
"prefix_len": 24,
"type": "auto"
}
}Subnet assignment from the non-default pools
To get an auto-assigned subnet from the non-default pools, user can specify the pool name in the create requests, e.g.
{
"name": "pod-network-2",
"description": "subnet assigned from non-default pool",
"allocator": {
"pool_selector": {
"type": "explicit",
"pool": "alternate-pool"
},
"prefix_len": 26,
"type": "auto"
}
}Subnet assignment for a specific CIDR
To create a subnet for a specific CIDR, one of the silo’s linked pools must cover the CIDR and the pool name should be included in the request, e.g.,
{
"name": "pod-network-3",
"description": "172.20.37.0/24 subnet slice",
"allocator": {
"type": "explicit",
"subnet": "172.20.37.128/26",
"pool": "alternate-pool"
}
}Attaching a subnet to instance
Once a subnet is allocated, it can be attached to any instance in the
project with the attach subcommand, e.g.,
oxide external-subnet attach \
--project oxdoc \
--external-subnet pod-network-1 \
--instance k8s-controller-1A subnet may only be attached to one instance at a time.
Service IP Pool
The service pool is created during rack setup and provides IP addresses exclusively for system services such as DNS, API, and NAT gateway functionality. The pool is managed separately from tenant IP pools via different API endpoints. It cannot be linked to silos and does not appear in the main IP pool list.
The service pool and tenant IP pools are independent. Both must be accessible from client networks, but they can use different subnets or address ranges. For example, the service pool might use 192.168.1.0/24 while tenant pools use 10.0.0.0/8 addresses.