Networking

Every network interface attached to an instance has a dedicated stateful firewall. By default, this firewall will:

Stateful refers to how the firewall behaves whenever a packet or flow is allowed. A decision to allow traffic will create a temporary exception for return-path traffic on the same network interface (i.e., with swapped source/destination ports and addresses).

We alter this behavior by installing firewall rules within the VPC. Each rule comprises:

Each type of filter stores a list of possible values, and a packet matches a rule if it has an entry in every non-empty list of filter entries. For example, a rule which contains the filters {"ports": [80, 443], "hosts": [{"type": "ip", "value": "192.168.1.1"}, {"type": "subnet", "value": "other-subnet"}]} will match all packets satisfying (Port 80 OR Port 443) AND (IP 192.168.1.1 OR Subnet other-subnet).

When a packet arrives at an instance firewall, it will use the highest priority rule which both matches and is enabled. If several rules of identical priority match a packet, then deny will take precedence over any allow actions.

Whenever a VPC is created, it includes a few rules designed to minimize the configuration needed to get started:

Firewall rules may be set for a VPC using the vpc_firewall_rules_update endpoint. Here is an example of a rule that allows SSH from anywhere to all instances on the rack:

Routes consist of two elements:

Destinations and targets can refer to explicit IPs and prefixes, or to named resources like instances and subnets. If a named destination or target does not correspond to any entity in the VPC, the route will still be created. However, it will not be applied until a match can be made — i.e., an instance/subnet is created or renamed. A route will also become a no-op in the event that its named subnet or instance is deleted or renamed.

System-only destinations and targets cannot be set or used in user-configured routes.

System routers are automatically created and managed by the control plane, and apply to all subnets within a VPC. Each VPC has a single system router, which will always be named "system". This router contains one route per subnet (which is immutable), and IPv4/v6 default outbound routes whose target can be changed.

We can inspect its routes using the vpc_router_route_list endpoint, which will contain routes like the below:

This means that:

The target of the 'default-v4' and 'default-v6' routes may be modified using the vpc_router_route_update endpoint. Example 3: Controlling access to upstream networks shows how you can do so — and instead prevent a VPC from sending any outbound traffic, allow outbound traffic from a single subnet, or allow a subnet to reach only a fixed set of upstream addresses.

Custom routers are collections of routes that are used to alter a subnet’s outbound routing behavior. VPC subnets have an optional custom_router field that can be used to assign a custom router. Each network interface on an instance combines its VPC-wide system router with the custom router applied to its subnet into a single routing table. A custom router may be used by many subnets.

Custom routes have higher priority than system routes when matching a destination subnet of identical prefix length.

How would we configure hoenn such that it cannot reach kanto or johto? Custom routers allow for coarse filtering on instances' outbound traffic, and can be used as a complementary tool to firewall rules when per-instance exceptions aren’t required.

By default, our project contains a fully populated system router:

We first create a new custom router:

Note that this router has "kind": "custom". We can then attach it to the hoenn subnet:

Finally, we insert routes to override the system router:

We can verify by attempting to ping between two instances: from mossdeep (192.168.0.5) to goldenrod (10.0.1.5). No packets from mossdeep (hoenn) will arrive at goldenrod (johto).

To detach island-routes from hoenn, we update the subnet via vpc_subnet_update, leaving the --custom-router field unset:

In a system router, we are able to modify the target of the 'default-v4' and 'default-v6' routes. We can use this to prevent outbound access to the wider Internet:

A natural consequence is that we can no longer ping or SSH into any of our instances! We can selectively re-enable access, possibly on an arbitrary upstream prefix, by using a custom router:

We can now SSH into goldenrod, but not mossdeep.

It’s worth noting that 0.0.0.0/0 and ::/0 are not the only valid choices of destination for a custom internet_gateway route — in custom routers, this target is compatible with all non-system destinations. This can be used to limit outbound access to a specific upstream subnet, or to specific IP addresses.

To restore the default system route’s state:

Custom routes allow you to redirect entire subnets towards individual instances or internal IP addresses. This allows you to use an instance as a software router — either for local traffic processing, or to use it as a VPN tunnel endpoint (TEP).

Suppose we want goldenrod to be responsible for all traffic between johto and 172.30.0.0/22. We can add a custom route to serve this function:

However, goldenrod will not yet receive any traffic bound for this subnet. OPTE, which handles per-guest packet processing, will drop any traffic on an interface which does not match its assigned IP address. This prevents instances from sending or receiving unwanted traffic. To allow an instance to opt-in on a wider address range, we must set a list of transit IPs on the relevant NICs.

First, we need to fetch the name or ID of the primary NIC assigned to the instance.

We can now update the network interface as needed:

If we attempt to contact an address in this range from ecruteak, then we can see that goldenrod will now receive all matching packets.

In every VPC, there is a pair of system-generated entities that govern the default routing behavior:

All subnets within a VPC share this default internet gateway.

Internet gateways can be used to ensure that instances only use certain external IP addresses when sending traffic to a given network. To override the default traffic flow with a different IP pool available to your silo, you can create a custom internet gateway and attach it to the desired IP pool or a single address in the pool, set it as the route target in a custom router, and attach the router to a VPC subnet. The custom internet gateway will be used for route determination only for instances with their primary NIC on that subnet.

Because all instances still have their Source NAT (SNAT) addresses allocated from your silo’s default IP pool, to leverage the custom internet gateway and route setup, you’ll need to attach ephemeral or floating IP addresses from an IP pool linked to the custom gateway.

Here is an example that illustrates how to achieve custom routing for a specific IP address prefix:

We’ll start by making two different IP pools. If you are setting up a custom internet gateway in an existing silo, it is likely that there is already a default IP pool created for the silo and used by the default gateways.

Next, create a custom internet gateway linked to the secondary IP pool

Next, create a custom router that uses the internet gateway as the route target for traffic going to 45.154.216.0/24.

Finally, attach the custom router to the desired VPC subnet (in this case, we simply use the system-created "default" subnet)

The set up is complete at this point. To make use of it, provision a floating IP from the "dc2" IP pool:

Then, create an instance with an ephemeral IP auto-assigned from the default pool and the "dc2-igw" floating IP (attached at create time or afterwards). Packets matching the 45.154.216.0/24 prefix will select any of the addresses in the "dc2" IP pool whereas other packets will select from the ephemeral IP and SNAT assigned to the instance.