Expanding Connectivity

This guide describes how to connect an Oxide rack to additional upstream networks through the API. It assumes that the rack is already accessible through a primary network, and the goal is to connect the rack to additional networks while maintaining primary network connectivity.

The example in this guide will walk through taking a rack that is connected to a transit provider in a data center, and expand connectivity to a direct connection with a cloud provider. In this example both forms of connectivity are using BGP. However, the concepts in this guide apply to static networking as well.

Initial Configuration

This guide will use the oxide CLI. To get started we’re going to look at the current network configuration of the rack.

$ oxide system networking switch-port-settings show
switch1/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.20.2/30  initial-infra  None

BGP Peer      Config   Export          Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.20.1  as65547  [no filtering]  [no filtering]  []           3              3           false             6          0               2          None        None      None     None  None        None

switch0/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.10.2/30  initial-infra  None

BGP Peer      Config   Export          Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.10.1  as65547  [no filtering]  [no filtering]  []           3              3           false             6          0               2          None        None      None     None  None        None

This shows that each switch has a port configured with a BGP session. This configuration lines up with the diagram on the left side of the figure above. We can get the status of each BGP session as follows.

$ oxide system networking bgp show-status
switch0
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.10.1  65547      64500       Established    15h 12m 56s 859ms

switch1
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.20.1  65547      64500       Established    15h 12m 58s 669ms

Expanding Connectivity

Expanding connectivity in this example involves peering with BGP routers over a VLAN segmented network. To do that, an address lot needs to be created for holding the interface addresses that will be used for peering. Addresses from that lot will be assigned to switch ports. BGP sessions and announcements will be configured. And finally, an IP pool will be created for instances to utilize the address space being announced on the expanded portion of the network.

Address Lot Creation

Configuration of the expanded network starts with defining an address lot. This address lot will contain the range of addresses that will be used for peering with upstream routers.

oxide system networking address-lot create \
    --json-body cloud-peering-lot.json

cloud-peering-lot.json

{
    "name": "cloud-peering",
    "description": "a lot for cloud peering",
    "kind": "infra",
    "blocks": [
        {
            "first_address": "169.254.30.1",
            "last_address": "169.254.40.255"
        }
    ]
}

We also create a lot for the address range that will be used later for IP pools announced to the cloud.

oxide system networking address-lot create \
    --json-body cloud-pool-lot.json

cloud-pool-lot.json

{
    "name": "cloud-pool",
    "description": "a lot for cloud communications",
    "kind": "pool",
    "blocks": [
        {
            "first_address": "203.0.113.1",
            "last_address": "203.0.113.254"
        }
    ]
}

Address Assignment

Next, addresses are assigned to the switch ports. In this example we are re-using existing links for the connectivity expansion. The new connections are going to be made over a VLAN segment. It’s common when deploying to colocation environments to have network services such as cloud or ISP access provisioned over VLAN segments.

The commands below add the address 169.254.30.2/30 to switch0/qsfp0 on VLAN 300 and the address 169.254.40.2/30 to switch1/qsfp0 on VLAN 400.

To update addresses, the target rack needs to be identified. For single rack deployments this can be done as follows.

export rack=`oxide system hardware rack list | jq -r .[0].id`
echo $rack

For multi-rack deployments, run the oxide system hardware rack list command and choose the appropriate rack UUID.

The following commands add the addresses we’ll need on the switch ports to peer with upstream cloud routers.

oxide system networking addr add \
    --rack $rack \
    --switch switch0 \
    --port qsfp0 \
    --addr 169.254.30.2/30 \
    --lot cloud-peering \
    --vlan 300

oxide system networking addr add \
    --rack $rack \
    --switch switch1 \
    --port qsfp0 \
    --addr 169.254.40.2/30 \
    --lot cloud-peering \
    --vlan 400

BGP Configuration

Next, BGP sessions are created to peer with routers in the new upstream network. Note that VLANs are also configured for the BGP sessions. This ensures that when the Oxide rack BGP daemons receive announcements from peers on these sessions, the routes that are added will have the corresponding VLAN association. This means that traffic egressing the rack for prefixes learned from these peers will have the specified VLAN tag.

oxide system networking bgp peer set \
    --rack $rack \
    --switch switch0 \
    --port qsfp0 \
    --addr 169.254.30.1 \
    --bgp-config as65547 \
    --vlan-id 300

oxide system networking bgp peer set \
    --rack $rack \
    --switch switch1 \
    --port qsfp0 \
    --addr 169.254.40.1 \
    --bgp-config as65547 \
    --vlan-id 400

The new sessions should eventually reach the Established state as observed through the CLI.

oxide system networking bgp show-status
switch0
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.10.1  65547      64500       Established    20s 83ms
169.254.30.1  65547      64502       Established    10s 811ms

switch1
=======
Peer Address  Local ASN  Remote ASN  Session State  State Duration
169.254.20.1  65547      64500       Established    12s 831ms
169.254.40.1  65547      64502       Established    3s 939ms

Controlling BGP Announcements

In some situations, it’s necessary to limit what announcements go to what peers. For the Oxide rack, this is configurable through the CLI. Consider the case where the intent is to only export 198.51.100.0/24 to transit providers and 203.0.113.0/24 to cloud providers. For the example topology in this guide, this filtering is accomplished as follows.

oxide system networking bgp filter \
    --rack $rack \
    --switch switch0 \
    --port qsfp0 \
    --peer 169.254.10.1 \
    --direction 'export' \
    --allowed 198.51.100.0/24

oxide system networking bgp filter \
    --rack $rack \
    --switch switch1 \
    --port qsfp0 \
    --peer 169.254.20.1 \
    --direction 'export' \
    --allowed 198.51.100.0/24

oxide system networking bgp filter \
    --rack $rack \
    --switch switch0 \
    --port qsfp0 \
    --peer 169.254.30.1 \
    --direction 'export' \
    --allowed 203.0.113.0/24

oxide system networking bgp filter \
    --rack $rack \
    --switch switch1 \
    --port qsfp0 \
    --peer 169.254.40.1 \
    --direction 'export' \
    --allowed 203.0.113.0/24

Note

The --allowed flag may be specified multiple times to allow multiple prefixes per-peer. The set of prefixes defined by the --allowed flag describe the entire set of allowed prefixes for the peer. Any pre-existing allowed prefixes that are omitted from the list will be removed. Prefixes are exact matches and do not follow subnet inclusion rules. Thus, allowing 10.10.0.0/16 does not imply allowing 10.10.0.0/24.

The new live configuration can be seen through the CLI.

oxide system networking switch-port-settings show
switch0/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.10.2/30  initial-infra  None
169.254.30.2/30  cloud-peering  Some(300)

BGP Peer      Config   Export             Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.10.1  as65547  [198.51.100.0/24]  [no filtering]  []           3              3           false             6          3               2          None        None      None     None  None        None
169.254.30.1  as65547  [203.0.113.0/24]   [no filtering]  []           0              0           false             6          0               2          None        None      None     None  None        Some(300)

switch1/qsfp0
=============
Autoneg  Fec   Speed
false    None  Speed100G

Address          Lot            VLAN
169.254.20.2/30  initial-infra  None
169.254.40.2/30  cloud-peering  Some(400)

BGP Peer      Config   Export             Import          Communities  Connect Retry  Delay Open  Enforce First AS  Hold Time  Idle Hold Time  Keepalive  Local Pref  Md5 Auth  Min TTL  MED   Remote ASN  VLAN
169.254.20.1  as65547  [198.51.100.0/24]  [no filtering]  []           3              3           false             6          3               2          None        None      None     None  None        None
169.254.40.1  as65547  [203.0.113.0/24]   [no filtering]  []           0              0           false             6          0               2          None        None      None     None  None        Some(400)

The prefixes that have been imported on a peer-by-beer basis can also be viewed through the CLI.

oxide system networking bgp imported ipv4 --asn 65547
[
  {
    "id": 3232235963,
    "nexthop": "169.254.20.1",
    "prefix": "0.0.0.0/0",
    "switch": "switch1"
  },
  {
    "id": 3232235882,
    "nexthop": "169.254.40.1",
    "prefix": "240.0.0.0/4",
    "switch": "switch1"
  },
  {
    "id": 3232235801,
    "nexthop": "169.254.30.1",
    "prefix": "240.0.0.0/4",
    "switch": "switch0"
  },
  {
    "id": 3232235887,
    "nexthop": "169.254.10.1",
    "prefix": "0.0.0.0/0",
    "switch": "switch0"
  }
]

Here we can see that the transit providers are announcing a default route to the rack, and the cloud providers are announcing the class-E prefix 240.0.0.0/4.

To round out the guide, the prefix 203.0.113.0/24 is announced to the cloud providers.

oxide system networking bgp announce \
    --announce-set as65547-announce \
    --address-lot cloud-pool \
    --prefix 203.0.113.0/24

Then an IP pool is created for instances to be created that draw addresses from the pool being announced to the cloud.

cloud_pool_name='cloud-pool'
oxide ip-pool create \
    --name $cloud_pool_name \
    --description 'ip pool for talking to the cloud'

oxide ip-pool range add \
    --pool $cloud_pool_name \
    --first 203.0.113.1 \
    --last 203.0.113.254

Last updated

Expanding Connectivity

Table of Contents

Initial Configuration

Expanding Connectivity

Address Lot Creation

Address Assignment

BGP Configuration

Controlling BGP Announcements