This guide shows how to connect an Oxide rack to an upstream network via the Border Gateway Protocol (BGP). The BGP setup this guide will go over is depicted below.
Address Lot Creation
The first part of establishing a peer session with an upstream router is
assigning an address to the switch port over which peering will take place. On
the Oxide platform, addresses cannot be arbitrarily assigned, they must be
allocated from an address lot. The oxide CLI invocation and JSON specification
below set up an address lot for peering.
oxide system networking address-lot create \
--name peering \
--description 'a lot for bgp peering' \
--json-body peering-address-lot.json{
"name": "peering",
"description": "a lot for bgp peering",
"kind": "infra",
"blocks": [
{
"first_address": "192.168.22.1",
"last_address": "192.168.22.6"
}
]
}This configuration creates a single address log of type infra. This means the
address lot is used for rack infrastructure. The alternative type is pool,
which means the lot is used for allocating instance IP pools from. These types
prevent address lots from being used for unintended purposes. The lot contains 7
addresses in the inclusive range 192.168.22.1 - 192.168.22.6.
Configure Announcements
The next thing to do is configure what network prefixes the rack will announce to upstream BGP peers.
oxide system networking bgp-announce-set create \
--name instances \
--description 'the instance block' \
--json-body announce-set.json{
"name": "instances",
"description": "the instance block",
"announcement": [{
"address_lot_block": "instances",
"network": "192.168.20.0/24"
}]
}This creates an object called an announce set, which is a collection of
announcements. The example above shows a single announcement for the prefix
192.168.20.0/24.
Create a BGP Configuration
Next, a BGP configuration must be created. BGP configurations are named objects that carry a BGP autonomous system number (ASN). BGP routers in the same ASN peer using the internal variant of the BGP protocol (iBGP). BGP routers in different ASNs peer using the external variant of the BGP protocol (eBGP). A common deployment configuration is one BGP configuration with a distinct ASN per switch. This ensures that the Oxide rack peers over eBGP with upstream routers.
oxide system networking bgp-config create \
--name as47 \
--description 'autonomous system 47' \
--asn 47 \
--bgp-announce-set-id instancesBGP configurations are tied to an announce set. In the example above, the
instances announce set was created in the previous section.
Activating a BGP Configuration
After the BGP configuration is created, the switch ports will need to be configured to use it.
The Oxide network model is organized around switch ports and port settings. Port
settings are reusable configurations that are referenced by switch ports. The
CLI invocations below will update an existing port settings object that is
referenced by the port qsfp0 on switch0 in an Oxide rack.
The current setup can be queried as follows.
oxide system networking switch-port-settings list
SwitchPortSettings {
description: "initial uplink configuration",
id: b6e36cc6-2974-4946-903f-b13fc2a011cf,
name: Name(
"default-uplink0",
),
time_created: 2023-08-30T18:59:13.214659Z,
time_modified: 2023-08-30T18:59:13.214659Z,
}oxide system hardware switch-port list
[/snip]
SwitchPort {
id: a4bfd575-90e4-4d4e-9125-4573d48109e1,
port_name: "qsfp0",
port_settings_id: Some(
b6e36cc6-2974-4946-903f-b13fc2a011cf,
),
rack_id: de608e01-b8e4-4d93-b972-a7dbed36dd22,
switch_location: "switch0",
}
[\snip]These two commands show us that the port qsfp0 references has a configuration
by the name default-uplink0. We are now going to update that configuration to
use BGP.
oxide system networking switch-port-settings create \
--name portofino \
--description 'just a port' \
--json-body bgp-port-settings.jsonThe port settings configuration is shown in its entirety below, but the BGP-relevant bits will be highlighted first.
In the first part of the BGP configuration, the peers that are expected to be to
be reachable over the port qsfp0 are defined. The peer configuration
references the overarching BGP configuration defined above via the name
as47.
bgp_announce_set parameter here is the same as the one in the
overarching BGP configuration. The redundant occurrence will be removed from
this data structure in a future release.The interface name indicates what interface on the port peering will take place
on. For ports without breakout cables this is always phy0.
The addr field is the address of the router to peer with.
The remaining parameters are timers from the BGP specification.
Hold Time | How long to keep a peer session alive in the absence of keep alive or update messages. (seconds) |
Idle Hold Time | How long to keep a peer in idle after session tear down. (seconds) |
Delay Open | How long to delay sending open messages to the peer for initial connection attempts. (seconds) |
Connect Retry | How often to retry connecting to a peer. (seconds) |
Keepalive | How often to send keep alive messages to the peer. |
"bgp_peers": {
"qsfp0": {
"bgp_announce_set": "instances",
"bgp_config": "as47",
"interface_name": "phy0",
"addr": "192.168.22.1",
"hold_time": 6,
"idle_hold_time": 6,
"delay_open": 0,
"connect_retry": 3,
"keepalive": 2
}
},The next thing to do is add the source address the switch will use for peering to the port. In this example, it’s placed along side an address that was already there for static routing so static routing and BGP can run side-by-side initially.
"addresses": {
"phy0": {
"addresses": [
{
"address_lot": "initial-infra",
"address": "192.168.20.3/29"
},
{
"address_lot": "peering",
"address": "192.168.22.3/29"
}
]
}
},Here is the complete config.
{
"name": "default-uplink0",
"description": "just a port",
"port_config": {
"geometry": "qsfp28x1"
},
"groups": [],
"links": {
"phy0": {
"mtu": 1500,
"lldp": {
"enabled": false,
"lldp_config": null
},
"fec": "none",
"speed": "speed100_g"
}
},
"interfaces": {},
"bgp_peers": {
"qsfp0": {
"bgp_announce_set": "instances",
"bgp_config": "as47",
"interface_name": "phy0",
"addr": "192.168.22.1",
"hold_time": 6,
"idle_hold_time": 6,
"delay_open": 0,
"connect_retry": 3,
"keepalive": 2
}
},
"addresses": {
"phy0": {
"addresses": [
{
"address_lot": "initial-infra",
"address": "192.168.20.3/29"
},
{
"address_lot": "peering",
"address": "192.168.22.3/29"
}
]
}
},
"routes": {
"phy0": {
"routes": [{
"dst": "0.0.0.0/0",
"gw": "192.168.20.1",
"interface_name": "phy0"
}]
}
}
}Checking BGP Status
Once BGP has been configured on a port, its status is visible. It may take a few seconds for the peer to become established, so initial states might look like this:
oxide system networking bgp status
[
{
"addr": "172.20.15.43",
"local_asn": 65002,
"remote_asn": 64601,
"state": "open_sent",
"state_duration_millis": 4593,
"switch": "switch0"
},
{
"addr": "172.20.15.51",
"local_asn": 65002,
"remote_asn": 64601,
"state": "open_sent",
"state_duration_millis": 4573,
"switch": "switch1"
}
]This output indicates the first peer session is in the OpenSent state and has
been there for 4593 milliseconds.
After a few seconds the connection make it to established.
oxide system networking bgp status
[
{
"addr": "172.20.15.43",
"local_asn": 65002,
"remote_asn": 64601,
"state": "established",
"state_duration_millis": 47,
"switch": "switch0"
},
{
"addr": "172.20.15.51",
"local_asn": 65002,
"remote_asn": 64601,
"state": "established",
"state_duration_millis": 24,
"switch": "switch1"
}
]The Oxide rack provides an API endpoint for retrieving the last 1024 BGP messages sent and received by reach rack switch.
oxide system networking bgp history --asn 65002
{
"switch_histories": [
{
"history": {
"172.20.15.51": {
"received": [
{
"message": {
"type": "update",
"value": {
"nlri": [
{
"length": 0,
"value": []
}
],
"path_attributes": [
{
"typ": {
"flags": 64,
"type_code": "origin"
},
"value": {
"origin": "igp"
}
},
{
"typ": {
"flags": 64,
"type_code": "as_path"
},
"value": {
"as4_path": [
{
"typ": "as_sequence",
"value": [
64601
]
}
]
}
},
{
"typ": {
"flags": 64,
"type_code": "next_hop"
},
"value": {
"next_hop": "172.20.15.51"
}
}
],
"withdrawn": []
}
},
"timestamp": "2024-05-09T04:24:02.699067710Z"
},
...
],
"sent": [
{
"message": {
"type": "update",
"value": {
"nlri": [
{
"length": 24,
"value": [
172,
20,
26,
0
]
}
],
"path_attributes": [
{
"typ": {
"flags": 64,
"type_code": "origin"
},
"value": {
"origin": "igp"
}
},
{
"typ": {
"flags": 64,
"type_code": "as_path"
},
"value": {
"as4_path": [
{
"typ": "as_sequence",
"value": [
47
]
}
]
}
},
{
"typ": {
"flags": 64,
"type_code": "next_hop"
},
"value": {
"next_hop": "172.20.15.53"
}
}
],
"withdrawn": []
}
},
"timestamp": "2024-05-09T04:24:02.698739718Z"
},
...
]
}
},
"switch": "switch1"
},
...
]
}