Kubernetes Cluster API

The Oxide infrastructure provider for Cluster API (CAPOx) enables Kubernetes Cluster API (CAPI) to provision Oxide instances as Kubernetes (K8s) nodes. This guide walks through creating a local management cluster that runs CAPI and CAPOx and using it to provision a workload cluster on Oxide.

Requirements

To follow this guide, you need the following.

  • API credentials for your Oxide silo. Refer to Authentication for instructions on generating API credentials.

  • An Ubuntu 24.04 Oxide image in your project. This is the base image used to build a K8s-conformant image.

  • oxide to create prerequisite Oxide resources.

  • docker to build the K8s-conformant image.

  • kind to run the local management cluster.

  • kubectl to connect to the K8s clusters created in this guide.

  • clusterctl to install the CAPI and CAPOx resources on the local management cluster.

  • cilium CLI to install Cilium, a Container Network Interface (CNI) plugin, on the workload cluster.

  • helm to install the Oxide Cloud Controller Manager.

Create VPC infrastructure

The workload cluster needs a VPC and subnet for its instances, along with VPC firewall rules that allow access to the Kubernetes API and to a web server we’ll deploy later in the guide.

  1. Set environment variables used throughout the guide.

    export OXIDE_HOST=<oxide-host>
    export OXIDE_TOKEN=<oxide-token>
    export OXIDE_PROJECT=<project>
    export OXIDE_BOOT_DISK_IMAGE_ID=<ubuntu-24-04-base-image-id>
    export OXIDE_VPC=capox
    export OXIDE_SUBNET=capox

  2. Create the VPC.

    oxide vpc create \
      --project "$OXIDE_PROJECT" \
      --name "$OXIDE_VPC" \
      --description capox \
      --dns-name capox

  3. Create the subnet.

    oxide vpc subnet create \
      --project "$OXIDE_PROJECT" \
      --vpc "$OXIDE_VPC" \
      --name "$OXIDE_SUBNET" \
      --description capox \
      --ipv4-block "192.168.0.0/16"

  4. Create firewall rules that allow ICMP, internal VPC traffic, and inbound TCP ports 80 (for a workload web server we will deploy) and 6443 (for kube-apiserver).

    FIREWALL_RULES=$(mktemp)

    cat <<EOF > "$FIREWALL_RULES"
    {
      "rules": [
        {
          "name": "allow-icmp",
          "description": "Allow ICMP.",
          "action": "allow",
          "direction": "inbound",
          "priority": 65534,
          "status": "enabled",
          "filters": {
            "protocols": [{ "type": "icmp" }]
          },
          "targets": [{ "type": "vpc", "value": "$OXIDE_VPC" }]
        },
        {
          "name": "allow-internal",
          "description": "Allow internal VPC traffic.",
          "action": "allow",
          "direction": "inbound",
          "priority": 65534,
          "status": "enabled",
          "filters": {
            "hosts": [{ "type": "vpc", "value": "$OXIDE_VPC" }]
          },
          "targets": [{ "type": "vpc", "value": "$OXIDE_VPC" }]
        },
        {
          "name": "allow-cluster-ports",
          "description": "HTTP and the K8s HTTPS API",
          "action": "allow",
          "direction": "inbound",
          "priority": 65534,
          "status": "enabled",
          "filters": {
            "ports": ["80", "6443"],
            "protocols": [{ "type": "tcp" }]
          },
          "targets": [{ "type": "vpc", "value": "$OXIDE_VPC" }]
        }
      ]
    }
    EOF

    oxide vpc firewall-rules update \
      --project "$OXIDE_PROJECT" \
      --vpc "$OXIDE_VPC" \
      --json-body "$FIREWALL_RULES"

Build the Kubernetes node image

CAPI’s built-in kubeadm bootstrap provider requires an instance with K8s components pre-installed (e.g., kubeadm, kubelet, containerd).

We’ll use the CAPI-maintained image-builder to build a K8s-conformant Oxide image based on Ubuntu 24.04.

  1. Build the Oxide K8s-conformant Oxide image with image-builder.

    Note
    The image build can take up to 40 minutes.
    docker run -it --rm --platform linux/amd64 \
      -e OXIDE_HOST \
      -e OXIDE_TOKEN \
      -e OXIDE_PROJECT \
      -e OXIDE_BOOT_DISK_IMAGE_ID \
      registry.k8s.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.55 \
      build-oxide-ubuntu-2404

  2. Export the image ID from the previous command’s output. This environment variable is used in the clusterctl generate cluster step below.

    export OXIDE_IMAGE_ID=<k8s-image-id>

Create the management cluster

CAPI uses a management cluster to configure and deploy workload clusters. Any K8s cluster can serve as the management cluster, but we’ll use kind to deploy a local management cluster.

  1. Create a kind K8s cluster.

    kind create cluster --name capi-oxide-management --wait 30s

  2. Save the kubeconfig.

    OXIDE_MANAGEMENT_KUBECONFIG="$(mktemp)"
    kind get kubeconfig --name capi-oxide-management > "$OXIDE_MANAGEMENT_KUBECONFIG"

  3. Add Oxide credentials to the K8s management cluster.

    kubectl create secret generic cluster-api-provider-oxide \
        --from-literal=oxide-host="$OXIDE_HOST" \
        --from-literal=oxide-token="$OXIDE_TOKEN" \
        --kubeconfig "$OXIDE_MANAGEMENT_KUBECONFIG"

  4. Install the CAPI and CAPOx resources into the K8s management cluster.

    clusterctl init --kubeconfig "$OXIDE_MANAGEMENT_KUBECONFIG" --infrastructure oxide

Create a workload cluster

  1. Create a K8s workload cluster called quickstart.

    Note
    The environment variables set earlier are used to generate the cluster manifests.
    clusterctl generate cluster \
        quickstart \
        --kubernetes-version "v1.36.3" \
        --control-plane-machine-count 3 \
        --worker-machine-count 2 | kubectl apply --kubeconfig "$OXIDE_MANAGEMENT_KUBECONFIG" -f -

  2. Wait for the quickstart cluster’s control plane to be initialized. It will take a few minutes for a control plane node to start and have a floating IP attached to it.

    kubectl wait cluster \
        quickstart \
        --for=condition=ControlPlaneInitialized \
        --timeout=5m \
        --kubeconfig "$OXIDE_MANAGEMENT_KUBECONFIG"

  3. Save the quickstart cluster’s kubeconfig.

    OXIDE_QUICKSTART_KUBECONFIG="$(mktemp)"
    clusterctl get kubeconfig quickstart --kubeconfig "$OXIDE_MANAGEMENT_KUBECONFIG" > "$OXIDE_QUICKSTART_KUBECONFIG"

  4. Install Cilium.

    cilium install --kubeconfig "$OXIDE_QUICKSTART_KUBECONFIG"

  5. Create a Secret to store Oxide credentials for the Oxide Cloud Controller Manager (CCM).

    kubectl create secret -n kube-system generic quickstart-oxide-cloud-controller-manager \
        --from-literal=oxide-host="$OXIDE_HOST" \
        --from-literal=oxide-token="$OXIDE_TOKEN" \
        --from-literal=oxide-project="$OXIDE_PROJECT" \
        --kubeconfig "$OXIDE_QUICKSTART_KUBECONFIG"

  6. Install the Oxide CCM.

    helm upgrade --install quickstart \
        oci://ghcr.io/oxidecomputer/helm-charts/oxide-cloud-controller-manager \
        --namespace kube-system \
        --wait \
        --kubeconfig "$OXIDE_QUICKSTART_KUBECONFIG"

  7. Wait for the nodes to report Ready.

    kubectl get nodes --kubeconfig "$OXIDE_QUICKSTART_KUBECONFIG"

Deploy a workload

We’ll deploy a web server to test functionality of our workload cluster.

  1. Create a Deployment of traefik’s whoami web server.

    kubectl create deployment hello-oxide \
        --image ghcr.io/traefik/whoami \
        --port 80 \
        --kubeconfig "$OXIDE_QUICKSTART_KUBECONFIG"

  2. Create a Service by exposing port 80 of the Deployment.

    kubectl expose deployment hello-oxide \
        --port 80 \
        --target-port 80 \
        --type LoadBalancer \
        --kubeconfig "$OXIDE_QUICKSTART_KUBECONFIG"

  3. Wait for the Deployment to be Available.

    kubectl wait deployment \
        hello-oxide \
        --for=condition=Available \
        --timeout=5m \
        --kubeconfig "$OXIDE_QUICKSTART_KUBECONFIG"

  4. Watch the service until an external IP is populated.

    kubectl get service hello-oxide --watch --kubeconfig "$OXIDE_QUICKSTART_KUBECONFIG"

  5. Access the web server with a browser or curl. Once the web server comes up, it will return some headers of the request.

    $ curl http://<external-ip>
    Hostname: hello-oxide-68bd4d9cf7-hg6hm
    IP: 127.0.0.1
    IP: ::1
    IP: 10.0.1.153
    IP: fe80::28f5:b4ff:feed:70f9
    RemoteAddr: 172.21.252.244:32858
    GET / HTTP/1.1
    Host: 172.21.253.73
    User-Agent: curl/8.5.0
    Accept: */*

Clean up

  1. Delete the workload cluster by deleting the cluster resource from the management cluster.

    kubectl delete cluster quickstart --kubeconfig "$OXIDE_MANAGEMENT_KUBECONFIG"
    Note
    If the quickstart cluster was not found, make sure you are using the management cluster kubeconfig.

  2. Ensure that all cluster resources in the management cluster have been deleted.

    kubectl get clusters --all-namespaces --kubeconfig "$OXIDE_MANAGEMENT_KUBECONFIG"
    Caution
    Always delete the workload cluster first. Deleting the management cluster before the workload cluster will leave orphaned Oxide resources that will need to be manually deleted.

  3. Delete the management cluster.

    kind delete cluster --name capi-oxide-management

  4. Delete the VPC infrastructure.

    oxide vpc subnet delete --project "$OXIDE_PROJECT" --vpc "$OXIDE_VPC" --subnet "$OXIDE_SUBNET"
    oxide vpc subnet delete --project "$OXIDE_PROJECT" --vpc "$OXIDE_VPC" --subnet default
    oxide vpc delete --project "$OXIDE_PROJECT" --vpc "$OXIDE_VPC"

References