Kubernetes management

Overview

This page describes the creation and management of Kubernetes clusters in our OpenStack-based stoney cloud using the Magnum container orchestration engine (COE).

Note: This page is a work-in-progress, so if you face any challenges, please contact us with the specific issue.

Images

To list available Kubernetes images, run the following command:

openstack image list -f yaml | yq '[ .[] | select(.Name | match("(?i)kubernetes.*raw")) ]'

Example output:

- ID: 4c6f82a1-1ebc-451d-b989-947cffe72f0e
  Name: 'Ubuntu 22.04 (20240702): Kubernetes v1.28.11 (raw)'
  Status: active
- ID: c2ac4670-1422-4de6-864c-a5b813cddf61
  Name: 'Ubuntu 22.04 (20240702): Kubernetes v1.29.6 (raw)'
  Status: active
- ID: 21ad55f9-21fa-4586-932f-c9ecd20b403d
  Name: 'Ubuntu 22.04 (20240702): Kubernetes v1.30.2 (raw)'
  Status: active

Creation

Creation - Cluster template

Use openstack coe cluster template create to create a cluster template:

# ID or name of the OpenStack Kubernetes image to use:
image=c539d525-d912-4acb-a7c3-bfcaf5f533c5 # Ubuntu 22.04 (20240605): Kubernetes v1.30.1

openstack coe cluster template create my_k8s_template \
  --coe kubernetes \
  --image "$image" \
  --external-network public \
  --master-flavor 'Standard Düdingen c002m0004' \
  --flavor 'Standard Düdingen c002m0004' \
  --public \
  --master-lb-enabled

For a list of all labels see Magnum User Guide - Labels.

Creation - Cluster

Use openstack coe cluster create to create a cluster:

# ID or name of the cluster template to use:
template=my_k8s_template

openstack coe cluster create my_k8s_cluster \
  --cluster-template "$template_id" \
  --master-count 1 \
  --node-count 2

Options:

• --master-count: number of control plane nodes.
• --node-count: number of worker nodes.

After creating a cluster, the cluster should show up in the list returned by openstack coe cluster list:

openstack coe cluster list

+--------------------------------------+----------------+------------+------------+--------------+--------------------+---------------+
| uuid                                 | name           | keypair    | node_count | master_count | status             | health_status |
+--------------------------------------+----------------+------------+------------+--------------+--------------------+---------------+
| e41cc55c-9f63-4880-9c18-0c021545efa4 | sst-yde-test-1 | None       |          1 |            1 | CREATE_IN_PROGRESS | None          |
+--------------------------------------+----------------+------------+------------+--------------+--------------------+---------------+

After 5-10 minutes, the cluster property status should turn from CREATE_IN_PROGRESS to CREATE_COMPLETE and health_status should be HEALTHY:

openstack coe cluster list

+--------------------------------------+----------------+------------+------------+--------------+--------------------+---------------+
| uuid                                 | name           | keypair    | node_count | master_count | status             | health_status |
+--------------------------------------+----------------+------------+------------+--------------+--------------------+---------------+
| e41cc55c-9f63-4880-9c18-0c021545efa4 | sst-yde-test-1 | None       |          1 |            1 | CREATE_COMPLETE    | HEALTHY       |
+--------------------------------------+----------------+------------+------------+--------------+--------------------+---------------+

Creation - Cluster - Access

To access a created cluster with kubectl, helm etc., use openstack coe cluster config to retrieve the cluster's configuration:

# ID or name of the cluster:
cluster=''

# Path to the directory where the Kubeconfig file will be stored:
directory="$HOME"/.kube

# Make sure the directory exists:
mkdir --parent "$directory"

# Download the Kubeconfig.
# This will create the Kubeconfig file at $directory/config
openstack coe cluster config --dir "$directory" "$cluster"

Depending on where the Kubeconfig has been store, you will need to set the environment variable KUBECONFIG accordingly (if you used the directory .kube in your home directory, this step is not required as it is the default path used by kubectl):

export KUBECONFIG=/path/to/config

You will now be able to access the cluster using kubectl:

kubectl get node -o wide

NAME                                                     STATUS   ROLES           AGE   VERSION    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
sst-yde-test-1-fj6ladv5sne3-control-plane-jl86d          Ready    control-plane   45m   v1.28.11   10.0.0.149    <none>        Ubuntu 22.04.4 LTS   5.15.0-113-generic   containerd://1.7.13
sst-yde-test-1-fj6ladv5sne3-default-worker-b2c7j-5bh2w   Ready    <none>          44m   v1.28.11   10.0.0.150    <none>        Ubuntu 22.04.4 LTS   5.15.0-113-generic   containerd://1.7.13

Management

Management - Control plane

TBD

Management - Worker nodes

# ID or name of the cluster:
cluster=''

# Number of worker nodes:
nodes=''

openstack coe cluster resize "$cluster" "$nodes"

Upgrade

To upgrade a cluster (upgrading the version of Kubernetes), a new cluster template needs to be created first.

After creating the cluster template, use openstack coe cluster upgrade to assign the new cluster template to your cluster:

# ID or name of the cluster to upgrade:
cluster=''

# ID or name of the new template to use:
template=''

openstack coe cluster upgrade "$cluster" "$template"

Deletion

Deletion - Cluster template

Use openstack coe cluster template delete to delete a cluster template:

# ID or name of the cluster template to delete:
template=''

openstack coe cluster template delete "$template"

Deletion - Cluster

Use openstack coe cluster delete to delete a cluster:

# ID or name of the cluster to delete:
cluster=''

openstack coe cluster delete "$cluster"

Troubleshooting

Troubleshooting - Adjusting the Kubernetes image label kube_version

When creating a cluster using openstack coe cluster create, you might encounter the following error:

Image c539d525-d912-4acb-a7c3-bfcaf5f533c5 does not have a kube_version property.

To fix this problem, add the kube_version property to the image:

# ID or name of the OpenStack Kubernetes image:
image_id=c539d525-d912-4acb-a7c3-bfcaf5f533c5 # Ubuntu 22.04 (20240605): Kubernetes v1.30.1

# The Kubernetes version of the image:
k8s_image_version=v1.30.1

# Adjust the image and add the "kube_version" property:
openstack image set "$image_id" --property kube_version="$k8s_image_version"

After adjust the image property you will have to delete the existing cluster and re-create it using openstack coe cluster create.