Skip to main content

Installing an OpenShift cluster on OpenStack

We'll be using the following guide to install a OKD Kubernetes cluster on an OpenMetal provisioned OpenStack cloud: Installing a cluster on OpenStack with customizations. For more information about configuration options please refer to the OKD documentation. The intent of this guide is to quickly validate the installation process on an OpenMetal cloud.

Prerequisites

We'll be performing these installation steps on one of the hardware nodes provisioned for your cloud. Make sure that you have deployed a cloud and have access ssh access to the nodes. If deploying from another machine, you may need to preform additional steps.

Setup OpenStack Access

Installing OpenStack CLI

Complete the following steps to install the OpenStack CLI:.

Update quotas

Remove the following quota limits for the project you plan on using for this installation by setting the value to -1.

  • RAM: -1
  • VCPU: -1
openstack quota set --cores -1 PROJECT_NAME
openstack quota set --ram -1 PROJECT_NAME

Save Password

During the setup process of the OpenStack CLI, you should have created a clouds.yaml. Update that file by adding your password in the auth field.

vim ~/.config/openstack/clouds.yaml
clouds:
openstack:
auth:
auth_url: http://my-hostname:5000
username: "username"
password: "password"
...

Note: You can also keep secrets in a separate file: https://docs.openstack.org/os-client-config/latest/user/configuration.html#splitting-secrets.

Prepare Installation

Download OKD Installer

mkdir ~/okd/ && cd ~/okd/
curl -o openshift-install-linux.tar.gz -L https://github.com/openshift/okd/releases/download/4.11.0-0.okd-2022-07-29-154152/openshift-install-linux-4.11.0-0.okd-2022-07-29-154152.tar.gz
tar -xvf openshift-install-linux.tar.gz

Generate a key pair

ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_okd && chmod 600 /root/.ssh/id_okd

Add the SSH private key identity to the SSH agent for your local user

eval "$(ssh-agent -s)"
ssh-add /root/.ssh/id_okd

Create floating IPs

Record the IP addresses it returns, we'll pass these IPs to the installer later.

openstack floating ip create --description "API okd test cluster" External
openstack floating ip create --description "APPS okd test cluster" External

Output:

+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| created_at | 2022-08-12T22:49:31Z |
| description | API okd test cluster |
| dns_domain | None |
| dns_name | None |
| fixed_ip_address | None |
| floating_ip_address | 127.0.0.1 |
| floating_network_id | 4e08a887-5fa7-4cc1-b106-26ade52e3e5f |
| id | c5c9c416-5ea4-4ebe-8b02-307f6c371b06 |
| name | 127.0.0.1 |
| port_details | None |
| port_id | None |
| project_id | a8df8d1113b444a694f59653b830c1df |
| qos_policy_id | None |
| revision_number | 0 |
| router_id | None |
| status | DOWN |
| subnet_id | None |
| tags | [] |
| tenant_id | a8df8d1113b444a694f59653b830c1df |
| updated_at | 2022-08-12T22:49:31Z |
+---------------------+--------------------------------------+

DNS Configuration

Option 1

Setup DNS records. Recommended for production clusters.

  • api.<cluster_name>.<base_domain>. IN A <API_FLOATING_IP>
  • *.apps.<cluster_name>.<base_domain>. IN A <APPS_FLOATING_IP>
  • api-int.<cluster_name>.<base_domain>. IN A <API_FLOATING_IP>

Note: OKD will fail if it cannot resolve these DNS records to the floating IP addresses you selected. If the install does fail for this reason, it will tell you the host is unreachable or no route to host.

Option 2: Add hostfile mods

Add the following to /etc/hosts. Replace the floating IPs with the IPs you created above. You'll need to use this block on several servers. Only recommended for testing purposes.

API_FLOATING_IP api.okd.testing-okd.com
API_FLOATING_IP api-int.okd.testing-okd.com
APPS_FLOATING_IP grafana-openshift-monitoring.apps.okd.testing-okd.com
APPS_FLOATING_IP prometheus-k8s-openshift-monitoring.apps.okd.testing-okd.com
APPS_FLOATING_IP oauth-openshift.apps.okd.testing-okd.com
APPS_FLOATING_IP console-openshift-console.apps.okd.testing-okd.com
APPS_FLOATING_IP integrated-oauth-server-openshift-authentication.apps.okd.testing-okd.com

Generate Installation Configs

Create Manifest Files

mkdir ~/okd/install-directory
./openshift-install --dir ~/okd/install-directory create manifests

You'll be prompted for information about the cluster. As a reference, we used the following values for each prompt.

  • Base domain will be the domain you set up DNS or the domain in your hosts file mod.
  • OKD wants a pull secret. The pull secret is used for making pull requests from private container image registries. If you have a private registry secret you can provide that. Otherwise, you can use a fake secret the OKD documentation provides: {"auths":{"fake":{"auth":"aWQ6cGFzcwo="}}}. Copy and paste that entire string into the prompt for pull secret. It will show as a series of asterisks. OKD will fail if the pull secret is not a valid pull secret or the fake secret. If you use the fake secret, OKD will pull its container images from public repositories. OKD does cache container images, so you do not have to repull them for every run of OKD. If you use the pull secret provided, Red Hat operators will be unavailable. For more information see their documentation:
  • You must choose a flavor with at least 16 GB memory, 4 vCPUs, and 100 GB storage space. The gp1.xlarge flavor has enough resources. OKD will fail with an error about the flavor not having enough RAM or vCPUs if you pick a small flavor.
? SSH Public Key /root/.ssh/id_okd.pub
? Platform openstack
? Cloud openstack
? ExternalNetwork External
? APIFloatingIPAddress <API floating IP address you generated earlier>
? FlavorName gp1.large
? Base Domain testing-okd.com
? Cluster Name okd
? Pull Secret [? for help] {"auths":{"fake":{"auth":"aWQ6cGFzcwo="}}}

Generate Install Configuration

./openshift-install create install-config --dir ~/okd/install-directory

Create Security Group

Create a security group that allows for SSH access from the host on which you are running the openshift-install application. You can get failures if the host cannot access the bootstrap node via SSH Provision floating IPs for the project that will host the OKD infrastructure.

openstack security group create okd-deploy
openstack security group rule create okd-deploy \
--protocol TCP --remote-ip YOUR_IP_ADDRESS/32 --description "Allow deployment host"

Update the Install Config

vim ~/okd/install-directory/install-config.yaml

Add security group UUID so the installer can access the cluster. Replace the UUID with the UUID of the security group you created above.

controlPlane:
platform:
openstack:
additionalSecurityGroupIDs: ["7af89a64-3cc0-444f-9273-63f309a003c2"]

Specify the IP of the floating IPs you created earlier. The API IP should already be filled in.

platform:
openstack:
apiFloatingIP: 127.0.0.1
ingressFloatingIP: 127.0.0.2

The complete config looks like the following example. You can customize this config to your needs.

apiVersion: v1
baseDomain: okd.testing-okd.com
compute:
- architecture: amd64
hyperthreading: Enabled
name: worker
replicas: 3
controlPlane:
architecture: amd64
hyperthreading: Enabled
name: master
platform:
openstack:
additionalSecurityGroupIDs: ["7af89a64-3cc0-444f-9273-63f309a003c2"]
replicas: 3
metadata:
creationTimestamp: null
name: okd
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
networkType: OVNKubernetes
serviceNetwork:
- 172.30.0.0/16
platform:
openstack:
apiFloatingIP: 127.0.0.1
ingressFloatingIP: 127.0.0.2
apiVIP: 10.0.0.5
cloud: openstack
defaultMachinePlatform:
type: gp1.large
externalDNS: null
externalNetwork: External
ingressVIP: 10.0.0.7
publish: External
pullSecret: '{"auths":{"fake":{"auth":"aWQ6cGFzcwo="}}}'
sshKey: |
ssh-ed25519 AAAAC3Nz...

Save a copy of the config

The installation process will delete your configuration file. Save a copy of the config before running the installer.

cp ~/okd/install-directory/install-config.yaml .

Install OKD

This process can take up to an hour. If the process fails, you'll need to delete the cluster and start again. Please see troubleshooting for more information.

./openshift-install create cluster --dir ~/okd/install-directory/ --log-level=info

If everything is successful, you should see URLs to access the API and the console. There will also be a username and password for the console. Save the username and password from the log output somewhere safe.

NOTE! If you are deploying with hosts file mods, you'll need to add the same entries to the bootstrapping VM after it's created.

watch openstack server list

Verify Installation

That's it! You've installed OKD onto your OpenStack cloud. Now you can verify that the installation was successful and deploy your own workloads.

Open Web Console

Optional - Add the hosts file mods to your local machine

Use the same entries you've added to the previous servers and add them to your local machine.

Web Console Login

Navigate to https://console-openshift-console.apps.okd.testing-okd.com/dashboards

Use the username and password that were output after you completed the installation. If you forget to save the password, the password is stored in the auth folder in the current directory in the kubeadmin-password file.

OKD Web Console on OpenMetal

Test Kubectl

Install Kubectl

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

Load configuration

 export KUBECONFIG=/root/okd/install-directory/auth/kubeconfig

List Pods

[root@silly-quokka okd]# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
openshift-apiserver-operator openshift-apiserver-operator-5686496c75-b6ngv 1/1 Running 1 (84m ago) 88m
openshift-apiserver apiserver-55b764f57-cp7w9 2/2 Running 0 76m
openshift-apiserver apiserver-55b764f57-ktq9c 2/2 Running 0 77m
openshift-apiserver apiserver-55b764f57-tf96q 2/2 Running 0 75m
openshift-authentication-operator authentication-operator-79647548ff-9pv9j 1/1 Running 1 (84m ago) 88m
openshift-authentication oauth-openshift-75cb6567b6-6ltrr 1/1 Running 0 69m
openshift-authentication oauth-openshift-75cb6567b6-n845d 1/1 Running 0 68m
openshift-authentication oauth-openshift-75cb6567b6-qtfcf 1/1 Running 0 69m
openshift-cloud-controller-manager-operator cluster-cloud-controller-manager-operator-54f49d867f-gzjlh 2/2 Running 0 87m
openshift-cloud-credential-operator cloud-credential-operator-7c4bbc8654-c6ckq 2/2 Running 0 88m
openshift-cluster-csi-drivers manila-csi-driver-operator-b4dfc5874-frvlj 1/1 Running 0 84m
openshift-cluster-csi-drivers openstack-cinder-csi-driver-controller-6b56dfff86-f5zfb 10/10 Running 0 82m
openshift-cluster-csi-drivers openstack-cinder-csi-driver-controller-6b56dfff86-wdh98 10/10 Running 0 85m
openshift-cluster-csi-drivers openstack-cinder-csi-driver-node-7v6sj 3/3 Running 0 76m
...

OKD Client

Install OKD Client

To verify operation, download the OKD client from the OKD GitHub releases page:

curl -OL https://github.com/openshift/okd/releases/download/4.11.0-0.okd-2022-07-29-154152/openshift-client-linux-4.11.0-0.okd-2022-07-29-154152.tar.gz
tar -xvf openshift-client-linux-4.11.0-0.okd-2022-07-29-154152.tar.gz

Fetch Kubernetes resources using OKD Client

Run the following OKD client commands using the oc binary:

  • ./oc get nodes
  • ./oc get clusterversion
  • ./oc get clusteroperator
  • ./oc get pods -A

Output:

(venv) [root@silly-quokka test]# ./oc get nodes
NAME STATUS ROLES AGE VERSION
okd-dstmh-master-0 Ready master 3h35m v1.24.0+9546431
okd-dstmh-master-1 Ready master 3h35m v1.24.0+9546431
okd-dstmh-master-2 Ready master 3h35m v1.24.0+9546431
okd-dstmh-worker-0-gbr8f Ready worker 3h24m v1.24.0+9546431
okd-dstmh-worker-0-l868x Ready worker 3h24m v1.24.0+9546431
okd-dstmh-worker-0-mjzlr Ready worker 3h24m v1.24.0+9546431

Troubleshooting

Restart OKD Installation

Delete the cluster

./openshift-install destroy cluster --dir ~/okd/install-directory/ --log-level=info

Copy the config

cp install-config.yaml ~/okd/install-directory/

Create the manifests

./openshift-install --dir ~/okd/install-directory create manifests

Start Install

./openshift-install create cluster --dir ~/okd/install-directory/ --log-level=info