Skip to content

Managing the Kubernetes Cluster

This article will outline methods and commands to manage our Kubernetes cluster. I'll walk you through the basics of SSHing into the server. Then an overview of some tools that you can use to manage the cluster.

SSHing into the Server.

Note

It is assumed that you are already aware of what SSH is and how to use it.

This is the most basic way to interact with the server. You can use it get a look at things like the mounted disks, running processes, trigger builds on the system etc.

The server has no password login method. Thus, to get access to the server, you need to ask someone who already has access to it to add you to its list of trusted hosts.

In order to add someone to the server, you need to first generate a SSH key-pair. To do that make sure that you have a ssh on your computer. Figure out how to install that if you don't. Then you can run

ssh-keygen # (1)!
  1. Follow the steps to generate the files correctly.

This should create a public-private key pair in your ~/.ssh folder. You then need to send your public key (the file ending with .pub) to a server admin.

Warning

It is very bad practice to share your private ssh key. Make sure that you never do that!

When sharing your public key

SSH verification doesn't work if you have trailing \n or \t or even \0 characters when you paste it. Make sure that you have correctly copied your public key when sharing it.

The server admin then has to add the public key in a newline in the ~/.ssh/authorized_keys file. The easiest way to do this is to open the file in nano or vim on the server and paste it as a newline. I would not recommend directly piping it into the file using the >> operator

You can then connect to the server by running: 

ssh ubuntu@118.138.243.12
# OR
ssh ubuntu@monashmotorsport-ci.cloud.edu.au

Make sure that you type yes when prompted.

Basic Inspection Through SSH

There are some basic thing that you can do via SSH to interact with the server and the cluster on it. Some things that i've found usefull are:

df -h # (1)!

htop # (2)!

nproc # (3)!

sudo journalctl # (4)!
  1. This lets you see free disk space. The -h makes the output human readable
  2. htop is a pretty way to see the usage of the server. It's a nicer formatted version of the top command that is standard on LINUX systems.
  3. This lets you see the Number of PROCesses on the machine
  4. This lets you see logs from the servers system. Letting you spot thing like compiler invocations and system calls that sometimes tell you why things might be crashing.

Use tmux for longer tasks

If you are doing any sort of monitoring, or longer tasks like building containers whilst SSH'd into the server. I would recommend using tmux which is preinstalled on the server. Use it to have persistent 'sessions' that will run even if your ssh disconnects. 

tmux # Start a new tmux session

tmux attach # Attach to your previous tmux session

man tmux # Bring up the tmux manual
I'll leave it to you to go look at the tmux documenation to figure out how to use it.

kubectl

This is perhaps the most important tool that you will have in order to interact with the cluster. It allows for you to send requests to the Kubernetes API through your command line. Follow the installation instructions to install it on your personal computer

Setting Up kubectl

There are then some steps that you have to do in order to get your local installation of kubectl to talk to the cluster.

  1. SSH into the cluster and copy the contents of the /etc/rancher/k3s/k3s.yaml file. You can do this with the following: 
    sudo cat /etc/rancher/k3s/k3s.yaml # Copy this from your terminal
  2. Run mkdir ~/.kube && touch ~/.kube/config to make a file called config.
  3. Paste the content you copied into the config file.

You can then check that kubectl is properly installed by running:

kubectl version
If you get any kind of error, you have probably done something wrong.

Alias kubectl to something easier

You are going to be using kubectl a lot. It's best to alias it something easier to type. Here is what I have: 

alias k="kubectl"
You can put that in your .bashrc or .zshrc to have it persistent.

Also make sure that your kubectl completions are properly setup. They are a lifesaver. 

# If you are using bash
echo "source <(kubectl completion bash) >> ~/.bashrc

# If you are using zsh
echo '[[ $commands[kubectl] ]] && source <(kubectl completion zsh)' >> ~/.zshrc

Using kubectl & Basic Patterns You'll See

Some of the basic commands you'll see yourself doing often are:

get

This is a command to list objects that you specify.

k -A get  services  # (1)!
k get pods
k -A get pods -o wide # (2)!
1. The -A is shorthand for --all-namespaces you will be using it a lot, or you will be specifying the namespace you want to look inside with -n <namespace> 2. List pods with more information, equivalent to the -v flag in most other commands.

describe

This is a command that lets you get the description of objects in your cluster.

k describe -n jenkins pods jenkins-0

k describe -n container-registry pod container-registry-zot-0 

k describe -n ingress ingressroutes.traefik.io http-catch-all-redirect

log

This lets you see the logs of k8s object.

k logs -n jenkins jenkins-0

k logs -n kube-system services/traefik

K9s

K9s is a cool little TUI to visually inspect the state of your cluster and perform actions such as viewing logs and descriptions more ergonomically. Read the installation instructions here

Further Reading