Add a GPU host to your NUSAPOD fleet by running the join command generated by your control plane. This guide covers everything to check before and after running that command.
Prerequisites
NVIDIA driver: the join script installs the NVIDIA container toolkit but expects the kernel driver to already be present. Verify before proceeding:
nvidia-smiIf nvidia-smi fails, install the driver first (Ubuntu: sudo ubuntu-drivers install; for other distros follow NVIDIA's guide for your kernel version). A reboot is usually required after driver installation.
Operating system: Ubuntu 22.04 / 24.04 or Debian 12 (uses apt-get); RHEL / Rocky 9 (uses dnf). x86_64 or arm64. Other distros are not supported by the join script.
Network: the node dials out — no inbound ports need to be opened on the GPU host itself. Required outbound access:
- To the control plane: the console/backend port (
80by default, or443if you configured TLS) to redeem the join token, and6443for the K3s API. Both must match the address in your join command. - Port
443toget.k3s.ioand the NVIDIA toolkit repositories to fetch packages.
Access: run the join command as root (sudo).
Multiple GPUs: nothing extra to do — every GPU on the host is detected and registered. Repeat this guide on each additional GPU host to grow the fleet.
Clock: keep the node NTP-synced — the enrollment token is single-use and time-limited (default 1 hour). If it expires, generate a fresh join command from the console (Fleet → Connect node).
Two ways to join
From the console (Fleet → Connect node) you can add a GPU host two ways — both end with the same node joined to the cluster:
- SSH install — the control plane connects to the GPU host and does everything for you (push). Best when you have SSH access to the box.
- Agent command — the console gives you a
curl … | bashcommand that you run on the GPU host yourself (pull). Best when you'd rather not hand the control plane SSH credentials, or SSH isn't available.
SSH install (push from the console)
In Fleet → Connect node, pick SSH installand fill in the GPU host's details. This is the normal case for a host on the same network as the control plane but at a different IP:
| Field | Value |
|---|---|
| Host / IP | the GPU host's address (e.g. 192.168.1.30) — not the control plane's |
| SSH port | 22 |
| SSH user | root, or a user with passwordless sudo |
| Auth | the SSH private key or password for that host |
Click Test connection first — the control plane SSHes in, runs nvidia-smi, and checks sudo, then shows the detected GPU model, count, and driver before you commit. If it looks right, click Connect and the control plane installs the toolkit and joins the node for you.
Prerequisites
The SSH user must be root or have passwordless sudo (sudo -n) — the installer needs root. If sudo prompts for a password the install fails; Test connection surfaces this as a failed sudo check. The GPU host also needs the NVIDIA driver already installed (nvidia-smi working) and must be able to reach the control plane on port 80/443 and 6443.
Agent command (run it yourself)
Alternatively, copy the join command from the console (Fleet → Connect node → Agent command) and run it on the GPU host:
curl -sfL https://<control-plane>/install.sh | bash -s -- --token <token> --server https://<control-plane>The token is single-use and time-limited (~1 hour). If it expires before you run the command, regenerate a new one from the console — the old token cannot be reused.
The script:
- Verifies the NVIDIA driver (
nvidia-smi). - Redeems the enrollment token for the K3s join token.
- Installs the NVIDIA container toolkit and sets
nvidiaas the default containerd runtime (see below). - Installs and joins the K3s agent, labelling the node
gpu=on.
nvidia as the default containerd runtime
Warning
K3s registers an nvidia runtime when the toolkit is present, but leaves the default as runc. NUSAPOD pods carry no runtimeClassName, so they use the default runtime — under runc a pod runs but cannot see the GPU (VRAM limits won't be enforced, nvidia-smi inside the pod fails). The join script sets nvidia as the default automatically.
The join script sets nvidia as the default automatically and rolls back if it would break the runtime (so the node still joins safely). Verify after the script completes:
sudo grep default_runtime_name /var/lib/rancher/k3s/agent/etc/containerd/config.tomlExpected output: default_runtime_name = "nvidia". If it still shows runc, set it manually — contact support or see the troubleshooting section in the control-plane setup guide.
Verify
Within about one minute of the script completing, the node should appear as online in the console under Fleet. The platform registers the GPU(s) automatically. You can also check from the control plane:
kubectl get nodesThe new node should show Ready status. GPU inventory and telemetry (utilisation, memory, temperature, power) will populate in the console within ~1 minute of the GPU device-plugin starting on the node.
The K3s agent runs as a systemd service, so the node rejoins automatically after a reboot — no need to re-run the join command.
Troubleshooting
nvidia-smi not found / fails:the kernel driver isn't installed or the host needs a reboot after installing it. The join script installs only the container toolkit, not the driver.
“token expired / already used”: the enrollment token is single-use and lasts ~1 hour. Generate a fresh join command from the console (Fleet → Connect node) and run the new one.
Redeem fails / cannot reach the server: confirm the GPU host can reach the control plane at the --server address on port 80/443, and the K3s API on 6443:
curl -sf https://<control-plane>/healthz # backend reachable?
curl -skf https://<control-plane>:6443/readyz # K3s API reachable?Node joined but doesn't appear in the console: check the agent is running and the node is labelled gpu=on:
sudo systemctl status k3s-agent # on the GPU node
kubectl get nodes --show-labels # on the control plane — expect gpu=onGPUs don't show up (node Ready, but no VRAM): the default containerd runtime is probably still runc — re-check the runtime step above.
Duplicate hostnames: two hosts with the same hostname are handled automatically — the control plane assigns a unique node name on join, so cloned VMs join cleanly without renaming.
Remove a node
Remove a GPU node from the console (Fleet → open the node → Remove), which de-registers it and reschedules its pods. Then, on the GPU host itself, uninstall the K3s agent to fully clean up:
sudo /usr/local/bin/k3s-agent-uninstall.sh