James Lab

Phase 3 - Post-Install Hardening & Lessons Learned

3 min read

The cluster is running. This final phase covers hardening your setup — running kubectl without root, securing credentials with Ansible Vault — and the key lessons learned from building this lab.

1. Non-Root Cluster Administration

By default, K3s places its kubeconfig at /etc/rancher/k3s/k3s.yaml owned by root:root with mode 600. Only root can use kubectl. Here’s how to fix that.

Create a Dedicated Group

sudo groupadd k3s-admin
sudo usermod -a -G k3s-admin jmartinez

Log out and back in (or run newgrp k3s-admin) for the group change to take effect.

Update the K3s Service

Tell K3s to set the kubeconfig group and permissions on startup.

  1. Edit the service file:
sudo nano /etc/systemd/system/k3s.service
  1. Find the ExecStart line and append:
--write-kubeconfig-group k3s-admin --write-kubeconfig-mode 640

640 means: owner (root) read/write, group (k3s-admin) read, others nothing.

  1. Reload and restart:
sudo systemctl daemon-reload
sudo systemctl restart k3s

Set the KUBECONFIG Variable

K3s uses a non-standard path. Add this to ~/.bashrc:

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

Then refresh:

source ~/.bashrc

Permission Summary

BeforeAfter
Ownerroot (rw)root (rw)
Grouproot (none)k3s-admin (r—)
Othersnonenone

Now kubectl get nodes works without sudo.

2. Ansible Vault Integration

The AAP install script references --ask-vault-pass for encrypted credentials. To encrypt sensitive inventory values:

ansible-vault encrypt inventory-growth

To edit the encrypted file:

ansible-vault edit inventory-growth

To run with a vault password:

ansible-playbook -i inventory-growth playbook.yml --ask-vault-pass

In production, use a vault password file or AAP’s built-in credential management instead of interactive prompts.

3. Lessons Learned

Beating Hardware Limits

  • Problem: AAP requires 16 GB of RAM; the lab host has 4–8 GB.
  • Lesson: Enterprise tools often have soft requirements you can override.
  • Fix: Disabled unused services (Hub, EDA), capped memory at 50%, added ignore_preflight_errors=true, and spoofed RAM with -e "{'ansible_memtotal_mb': 16000}".

Automating Server Communication

  • Problem: Worker nodes couldn’t access the master’s join token.
  • Lesson: Ansible delegate_to lets one host read another host’s files — this is “fact sharing” across nodes.
  • Fix: Used slurp to read the token from the master, then passed it to agents via b64decode | trim.

Idempotent Automation

  • Problem: Re-running the installer caused errors.
  • Lesson: Good automation is idempotent — it only changes what needs changing.
  • Fix: Added args: creates: /usr/local/bin/k3s so Ansible skips the install if K3s is already present.

Managing Secure Access

  • Problem: Scripts failed due to missing sudo privileges.
  • Lesson: Credentials should be stored in a vault, not hardcoded in playbooks.
  • Fix: Used AAP’s credential management and --ask-vault-pass for Ansible Vault.

Manual vs. Automation Speed

  • Problem: Manual 4-node setup takes ~20 minutes with copy-paste across 4 terminals.
  • Lesson: Automation eliminates human error and cuts deployment to 2 minutes.
  • Fix: Replaced manual SSH steps with a single AAP Job Template that handles master install, token retrieval, and agent joins in one run.

AAP Inventory Naming

  • Problem: AAP installer failed with localhost and raw IPs on single-host installs.
  • Lesson: The AAP 2.6 installer requires a proper FQDN for certificate generation and container linking.
  • Fix: Configured ansible-host.local in /etc/hosts and used that FQDN consistently across the entire inventory.

What’s Next

Your lab is fully operational:

  • K3s cluster — 4 nodes, deployed via AAP with two clicks
  • Non-root access — kubectl works without sudo
  • AAP — orchestrates all automation from a web UI
  • Credentials — secured with Ansible Vault

From here you can extend the lab: deploy workloads, add monitoring with Prometheus/Grafana, set up CI/CD pipelines, or integrate with ArgoCD for GitOps.

Backlinks