AI and homelab

In my previous homelab post, I wrote that the main reason I have these little computers is to learn, so it was time to put some work on them.

My background

For context, I am a software engineer with over 15 years of experience. I started with PHP and MySQL, moved through jQuery toward React and Next.js, and eventually worked with React Native. I used WAMP, then LAMP, and more recently AWS, Kubernetes, and related technologies. But with most of these technologies, I was just a user—some DevOps person configured them.

I was still curious about it, so now it was my turn.

Installing the core

I already mentioned that I was playing with Proxmox, but for these small PCs, it was a precious resource and a little overkill—at least according to ChatGPT and Gemini. So I installed regular Ubuntu on the devices. I called them Antman and Wasp. I like Marvel, and they are tiny, you know...

One reason I chose Ubuntu over Proxmox is travel. Turning off a device, moving it to a new location, and turning it back on is something Proxmox does not handle very well.

Installing the k3s cluster was simple; I followed the instructions on its website. I do not think it makes sense to reproduce them here, as they can change at any time. You should always use the latest available versions.

To access the mini PCs, I used SSH with private keys, of course. I also installed Tailscale so I could access them from anywhere in the world.

How to run the apps?

In many tutorials, I saw a yaml file containing applications; someone ran a kubectl command, and the application was deployed. But this was not how I imagined working.

Before k3s, I was thinking of using Docker and Docker Compose, installed with Ansible. But once k3s was in place, the discussion moved to Kubernetes resources, Helm charts, and GitOps.

This was a discussion with an AI agent. I asked questions; it explained the options and prepared configuration, and I ran commands on the server and brought back the results. For me, it is a great way to learn. On one hand, I am doing the work; on the other, I need to consider its proposals, ask the right questions, and steer the whole conversation toward my goal.

Why I chose Argo CD over Flux

I wanted configuration in GitLab, but I also wanted to see what was running. I considered two options: Argo CD and Flux.

I already had experience using Argo CD. Based on that, I knew I wanted a dashboard where I could inspect applications, their resources, their health, and whether they matched Git. Its trade-offs include maintaining dashboard access and choosing a separate secret-management approach. There is also a Helm distinction: Argo CD renders charts into Kubernetes resources rather than installing regular Helm releases. Argo CD Helm documentation.

As it was the only tool I knew, I had to ask about the alternative, and the AI suggested Flux.

Flux has built-in support for SOPS-encrypted secrets and manages actual Helm releases, including their upgrade and remediation behavior. The tradeoff for me was visibility: a comparable dashboard would be another integration, and troubleshooting would involve more CLI output and controller logs. Flux Helm documentation.

I chose Argo CD because I wanted that visual feedback. I did not run both tools or benchmark them; this was a choice based on how I wanted to work.

Installing Argo CD and giving it a starting point

I installed Argo CD into its own namespace using the official manifests. Initially, I accessed the dashboard through port-forwarding and an SSH tunnel, retrieved the generated admin password, and connected my GitLab repository with a read-only deploy token. Official getting-started guide.

There were a few basic things I had to understand first. An empty GitLab repository cannot provide deployment configuration. And a command referring to bootstrap/root.yaml does not magically fetch that file from Git: the file has to exist where I run the command, or I need to pass its contents directly.

I used the App of Apps pattern. A parent Application called homelab watches an applications/ directory in GitLab. The files in that directory define child Applications, each pointing to the configuration for a service. Argo CD's bootstrapping example.

The names of my two main folders initially confused me. They are a convention in my repository, not special Kubernetes directory names. The applications/ folder contains Argo CD Application definitions. Each file tells Argo CD what source to read, which revision to follow, and which namespace to deploy into. The apps/ folder contains the actual configuration for those services: Kubernetes manifests, Kustomize files, Helm values, and application settings.

The flow looks like this:

bootstrap/root.yaml
    → watches applications/
applications/homepage.yaml
    → points to apps/homepage/
apps/homepage/
    → creates the Deployment, Service, Ingress, and ConfigMap

In other words, applications/ tells Argo CD what to manage, while apps/ describes how each service should run.

I manually created the parent once. That was the bootstrap step: Argo CD needed to know which repository, branch, and directory to watch. After that, adding a child definition to Git was enough for the parent to discover it.

The agent also created a small Nginx Helm chart as a demo. I asked why it was needed. I did not need Nginx for Argo CD; it was just a harmless application to verify the whole deployment flow before adding something important.

When both homelab and demo showed Healthy and Synced, I could see the idea working. Configuration reached main, Argo CD read it, and Kubernetes ran the application.

Choosing ingress for local access

The next problem was access. Port-forwarding worked, but I did not want to keep a terminal and an SSH tunnel open every time I visited a service.

I asked the AI what my options were. I had already seen on YouTube that a load balancer could be the answer, but it suggested ingress. I knew the word from past jobs, but nothing more than that, so it was another opportunity to learn and a new decision to make.

• A LoadBalancer exposes a service at an address and port.
• Ingress adds HTTP routing, so several web applications can share an entry point and use different hostnames.

With default k3s ServiceLB, services use node addresses and available host ports; dedicated LAN addresses would require another setup, such as MetalLB. k3s networking documentation.

I chose ingress for the web apps. k3s already included Traefik, so I could use it rather than install another controller. I learned that an Ingress is the routing configuration, while Traefik is the software that implements it.

I added an Ingress to the demo chart and a separate HTTPS passthrough route for Argo CD. The latter forwards the encrypted connection to Argo CD, keeping its existing TLS setup. That also means the self-signed certificate warning remains; successful routing does not create a trusted certificate. Traefik TLS passthrough.

One DNS rewrite for the web apps

I already use AdGuard Home on my router, and all my network traffic goes through it. So I added a wildcard DNS rewrite:

*.bootfort.arpa → K3S_LOCAL_IP_ADDRESS

The wildcard means I do not need a new DNS entry for every web app. Each app still needs its own routing rule, but names such as demo.bootfort.arpa and argocd.bootfort.arpa point to the same address.

I was advised to use home.arpa, as it is a reserved home-network domain (RFC 8375), but I went with bootfort.arpa.

Tailscale needed DNS and a route

I had Tailscale installed and assumed that might be enough to use the same hostnames outside my home. It was not.

The first clue was a DNS lookup using 100.100.100.100, Tailscale's resolver, which returned NXDOMAIN. Asking AdGuard directly returned the correct server address. I configured Tailscale split DNS so queries for bootfort.arpa go to AdGuard. Tailscale DNS documentation.

Later, when testing outside the home network, both DNS requests to AdGuard and connections to the server timed out. Knowing a name's address is not the same as being able to reach it.

Tailscale was running on my Ubuntu server, antman, and IP forwarding was already enabled. I advertised the home subnet from that server:

sudo tailscale set --advertise-routes=192.168.X.0/24

Then I approved the route in the Tailscale admin console. That made the LAN addresses reachable through the subnet router. The device must stay online, and Tailscale access rules must permit the traffic. Subnet-router documentation.

After that, AdGuard answered remotely and Argo CD returned HTTP 200. I could use the same hostname away from home with Tailscale connected, without an SSH tunnel or public router port forwarding.

What I like and what I do not

I like how quickly everything moves. If I get stuck on something, I can resolve it pretty quickly. Learning happens precisely at the points where I want to go deeper.

The downside is that I am not forced to learn about things I do not really want to explore. For example, I did not look closely at any of the application's YAML files. I know roughly how they work, but I would not be able to write one from scratch.

I cannot tell whether that is good or bad, but I think it is better than getting stuck on something, being unable to deploy anything, and abandoning homelabbing until the frustration fades.

See you next Monday.