A Prometheus alert woke up my dashboard: CPU above 80 percent for more than five minutes on my primary Pi-hole. DNS still resolved, so nothing was visibly broken, but a DNS server running that hot is a DNS server about to fall over. The fix turned out to be one line. Finding the right size took two tries, and the interesting part was reading what the numbers were actually telling me.

The number that explained everything

I SSH’d in and ran uptime. The load average was 19.72.

That one number is the whole story, if you read it right. Load average measures something different from CPU percentage: it is the length of the run queue, roughly how many processes are running or waiting to run at a given moment. On a container with one CPU core, a load average of 19.72 means about twenty things all want that single core at once. The core is gridlocked, and everything is taking turns crawling through it.

vmstat 1 filled in the rest:

procs: 19-27 runnable
cpu:   81% user, 0% idle
context switches: ~17,894/sec

Zero percent idle and nearly eighteen thousand context switches a second is the signature of thrashing. The CPU was spending much of its time switching between tasks rather than doing any of them.

39 threads, one core

ps showed the culprit, and it was something mundane rather than a runaway process or a leak: pihole-FTL, Pi-hole’s DNS engine, running 39 threads. Pi-hole-FTL is multi-threaded by design, and at the time it was serving 1.1 million DNS queries a day. Thirty-nine threads fanning out across that query volume is fine when there are cores to run them on. On a single core they just pile into the run queue and fight, which is exactly what the load average was showing.

Memory was tight too: of the 512MB the container had, pihole-FTL was using 389MB, about 74 percent. Not the immediate problem, but not comfortable either.

The root cause had nothing to do with Pi-hole. It was me. I had given this container a single core back when it handled far less traffic, and a multi-threaded resolver at 1.1 million queries a day had quietly outgrown the box I put it in.

The fix that wasn’t enough, and the one that was

The fix lives on the Proxmox host, and it does not even need a reboot for an LXC container:

pct set <vmid> --cores 2

Two cores took the edge off. The load average came down and the alert stopped firing, but the container was still running hot under bursts, more loaded than I wanted a core DNS server to be. Doubling had helped without actually fixing it.

So I kept going to four:

pct set <vmid> --cores 4

That was the floor. With 39 to 40 FTL threads at this query volume, four cores is where it settled: CPU usage dropped from pinned to 25-30 percent with most of the capacity idle, and the load average fell from the high teens to around 2.4-3.5 and stayed there. I also bumped the container’s RAM from 512MB to 1GB while I was at it, since FTL was already sitting at three-quarters of the old limit.

Adding cores treats the symptom. The other lever is reducing the query volume itself, which for me meant fixing some chatty internal DNS traffic, but that is a separate story.

Lessons

• Read load average against core count. It is a run-queue length, so 19.72 is catastrophic on one core and unremarkable on twenty. The number only means something next to the number of cores.
• A multi-threaded process on a single core is a thrash waiting for load. It looks fine until the query volume shows up, and then context switching eats the CPU before any real work gets done.
• Right-size as the workload grows. I under-provisioned this container during an earlier migration and the traffic growth caught up with it. In a homelab the cheapest fix is usually just more cores.
• The first bump may not be enough. Doubling to two cores quieted the alert but left it loaded; the honest move was to keep going to four rather than declare victory the moment the alert cleared.