I run a Pi-hole master on Proxmox with replicas elsewhere in the lab. Nebula-Sync pushes configuration from the master on a schedule. That pattern worked until I rotated credentials and bumped into Pi-hole v6’s permission model.

Pi-hole separates application passwords (session login and /api/auth in my Ansible playbooks) from API keys (separate tokens in the API settings UI). I put API keys into Nebula-Sync’s PRIMARY|… / REPLICAS|… env file. Nebula-Sync’s README uses the same URL|password shape and documents app_sudo on replicas in App passwords and authentication errors (Pi-hole’s wording there is “app passwords”; I use API keys after the pipe, not the master application password string). Conflating application password, API key, and stale env values makes debugging slower.

Sync jobs failed, logs pointed at the teleporter path, and the fix was not “paste the new secret and restart.”

The problem

After rotation, Nebula-Sync runs started failing. Some failures looked like plain authentication problems. Others surfaced as HTTP 403 responses in Nebula-Sync’s output when it hit Pi-hole’s teleporter-related API surface. I am describing what showed up in my logs and Nebula-Sync’s behavior, not paraphrasing Pi-hole’s auth documentation. I had already updated the API keys in the env file Nebula-Sync reads, so it was tempting to blame the tool or TLS. The replica itself was still answering DNS. The breakage was confined to authenticated API actions that need the right permission mode for API keys (separate from the application password flow on the master).

Investigation

Pi-hole v6’s web API is stricter than the old password-in-api.php world. For automation you pick a credential type and stick to it per integration: application password (session flow on the master for Ansible) versus API keys on replicas for Nebula-Sync. They are not interchangeable strings; the UI labels them separately for a reason.

Not every endpoint grants the same capability to the credential you use after the pipe. What I observed after rotation: with new API keys in Nebula-Sync’s env, calls still returned 403 on routes tied to teleporter-style work until webserver.api.app_sudo was true on the replica. That behavior showed up in my Nebula-Sync logs, not as a sentence I copied from Pi-hole’s docs. Nebula-Sync’s own README documents the app_sudo fix in App passwords and authentication errors (tool-side guidance for Pi-hole v6 and replicas, distinct from Pi-hole’s auth reference pages). The FTL setting webserver.api.app_sudo is the knob that matched my reproduction (toggle it off, 403-style failures return; toggle it on, sync succeeds).

Separately, Ansible on the master uses /api/auth with the application password to obtain a SID. Nebula-Sync on the replicas uses the API keys I store in its env. I had to keep those flows separate so I did not mix “session SID on the master” debugging with “403 on the replica during Nebula-Sync.”

Solution

1. Put the new API keys where Nebula-Sync actually reads them.

Nebula-Sync loads primary and replica API keys from its env file (in my setup under /etc/nebula-sync/). Rotating or regenerating keys in the Pi-hole UI but leaving the old values in that file guarantees 401s until you update the file and restart the sync service. If you also rotated the master application password, update Ansible or any other consumer that uses /api/auth separately; that password does not belong in Nebula-Sync’s env.

2. Enable app_sudo on replicas that receive sync.

On each replica:

sudo pihole-FTL --config webserver.api.app_sudo true
sudo systemctl restart pihole-FTL

You can confirm the current value with:

pihole-FTL --config webserver.api.app_sudo

3. Encode it in automation so replicas do not drift.

In the homelab repo, the Pi-hole role enables webserver.api.app_sudo for replica nodes during configuration, and the DNS-records playbook sets it on the master when automated host entries need the API. That way a rebuild or a fresh container does not silently drop the flag that Nebula-Sync depends on.

4. Re-run sync and read the unit journal if anything is still off.

sudo systemctl start nebula-sync.service
journalctl -u nebula-sync.service -n 80 --no-pager

A clean run should show successful replication without 403 lines on teleporter-related calls in the journal.

Verification

After updating Nebula-Sync’s API keys, enabling app_sudo on the replica, and restarting FTL, manual systemctl start nebula-sync.service completed with exit code 0. DNS lists and related settings converged across instances on the next timer-fired run. Breaking app_sudo back to false reproduced the permission-style failures I had seen earlier in Nebula-Sync logs.

What I learned

Rotating credentials is a checklist, not one field. New API keys in Pi-hole and updated values in Nebula-Sync’s env, plus a separate update if the master application password changed for Ansible.

403 in my Nebula-Sync runs was not always “wrong API key.” In my case the key was accepted for lighter calls but teleporter-related routes returned 403 until app_sudo was enabled on the replica. That is my observed pattern, consistent with Nebula-Sync’s README (App passwords and authentication errors), not a sentence lifted from Pi-hole’s auth docs.

Keep master and replica debugging separate. Application password + SID on the master for Ansible, API keys on replicas for Nebula-Sync; mixing them wastes time.

I am not claiming this is the only reason Nebula-Sync can fail on v6, but if you rotate API keys and suddenly see teleporter 403s on replicas, webserver.api.app_sudo belongs on the short list before you rip out TLS or downgrade the tool.