Node Exporter Sidecar Troubleshooting

No filesystem metrics

Problem Definition

Metrics under node_filesystem_... (disk usage, available space) return no data even though the sidecar is running and /metrics is accessible.

Missing sub-mounts

The host root filesystem mount (/) must use shared propagation for the :rslave binding to expose sub-mounts (e.g. /data, /var, /opt) inside the container. If / uses the default private propagation, those sub-mounts are invisible to the container and node-exporter reports no filesystem metrics.

Solution

Enable shared propagation on the host

On systemd-based Linux, run:

sudo mount --make-rshared /

This command is idempotent — running it again is harmless.

Verify sub-mount visibility

From a shell inside the sidecar container, list the filesystem tree:

podman exec node-exporter ls /hostfs

You should see your data and OS partitions listed (e.g. data, var, opt). If they are missing, the host root mount is not shared.

Persistence

The rshared setting does not survive a reboot. On systemd systems you can make it persistent:

# Create a systemd drop-in
sudo systemctl edit local-fs.target

Add the following:

[Unit]
Before=local-fs.target

[Service]
ExecStart=
ExecStart=mount --make-rshared /

Or add it to /etc/fstab (non-systemd systems):

none / overlayfs defaults,rshared 0 0

Endpoint unreachable

Problem Definition

curl http://<host-ip>:9100/metrics times out or returns Connection refused when the sidecar container is running but the /metrics endpoint is inaccessible.

Causes

The container uses network_mode: host and another process already owns port 9100, or it is on a bridge network without an exposed port, or the sidecar is in a crash-loop.

Solution

Step 1 — Check the container logs

podman logs node-exporter

If logs are empty or show only startup info, the container is healthy. If you see errors, fix the underlying issue first.

Step 2 — Confirm port binding

When using network_mode: host, the sidecar binds directly on the host's network interface. Check that nothing else owns port 9100:

ss -tlnp | grep 9100
# or
lsof -i :9100

If another process is using the port, either stop that process or change the sidecar to bridge mode with a different port mapping:

# Remove: network_mode: host
# Add:
ports:
  - "9101:9100"
networks:
  - monitoring

Step 3 — Bridge mode test

If you are running in bridge mode, verify the port is mapped:

podman port node-exporter
# Expected: 9101/tcp -> 0.0.0.0:9101  (if using the example above)

Then access via:

http://<host-ip>:9101/metrics

Container refuses to start

Problem Definition

The sidecar container exits immediately or fails to start due to a security policy violation.

Cause

Some container runtimes restrict the privileged flag. Kubernetes clusters with PodSecurity Admission or OPA/Gatekeeper may reject privileged pods.

Solution

Docker / Podman

On the host, verify the runtime permits privileged mode:

podman info --format "{{.Security.Rootless}}"
# 0 = not running as an unprivileged user (privileged allowed)
# 1 = rootless (some capabilities are restricted)

When running rootless with Podman, privileged: true maps to cap_add: ALL with additional restrictions. If you see permission denied errors on mount or cgroup access:

1. Use network_mode: host with --cap-add:

   podman run --rm -d \
     --network host \
     --cap-add SYS_ADMIN \
     -v /:/hostfs:ro \
     --name node-exporter \
     prom/node-exporter:v1.11.1 \
     --path.rootfs=/hostfs

2. Or use --privileged (less selective but allows all capabilities).

Kubernetes

Check your cluster's namespace for PodSecurity policies:

kubectl get namespace <namespace> --show-labels

If the namespace has pod-security.kubernetes.io/enforce: restricted or baseline, privileged pods are blocked. Create an exception:

kubectl annotate namespace <namespace> \
  pod-security.kubernetes.io/privileged:audit \
  pod-security.kubernetes.io/privileged:latest

Or create a dedicated namespace without enforcement:

kubectl create namespace monitoring-privileged