How do you make Prometheus highly available? [Intermediate]
Answer
I make Prometheus highly available by running two or more identical replicas scraping the same targets, giving each replica a unique external label, and deduplicating results in the query layer or long-term backend. I also keep Alertmanager highly available.
Technical explanation
Prometheus replicas do not coordinate scraping; they independently scrape the same targets.
Deduplication is handled by tools such as Thanos Query or Mimir, based on replica labels.
Alertmanager clustering prevents duplicate notifications when multiple Prometheus replicas fire the same alert.
Hands-on example
Hands-on: deploy prometheus-k8s-0 and prometheus-k8s-1 with external_labels cluster='prod-a' and replica='0'/'1'. Configure both to send alerts to a clustered Alertmanager. In Thanos Query, set replica-label=replica so dashboards show one deduplicated series.
Check how well your resume matches the role with our free resume checker— match score, ATS check, and the skills you're missing.
More Observability interview questions
- What is observability, and how is it different from traditional monitoring? [Basic]
- What are the three pillars of observability (metrics, logs, traces)? [Basic]
- What is the difference between monitoring and observability in practice? [Basic]
- What are the four golden signals of monitoring? [Basic]
- What is the difference between the USE method and the RED method? [Basic]
- When would you use the USE method versus the RED method? [Basic]
- What is an SLI, an SLO, and an SLA, and how do they relate? [Basic]
- How do you choose good SLIs for a service? [Basic]