How does Prometheus integrate with Kubernetes service discovery? [Intermediate]
Answer
Prometheus integrates with Kubernetes service discovery by watching Kubernetes API resources and turning pods, services, endpoints, EndpointSlices, nodes, and ingresses into scrape targets with metadata labels.
Technical explanation
Prometheus can use native kubernetes_sd_configs or higher-level Prometheus Operator objects.
Relabeling maps Kubernetes metadata into stable labels and filters targets based on annotations or selectors.
RBAC must allow Prometheus to list and watch the required resources.
Hands-on example
Example: with the Prometheus Operator, create a Service exposing port name 'metrics' and a ServiceMonitor selecting app='checkout'. Verify targets in Prometheus UI. If missing, check ServiceMonitor namespaceSelector, service labels, endpoint port name, and Prometheus serviceMonitorSelector.
Check how well your resume matches the role with our free resume checker— match score, ATS check, and the skills you're missing.
More Observability interview questions
- What is observability, and how is it different from traditional monitoring? [Basic]
- What are the three pillars of observability (metrics, logs, traces)? [Basic]
- What is the difference between monitoring and observability in practice? [Basic]
- What are the four golden signals of monitoring? [Basic]
- What is the difference between the USE method and the RED method? [Basic]
- When would you use the USE method versus the RED method? [Basic]
- What is an SLI, an SLO, and an SLA, and how do they relate? [Basic]
- How do you choose good SLIs for a service? [Basic]