How does Prometheus storage (TSDB) work, and what is the retention model? [Intermediate]
Answer
Prometheus stores samples in its local time-series database, using in-memory head blocks for recent data and immutable blocks on disk for older data. Retention is controlled by time and/or size, after which old blocks are deleted.
Technical explanation
The TSDB is optimized for recent local queries and rule evaluation.
Retention flags such as storage.tsdb.retention.time and storage.tsdb.retention.size define how much data stays locally.
For long-term historical data, remote write or object-storage-based systems such as Thanos, Cortex, or Mimir are commonly used.
Hands-on example
Hands-on: set Prometheus args to --storage.tsdb.retention.time=15d and --storage.tsdb.retention.size=100GB. Monitor prometheus_tsdb_head_series, prometheus_tsdb_storage_blocks_bytes, and disk usage. If head series grows sharply, investigate cardinality before increasing disk.
Check how well your resume matches the role with our free resume checker— match score, ATS check, and the skills you're missing.
More Observability interview questions
- What is observability, and how is it different from traditional monitoring? [Basic]
- What are the three pillars of observability (metrics, logs, traces)? [Basic]
- What is the difference between monitoring and observability in practice? [Basic]
- What are the four golden signals of monitoring? [Basic]
- What is the difference between the USE method and the RED method? [Basic]
- When would you use the USE method versus the RED method? [Basic]
- What is an SLI, an SLO, and an SLA, and how do they relate? [Basic]
- How do you choose good SLIs for a service? [Basic]