Monitoring and Debugging Prometheus

Training Overview
Introduction
Prerequisites
Monitoring and Debugging Overview
Motivation
Overview
Web Status Pages
Overview
Target Scrape Status
Service Discovery Status
TSDB Status Information
Knowledge Check
Logs
Overview
Standard Error Logs
Query Log
Active Queries Log
Knowledge Check
Metrics-Based Meta-Monitoring
Overview
Prometheus' Own Metrics
Meta-Monitoring Architecture
Alerting Rule Examples
End-to-End Watchdog Alerts
Knowledge Check
Profiling
Overview
Profiling Heap Memory Usage
Profiling CPU Usage
Inspecting Goroutine States
Knowledge Check
Review and Outlook
Summary
Further Steps

Knowledge Check

What should you watch out for if you want to build reliable meta-monitoring for your Prometheus servers?

While you can let a Prometheus server scrape itself, this will lead to brittle monitoring if that Prometheus server breaks. It's better to run a dedicated meta-monitoring Prometheus server (pair) that only monitors you monitoring.

It's sufficient to let two highly-available production Prometheus servers monitor each other, as it's unlikely that both are down at the same time.

Depending on your needs, you may or may not run a completely separate Alertmanager setup for meta-monitoring as well.

Even with meta-monitoring in place, it's a good idea to have a way to test the entire alerting and notification pipeline end-to-end.

How can you get notified if your entire alerting setup (including your meta-monitoring) breaks somehow?

There is no general way to catch this situation, since your entire organization's network might be unreachable.

You can configure a continuously firing watchdog alerting rule that an external service expects to receive notifications for every couple of minutes, and let that external service notify you in case it doesn't receive those notifications.

You can configure a Prometheus server to directly send a special heartbeat signal to an external service that notifies you if that heartbeat signal stops.

None of the above.