Knowledge Check

What should you watch out for if you want to build reliable meta-monitoring for your Prometheus servers?

While you can let a Prometheus server scrape itself, this will lead to brittle monitoring if that Prometheus server breaks. It's better to run a dedicated meta-monitoring Prometheus server (pair) that only monitors you monitoring.

It's sufficient to let two highly-available production Prometheus servers monitor each other, as it's unlikely that both are down at the same time.

Depending on your needs, you may or may not run a completely separate Alertmanager setup for meta-monitoring as well.

Even with meta-monitoring in place, it's a good idea to have a way to test the entire alerting and notification pipeline end-to-end.

How can you get notified if your entire alerting setup (including your meta-monitoring) breaks somehow?

There is no general way to catch this situation, since your entire organization's network might be unreachable.

You can configure a continuously firing watchdog alerting rule that an external service expects to receive notifications for every couple of minutes, and let that external service notify you in case it doesn't receive those notifications.

You can configure a Prometheus server to directly send a special heartbeat signal to an external service that notifies you if that heartbeat signal stops.

None of the above.

Training Overview
Introduction
Prerequisites

Monitoring and Debugging Overview
Motivation
Overview

Review and Outlook
Summary
Further Steps