Monitoring and Debugging Prometheus

Training Overview
Introduction
Prerequisites
Monitoring and Debugging Overview
Motivation
Overview
Web Status Pages
Overview
Target Scrape Status
Service Discovery Status
TSDB Status Information
Knowledge Check
Logs
Overview
Standard Error Logs
Query Log
Active Queries Log
Knowledge Check
Metrics-Based Meta-Monitoring
Overview
Prometheus' Own Metrics
Meta-Monitoring Architecture
Alerting Rule Examples
End-to-End Watchdog Alerts
Knowledge Check
Profiling
Overview
Profiling Heap Memory Usage
Profiling CPU Usage
Inspecting Goroutine States
Knowledge Check
Review and Outlook
Summary
Further Steps

Knowledge Check

What do you need to profile a Prometheus server or Alertmanager?

You need to set the --web.enable-profiling flag to enable the profiling HTTP endpoints.

You don't need to enable the profiling web endpoints, as they are enabled by default and available under /debug/pprof.

You need the Go tool to help you fetch and better interpret and visualize some of the profiling information, like CPU and heap statistics.

Fetching profiling information is usually quite expensive and disruptive for the profiled server, so you should always confirm first that it's currently ok to do so.

Which kinds of profiling information is available under the /debug/pprof endpoints?

The call stacks and execution states for all currently running goroutines.

Heap allocation statistics.

Filesystem usage statistics.

All of the above.