Standard Error Logs

Prometheus logs a number of lifecycle events (like startup and shutdown information), as well as certain errors and other unusual activity, to the standard error (stderr) file descriptor. For example, when you start Prometheus in a terminal, you will see something like the following being logged:

ts=2023-04-09T17:51:04.836Z caller=main.go:520 level=info msg="No time or size retention was set so using the default time retention" duration=15d
ts=2023-04-09T17:51:04.836Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)"
ts=2023-04-09T17:51:04.836Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)"
ts=2023-04-09T17:51:04.836Z caller=main.go:570 level=info host_details="(Linux 6.2.9-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 30 Mar 2023 14:51:14 +0000 x86_64 workstation (none))"
ts=2023-04-09T17:51:04.836Z caller=main.go:571 level=info fd_limits="(soft=524288, hard=524288)"
ts=2023-04-09T17:51:04.836Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2023-04-09T17:51:04.839Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2023-04-09T17:51:04.840Z caller=main.go:1005 level=info msg="Starting TSDB ..."
ts=2023-04-09T17:51:04.842Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9090
ts=2023-04-09T17:51:04.842Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9090
ts=2023-04-09T17:51:04.848Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2023-04-09T17:51:04.848Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=4.33µs
ts=2023-04-09T17:51:04.848Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while"
ts=2023-04-09T17:51:04.849Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
ts=2023-04-09T17:51:04.849Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=35.14µs wal_replay_duration=984.342µs wbl_replay_duration=412ns total_replay_duration=1.081904ms
ts=2023-04-09T17:51:04.851Z caller=main.go:1026 level=info fs_type=TMPFS_MAGIC
ts=2023-04-09T17:51:04.852Z caller=main.go:1029 level=info msg="TSDB started"
ts=2023-04-09T17:51:04.852Z caller=main.go:1209 level=info msg="Loading configuration file" filename=prometheus.yml
ts=2023-04-09T17:51:04.852Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=prometheus.yml totalDuration=828.63µs db_storage=2.275µs remote_storage=2.77µs web_handler=876ns query_engine=1.539µs scrape=294.164µs scrape_sd=39.016µs notify=47.754µs notify_sd=16.983µs rules=2.835µs tracing=9.077µs
ts=2023-04-09T17:51:04.852Z caller=main.go:990 level=info msg="Server is ready to receive web requests."
ts=2023-04-09T17:51:04.852Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..."

Interpreting log lines

Prometheus logs information in a structured key/value output format.

Some keys are always present in a log line:

  • ts: The timestamp at which the line was logged.
  • caller: The source code file that contained the logging statement.
  • level: The severity level of the log message (debug, info, warn, or error).

Other keys depend on the specific component and logging statement. Often you will find:

  • component: The component inside Prometheus that is emitting the log line.
  • msg: A human-readable message of what happened.
  • err: An error message associated with something that went wrong.

You will also find other custom fields on some messages that provide extra statistics about the logged event.

Especially lines with a level=error field are useful for finding things that have gone wrong, like:

  • Errors while loading a new configuration file,
  • Errors writing to the TSDB,
  • Errors writing samples to a remote storage endpoint,
  • ...etc.

Note that Prometheus does not log individual target scrape failures, as scrape failures don't usually point to a problem with Prometheus itself (but with the network or the target), and those kinds of logs would be overly verbose.

Configuring logging

By default, Prometheus only outputs log lines that have a severity level of info and above. This means that any debug-level log statements will not be visible. To change this behavior, you can use the --log.level flag to set the minimum shown log level to something else than info (allowed values are debug, info, warn, and error).

For example, to also show debug log lines, you could start Prometheus like this:

./prometheus --log.level=debug

NOTE: This will produce excessively noisy debug output, so you will likely not want to run Prometheus with a debug log level in normal operation. However, it can sometimes be useful to temporarily turn this on to diagnose specific problems.

Prometheus also allows you to configure the log output format by supplying a --log.format flag. The default value of logfmt produces the structured log output we saw above, while a value of json produces JSON-formatted logs:

{"caller":"main.go:520","duration":"15d","level":"info","msg":"No time or size retention was set so using the default time retention","ts":"2023-04-09T17:52:15.981Z"}
{"caller":"main.go:564","level":"info","mode":"server","msg":"Starting Prometheus Server","ts":"2023-04-09T17:52:15.981Z","version":"(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)"}
{"build_context":"(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)","caller":"main.go:569","level":"info","ts":"2023-04-09T17:52:15.982Z"}
{"caller":"main.go:570","host_details":"(Linux 6.2.9-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 30 Mar 2023 14:51:14 +0000 x86_64 workstation (none))","level":"info","ts":"2023-04-09T17:52:15.982Z"}
{"caller":"main.go:571","fd_limits":"(soft=524288, hard=524288)","level":"info","ts":"2023-04-09T17:52:15.982Z"}
{"caller":"main.go:572","level":"info","ts":"2023-04-09T17:52:15.982Z","vm_limits":"(soft=unlimited, hard=unlimited)"}
{"address":"0.0.0.0:9090","caller":"web.go:561","component":"web","level":"info","msg":"Start listening for connections","ts":"2023-04-09T17:52:15.985Z"}
{"caller":"main.go:1005","level":"info","msg":"Starting TSDB ...","ts":"2023-04-09T17:52:15.986Z"}
{"address":"[::]:9090","caller":"tls_config.go:232","component":"web","level":"info","msg":"Listening on","ts":"2023-04-09T17:52:15.988Z"}
{"address":"[::]:9090","caller":"tls_config.go:235","component":"web","http2":false,"level":"info","msg":"TLS is disabled.","ts":"2023-04-09T17:52:15.988Z"}
{"caller":"head.go:587","component":"tsdb","level":"info","msg":"Replaying on-disk memory mappable chunks if any","ts":"2023-04-09T17:52:15.992Z"}
{"caller":"head.go:658","component":"tsdb","duration":"10.145µs","level":"info","msg":"On-disk memory mappable chunks replay completed","ts":"2023-04-09T17:52:15.993Z"}
{"caller":"head.go:664","component":"tsdb","level":"info","msg":"Replaying WAL, this may take a while","ts":"2023-04-09T17:52:15.993Z"}
{"caller":"head.go:735","component":"tsdb","level":"info","maxSegment":0,"msg":"WAL segment loaded","segment":0,"ts":"2023-04-09T17:52:15.995Z"}
{"caller":"head.go:772","checkpoint_replay_duration":"116.589µs","component":"tsdb","level":"info","msg":"WAL replay completed","total_replay_duration":"1.927192ms","ts":"2023-04-09T17:52:15.995Z","wal_replay_duration":"1.063949ms","wbl_replay_duration":"383ns"}
{"caller":"main.go:1026","fs_type":"TMPFS_MAGIC","level":"info","ts":"2023-04-09T17:52:15.997Z"}
{"caller":"main.go:1029","level":"info","msg":"TSDB started","ts":"2023-04-09T17:52:15.997Z"}
{"caller":"main.go:1209","filename":"prometheus.yml","level":"info","msg":"Loading configuration file","ts":"2023-04-09T17:52:15.998Z"}
{"caller":"main.go:1246","db_storage":"1.958µs","filename":"prometheus.yml","level":"info","msg":"Completed loading of configuration file","notify":"49.627µs","notify_sd":"18.184µs","query_engine":"1.622µs","remote_storage":"2.876µs","rules":"2.485µs","scrape":"284.359µs","scrape_sd":"42.35µs","totalDuration":"856.846µs","tracing":"8.395µs","ts":"2023-04-09T17:52:15.998Z","web_handler":"919ns"}
{"caller":"main.go:990","level":"info","msg":"Server is ready to receive web requests.","ts":"2023-04-09T17:52:15.998Z"}
{"caller":"manager.go:974","component":"rule manager","level":"info","msg":"Starting rule manager...","ts":"2023-04-09T17:52:15.998Z"}

This can be helpful in case you are ingesting your Prometheus logs into a log-processing system that prefers JSON-based input.