TrainingsPCA CertificationTeam PlansSign In

Standard Error Logs

Prometheus logs a number of lifecycle events (like startup and shutdown information), as well as certain errors and other unusual activity, to the standard error (stderr) file descriptor. For example, when you start Prometheus in a terminal, you will see something like the following being logged:

ts=2024-07-09T10:54:53.476Z caller=main.go:601 level=info msg="No time or size retention was set so using the default time retention" duration=15d
ts=2024-07-09T10:54:53.476Z caller=main.go:645 level=info msg="Starting Prometheus Server" mode=server version="(version=2.53.0, branch=main, revision=5646c315bd076d0df7ad2312c2fea5caafb5b34f)"
ts=2024-07-09T10:54:53.476Z caller=main.go:650 level=info build_context="(go=go1.22.4, platform=linux/amd64, user=julius@workstation, date=20240709-10:53:09, tags=netgo,builtinassets,stringlabels)"
ts=2024-07-09T10:54:53.476Z caller=main.go:651 level=info host_details="(Linux 6.9.6-arch1-1 #1 SMP PREEMPT_DYNAMIC Fri, 21 Jun 2024 19:49:19 +0000 x86_64 workstation (none))"
ts=2024-07-09T10:54:53.476Z caller=main.go:652 level=info fd_limits="(soft=524288, hard=524288)"
ts=2024-07-09T10:54:53.476Z caller=main.go:653 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2024-07-09T10:54:53.478Z caller=web.go:571 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2024-07-09T10:54:53.478Z caller=main.go:1160 level=info msg="Starting TSDB ..."
ts=2024-07-09T10:54:53.478Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1720363034692 maxt=1720368000000 ulid=01J274B2H15XP0Z82A43C00P3N
ts=2024-07-09T10:54:53.478Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1720368005409 maxt=1720375200000 ulid=01J2792M24YVKDSQPVQ7H760B9
ts=2024-07-09T10:54:53.478Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1719997228600 maxt=1720036800000 ulid=01J2792M3VA596NZ9ZTKTZ94WA
ts=2024-07-09T10:54:53.478Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9090
ts=2024-07-09T10:54:53.478Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9090
ts=2024-07-09T10:54:53.480Z caller=head.go:626 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2024-07-09T10:54:53.480Z caller=head.go:713 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=755.691µs
ts=2024-07-09T10:54:53.480Z caller=head.go:721 level=info component=tsdb msg="Replaying WAL, this may take a while"
ts=2024-07-09T10:54:53.482Z caller=head.go:758 level=info component=tsdb msg="WAL checkpoint loaded"
ts=2024-07-09T10:54:53.499Z caller=head.go:793 level=info component=tsdb msg="WAL segment loaded" segment=44 maxSegment=48
ts=2024-07-09T10:54:53.517Z caller=head.go:793 level=info component=tsdb msg="WAL segment loaded" segment=45 maxSegment=48
ts=2024-07-09T10:54:53.532Z caller=head.go:793 level=info component=tsdb msg="WAL segment loaded" segment=46 maxSegment=48
ts=2024-07-09T10:54:53.533Z caller=head.go:793 level=info component=tsdb msg="WAL segment loaded" segment=47 maxSegment=48
ts=2024-07-09T10:54:53.533Z caller=head.go:793 level=info component=tsdb msg="WAL segment loaded" segment=48 maxSegment=48
ts=2024-07-09T10:54:53.533Z caller=head.go:830 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=1.695238ms wal_replay_duration=50.845175ms wbl_replay_duration=154ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=755.691µs total_replay_duration=53.312691ms
ts=2024-07-09T10:54:53.534Z caller=main.go:1181 level=info fs_type=EXT4_SUPER_MAGIC
ts=2024-07-09T10:54:53.534Z caller=main.go:1184 level=info msg="TSDB started"
ts=2024-07-09T10:54:53.534Z caller=main.go:1367 level=info msg="Loading configuration file" filename=demo.yml
ts=2024-07-09T10:54:53.535Z caller=main.go:1404 level=info msg="updated GOGC" old=100 new=75
ts=2024-07-09T10:54:53.535Z caller=main.go:1415 level=info msg="Completed loading of configuration file" filename=demo.yml totalDuration=1.518972ms db_storage=732ns remote_storage=817ns web_handler=290ns query_engine=555ns scrape=86.155µs scrape_sd=17.352µs notify=702ns notify_sd=507ns rules=1.274205ms tracing=3.2µs
ts=2024-07-09T10:54:53.535Z caller=main.go:1145 level=info msg="Server is ready to receive web requests."
ts=2024-07-09T10:54:53.535Z caller=manager.go:164 level=info component="rule manager" msg="Starting rule manager..."

Interpreting log lines

Prometheus logs information in a structured key/value output format.

Some keys are always present in a log line:

  • ts: The timestamp at which the line was logged.
  • caller: The source code file that contained the logging statement.
  • level: The severity level of the log message (debug, info, warn, or error).

Other keys depend on the specific component and logging statement. Often you will find:

  • component: The component inside Prometheus that is emitting the log line.
  • msg: A human-readable message of what happened.
  • err: An error message associated with something that went wrong.

You will also find other custom fields on some messages that provide extra statistics about the logged event.

Especially lines with a level=error field are useful for finding things that have gone wrong, like:

  • Errors while loading a new configuration file,
  • Errors writing to the TSDB,
  • Errors writing samples to a remote storage endpoint,
  • ...etc.

Note that Prometheus does not log individual target scrape failures, as scrape failures don't usually point to a problem with Prometheus itself (but with the network or the target), and those kinds of logs would be overly verbose.

Configuring logging

By default, Prometheus only outputs log lines that have a severity level of info and above. This means that any debug-level log statements will not be visible. To change this behavior, you can use the --log.level flag to set the minimum shown log level to something else than info (allowed values are debug, info, warn, and error).

For example, to also show debug log lines, you could start Prometheus like this:

./prometheus --log.level=debug

NOTE: This will produce excessively noisy debug output, so you will likely not want to run Prometheus with a debug log level in normal operation. However, it can sometimes be useful to temporarily turn this on to diagnose specific problems.

Prometheus also allows you to configure the log output format by supplying a --log.format flag. The default value of logfmt produces the structured log output we saw above, while a value of json produces JSON-formatted logs:

{"caller":"main.go:601","duration":"15d","level":"info","msg":"No time or size retention was set so using the default time retention","ts":"2024-07-09T10:56:44.719Z"}
{"caller":"main.go:645","level":"info","mode":"server","msg":"Starting Prometheus Server","ts":"2024-07-09T10:56:44.719Z","version":"(version=2.53.0, branch=main, revision=5646c315bd076d0df7ad2312c2fea5caafb5b34f)"}
{"build_context":"(go=go1.22.4, platform=linux/amd64, user=julius@workstation, date=20240709-10:53:09, tags=netgo,builtinassets,stringlabels)","caller":"main.go:650","level":"info","ts":"2024-07-09T10:56:44.719Z"}
{"caller":"main.go:651","host_details":"(Linux 6.9.6-arch1-1 #1 SMP PREEMPT_DYNAMIC Fri, 21 Jun 2024 19:49:19 +0000 x86_64 workstation (none))","level":"info","ts":"2024-07-09T10:56:44.719Z"}
{"caller":"main.go:652","fd_limits":"(soft=524288, hard=524288)","level":"info","ts":"2024-07-09T10:56:44.719Z"}
{"caller":"main.go:653","level":"info","ts":"2024-07-09T10:56:44.719Z","vm_limits":"(soft=unlimited, hard=unlimited)"}
{"address":"0.0.0.0:9090","caller":"web.go:571","component":"web","level":"info","msg":"Start listening for connections","ts":"2024-07-09T10:56:44.721Z"}
{"caller":"main.go:1160","level":"info","msg":"Starting TSDB ...","ts":"2024-07-09T10:56:44.721Z"}
{"caller":"repair.go:56","component":"tsdb","level":"info","maxt":1720368000000,"mint":1720363034692,"msg":"Found healthy block","ts":"2024-07-09T10:56:44.721Z","ulid":"01J274B2H15XP0Z82A43C00P3N"}
{"caller":"repair.go:56","component":"tsdb","level":"info","maxt":1720375200000,"mint":1720368005409,"msg":"Found healthy block","ts":"2024-07-09T10:56:44.721Z","ulid":"01J2792M24YVKDSQPVQ7H760B9"}
{"caller":"repair.go:56","component":"tsdb","level":"info","maxt":1720036800000,"mint":1719997228600,"msg":"Found healthy block","ts":"2024-07-09T10:56:44.721Z","ulid":"01J2792M3VA596NZ9ZTKTZ94WA"}
{"address":"[::]:9090","caller":"tls_config.go:313","component":"web","level":"info","msg":"Listening on","ts":"2024-07-09T10:56:44.721Z"}
{"address":"[::]:9090","caller":"tls_config.go:316","component":"web","http2":false,"level":"info","msg":"TLS is disabled.","ts":"2024-07-09T10:56:44.721Z"}
{"caller":"head.go:626","component":"tsdb","level":"info","msg":"Replaying on-disk memory mappable chunks if any","ts":"2024-07-09T10:56:44.724Z"}
{"caller":"head.go:713","component":"tsdb","duration":"1.043987ms","level":"info","msg":"On-disk memory mappable chunks replay completed","ts":"2024-07-09T10:56:44.725Z"}
{"caller":"head.go:721","component":"tsdb","level":"info","msg":"Replaying WAL, this may take a while","ts":"2024-07-09T10:56:44.725Z"}
{"caller":"head.go:758","component":"tsdb","level":"info","msg":"WAL checkpoint loaded","ts":"2024-07-09T10:56:44.726Z"}
{"caller":"head.go:793","component":"tsdb","level":"info","maxSegment":49,"msg":"WAL segment loaded","segment":44,"ts":"2024-07-09T10:56:44.773Z"}
{"caller":"head.go:793","component":"tsdb","level":"info","maxSegment":49,"msg":"WAL segment loaded","segment":45,"ts":"2024-07-09T10:56:44.799Z"}
{"caller":"head.go:793","component":"tsdb","level":"info","maxSegment":49,"msg":"WAL segment loaded","segment":46,"ts":"2024-07-09T10:56:44.815Z"}
{"caller":"head.go:793","component":"tsdb","level":"info","maxSegment":49,"msg":"WAL segment loaded","segment":47,"ts":"2024-07-09T10:56:44.816Z"}
{"caller":"head.go:793","component":"tsdb","level":"info","maxSegment":49,"msg":"WAL segment loaded","segment":48,"ts":"2024-07-09T10:56:44.816Z"}
{"caller":"head.go:793","component":"tsdb","level":"info","maxSegment":49,"msg":"WAL segment loaded","segment":49,"ts":"2024-07-09T10:56:44.816Z"}
{"caller":"head.go:830","checkpoint_replay_duration":"803.349µs","chunk_snapshot_load_duration":"0s","component":"tsdb","level":"info","mmap_chunk_replay_duration":"1.043987ms","msg":"WAL replay completed","total_replay_duration":"91.574255ms","ts":"2024-07-09T10:56:44.816Z","wal_replay_duration":"89.701335ms","wbl_replay_duration":"133ns"}
{"caller":"main.go:1181","fs_type":"EXT4_SUPER_MAGIC","level":"info","ts":"2024-07-09T10:56:44.817Z"}
{"caller":"main.go:1184","level":"info","msg":"TSDB started","ts":"2024-07-09T10:56:44.817Z"}
{"caller":"main.go:1367","filename":"demo.yml","level":"info","msg":"Loading configuration file","ts":"2024-07-09T10:56:44.817Z"}
{"caller":"main.go:1404","level":"info","msg":"updated GOGC","new":75,"old":100,"ts":"2024-07-09T10:56:44.818Z"}
{"caller":"main.go:1415","db_storage":"739ns","filename":"demo.yml","level":"info","msg":"Completed loading of configuration file","notify":"532ns","notify_sd":"707ns","query_engine":"436ns","remote_storage":"1.188µs","rules":"927.383µs","scrape":"81.185µs","scrape_sd":"17.292µs","totalDuration":"1.357385ms","tracing":"2.352µs","ts":"2024-07-09T10:56:44.818Z","web_handler":"293ns"}
{"caller":"main.go:1145","level":"info","msg":"Server is ready to receive web requests.","ts":"2024-07-09T10:56:44.818Z"}
{"caller":"manager.go:164","component":"rule manager","level":"info","msg":"Starting rule manager...","ts":"2024-07-09T10:56:44.818Z"}

This can be helpful in case you are ingesting your Prometheus logs into a log-processing system that prefers JSON-based input.