Introduction

In one of our earlier articles, we explained the basics of syslog/rsyslog log management system in Linux and provided a quick demonstration on how we could modify the rsyslog configuration to redirect logs to a particular destination. Systemd maintains a binary log of services and programs that it manages in a system journal. This binary log is written to and managed by the systemd-journald service. The journalctl command provides an interface to users to be able to query this log and read messages from it. In this article, we will demonstrate with examples how you could use the journalctl command line interface to extract desired information from the systemd journal.

Example 1: Using journalctl without any options
When used without any accompanying options, the journalctl command will print the entire systemd journal log. This would present a lengthy output to the terminal screen and would prove to be rather tedious to interpret.
Given below is a quick snippet from the log:

[root@linuxnix ~]# journalctl
-- Logs begin at Wed 2018-02-21 12:56:32 IST, end at Wed 2018-02-21 13:01:01 IST. --
Feb 21 12:56:32 linuxnix systemd-journal[92]: Runtime journal is using 8.0M (max allowed 91.1M, trying to leave 136.6M free of 903.1M a
Feb 21 12:56:32 linuxnix kernel: Initializing cgroup subsys cpuset
Feb 21 12:56:32 linuxnix kernel: Initializing cgroup subsys cpu
Feb 21 12:56:32 linuxnix kernel: Initializing cgroup subsys cpuacct
Feb 21 12:56:32 linuxnix kernel: Linux version 3.10.0-693.11.6.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623
Feb 21 12:56:32 linuxnix kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-693.11.6.el7.x86_64 root=/dev/mapper/cl-root ro crashkernel=a
Feb 21 12:56:32 linuxnix kernel: Disabled fast string operations
Feb 21 12:56:32 linuxnix kernel: e820: BIOS-provided physical RAM map:
Feb 21 12:56:32 linuxnix kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009ebff] usable
---------------------------------------------output truncated for brevity

One very useful feature of the journalctl command is that its output is paged (similar to using the pagers more/less) and can be searched through i.e. in a manner similar to vi or less, we could press the forward slash followed by the desired keyword and press enter to search for it in the journal log. This makes the journal log easier to work with.

Example 2: View system boot related messages
To view messages related to system boot, use the journalctl command with the -b option.

[root@linuxnix ~]# journalctl -b
-- Logs begin at Wed 2018-02-21 12:56:32 IST, end at Wed 2018-02-21 13:12:16 IST. --
Feb 21 12:56:32 linuxnix systemd-journal[92]: Runtime journal is using 8.0M (max allowed 91.1M, trying to leave 136.6M free of 903.1M a
Feb 21 12:56:32 linuxnix kernel: Initializing cgroup subsys cpuset
Feb 21 12:56:32 linuxnix kernel: Initializing cgroup subsys cpu
Feb 21 12:56:32 linuxnix kernel: Initializing cgroup subsys cpuacct
Feb 21 12:56:32 linuxnix kernel: Linux version 3.10.0-693.11.6.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623
Feb 21 12:56:32 linuxnix kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-693.11.6.el7.x86_64 root=/dev/mapper/cl-root ro crashkernel=a
Feb 21 12:56:32 linuxnix kernel: Disabled fast string operations

This shows boot messages corresponding to the current system boot. We could view historic messages going back to previous system boot messages if there are still available within the journal. To view messages related to the last system boot we would use journalctl -b -1, to view messages related to two boots ago we would use -2 and so on. In order to list the available journal messages corresponding to the last system boots use the following command:

[root@linuxnix ~]# journalctl -b --list-boots
0 e94d75821f704574b8da2869d1e4dbac Wed 2018-02-21 12:56:32 IST—Wed 2018-02-21 13:12:16 IST
[root@linuxnix ~]#

Apparently on this system only the current boot messages are available.

Example 3: Using a time range
We can specify a range of time from which we would like the log messages to be printed while using the journalctl command. We add the since keyword along with the time frame. In the below example we print messages that have been logged since the last 30 minutes.

[root@linuxnix ~]# journalctl --since "30 minutes ago"
-- Logs begin at Wed 2018-02-21 12:56:32 IST, end at Wed 2018-02-21 13:30:01 IST. --
Feb 21 13:07:16 linuxnix systemd[1]: Starting dnf makecache...
Feb 21 13:07:18 linuxnix dnf[1706]: cachedir: /var/cache/dnf/x86_64/7/x86_64/7
Feb 21 13:07:18 linuxnix dnf[1706]: DNF version: 0.6.4
Feb 21 13:07:18 linuxnix dnf[1706]: Making cache files for all metadata files.
Feb 21 13:07:18 linuxnix dnf[1706]: epel: will expire after 11446 seconds.
Feb 21 13:07:18 linuxnix dnf[1706]: puppetlabs-pc1: will expire after 11476 seconds.

Tp print log messages going as far as the last 24 hours, we could type:

[root@linuxnix ~]# journalctl --since "1 day ago"
-- Logs begin at Wed 2018-02-21 12:56:32 IST, end at Wed 2018-02-21 13:35:55 IST. --
Feb 21 12:56:32 linuxnix systemd-journal[92]: Runtime journal is using 8.0M (max allowed 91.1M, trying to leave 136.6M free of 903.1M a
Feb 21 12:56:32 linuxnix kernel: Initializing cgroup subsys cpuset
Feb 21 12:56:32 linuxnix kernel: Initializing cgroup subsys cpu
Feb 21 12:56:32 linuxnix kernel: Initializing cgroup subsys cpuacct
Feb 21 12:56:32 linuxnix kernel: Linux version 3.10.0-693.11.6.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623
Feb 21 12:56:32 linuxnix kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-693.11.6.el7.x86_64 root=/dev/mapper/cl-root ro crashkernel=a
Feb 21 12:56:32 linuxnix kernel: Disabled fast string operations
------------------------------------------------------------------output truncated for brevity

Example 4: View messages for a particular service unit
To view log messages for a particular service unit or daemon, we add the -u option with the journalctl command followed by the service unit name. For example, to view journal log messages available for the cron daemon, we would type:

[root@linuxnix ~]# journalctl -u crond.service
-- Logs begin at Wed 2018-02-21 12:56:32 IST, end at Wed 2018-02-21 13:35:55 IST. --
Feb 21 12:56:46 linuxnix systemd[1]: Started Command Scheduler.
Feb 21 12:56:46 linuxnix systemd[1]: Starting Command Scheduler...
Feb 21 12:56:46 linuxnix crond[935]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 84% if used.)
Feb 21 12:56:46 linuxnix crond[935]: (CRON) INFO (running with inotify support)
[root@linuxnix ~]#

Example 5: Tail or follow journal log
Similar to the -f option available with the tail command, the journalctl utility also provides a -f option which we could use to follow journal log messages in real time as they get updated or appended to the journal. To do so we would type:

journalctl -f

Along with the -f option, the journalctl command also provides the -n option allowing users to view the last n lines of the journal log. For example, to view the last 10 lines of the systemd journal, we would type the following command:

[root@linuxnix ~]# journalctl -n 10
-- Logs begin at Wed 2018-02-21 12:56:32 IST, end at Wed 2018-02-21 13:40:01 IST. --
Feb 21 13:35:55 linuxnix systemd[1]: Starting Network Manager Script Dispatcher Service...
Feb 21 13:35:55 linuxnix dhclient[1915]: bound to 192.168.188.131 -- renewal in 714 seconds.
Feb 21 13:35:55 linuxnix dbus[891]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Feb 21 13:35:55 linuxnix dbus-daemon[891]: dbus[891]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Feb 21 13:35:55 linuxnix systemd[1]: Started Network Manager Script Dispatcher Service.
Feb 21 13:35:55 linuxnix nm-dispatcher[2119]: req:1 'dhcp4-change' [ens33]: new request (3 scripts)
Feb 21 13:35:55 linuxnix nm-dispatcher[2119]: req:1 'dhcp4-change' [ens33]: start running ordered scripts...
Feb 21 13:40:01 linuxnix systemd[1]: Started Session 13 of user root.
Feb 21 13:40:01 linuxnix systemd[1]: Starting Session 13 of user root.
Feb 21 13:40:01 linuxnix CROND[2146]: (root) CMD (/usr/lib64/sa/sa1 1 1)

Example 6: Journalctl output formats
The -o option controls the formatting of the journal entries that are shown. IT takes one of the following options:

  • short is the default and generates an output that is mostly identical to the formatting of classic syslog files, showing one line per journal entry.
  • short-iso is very similar, but shows ISO 8601 wallclock timestamps.
  • short-precise is very similar, but shows timestamps with full microsecond precision.
  • short-monotonic is very similar, but shows monotonic timestamps instead of wall clock timestamps.
  • verbose shows the full-structured entry items with all fields.
  • export serializes the journal into a binary (but mostly text-based) stream suitable for backups and network transfer json formats entries as JSON data structures, one per line.
  • json-pretty formats entries as JSON data structures, but formats them in multiple lines in order to make them more readable by humans.
  • json-sse formats entries as JSON data structures, but wraps them in a format suitable for Server-Sent Events.
  • cat generates a very terse output, only showing the actual message of each journal entry with no metadata, not even a timestamp.

Given below is a snippet from the output when I printed the journal logs for the cron service in json-pretty format:

[root@linuxnix ~]# journalctl -o json-pretty -u crond.service
{
"__CURSOR" : "s=cbb7ec93deef44ddb997a39b714573e7;i=806;b=e94d75821f704574b8da2869d1e4dbac;m=f3e9ef;t=565b3d982891d;x=9a4ddf9
"__REALTIME_TIMESTAMP" : "1519198006315293",
"__MONOTONIC_TIMESTAMP" : "15985135",
"_BOOT_ID" : "e94d75821f704574b8da2869d1e4dbac",
"PRIORITY" : "6",
"_UID" : "0",
"_GID" : "0",
"_MACHINE_ID" : "bca7534af004465fa35d176e2a1018c7",
"_HOSTNAME" : "linuxnix",
"SYSLOG_FACILITY" : "3",
"SYSLOG_IDENTIFIER" : "systemd",
"CODE_FILE" : "src/core/job.c",
"CODE_LINE" : "776",
"CODE_FUNCTION" : "job_log_status_message",
"MESSAGE_ID" : "39f53479d3a045ac8e11786248231fbf",
"RESULT" : "done",
"_TRANSPORT" : "journal",
"_PID" : "1",
"_COMM" : "systemd",
"_EXE" : "/usr/lib/systemd/systemd",
"_CAP_EFFECTIVE" : "1fffffffff",
"_SYSTEMD_CGROUP" : "/",
"_CMDLINE" : "/usr/lib/systemd/systemd --switched-root --system --deserialize 21",
"UNIT" : "crond.service",
"MESSAGE" : "Started Command Scheduler.",
"_SOURCE_REALTIME_TIMESTAMP" : "1519198006314964"
}

Example 7: View logs for a particular user id
We could add the _UID option followed by the user id of the operating system user whose journal logs we are interested in viewing. In the given example, we view journal logs for a user named sahil having user ID 1001.

[root@linuxnix ~]# id -a sahil
uid=1001(sahil) gid=1001(sahil) groups=1001(sahil)
[root@linuxnix ~]#
[root@linuxnix ~]# journalctl _UID=1001
-- Logs begin at Wed 2018-02-21 12:56:32 IST, end at Wed 2018-02-21 13:53:29 IST. --
Feb 21 13:53:29 linuxnix sshd[2207]: Received disconnect from 192.168.188.131: 11: disconnected by user
[root@linuxnix ~]#

Conclusion

This concludes our exploration of the systemd journal logging system. We hope that you’ve found the examples described in this article to be useful and we look forward to your feedback.

Post Views: 8,066
The following two tabs change content below.

Sahil Suri

He started his career in IT in 2011 as a system administrator. He has since worked with HP-UX, Solaris and Linux operating systems along with exposure to high availability and virtualization solutions. He has a keen interest in shell, Python and Perl scripting and is learning the ropes on AWS cloud, DevOps tools, and methodologies. He enjoys sharing the knowledge he's gained over the years with the rest of the community.