Starting with systemd Journal logging system in Linux with 20 examples

Introduction

Systemd Journal Logging system made life easier while troubleshooting something in a production environment. Today in our guide, let us see how to use the journalctl command to verify logging information.

Read more logging guides:

Persistently saving journal logs

By default, the Journal logs will not be saved persistently. To save the journal logs we need to do a few configurations. Let’s start to configure the same.

create the directory to store the journal logs

# mkdir -p /var/log/journal/

Change the permission and ownership as follows

# chmod 2755 /var/log/journal/
# chown root:systemd-journal /var/log/journal

Edit the journal configuration and change the parameter “Storage=auto” to “Storage=persistent” or use sed to search and replace.

# vim /etc/systemd/journald.conf
# sed -i 's/#Storage=auto/Storage=persistent/g' /etc/systemd/journald.conf

To make the changes restart the journal service.

# systemctl restart systemd-journald.service 
# systemctl status systemd-journald.service 

Finally, verify the status of the service. By following, let’s start on how to use systemd journal logs with examples.

Similar to tail

To list end of journal log and follow mode (f) same like tail -f

# journalctl -ef

Listing latest at top

To list the newest entries at first.

# journalctl -r

Checking for specific service

To list only the logging information about sshd.service unit.

# journalctl -u httpd.service

Checking for the specific Systemd unit

To list all the logging information about sshd.service system unit.

# journalctl _SYSTEMD_UNIT=httpd.service

Listing with priority in Systemd Journal Logging

Listing logs with priority, In our example we are listing level 0 and level 3 logs. We have 8 levels of priority in the journal logging system.

# journalctl -p emerg..err

Find the list of priority level

S:NO: Priority Level Priority No
1. emerg 0
2. alert 1
3. crit 2
4. err 3
5. warning 4
6. notice 5
7. info 6
8. debug 7

Looking for SELinux context

In some case, if you are creating an interface file manually and facing some permission related issue or facing any challenge in bringing up the interface it’s good to look for SELinux context related to network as shown below.

# journalctl _SELINUX_CONTEXT=system_u:system_r:NetworkManager_t:s0

The output for the above command will too long.

Printing for specific UID

List the logs for any specific user ID, to sort the logs only for specific user this will be handy.

# journalctl _UID=998

Checking for an executable file

List all executable files, in our example, one of new account has been created

# journalctl _EXE=/usr/sbin/useradd

Output for your reference

[root@webserver ~]# journalctl _EXE=/usr/sbin/useradd 
 -- Logs begin at Thu 2019-12-26 20:28:51 GST, end at Tue 2019-12-31 21:15:34 GST. --
 Dec 31 21:15:34 webserver.linuxsysadmins.local useradd[13700]: new group: name=testuser1, GID=1002
 Dec 31 21:15:34 webserver.linuxsysadmins.local useradd[13700]: new user: name=testuser1, UID=1002, GID=1002, home=/home/testuser1, shell=/bin/bash
[root@webserver ~]#

Troubleshooting Kernel Devices

In our example, we are listing for attached SCSI disks.

# journalctl _KERNEL_DEVICE=+scsi:0:0:0:0

Output for reference

[root@servera ~]# journalctl _KERNEL_DEVICE=+scsi:0:0:0:0
  -- Logs begin at Thu 2019-12-26 20:28:51 GST, end at Tue 2019-12-31 21:20:01 GST. --
  Dec 26 20:28:51 webserver.linuxsysadmins.local kernel: sd 0:0:0:0: [sda] 41943040 512-byte logical blocks: (21.4 GB/20.0 GiB)
  Dec 26 20:28:51 webserver.linuxsysadmins.local kernel: sd 0:0:0:0: [sda] Write Protect is off
  Dec 26 20:28:51 webserver.linuxsysadmins.local kernel: sd 0:0:0:0: [sda] Attached SCSI disk

Depends on the number of disks attached in server the output will be too long.

Show for a specific boot

To list the last system boot information.

# journalctl -b -1
# journalctl _BOOT_ID=-1

Listing for a specific range

To list journal logs for a range of period.

# journalctl --since "2017-07-25 13:00:00" --until "2017-09-03 15:00:00"

Print the dmesg

To list the kernel dmesg logs from the current boot time.

# jouralctl -k

Printing logs using PID

In case, if you need to print the journal logs for a specific PID we can use below. In our example, we are looking for one of cron service’s PID.

[root@webserver ~]# journalctl _PID=12532  -- Logs begin at Thu 2019-12-26 20:28:51 GST, end at Tue 2019-12-31 20:30:01 GST. --  Dec 27 23:10:01 webserver.linuxsysadmins.local CROND[12532]: (root) CMD (/usr/lib64/sa/sa1 1 1) [root@webserver ~]#

Looking for specific machine log

This help to look for a specific machines log using _MACHINE_ID

# journalctl _MACHINE_ID=a73f9d838205494593574145673376d1

Format the journal output

Print the output in any format, for instance, we are printing in JSON-pretty format.

# journalctl _SYSTEMD_UNIT=sshd.service -o json-pretty
# journalctl _SYSTEMD_UNIT=sshd.service -o short-iso

These are the supported formats.

short, short-iso, short-precise, short-montonic, verbose, json, export, json-pretty, json-sse and cat

Printing with more verbose

A short description about any issue won’t help us to find the root cause, to get more information it’s better to print with more verbosity.

# journalctl _SYSTEMD_UNIT=sshd.service -o json-sse -o verbose

Just by adding -o verbose after any options it will give you more information.

Disk usage of Journal logs

Show total disk usage of all journal files

# jouralctl --disk-usage

Verifying journal consistency

To verify the journal file consistency

# journalctl --verify

Listing message catalogue

To list the entries in the message catalogue

# journalctl --dump-catalog

One or more messages could not be forwarded to the Syslog service
running side-by-side with journald. This usually indicates that the
syslog implementation has not been able to keep up with the speed of
messages queued.

Learning more about Journal

Read more about journalctl command run below man page.

# man journalctl

Conclusion

Systemd Journal logging system is one of the easiest ways to read logs on any RHEL based Linux operating systems. Hope this guide helps you to walk through troubleshooting. Subscribe to our newsletter and stay tuned for more guides.