How to view past performance with sar in Linux

There are many tools/utilities that can be used to analyze the current system performance. But how do we gauge the historic system performance? For that you can use the System Activity Report (SAR) tool. Using the sar tool, you will be able to look back over a period of time and see how the server has been running.

In this post we will see how to install and use the sysstat tools; thus, allowing you to examine historical system statistics.

Installing sysstat

First thing first, install the “sysstat” which provides the sar utility.

# apt-get install sysstat       ### Debian-based distributions
# yum install sysstat           ### RedHat Based distribution

The historic data for various linux distribution is stored in below directories: 1. Red Hat, Fedora, CentOS, and Scientific Linux should use the /var/log/sa directory 2. Debian, Mint, and Ubuntu users should use the /var/log/sysstat directory

By default, sar stats are collected every 10 minutes. The data is collected using a simple cron job configured within /etc/cron.d/sysstat. This job can be amended to collect the data as frequently as you require. Example /etc/cron.d/sysstat file is provided below.

# cat /etc/cron.d/sysstat
# Run system activity accounting tool every 10 minutes
*/10 * * * * root /usr/lib64/sa/sa1 1 1
# 0 * * * * root /usr/lib64/sa/sa1 600 6 &
# Generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib64/sa/sa2 -A

Gathering historical sar data

1. Getting live sar data is easy with sar. By default sar operates in CPU mode. Simply using the command as named, we will receive CPU activity samples for every 10 minutes of the current day.

# sar
Linux 2.6.32-504.el6.x86_64 (geeklab)   09/28/2018      _x86_64_        (16 CPU)

12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:10:01 AM     all      7.28      0.00      2.94      0.02      0.00     89.77
12:20:01 AM     all      7.16      0.00      2.47      0.02      0.00     90.35
12:30:01 AM     all      6.96      0.00      2.43      0.01      0.00     90.59
12:40:01 AM     all      6.92      0.00      2.42      0.01      0.00     90.64
...

2. Lets suppose you want to analyse the system performance at some past time. For example view system performance between 10 A.M. to 12 A.M. when application team faced a slowness in their app.

# sar -r -s 10:00:00 -e 12:00:00

sar uses the -s arguement to specify the start time of a data extract and another (-e) to set the end time. These parameters must be written in HH:MM:SS format, or sar will ignore them with an error. The -r argument is used to display memory usage data.

3. sar historic data is stored in /var/log/sa directory in case of RedHat based distributions. Files are simply binary formats containing sar data for each retained date. Files are prefixed with sa. Thus, sa23 is the sar data for the 23rd of the month. For example, on a CenOS/RHEL system.

# ls /var/log/sa
sa01  sa04  sa07  sa10  sa13  sa16  sa19  sa22  sa25  sa28  sar01  sar04  sar07  sar10  sar13  sar16  sar19  sar22  sar25  sar30
sa02  sa05  sa08  sa11  sa14  sa17  sa20  sa23  sa26  sa30  sar02  sar05  sar08  sar11  sar14  sar17  sar20  sar23  sar26  sar31
sa03  sa06  sa09  sa12  sa15  sa18  sa21  sa24  sa27  sa31  sar03  sar06  sar09  sar12  sar15  sar18  sar21  sar24  sar27

Execute the following command to view past I/O statistics for the 10th of the month:

# sar -f /var/log/sysstat/sa10 -b

Changing sar historical data retention

One can change the historical sar data retention using the file /etc/sysconfig/sysstat or /etc/sysstat/sysstat. Change the HISTORY parameter setting to the desired amount of days to retain data. For example, to keep 28 days of records, we could use this:

# cat /etc/sysconfig/sysstat
# sysstat-9.0.4 configuration file.

# How long to keep log files (in days).
# If value is greater than 28, then log files are kept in
# multiple directories, one for each month.
HISTORY=28

# Compress (using gzip or bzip2) sa and sar files older than (in days):
COMPRESSAFTER=31

# Parameters for the system activity data collector (see sadc manual page)
# which are used for the generation of log files.
SADC_OPTIONS="-S DISK"

# Compression program to use.
ZIP="bzip2"