Sar
What is sar
Sar is a command line utility installed by default on the CentOS servers. It is used to collect, report of save system activity. It is most in tracking down the cause of disk I/O. Sar uses a series of files in /var/log/sa/ to keep track of its data.
Sar History
To see a list of the previous month's
ll /var/log/sa/
This should provide you with a list that looks like the following.
-rw-r--r-- 1 root root 484640 Aug 1 15:51 sa01 -rw-r--r-- 1 root root 484176 Aug 2 23:50 sa02 -rw-r--r-- 1 root root 228576 Aug 3 11:00 sa03 -rw-r--r-- 1 root root 490992 Jul 26 23:50 sa26 -rw-r--r-- 1 root root 490992 Jul 27 23:50 sa27 -rw-r--r-- 1 root root 490992 Jul 28 23:50 sa28 -rw-r--r-- 1 root root 481232 Jul 29 23:50 sa29 -rw-r--r-- 1 root root 490992 Jul 30 23:50 sa30 -rw-r--r-- 1 root root 485104 Jul 31 15:37 sa31 -rw-r--r-- 1 root root 491873 Aug 1 23:53 sar01 -rw-r--r-- 1 root root 491907 Aug 2 23:53 sar02 -rw-r--r-- 1 root root 498737 Jul 25 23:53 sar25 -rw-r--r-- 1 root root 498737 Jul 26 23:53 sar26 -rw-r--r-- 1 root root 498737 Jul 27 23:53 sar27 -rw-r--r-- 1 root root 498737 Jul 28 23:53 sar28 -rw-r--r-- 1 root root 489709 Jul 29 23:53 sar29 -rw-r--r-- 1 root root 498737 Jul 30 23:53 sar30 -rw-r--r-- 1 root root 493163 Jul 31 23:53 sar31
Using sar
Without any flags, the sar command will show you breakdowns of CPU usage. This is similar to output given by the iostat command. Output will look something like this:
sar
12:30:01 PM CPU %user %nice %system %iowait %steal %idle 12:40:01 PM all 0.13 0.01 0.08 0.02 0.00 99.76 12:50:01 PM all 0.13 0.02 0.07 0.02 0.00 99.76 01:00:01 PM all 0.12 0.01 0.08 0.03 0.00 99.77 01:10:01 PM all 0.22 0.01 0.09 0.05 0.00 99.63 Average: all 0.15 0.01 0.08 0.03 0.00 99.73
Sar by itself is useful for tracking down CPU usage problems.
For memory related problems, run sar with the -r flag. -r show historical memory and swap usage, both in percentages and kilobytes. Output will look something like this.
sar -r
12:30:01 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad 12:40:01 PM 70076 1687976 96.01 191540 984628 1959852 68 0.00 0 12:50:01 PM 69432 1688620 96.05 191540 984648 1959852 68 0.00 0 01:00:01 PM 69984 1688068 96.02 191548 984672 1959852 68 0.00 0 01:10:01 PM 67744 1690308 96.15 191564 985536 1959852 68 0.00 0 Average: 69309 1688743 96.06 191548 984871 1959852 68 0.00 0
Sar with the -W flag shows actual change in swapfile usage. Output looks like this following
sar -W 12:30:01 PM pswpin/s pswpout/s 12:40:01 PM 0.00 0.00 12:50:01 PM 0.00 0.00 01:00:01 PM 0.00 0.00 01:10:01 PM 0.00 0.00 01:20:01 PM 0.00 0.00 Average: 0.00 0.00
The -s and -e flags can be used to specify start and end points with the sar output. Both are follow the format of HH:MM:SS.
sar -s 09:00:00 -e 10:30:00
09:00:01 AM CPU %user %nice %system %iowait %steal %idle 09:10:01 AM all 0.22 0.01 0.09 0.06 0.00 99.62 09:20:01 AM all 0.13 0.01 0.07 0.06 0.00 99.73 09:30:01 AM all 0.12 0.01 0.07 0.02 0.00 99.76 09:40:01 AM all 0.13 0.01 0.07 0.03 0.00 99.76 09:50:01 AM all 0.13 0.01 0.07 0.02 0.00 99.77 10:00:01 AM all 0.13 0.01 0.07 0.05 0.00 99.73 10:10:01 AM all 0.23 0.01 0.09 0.04 0.00 99.63 10:20:01 AM all 0.13 0.02 0.08 0.03 0.00 99.75 Average: all 0.15 0.01 0.08 0.04 0.00 99.72
There are more flags available to sar, and it is recommended that you search through the man page for more options.
Checking the log of a different day
Looking at the current day's files are helpful, but more often than not, you'll have to go through the older logs. You can do this with the -f flag.
Template:P sar -f /var/log/sa/sa01
Linux 2.6.18-164.11.1.el5 (training.p-hawk.com) 08/01/2010 12:00:01 AM CPU %user %nice %system %iowait %steal %idle 12:10:01 AM all 0.24 0.01 0.09 0.54 0.00 99.11 12:20:01 AM all 0.14 0.01 0.08 0.21 0.00 99.57 12:30:01 AM all 0.13 0.01 0.07 0.03 0.00 99.76 12:40:01 AM all 0.14 0.01 0.07 0.07 0.00 99.71 12:50:01 AM all 0.13 0.02 0.09 0.03 0.00 99.74 01:00:01 AM all 0.13 0.01 0.08 0.40 0.00 99.39 01:10:01 AM all 0.22 0.01 0.09 0.11 0.00 99.57 01:20:01 AM all 0.12 0.01 0.08 0.05 0.00 99.74 01:30:01 AM all 0.12 0.01 0.07 0.02 0.00 99.79 01:40:01 AM all 0.12 0.02 0.08 0.03 0.00 99.76 01:50:01 AM all 1.35 0.01 0.20 0.13 0.00 98.31 02:00:01 AM all 0.14 0.01 0.08 0.05 0.00 99.73 02:10:01 AM all 0.23 0.01 0.09 0.04 0.00 99.64
The -f can be combined with other sar flags, to get you the information you require.
Template:Psar -f /var/log/sa/sa01 -W
Linux 2.6.18-164.11.1.el5 (training.p-hawk.com) 08/01/2010 12:00:01 AM pswpin/s pswpout/s 12:10:01 AM 0.00 0.00 12:20:01 AM 0.00 0.00 12:30:01 AM 0.00 0.00 12:40:01 AM 0.00 0.00 12:50:01 AM 0.00 0.00 01:00:01 AM 0.00 0.00 01:10:01 AM 0.00 0.00 01:20:01 AM 0.00 0.00 01:30:01 AM 0.00 0.00 01:40:01 AM 0.00 0.00 01:50:01 AM 0.00 0.00
Adjusting Sar
You can also adjust the frequency that sysstat collects data at. All you have to do is modify the time on the cron.
cat /etc/cron.d/sysstat # run system activity accounting tool every 10 minutes */10 * * * * root /usr/lib/sa/sa1 1 1
Change the */10 to however many minutes you want it to run and then restart crond.
/etc/init.d/crond restart
Please remember to revert your changes after you have aquired the information that you needed.
Troubleshooting
Sar is a pretty straight forward command, but every once in a while you will notice that sar is no longer updating. You may see something like the following
Template:Pll /var/log/sa
total 2932 drwxr-xr-x 2 root root 4096 Aug 3 11:36 ./ drwxr-xr-x 10 root root 4096 Dec 13 2009 ../ -rw-r--r-- 1 root root 143088 Dec 8 2009 sa08 -rw-r--r-- 1 root root 143088 Dec 9 2009 sa09 -rw-r--r-- 1 root root 143088 Dec 10 2009 sa10 -rw-r--r-- 1 root root 143088 Dec 11 2009 sa11 -rw-r--r-- 1 root root 143088 Dec 12 2009 sa12 -rw-r--r-- 1 root root 143088 Dec 13 2009 sa13 -rw-r--r-- 1 root root 143088 Dec 14 2009 sa14 -rw-r--r-- 1 root root 143088 Dec 15 2009 sa15 -rw-r--r-- 1 root root 134160 Dec 16 2009 sa16 -rw-r--r-- 1 root root 202085 Dec 8 2009 sar08 -rw-r--r-- 1 root root 202085 Dec 9 2009 sar09 -rw-r--r-- 1 root root 202085 Dec 10 2009 sar10 -rw-r--r-- 1 root root 202085 Dec 11 2009 sar11 -rw-r--r-- 1 root root 202085 Dec 12 2009 sar12 -rw-r--r-- 1 root root 202085 Dec 13 2009 sar13 -rw-r--r-- 1 root root 202085 Dec 14 2009 sar14 -rw-r--r-- 1 root root 202085 Dec 15 2009 sar15
The dates on those are old, so you'll want to restart sysstat using the following method:
rm /var/lock/subsys/sysstat /etc/init.d/sysstat restart
This will force sysstat to restart and in turn force sar updates