sar – System Activity Recorder
sar
From the earlier discussions, one common thread emerges: Getting real time metrics is not the only important thing; the historical trend is equally important.
Furthermore, consider this situation: how many times has someone reported a performance problem, but when you dive in to investigate, everything is back to normal? Performance issues that have occurred in the past are difficult to diagnose without any specific data as of that time. Finally, you will want to examine the performance data over the past few days to decide on some settings or to make adjustments.
The sar utility accomplishes that goal. sar stands for System Activity Recorder, which records the metrics of the key components of the Linux system—CPU, Memory, Disks, Network, etc.—in a special place: the directory /var/log/sa. The data is recorded for each day in a file named sa where is the two digit day of the month. For instance the file sa27 holds the data for the date 27th of that month. This data can be queried by the command sar.
The simplest way to use sar is to use it without any arguments or options. Here is an example:
# sar
Linux 2.6.9-55.0.9.ELlargesmp (prolin3)Â Â Â Â 12/27/2008
Â
12:00:01 AM      CPU    %user    %nice  %system  %iowait    %idle
12:10:01 AM      all    14.99     0.00     1.27     2.85    80.89
12:20:01 AM      all    14.97     0.00     1.20     2.70    81.13
12:30:01 AM      all    15.80     0.00     1.39     3.00    79.81
12:40:01 AM      all    10.26     0.00     1.25     3.55    84.93
… and so on …
The output shows the CPU related metrics collected in 10 minute intervals. The columns mean:
CPU                 The CPU identifier; “all†means all the CPUs
%user              The percentage of CPU used for user processes. Oracle processes come under this category.
%nice              The %ge of CPU utilization while executing under nice priority
%system          The %age of CPU executing system processes
%iowait            The %age of CPU waiting for I/O
%idle                The %age of CPU idle waiting for work
Â
From the above output, you can see that the system has been well balanced; actually severely under-utilized (as seen from the high degree of %age idle number). Going further through the output we see the following:
… continued from above …
03:00:01 AM      CPU    %user    %nice  %system  %iowait    %idle
03:10:01 AM      all    44.99     0.00     1.27     2.85    40.89
03:20:01 AM      all    44.97     0.00     1.20     2.70    41.13
03:30:01 AM      all    45.80     0.00     1.39     3.00    39.81
03:40:01 AM      all    40.26     0.00     1.25     3.55    44.93
… and so on …
This tells a different story: the system was loaded by some user processes between 3:00 and 3:40. Perhaps an expensive query was executing; or perhaps an RMAN job was running, consuming all that CPU. This is where the sar command is useful–it replays the recorded data showing the data as of a certain time, not now. This is exactly what you wanted to accomplish the three objectives outlined in the beginning of this section: getting historical data, finding usage patterns and understanding trends.
Â
If you want to see a specific day’s sar data, merely open sar with that file name, using the -f option as shown below (to open the data for 26th)
# sar -f /var/log/sa/sa26
It can also display data in real time, similar to vmstat or mpstat. To get the data every 5 seconds for 10 times, use:
Â
# sar 5 10
Linux 2.6.9-55.0.9.ELlargesmp (prolin3)Â Â Â Â 12/27/2008
Â
01:39:16 PM      CPU    %user    %nice  %system  %iowait    %idle
01:39:21 PM      all    20.32     0.00     0.18     1.00    78.50
01:39:26 PM      all    23.28     0.00     0.20     0.45    76.08
01:39:31 PM      all    29.45     0.00     0.27     1.45    68.83
01:39:36 PM      all    16.32     0.00     0.20     1.55    81.93
… and so on 10 times …
Â
Did you notice the “all†value under CPU? It means the stats were rolled up for all the CPUs. In a single processor system that is fine; but in multi-processor systems you may want to get the stats for individual CPUs as well as an aggregate one. The -P ALL option accomplishes that.
Â
#sar -P ALL 2 2
Linux 2.6.9-55.0.9.ELlargesmp (prolin3)Â Â Â Â 12/27/2008
Â
01:45:12 PM      CPU    %user    %nice  %system  %iowait    %idle
01:45:14 PM      all    22.31     0.00    10.19     0.69    66.81
01:45:14 PMÂ Â Â Â Â Â Â Â 0Â Â Â Â Â 8.00Â Â Â Â Â 0.00Â Â Â Â 24.00Â Â Â Â Â 0.00Â Â Â Â 68.00
01:45:14 PMÂ Â Â Â Â Â Â Â 1Â Â Â Â 99.00Â Â Â Â Â 0.00Â Â Â Â Â 1.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00
01:45:14 PMÂ Â Â Â Â Â Â Â 2Â Â Â Â Â 6.03Â Â Â Â Â 0.00Â Â Â Â 18.59Â Â Â Â Â 0.50Â Â Â Â 74.87
01:45:14 PMÂ Â Â Â Â Â Â Â 3Â Â Â Â Â 3.50Â Â Â Â Â 0.00Â Â Â Â Â 8.50Â Â Â Â Â 0.00Â Â Â Â 88.00
01:45:14 PMÂ Â Â Â Â Â Â Â 4Â Â Â Â Â 4.50Â Â Â Â Â 0.00Â Â Â Â 14.00Â Â Â Â Â 0.00Â Â Â Â 81.50
01:45:14 PMÂ Â Â Â Â Â Â Â 5Â Â Â Â 54.50Â Â Â Â Â 0.00Â Â Â Â Â 6.00Â Â Â Â Â 0.00Â Â Â Â 39.50
01:45:14 PMÂ Â Â Â Â Â Â Â 6Â Â Â Â Â 2.96Â Â Â Â Â 0.00Â Â Â Â Â 7.39Â Â Â Â Â 2.96Â Â Â Â 86.70
01:45:14 PMÂ Â Â Â Â Â Â Â 7Â Â Â Â Â 0.50Â Â Â Â Â 0.00Â Â Â Â Â 2.00Â Â Â Â Â 2.00Â Â Â Â 95.50
Â
01:45:14 PM      CPU    %user    %nice  %system  %iowait    %idle
01:45:16 PM      all    18.98     0.00     7.05     0.19    73.78
01:45:16 PMÂ Â Â Â Â Â Â Â 0Â Â Â Â Â 1.00Â Â Â Â Â 0.00Â Â Â Â 31.00Â Â Â Â Â 0.00Â Â Â Â 68.00
01:45:16 PMÂ Â Â Â Â Â Â Â 1Â Â Â Â 37.00Â Â Â Â Â 0.00Â Â Â Â Â 5.50Â Â Â Â Â 0.00Â Â Â Â 57.50
01:45:16 PMÂ Â Â Â Â Â Â Â 2Â Â Â Â 13.50Â Â Â Â Â 0.00Â Â Â Â 19.00Â Â Â Â Â 0.00Â Â Â Â 67.50
01:45:16 PMÂ Â Â Â Â Â Â Â 3Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â 100.00
01:45:16 PMÂ Â Â Â Â Â Â Â 4Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.50Â Â Â Â Â 0.00Â Â Â Â 99.50
01:45:16 PMÂ Â Â Â Â Â Â Â 5Â Â Â Â 99.00Â Â Â Â Â 0.00Â Â Â Â Â 1.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00
01:45:16 PMÂ Â Â Â Â Â Â Â 6Â Â Â Â Â 0.50Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â 99.50
01:45:16 PMÂ Â Â Â Â Â Â Â 7Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 1.49Â Â Â Â 98.51
Â
Average:         CPU    %user    %nice  %system  %iowait    %idle
Average:         all    20.64     0.00     8.62     0.44    70.30
Average:Â Â Â Â Â Â Â Â Â Â Â 0Â Â Â Â Â 4.50Â Â Â Â Â 0.00Â Â Â Â 27.50Â Â Â Â Â 0.00Â Â Â Â 68.00
Average:Â Â Â Â Â Â Â Â Â Â Â 1Â Â Â Â 68.00Â Â Â Â Â 0.00Â Â Â Â Â 3.25Â Â Â Â Â 0.00Â Â Â Â 28.75
Average:Â Â Â Â Â Â Â Â Â Â Â 2Â Â Â Â Â 9.77Â Â Â Â Â 0.00Â Â Â Â 18.80Â Â Â Â Â 0.25Â Â Â Â 71.18
Average:Â Â Â Â Â Â Â Â Â Â Â 3Â Â Â Â Â 1.75Â Â Â Â Â 0.00Â Â Â Â Â 4.25Â Â Â Â Â 0.00Â Â Â Â 94.00
Average:Â Â Â Â Â Â Â Â Â Â Â 4Â Â Â Â Â 2.25Â Â Â Â Â 0.00Â Â Â Â Â 7.25Â Â Â Â Â 0.00Â Â Â Â 90.50
Average:Â Â Â Â Â Â Â Â Â Â Â 5Â Â Â Â 76.81Â Â Â Â Â 0.00Â Â Â Â Â 3.49Â Â Â Â Â 0.00Â Â Â Â 19.70
Average:Â Â Â Â Â Â Â Â Â Â Â 6Â Â Â Â Â 1.74Â Â Â Â Â 0.00Â Â Â Â Â 3.73Â Â Â Â Â 1.49Â Â Â Â 93.03
Average:Â Â Â Â Â Â Â Â Â Â Â 7Â Â Â Â Â 0.25Â Â Â Â Â 0.00Â Â Â Â Â 1.00Â Â Â Â Â 1.75Â Â Â Â 97.01
This shows the CPU identifier (starting with 0) and the stats for each. At the very end of the output you will see the average of runs against each CPU.
Â
The command sar is not only fro CPU related stats. It’s useful to get the memory related stats as well. The -r option shows the extensive memory utilization.
# sar -r
Linux 2.6.9-55.0.9.ELlargesmp (prolin3)Â Â Â Â 12/27/2008
Â
12:00:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
12:10:01 AMÂ Â Â 712264Â 32178920Â Â Â Â 97.83Â Â 2923884Â 25430452Â 16681300Â Â Â Â 95908Â Â Â Â Â 0.57Â Â Â Â Â Â 380
12:20:01 AMÂ Â Â 659088Â 32232096Â Â Â Â 98.00Â Â 2923884Â 25430968Â 16681300Â Â Â Â 95908Â Â Â Â Â 0.57Â Â Â Â Â Â 380
12:30:01 AMÂ Â Â 651416Â 32239768Â Â Â Â 98.02Â Â 2923920Â 25431448Â 16681300Â Â Â Â 95908Â Â Â Â Â 0.57Â Â Â Â Â Â 380
12:40:01 AMÂ Â Â 651840Â 32239344Â Â Â Â 98.02Â Â 2923920Â 25430416Â 16681300Â Â Â Â 95908Â Â Â Â Â 0.57Â Â Â Â Â Â 380
12:50:01 AMÂ Â Â 700696Â 32190488Â Â Â Â 97.87Â Â 2923920Â 25430416Â 16681300Â Â Â Â 95908Â Â Â Â Â 0.57Â Â Â Â Â Â 380
Let’s see what each column means:
kbmemfree                The free memory available in KB at that time
kbmemused               The memory used in KB at that time
%memused               %age of memory used
kbbuffers                   This %age of memory was used as buffers
kbcached                   This %age of memory was used as cache
kbswpfree                  The free swap space in KB at that time
kbswpused                 The swap space used in KB at that time
%swpused                 The %age of swap used at that time
kbswpcad                   The cached swap in KB at that time
At the very end of the output, you will see the average figure for time period.
Â
You can also get specific memory related stats. The -B option shows the paging related activity.
# sar -B
Linux 2.6.9-55.0.9.ELlargesmp (prolin3)Â Â Â Â 12/27/2008
Â
12:00:01 AM pgpgin/s pgpgout/s  fault/s majflt/s
12:10:01 AMÂ Â Â 134.43Â Â Â 256.63Â Â 8716.33Â Â Â Â Â 0.00
12:20:01 AMÂ Â Â 122.05Â Â Â 181.48Â Â 8652.17Â Â Â Â Â 0.00
12:30:01 AMÂ Â Â 129.05Â Â Â 253.53Â Â 8347.93Â Â Â Â Â 0.00
… and so on …
The column shows metrics at that time, not currently.
pgpgin/s           The amount of paging into the memory from disk, per second
pgpgout/s         The amount of paging out to the disk from memory, per second
fault/s                Page faults per second
majflt/s              Major page faults per second
Â
To get a similar output for swapping related activity, you can use the -W option.
# sar -W
Linux 2.6.9-55.0.9.ELlargesmp (prolin3)Â Â Â Â 12/27/2008
Â
12:00:01 AMÂ pswpin/s pswpout/s
12:10:01 AMÂ Â Â Â Â 0.00Â Â Â Â Â 0.00
12:20:01 AMÂ Â Â Â Â 0.00Â Â Â Â Â 0.00
12:30:01 AMÂ Â Â Â Â 0.00Â Â Â Â Â 0.00
12:40:01 AMÂ Â Â Â Â 0.00Â Â Â Â Â 0.00
… and so on …
The columns are probably self-explanatory; but here is the description of each anyway:
pswpin/s       Pages of memory swapped back into the memory from disk, per second
Â
pswpout/s     Pages of memory swapped out to the disk from memory, per second
Â
If you see a lot of swapping, you may be running low on memory. It’s not a foregone conclusion but rather something that may be a strong possibility.
To get the disk device statistics, use the -d option:
# sar -d
Linux 2.6.9-55.0.9.ELlargesmp (prolin3)Â Â Â Â 12/27/2008
Â
12:00:01 AM      DEV      tps rd_sec/s wr_sec/s
12:10:01 AMÂ Â Â dev1-0Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00
12:10:01 AMÂ Â Â dev1-1Â Â Â Â Â 5.12Â Â Â Â Â 0.00Â Â Â 219.61
12:10:01 AMÂ Â Â dev1-2Â Â Â Â Â 3.04Â Â Â Â 42.47Â Â Â Â 22.20
12:10:01 AMÂ Â Â dev1-3Â Â Â Â Â 0.18Â Â Â Â Â 1.68Â Â Â Â Â 1.41
12:10:01 AMÂ Â Â dev1-4Â Â Â Â Â 1.67Â Â Â Â 18.94Â Â Â Â 15.19
… and so on …
Average:Â Â Â Â Â dev8-48Â Â Â Â Â 4.48Â Â Â 100.64Â Â Â Â 22.15
Average:Â Â Â Â Â dev8-64Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00
Average:Â Â Â Â Â dev8-80Â Â Â Â Â 2.00Â Â Â Â 47.82Â Â Â Â Â 5.37
Average:Â Â Â Â Â dev8-96Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00
Average:Â Â Â Â dev8-112Â Â Â Â Â 2.22Â Â Â Â 49.22Â Â Â Â 12.08
Here is the description of the columns. Again, they show the metrics at that time.
tps                         Transfers per second. Transfers are I/O operations.
                             Note: this is just number of operations; each operation may be large or small.
                             So, this, by itself, does not tell the whole story.
Â
rd_sec/s                 Number of sectors read from the disk per second
Â
wr_sec/s                Number of sectors written to the disk per second
Â
To get the historical network statistics, you use the -n option:
# sar -n DEV | more
Linux 2.6.9-42.0.3.ELlargesmp (prolin3)Â Â Â Â 12/27/2008
Â
12:00:01 AM    IFACE  rxpck/s  txpck/s  rxbyt/s  txbyt/s  rxcmp/s  txcmp/s rxmcst/s
12:10:01 AM       lo     4.54     4.54   782.08   782.08     0.00     0.00     0.00
12:10:01 AMÂ Â Â Â Â eth0Â Â Â Â Â 2.70Â Â Â Â Â 0.00Â Â Â 243.24Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.99
12:10:01 AMÂ Â Â Â Â eth1Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00
12:10:01 AMÂ Â Â Â Â eth2Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00
12:10:01 AMÂ Â Â Â Â eth3Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00
12:10:01 AMÂ Â Â Â Â eth4Â Â Â 143.79Â Â Â 141.14Â 73032.72Â 38273.59Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.99
12:10:01 AMÂ Â Â Â Â eth5Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00
12:10:01 AMÂ Â Â Â Â eth6Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00
12:10:01 AMÂ Â Â Â Â eth7Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00
12:10:01 AMÂ Â Â Â bond0Â Â Â 146.49Â Â Â 141.14Â 73275.96Â 38273.59Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 1.98
… and so on …
Average:Â Â Â Â Â Â Â bond0Â Â Â 128.73Â Â Â 121.81Â 85529.98Â 27838.44Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 1.98
Average:Â Â Â Â Â Â Â Â eth8Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00
Average:Â Â Â Â Â Â Â Â eth9Â Â Â Â Â 3.52Â Â Â Â Â 6.74Â Â Â 251.63Â 10179.83Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00
Average:Â Â Â Â Â Â Â Â sit0Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00Â Â Â Â Â 0.00
In summary, you have these options for the sar command to get the metrics for the components:
Use this option …  … to get the stats on:
Â
-PÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Specific CPU(s)
-d                           Disks
-r                            Memory
-BÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Paging
-WÂ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Swapping
-n                           Network
What if you want to get the all the available stats on one output? Instead of calling sar with all these options, you can use the -A option which shows all the stats stored in the sar files.
Â
(Extracted from oracle technet notes author Arup Nanda)
Discussion ¬