UNIX Server monitoring commands

UNIX provides a wealth of commands that help the Oracle DBA understand what is happening on the server.  We will begin with a discussion of the most popular commands and utilities for monitoring UNIX including top, svmon, glance and vmstat. In Chapter 3 we will discuss how to use the output from these utilities to determine your overall server performance.

Using the UNIX top utility

The top utility is used to show CPU consumption, RAM memory consumption, and the top sessions on a UNIX server. The top utility is invoked by entering the top command from the UNIX prompt. The output from top is displayed in three sections.

The top Load Averages

At the very beginning of the top output we see a series of three numbers. These are the called load average metrics. The load average is an arbitrary number that describes the load on the system. The first load average value is the immediate load for the past minute. The next value represents the load average from 5 minutes in the past. The third value is the load average from 15 minutes ago. Whenever the load average rises above 1, you can assume that the processors are fully burdened and you should immediately run vmstat to check the runqueue values.

Here we start with information on the server load average.  The load average is an arbitrary number that describes the load on the server.  In general, a number less than one is ideal, and here we see three load average values.

The load average display shows the load averages for the past minute, the past 5 minutes, and the past 10 minutes.  A low load average is ideal, and the load average should stay below zero.  Whenever the value exceeds “1” there may be a CPU overload problem.

System: penguin                           Mon Sep 10 17:35:13
001
Load averages: 0.52, 0.38, 0.34
213 processes: 211 sleeping, 2 running

The top CPU Summary

The first output from top shows the load on each processor and the current top sessions in terms of CPU utilization. Top gives details on each CPU on the server, and we can immediately see from the listing below that this server has six CPUs, numbered 0–5:

root> top 

Cpu states:

CPU   LOAD   USER   NICE    SYS   IDLE  BLOCK  SWAIT   INTR   SSYS
 0    0.63  10.9%   0.0%   4.0%  85.1%   0.0%   0.0%   0.0%   0.0%
 1    0.40  27.4%   0.0%   7.4%  65.3%   0.0%   0.0%   0.0%   0.0%
---   ----  -  -  -  -  -  -  -  -----
avg   0.52  19.1%   0.0%   5.7%  75.3%   0.0%   0.0%   0.0%   0.0%

Memory: 100428K (25608K) real, 107920K (30120K) virtual, 328948K free 

Top Sessions

Now let’s look at the second section from the top command. The second section of top output details the current top sessions in terms of CPU utilization, appears as follows:

System: core-prod                             Mon Dec 15 08:19:56 2001
Load averages: 0.06, 0.08, 0.03
3172 processes: 3138 sleeping, 34 running

Cpu states:

CPU   LOAD   USER   NICE    SYS   IDLE  BLOCK  SWAIT   INTR   SSYS
 0    0.11   0.0%   0.0%   0.0% 100.0%   0.0%   0.0%   0.0%   0.0%
 1    0.02   0.0%   0.0%   0.0% 100.0%   0.0%   0.0%   0.0%   0.0%
 2    0.02   0.0%   0.0%   0.0% 100.0%   0.0%   0.0%   0.0%   0.0%
 3    0.03   0.0%   0.0%   0.0% 100.0%   0.0%   0.0%   0.0%   0.0%
 4    0.00   0.0%   0.0%   0.0% 100.0%   0.0%   0.0%   0.0%   0.0%
 5    0.01   0.0%   0.0%   0.0% 100.0%   0.0%   0.0%   0.0%   0.0%
---   ----  -----  -----  -----  -----  -----  -----  -----  -----
avg   0.03   0.0%   0.0%   0.0% 100.0%   0.0%   0.0%   0.0%   0.0% 

Memory: 736056K (417860K) real, 733560K (422192K) virtual, 1101512K free  Page# 1/54 

CPU TTY     PID USERNAME PRI NI   SIZE    RES STATE    TIME %WCPU  %CPU COMMAND

 3   ?    16664 oracle   154 20 20304K  1892K sleep   15:32  2.21  2.21 oracleTE
 5   ?       36 root     152 20     0K     0K run     57:52  1.65  1.65 vxfsd
 2   ?      477 root     154 20    32K    80K sleep  160:55  0.71  0.71 syncer
 3   ?    14963 oracle   154 20  4448K  2780K sleep    4:39  0.32  0.32 oraweb
 0   ?    15980 oracle   154 20  4704K  3020K sleep    4:41  0.31  0.31 oraweb
 0 pts/tb 21355 root     158 20   536K   184K sleep    0:00  0.77  0.30 sh

In this section of the top output, we see the process ID (PID), username, the dispatching priority (PRI), the nice value (NI), the size of each task's memory (SIZE), the state, the execution time, and the percentage of CPU being used by each process.

While top has many columns of information, there are only a few columns that are of interest to you as the Oracle DBA:

* Load averages?These are the load averages for the entire server. Values greater than 1 may indicate an overload problem on the server.

* CPU?The first section of the top output shows a load summary for each CPU. The CPU column in the detailed listing shows which CPU is servicing each individual task.

* LOAD?The LOAD column shows the load on each of the CPUs.

* NI?The NI (nice) value is the dispatching priority of the task, and refers to the rate that the task receives services from the CPUs.

* IDLE?This shows the percentage of time that each CPU has been idle.

Next, let’s take a look at using the svmon utility in the IBM AIX operating system.

Using svmon on IBM AIX

The svmon utility can be used in the HP/UX environment to display server values.  Here is an example listing from svmon.

root> svmon 

               size      inuse       free        pin    virtual          
memory      1048566    1023178       4976      55113     251293
pg space     524288      10871

               work       pers       clnt
pin           55116          0          0
in use       250952     772224          2

Here is a list of the column definitions for the above output:

size  the number of real memory frames (size of real memory)

inuse is the number of frames containing pages

pin   Number of frames containing pinned pages in use

The svmon -p command can also be used to display characteristics for a specific process ID (PID):

root> svmon -P 26060

----------------------------------------------------------------------
     Pid Command    Inuse      Pin   Pgsp  Virtual   64-bit    Mthrd
   26060 pr          6871     1607   1022     6001        N        N 

  Vsid  Esid Type Description         Inuse   Pin Pgsp Virtual Addr
 24029     d work shared library text  3992     0   22  2779   0..65535
     0     0 work kernel seg           2509  1606  926  2897   0..32767                                                                    

5475..65535

 105e4        2 work process private  188     1   48   230   0..273 :

5298..65535

 285ea   f work shared library data    92     0   26    95   0..919
 185e6   1 pers code,/dev/lvs001:301   81     0    -     -   0..149
 6c59b   - pers /dev/lvs001:92402       6     0    -     -   0..9
 744fd   - pers /dev/lvs001:763909      3     0    -     -   0..9
 7c5ff   - pers /dev/lvs001:1327130     0     0    -     -   0..29

The SAR Utility in UNIX

The sar utility is short for System Activity Reporter.  This system activity reporter is quite popular in HP/UX systems, and is now becoming available for AIX and Solaris dialects of UNIX.  SAR has much of the same functionality as the vmstat utility, but provides additional details.

There are four main ways to invoke sar, each producing a different output display as shown in Table 2-5.

Table 5: Arguments for the UNIX sar command

Note: Each dislect of UNIX has different display formats for the sar utility.  For example, some of the argument flags in the Solaris are not available on HP/UX.  Please check your UNIX-specific documentation for details on your UNIX server.

The output from sar reports usually shows a time-based snapshot of activity. This is true for all reports that you'll see in this section. When you issue the sar command, you pass two numeric arguments. The first represents the time interval between samples, and the second represents the number of samples to take. For example:

The sar command in this example is requesting five samples taken at 10-second intervals.

sar –u: The CPU Report

The sar –u command is very useful for seeing the overall CPU consumption over time. In the example that follows, I execute sar –u to see the state of the CPU. CPU time can be allocated into the following four sections: user mode, system mode, waiting on I/O, and idle.

root> sar -u 2 5 

HP-UX corp-hp1 B.11.00 U 9000/800    12/25/00

07:18:44    %usr    %sys    %wio   %idle
07:18:46       0       0       1      99
07:18:48       0       0       1      99
07:18:50       4       0      13      83
07:18:52       2       1       7      90
07:18:54       0       0       3      98 

Average        1       0       5      93

sar –w: The Memory Switching and Swapping Activity Report

The sar –w command is especially useful if you suspect that your database server is experiencing a memory shortage. The following example shows the swapping activity report that you get from sar:

root> sar -w 5 5

HP-UX corp-hp1 B.11.00 U 9000/800    12/25/00

07:19:33 swpin/s bswin/s swpot/s bswot/s pswch/s
07:19:38    0.00     0.0    0.00     0.0     261
07:19:43    0.00     0.0    0.00     0.0     231
07:19:48    0.00     0.0    0.00     0.0     326
07:19:53    0.00     0.0    0.00     0.0     403
07:19:58    0.00     0.0    0.00     0.0     264 

Average     0.00     0.0    0.00     0.0     297

The column descriptions for sar –w are as follows:

* swpin/s?Number of process swap-ins per second.

* swpot/s?Number of process swap-outs per second.

* bswin/s?Number of 512-byte swap-ins per second.

* bswot/s?Number of 512-byte swap-outs per second.

* pswch/s?Number of process context switches per second.

sar –b: The Buffer Activity Report

The sar -b command causes sar to report buffer activity, which equates to disk I/O activity and is especially useful if you suspect that your database is I/O bound. The report shows real disk I/O, and the interaction with the UNIX Journal File System (JFS) buffer. For example, below we see a sample of sar output over a 5-second interval:

>sar -b 1 5 

HP-UX corp-hp1 B.11.00 U 9000/800    12/25/00

07:20:40 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
07:20:41       0      72     100       6       7      14       0       0
07:20:42       0       3     100       3       3       0       0       0
07:20:43       0       3     100       0       9     100       0       0
07:20:44       0      26     100       6      12      50       0       0
07:20:45       0      19     100       3      15      80       0       0 

Average        0      25     100       4       9      61       0       0

In the output shown here, you see the following data columns:

* Bread/s Number of physical reads from disk per second.

* lread/s?Number of reads per second from the UNIX JFS buffer cache.

* %rcache?Buffer cache hit ratio (for the UNIX JFS buffer cache) for read requests.

* bwrit/s?Number of physical writes to disk per second. This gives the DBA an indication of the overall write activity on the server.

* lwrit/s?Number of writes per second to the UNIX JFS buffer cache.

* %wcache?Buffer cache hit ratio (for the UNIX JFS buffer cache) for write requests.

* pread/s?Number of reads per second from disk. This is an excellent measure of the load on the I/O subsystem.

* pwrit/s?Number of writes per second to disk

The sar –b command is often used in reactive tuning when you want to correlate what is happening inside Oracle with what is happening on the database server. Now let’s turn our attention to the UNIX sadc utility.