ss (socket statistics) introduction

Almost every sys-admin know the powerful tool called ‘netstat’ but it is time to say bye-bye to netstat as we have a more powerful and concise tool called ‘ss’.

1. ss introduction

ss is the abbreviation for socket statistics. It is part of the iproute tool. If you can’t use ss, you need to install iproute such as
       yum install iproute iproute-doc
while netstat is part of net-tools which has not been updated since 2001
when we compare the performance between ss and netstat, we can get the conclusion that ss is much faster than netstat, that is because ss uses TCP’s tcp_diag to get the statistics from linux kernel.

See when a linux handle 10000 TCP connections, netstat took over 1 second to get the output and only 0.01 second to get the ss status

[root@X003 bin]# time netstat -ant > /dev/null

real    0m1.334s
user    0m0.230s
sys     0m1.101s
[root@X003 bin]# time ss -s
Total: 10439 (kernel 10442)
TCP:   11073 (estab 9203, closed 1, orphaned 800, synrecv 0, timewait 0/0), ports 10007

Transport Total     IP        IPv6
*         10442     -         -
RAW       0         0         0
UDP       8         5         3
TCP       11072     10008     1064
INET      11080     10013     1067
FRAG      0         0         0


real    0m0.014s
user    0m0.000s
sys     0m0.012s
ss 

2. Basic usage:

1. ss -s #Get basic statistics information

[root@X003 bin]# ss -s
Total: 180 (kernel 197)
TCP:   15 (estab 3, closed 1, orphaned 0, synrecv 0, timewait 0/0), ports 7

Transport Total     IP        IPv6
*         197       -         -
RAW       0         0         0
UDP       8         5         3
TCP       14        8         6
INET      22        13        9
FRAG      0         0         0

2. ss -l #only list the listening port

[root@X003 bin]# ss -l
State      Recv-Q Send-Q      Local Address:Port          Peer Address:Port
LISTEN     0      128                    :::sunrpc                  :::*
LISTEN     0      128                     *:sunrpc                   *:*
LISTEN     0      128                    :::http                    :::*
LISTEN     0      128                    :::ssh                     :::*
LISTEN     0      128                     *:ssh                      *:*
LISTEN     0      128             127.0.0.1:ipp                      *:*
LISTEN     0      128                   ::1:ipp                     :::*
LISTEN     0      100                   ::1:smtp                    :::*
LISTEN     0      100             127.0.0.1:smtp                     *:*
LISTEN     0      128                    :::55878                   :::*
LISTEN     0      128                     *:38186                    *:*

3. ss -t #only list the tcp status

[root@X003 bin]# ss -t
State      Recv-Q Send-Q      Local Address:Port          Peer Address:Port
ESTAB      0      52         192.168.56.113:ssh           192.168.56.1:65433
ESTAB      0      0          192.168.56.113:ssh           192.168.56.1:65417
ESTAB      0      0          192.168.56.113:ssh           192.168.56.1:65418

4. ss -u #only list the udp status
5. ss -p #list the program associated with port

[root@X003 bin]# ss -p
State      Recv-Q Send-Q      Local Address:Port          Peer Address:Port
ESTAB      0      52         192.168.56.113:ssh           192.168.56.1:65433    users:(("sshd",1789,3))
ESTAB      0      0          192.168.56.113:ssh           192.168.56.1:65417    users:(("sshd",1734,3))
ESTAB      0      0          192.168.56.113:ssh           192.168.56.1:65418    users:(("sshd",1759,3))

6. ss -n #don’t convert the number to the hostname or known service name

[root@X003 bin]# ss -n
State      Recv-Q Send-Q        Local Address:Port          Peer Address:Port
ESTAB      0      52           192.168.56.113:22            192.168.56.1:65433
ESTAB      0      0            192.168.56.113:22            192.168.56.1:65417
ESTAB      0      0            192.168.56.113:22            192.168.56.1:65418

7. ss -e #show the extensive/detailed information.

[root@X003 bin]# ss -e
State      Recv-Q Send-Q      Local Address:Port          Peer Address:Port
ESTAB      0      52         192.168.56.113:ssh           192.168.56.1:65433    timer:(on,354ms,0) ino:74064 sk:ffff88002c8d5440
ESTAB      0      0          192.168.56.113:ssh           192.168.56.1:65417    timer:(keepalive,76min,0) ino:73782 sk:ffff8800299e4d80
ESTAB      0      0          192.168.56.113:ssh           192.168.56.1:65418    timer:(keepalive,76min,0) ino:73915 sk:ffff8800299e4700

Linux tool - screen

when you have an long running or important task, sometimes you have to pray that the tty session won't be timed out during the task. Now you can use a powerful Linux tool called screen to help you.

1. start an screen session:

stack@openstack:~$ screen -S mytest

2. run your test program in my test session:

stack@openstack:~$ cd /var/tmp/

stack@openstack:/var/tmp$ cat test.sh

#!/bin/bash

while true

do

echo "hello world"

sleep 1

done

the program will keep printing out 'hello world'

3. you can use CRTL+a d to exit the session

[detached from 2898.mytest]

4. quit the putty session

5．use putty to relogin the session and use 'screen –ls' to check the running sessions

stack@openstack:~$ screen -ls

There is a screen on:

        2898.mytest     (08/06/14 20:41:59)     (Detached)

1 Socket in /var/run/screen/S-stack.

6. use screen –r mytest back to session

stack@openstack:~$ screen -r mytest

hello world

hello world

hello world

hello world

hello world

hello world

hello world

man sections

'man' (manual) is the most useful tool for sysadmin. We can use man to understand the command, configuration file format, etc. but man has different sections. they are:

    1      User Commands
    2      System Calls
    3      C Library Functions
    4      Devices and Special Files
    5      File Formats and Conventions
    6      Games et. Al.
    7      Miscellanea
    8      System Administration tools and Deamons

we can use command whatis to check how many sections a command will be in
as

root@ip-172-31-7-84:~# whatis time

time (7)             - overview of time and timers

time (1)             - run programs and summarize system resource usage

time (2)             - get time in seconds

then you can use:

man 7 time
man 1 time
man 2 time

to check different means of time in linux

File Magic Number

In many OSes, different type of files have different starting bytes. The typical starting numbers are called magic number. Usually the command ‘file’ checks the magic number to decide what’s the file type

Examples of ‘file’ command:

[ec2-user@ip-172-31-16-100 webchecker]$ file dnstest.py
dnstest.py: a /usr/bin/python script text executable
[ec2-user@ip-172-31-16-100 webchecker]$ file /bin/ls
/bin/ls: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped
[ec2-user@ip-172-31-16-100 webchecker]$ file hostname.conf
hostname.conf: ASCII text

when we write the script, either with python, perl or bash. We actually set the magic number for the file. The script may start with a "shebang" (#!, 23 21) followed by the path to an interpreter,

we may use readelf command to get the magic number

[ec2-user@ip-172-31-16-100 webchecker]$ readelf -h /usr/bin/python
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x400620
  Start of program headers:          64 (bytes into file)
  Start of section headers:          7048 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         8
  Size of section headers:           64 (bytes)
  Number of section headers:         31

  Section header string table index: 30
  

Advanced linux keyborad Skills

Sysadmins use terminal or virtual terminal to control the Linux Systems. It is very usually for the system admin to master some terminal skills.

Ctrl-a	Move the cursor to the beginning of the line
Ctrl-e	Move the cursor to the end of the line
Ctrl-f	Move forward the cursor one character ahead
Ctrl-b	Move backward the cursor one character behind
Alt-f	Move forward the cursor one word ahead
Alt-b	Move backward the cursor one word behind
Ctrl-d	Clear the screen and keep the command
Alt-l	Change all the letters lower case from cursor to end of the word
Alt-u	Change all the letters upper case from cursor to end of the word
Ctrl-k	cut the command line from cursor to end of the line
Ctrl-u	cut the command line from cursor to beginning of the line
Alt-d	Cut the word from cursor to end of word
Alt-Backspace	Cut the word from cursor to beginning of word
Ctrl-y	Paste
Alt-?	Same as [tab][tab], list all possible command
history command	List all the history command
Ctrl-r	Search in the history command
!!	Execute the last command
!number	Execute the number command

Practice is the most efficient way to master the commands

Program compile and link in linux.

Program compile and link in linux.

Most of system software provides the source code and we can use compile to build it into the executable binaries. This blog will introduce what is inside the compilation process.

Compile tools

On linux platform, gcc is the most wide used tool for c, c++ and even java compiler.

You will use

#which gcc
to determine if gcc has been installed

Single file compilation

here is a single c program:

#include <stdio.h> 
void main(void) 
{ 
  printf("hello world!"); 
} 

now we can compile the program and run it

[root@X001 cprogram]# ls
hello.c
[root@X001 cprogram]# gcc hello.c
[root@X001 cprogram]# ls
a.out  hello.c
[root@X001 cprogram]# ./a.out
hello world!

or you may use 'gcc -o outputfile sourcefile' to assign an output executable file name

multiple files compilation

if we have more than one single source file, which is the most case in application development. source code A will refer to other files, when one of the files is changed,  do we need to re-compile all of the related files?
the answer is no.
we have two files

File:a.c 
#include <stdio.h> 
int main () 
{ 
    printf("this is from first file\n");
    method(); 
} 

File:b.c 
#include <stdio.h> 
void method(void) 
{ 
    printf("this is from second file:\n"); 
}

now we will do
1. compile the individual source files into object files
2. link the object files into binary

[root@X001 cprogram]# ll
total 8
-rw-r--r--. 1 root root 95 Apr 14 14:11 a.c
-rw-r--r--. 1 root root 87 Apr 14 14:05 b.c
======compile the files here ===========
[root@X001 cprogram]# gcc -c a.c b.c
[root@X001 cprogram]# ll
total 16
-rw-r--r--. 1 root root   95 Apr 14 14:11 a.c
-rw-r--r--. 1 root root 1568 Apr 14 14:11 a.o
-rw-r--r--. 1 root root   87 Apr 14 14:05 b.c
-rw-r--r--. 1 root root 1504 Apr 14 14:11 b.o
=======link the files here ==============
[root@X001 cprogram]# gcc -o result a.o b.o

[root@X001 cprogram]# ./result
this is from first file
this is from second file

if we change the print statement of the b.c to "this is the updated file" .
we only need re-compile the b.o file and then regenerate the executable binary. 

link library

none of the commercial software is developed from scratch. We will use some of common software component called library.  for example, you may use mathmatics library to get the sin value for PI. 
 

Linux file types

Linux supports multiple file types such as normal files, directory, socket files etc. we can use command ‘ll’ or ‘li –al ’ to list the file and check the file type by the first character of the output.

In general, linux system has the below file types:

-     normal files
d     directory file
l     link (symlink) file
s     socket file
p     pipe file
b     block file
c     character file

normal file. (marked as ‘-’)

Most of the files in linux system are normal files, it includes normal text file, library files, zip files, executable binaries

Directory: (marked as ‘d’)

Directory is a special kind of the file, it contains can contain other kind of files including directories.

Block device file(marked as ‘b’)

Block files are in /dev/ directory, they are some kind of presuedo files. The file can be visited randomly such as disk, usd stick.

Character device file (marked as ‘c’)

Character files are in /dev/ directory as well, the file has to be visited by sequence. It contains device such as mouse, keyboard, serial ports, console tty.

Both character and block files are called device file.

Socket file (marked as ‘s’)

Socket file is used for process communication. Note, it is not the network socket file, it is the unix socket file.

Link file (marked as ‘l’)

The file is linked file (symlink) and created by ‘ln -s’ command. 

Linux command tips - updating

how to view the process priority

ps -lef  --- l means long mode
you can view the priority and nice value

S   UID   PID  PPID  C PRI  NI   RSS    SZ WCHAN  TTY          TIME CMD
S     0     1     0  0  80   0  1540  4839 poll_s ?        00:00:01 init
S     0     2     0  0  80   0     0     0 kthrea ?        00:00:00 kthreadd
S     0     3     2  0 -40   -     0     0 migrat ?        00:00:00 migration/0
S     0     4     2  0  80   0     0     0 ksofti ?        00:00:00 ksoftirqd/0

how to list only just the first level of directories in a directory

1. find . -maxdepth -type d  (it will include the directory itself)
2. ls -ld */

how to use BASH to read a file line by line

[root@X001 tmp]# while read line
> do
> echo $line
> done < test.txt
this is a test file from rafa
the file is very good ha ha
could you give me the file here
[root@X001 tmp]# cat test.txt
this is a test file from rafa
the file is very good ha ha
could you give me the file here

linux hash command:

Linux uses hash table to speed up the commands finding process. when linux found a command, it stores it in the hash table so next time when it needs to check the command, it can get from hash table. the cost of looking via hash table is O(1)

linux command line {} expand:

when we use { } in bash, it will be expanded by bash.

for example: 

mkdir -pv /var/tmp/{a/b,c} = mkdir -pv /var/tmp/a/b /var/tmp/c

How to recover a file deleted by mistake when it is still opened in the process:

Find the file by ls -l /proc/<pid>/fd/, then copy it by cp /proc/<pid>/fd/<fd> ~/myfile.txt 

BASH command line intercept and procession

Bash command is is the interface for sysadmin to control the bash. It is very important for sysadmin to understand how BASH intercepts the command. Here is the brief introduction how it is working.

• split the command into tokens using delimiters.The delimiters include SPACE, TAB, NEWLINE, ; , (, ), <,  >, |, &
• build the command stack (complicated process, not discussed here)
• check if the first token of command is an alias, if it is, it will replace the alias with the value.
• expand the {}, eg. It will expand a{a,b} to aa and ab
• if the token is started with ~, it will replace with the home directory
• any expression started with $, it will replace it with expression value.
• execute the command in between ``
• calculate the $((expression)) and replace it with result
• wildcast expansion. Such as * ? , [ / ]
• find the exact commands (buildin, $PATH)
• IO redirection

Here is an example.

echo ~/i* $PWD `echo Yahoo Hadop` $((21*20)) > output

step 1. split the command into tokens

token[1] = echo

token[2] = ~/i*

token[3] = $PWD

token[4] = `echo Yahoo Hadop`

token[5] = $((21*20))

> output are not the tokens, they will be process in the IO rediretion.

Step 2,3,4 skipped

Step 5. replace ~ with /root. So the command is looking like

echo /root/i* $PWD `echo Yahoo Hadop` $((21*20))

step 6. replace $PWD with the current path for example it is:

              echo /root/i* /root `echo Yahoo Hadop` $((21*20))

step 7. excute the command in ``. so it would look like (iteriter process)

        echo /root/i* /root Yahoo Hadop $((21*20))

step 8. calculate the value in $(()). so it would look like

        echo /root/i* /root Yahoo Hadop 420

step 9: expand the wildcast.(take example)

        echo /root/indirect.sh /root/install.log /root/install.log.syslog /root Yahoo Hadop 420

now the BASH is ready to execute the commands as echo is a buidin command

        it will redirect the output to ouput file

Linux trace introduction- 1 strace command

Linux provides system admin quite a few useful tools for troubleshooting. Strace is one of the tools which can provide the details of syscalls including parameters, values, and the consumed time.

Strace is a very complicated command with quite a few options; we need to understand some common options for daily usage:

-c -- count time, calls, and errors for each syscall and report summary

-f -- follow forks, -ff -- with output into separate files

-r -- print relative timestamp, -t -- absolute timestamp, -tt -- with usecs

-e expr -- a qualifying expression: option=[!]all or option=[!]val1[,val2]...

   options: trace, abbrev, verbose, raw, signal, read, or write

-o file -- send trace output to FILE instead of stderr

-p pid -- trace process with process id PID, may be repeated

Some examples

Try to ls a non-existing file

[root@X001 tmp]# strace ls notexisting

execve("/bin/ls", ["ls", "notexisting"], [/* 29 vars */]) = 0

brk(0)                                  = 0x1b7b000

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5b87f51000

access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)

open("/etc/ld.so.cache", O_RDONLY)      = 3

fstat(3, {st_mode=S_IFREG|0644, st_size=38923, ...}) = 0

mmap(NULL, 38923, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f5b87f47000

close(3)                                = 0

-----omitted-----

ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0

ioctl(1, TIOCGWINSZ, {ws_row=63, ws_col=237, ws_xpixel=0, ws_ypixel=0}) = 0

stat("notexisting", 0x1b7c0e0)          = -1 ENOENT (No such file or directory)

lstat("notexisting", 0x1b7c0e0)         = -1 ENOENT (No such file or directory)

open("/usr/share/locale/locale.alias", O_RDONLY) = 3

fstat(3, {st_mode=S_IFREG|0644, st_size=2512, ...}) = 0

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5b87f50000

read(3, "# Locale name alias data base.\n#"..., 4096) = 2512

read(3, "", 4096)                       = 0

close(3)                                = 0

exit_group(2)                           = ?

try to open an non-listening port only with network syscall enabled

[root@X001 tmp]# strace -e trace=network telnet localhost 9999

socket(PF_NETLINK, SOCK_RAW, 0)         = 3

bind(3, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0

getsockname(3, {sa_family=AF_NETLINK, pid=2395, groups=00000000}, [12]) = 0

sendto(3, "\24\0\0\0\26\0\1\3\342\346\vS\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20

recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"0\0\0\0\24\0\2\0\342\346\vS[\t\0\0\2\10\200\376\1\0\0\0\10\0\1\0\177\0\0\1"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 108

recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"@\0\0\0\24\0\2\0\342\346\vS[\t\0\0\n\200\200\376\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 128

recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\0\342\346\vS[\t\0\0\0\0\0\0\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 20

socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3

connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)

socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3

connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)

socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 3

connect(3, {sa_family=AF_INET, sin_port=htons(9999), sin_addr=inet_addr("127.0.0.1")}, 16) = 0

getsockname(3, {sa_family=AF_INET, sin_port=htons(33896), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0

socket(PF_INET6, SOCK_DGRAM, IPPROTO_IP) = 3

connect(3, {sa_family=AF_INET6, sin6_port=htons(9999), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0

getsockname(3, {sa_family=AF_INET6, sin6_port=htons(57576), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0

Trying ::1...

socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 3

connect(3, {sa_family=AF_INET6, sin6_port=htons(9999), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 ECONNREFUSED (Connection refused)

telnet: connect to address ::1: Connection refused

Trying 127.0.0.1...

socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3

setsockopt(3, SOL_IP, IP_TOS, [16], 4)  = 0

connect(3, {sa_family=AF_INET, sin_port=htons(9999), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 ECONNREFUSED (Connection refused)

telnet: connect to address 127.0.0.1: Connection refused

[root@X001 tmp]#

try to get the summary of the syscalls
[root@X001 tmp]# strace -c -e trace=network telnet localhost 9999
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.022996        3285         7           socket
  0.00    0.000000           0         6         4 connect
  0.00    0.000000           0         1           sendto
  0.00    0.000000           0         3           recvmsg
  0.00    0.000000           0         1           bind
  0.00    0.000000           0         3           getsockname
  0.00    0.000000           0         1           setsockopt
------ ----------- ----------- --------- --------- ----------------
100.00    0.022996                    22         4 total
[root@X001 tmp]#
 

to understand the output of strace, we need to have a brief idea about the linux internal and syscalls

linux interview preparation - linux operation

Explain different runlevels:

How many ways of restarting linux:

How many file types:

List the pseudo files in /dev/

How to change the hostname of the linux server

Where is the DNS server configured.

How to review the routing table of the linux box:

How to add a route entry in the linux box

What does dynamic link, how to view the dynamic library.

How to install software by source file: what does these steps mean?

What is initrd file in /boot/ does?

How to check if the TCP port is open ? how to find which process is opening it.

When create a directory, how many harklink you will get?

Why can’t make hark links for directories

Where is the network interface eth0 configuration file and where is the DNS configuration file.

How to find the files containing ‘abc’ and replace with ‘edf’

linux performance monitoring

Linux Performance monitoring

Performance monitoring is a hard job for system admin. It depends on the application running on the box and also the experience of the sys admin. Here are some basic guide for performance measuring.

	Good	warning	Bad
CPU	user+sys < 70%	70%<user+sys<90%	user+sys > 70%
Memory	no swap	per CPU < 10 pages/s	lots of SI/SO
DISK	iowait < 20%	20% < iowait < 50%	iowait > 50%

Here we will show about the performance measuring tools and the output.

1.       vmstat: commands: vmstat [interval] [counters]

vmstat is a useful tool to analyse the performance.

Procs: process number:

                r: runnable process but waiting for CPU resource. If there are lots of r, it usually means the cpu resource is not enough (experience value < 2/cpu)

                b: blocked processes waiting for other resource (usually memory or IO)

Memory: memory information:

                swpd:  swaped memory measured by kb

                free:   usable memory

                buff:  buffered memory

                cache: cached memory

swap: swap information

                si: swap in. from disk to memory

                so: swap out. From memory to disk

IO: block IO information:

                Bi: block in. read from disk to memory

                Bo: block out. Write the memory information to disk.

System: 

                In: interruption numbers

                cs: content switch numbers

                there shouldn’t be many Interruptions and content switching for a stable system.

CPU: cpu usage by percentage

                Us: user level, sy: system level, id: idle. Wa: wait for IO.

free command:

Free command is very simple. Just need to understand the

Buffers: blocked IO related memory

Cached: file content cached.

Uptime: 

Showing the current time,  system running time since the last reboot and the current users. It will show the system load for the last min, 5 mins and 15 mins. It should consider the cpu cores when measuring the load.

iostat: check the disk IO status

The most important value is the %iowait: it is usually beblow 20%

%util. the lower the better.