netstat
The status of the input and output through a network interface is assessed via the command netstat. This command can provide the complete information on how the network interface is performing, down to even socket level. Here is an example:

# netstat
Active Internet connections  (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address  State     
tcp        0      0 prolin1:31027 prolin1:5500     TIME_WAIT
tcp        4      0 prolin1l:1521 applin1:40205    ESTABLISHED
tcp        0      0 prolin1l:1522 prolin1:39957    ESTABLISHED
tcp        0      0 prolin1l:3938 prolin1:31017    TIME_WAIT
tcp        0      0 prolin1l:1521 prolin1:21545    ESTABLISHED
… and so on …
The above output goes on to show all the open sockets. In very simplistic terms, a socket is akin to a connection between two processes. [Please note: strictly speaking, “sockets” and “connections” are technically different. A socket could exist without a connection. However, a discussion on sockets and connections is beyond the scope of this article. Therefore I have merely presented the concept in an easy-to-understand manner.] Naturally, a connection has to have a source and a destination, called local and remote address. The end points could be on the same server; or on different servers.

In many cases, the programs connect to the same server. For instance, if two processes communicate among each other, the local and remote addresses will be the same, as you can see in the first line – the local and remote addresses are both the sever “prolin1”. However, the processes communicate over a port, which will be different. This port is shown next to the host name after the “:” (colon) mark. The user program sends the data to be sent across the socket to a queue and the receiver reads from a queue at the remote end. Here are the columns of the output:

The leftmost column “Proto” shows the type of the connection – tcp in this case.
The column Recv-Q shows the bytes of data in the queue to be sent to the user program that established the connection. This value should be as close to 0 as possible. In busy servers this value will be more than 0 but shouldn’t be very high. A higher number may not mean much, unless you see a large number in Send-Q column, described below.
The Send-Q column denotes the bytes in the queue to be sent to the remote program, i.e. the remote program has not yet acknowledged receiving it. This should be close to 0. A large number may indicate a network bottleneck.
Local Address is source of the connection and the port number of the program.
Foreign Address is the destination host and port number. In the first line, both the source and destination are on the same host: prolin1. The connection is simply waiting. The second line shows and established connection between port 1521 of proiln1 going to the port 40205 of the host applin1. It’s most likely an Oracle connection coming from the client applin1 to the database server prolin1. The Oracle listener on prolin1 runs on port 1521; so the port of the source is 1521. In this connection, the server is sending the requested data to the client.
The column State shows the status of the connection. Here are some common values.
ESTABLISHED – that the connection has been established. It does not mean that any data is flowing between the end points; merely that the end points have talked to each other.
CLOSED – the connection has been closed, i.e. not used now.
TIME_WAIT – the connection is being closed but there are still packets in the network that are being handled.
CLOSE_WAIT – the remote end has shutdown and has asked to close the connection.
Well, from the foreign and local addresses, especially from the port numbers, you can probably guess that the connections are Oracle related, but won’t it be nice to know that for sure? Of course. The -p option shows the process information as well:

#  netstat -p
Proto  Recv-Q Send-Q Local Address Foreign Address State       PID/Program name  
tcp        0       0 prolin1:1521   prolin1:33303   ESTABLISHED  1327/oraclePROPRD1 
tcp        0       0 prolin1:1521   applin1:51324   ESTABLISHED 13827/oraclePROPRD1
tcp        0       0 prolin1:1521   prolin1:33298   ESTABLISHED  32695/tnslsnr      
tcp        0       0 prolin1:1521   prolin1:32544   ESTABLISHED  15251/oracle+ASM   
tcp        0       0 prolin1:1521   prolin1:33331   ESTABLISHED  32695/tnslsnr    
This clearly shows the process IP and the process name in the last column, which confirms it to be Oracle server processes, listener process, and ASM server processes.

The netstat command can have various options and parameters. Here are some key ones:

To find out the network statistics for various interfaces, use the -i option.

#  netstat -i
Kernel  Interface table
Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0       1500   0  6860659      0      0      0  2055833      0      0      0 BMRU
eth8       1500   0     2345      0      0      0      833      0      0      0 BMRU
lo        16436   0 14449079      0      0      0 14449079      0      0      0 LRU
This shows the different interfaces present in the server (eth0, eth8, etc.) and the metrics associated with the interface.

RX-OK shows the number of packets successfully sent (for this interface)
RX-ERR shows number of errors
RX-DRP shows packets dropped and had to be re-sent (either successfully or not)
RX-OVR shows packets overrun
The next sets of columns (TX-OK, TX-ERR, etc.) show the corresponding stats for send data.

Flg column is a composite value of the property of the interface. Each letter indicates a specific property being present. Here is an explanation of the letters.

B – Broadcast
M – Multicast
R – Running
U – Up
O – ARP Off
P – Point to Point Connection
L – Loopback
m – Master
s – Slave

You can use the –interface (note: there are two hyphens, not one) option to display the same for a specific interface.

# netstat –interface=eth0
Kernel Interface table
Iface       MTU Met    RX-OK  RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR  TX-DRP TX-OVR Flg
eth0       1500   0 277903459      0      0      0 170897632      0      0      0 BMsRU
Needless to say, the output is wide and is a little difficult to grasp at one shot. If you are comparing across interfaces, it makes sense to have a tabular output. If you want to examine the values in a more readable format, use the -e option to produce an extended output:

# netstat -i -e
Kernel Interface table
eth0      Link encap:Ethernet   HWaddr 00:13:72:CC:EB:00 
          inet addr:10.14.106.0   Bcast:10.14.107.255   Mask:255.255.252.0
          inet6 addr: fe80::213:72ff:fecc:eb00/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6861068 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2055956 errors:0 dropped:0 overruns:0  carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3574788558 (3.3 GiB)  TX bytes:401608995 (383.0 MiB)
          Interrupt:169
Does the output seem familiar? It should; it’s the same as the output of the ifconfig.

If you’d rather see the output showing IP addresses instead of host names, use the -n option.

The -s option shows the summary statistics of each protocol, rather than showing the details of each connection. This can be combined with the protocol specific flag. For instance -u shows the stats related to the UDP protocol.

# netstat -s -u
Udp:
    12764104 packets received
    600849 packets to unknown port received.
    0 packet receive errors
    13455783 packets sent
Similarly, to see the stats for tcp, use -t and for raw, -r.

One of the really useful options is the display of the routing table, the -r option.

#  netstat -r
Kernel  IP routing table
Destination     Gateway         Genmask          Flags   MSS Window  irtt Iface
10.20.191.0     *               255.255.255.128  U         0 0          0 bond0
172.22.13.0     *               255.255.255.0    U         0 0          0 eth9
169.254.0.0     *               255.255.0.0      U         0 0          0 eth9
default         10.20.191.1     0.0.0.0          UG        0 0          0 bond0
The second column of netstat output–Gateway–shows the gateway to which the routing entry points. If no gateway is used, an asterisk is printed instead. The third column–Genmask–shows the “generality” of the route, i.e., the network mask for this route. When given an IP address to find a suitable route for, the kernel steps through each of the routing table entries, taking the bitwise AND of the address and the netmask before comparing it to the target of the route.

The fourth column, Flags, displays the following flags that describe the route:

G means the route uses a gateway.
U means the interface to be used is up (available).
H means only a single host can be reached through the route. For example, this is the case for the loopback entry 127.0.0.1.
D means this route is dynamically created.
! means the route is a reject route and data will be dropped.
The next three columns show the MSS, Window, and irtt that will be applied to TCP connections established via this route.

MSS stands for Maximum Segment Size – the size of the largest datagram for transmission via this route.
Window is the maximum amount of data the system will accept in a single burst from a remote host for this route.
irtt stands for Initial Round Trip Time. It’s a little complicated to explain. Let me explain that separately.
The TCP protocol has a built-in reliability check. If a data packet fails during transmission, it’s re-transmitted. The protocol keeps track of how long the takes for the data to reach the destination and acknowledgement to be received. If the acknowledgement does not come within that timeframe, the packet is retransmitted. The amount of time the protocol has to wait before re-transmitting is set for the interface once (which can be changed) and that value is known as initial round trip time. A value of 0 means the default value is used.

Finally, the last field displays the network interface that this route will use.

 
(Extracted from oracle technet notes author Arup Nanda)