Cloud DevOps Admin Guide

PowerPath Powermt Commands - EMC

Below are the 10 major commands to check the POWER PATH config on unix servers.
Please follow the below commands

1.powermt display ====>Display High Level HBA I/O Paths
2.powermt display dev=emcpowera ===>Display for specific LUN
3.powermt display dev=all ====> Display All Attached LUNs
4.powermt check_registration ===> Display PowerPath Registration Key / Status
5.powermt display options ===> Display EMC PowerPath Options
6.powermt display hba_mode ====> Display PowerPath HBA Mode
7.powermt display paths – Display available I/O Paths.
8.powermt displays port_mode ===>Display Port Status
9.powermt version ====> Display EMC PowerPath Version
10.powermt check ===>Check the I/O Paths

1. #powermt display ===>Display High Level HBA I/O Paths

Example output:

Symmetrix logical device count=212
CLARiiON logical device count=0
Hitachi logical device count=0
Invista logical device count=0
HP xp logical device count=0
Ess logical device count=0
HP HSx logical device count=0
==============================================================================
----- Host Bus Adapters --------- ------ I/O Paths ----- ------ Stats ------
### HW Path                       Summary   Total   Dead IO/Sec Q-IOs Errors
==============================================================================
   3 0/4/0/0/0/1                   optimal     424      0       -     0    848
   5 0/5/0/0/0/1                   optimal     424      0       -     0    848

2. #powermt display dev=emcpowera ===>Display specific LUN

When there are multiple LUNs connected to a server, you might want to view information about a specific LUN by providing the logical name of the LUN as shown below.

3.#powermt display dev=all ====> Display All Attached LUNs

Mostly we used to run this command powermt, which will display all the attached logical devices to the server.

Pseudo name=disk915
Symmetrix ID=000290103691
Logical device ID=06B8
state=alive; policy=SymmOpt; priority=0; queued-IOs=0;
==============================================================================
--------------- Host ---------------   - Stor -   -- I/O Path -- -- Stats ---
### HW Path               I/O Paths    Interf.   Mode    State   Q-IOs Errors
==============================================================================
   3 0/4/0/0/0/1.0x5006048c52a862e7.0x40a6000000000000 c14t4d6   FA 8cB   active alive       0      2
   3 0/4/0/0/0/1.0x5006048c52a862f7.0x40a6000000000000 c15t4d6   FA 8dB   active alive       0      2
   5 0/5/0/0/0/1.0x5006048c52a862e8.0x40a6000000000000 c16t4d6   FA 9cB   active alive       0      2
   5 0/5/0/0/0/1.0x5006048c52a862f8.0x40a6000000000000 c17t4d6   FA 9dB   active alive       0      2

Pseudo name=disk988
Symmetrix ID=000290103691
Logical device ID=074B
state=alive; policy=SymmOpt; priority=0; queued-IOs=0;
==============================================================================
--------------- Host ---------------   - Stor -   -- I/O Path -- -- Stats ---
### HW Path               I/O Paths    Interf.   Mode    State   Q-IOs Errors
==============================================================================
   5 0/5/0/0/0/1.0x5006048c52a862e8.0x40dc000000000000 c16t11d4 FA 9cB   active alive       0      2
   3 0/4/0/0/0/1.0x5006048c52a862e7.0x40dc000000000000 c14t11d4 FA 8cB   active alive       0      2
   3 0/4/0/0/0/1.0x5006048c52a862f7.0x40ce000000000000 c15t9d6   FA 8dB   active alive       0      2
   5 0/5/0/0/0/1.0x5006048c52a862f8.0x40ce000000000000 c17t9d6   FA 9dB   active alive       0      2

Details:

a. Pseudo name=emcpowera – The device name that can be used by the server. For example,
/dev/emcpowera.
b. CLARiiON ID=AAA00000000000 [dev-server] - EMC CLARiiON CX3 serial number and
the server name.
c. Logical device ID=11111111 [LUN 1] – LUN number. For example, LUN 1.
d. state=alive; policy=CLAROpt; – This displays that this particular LUN is valid and using
the CLAROpt policy.
e. Owner: default=SP B, current=SP B – This indicates that the default (and current) owner for
this LUN is storage processor SP B.

4. powermt check_registration – Display PowerPath Registration Key / Status

If you’ve lost the PowerPath registration key that you’ve used during the EMC PowerPath installation, you can retrieve it using the following command.

# powermt check_registration
Key AAAA-BBBB-CCCC-DDDD-EEEE-FFFF
Product: PowerPath
Capabilities: All

5. #powermt display options ===> Display EMC PowerPath Options

Displays the high level EMC SAN array options.

6.#powermt display hba_mode ====> Display PowerPath HBA Mode

This is similar to #1, but displays whether hba is enabled or not, as shown in the last column of the output.

Examble output:

Symmetrix logical device count=212
CLARiiON logical device count=0
Hitachi logical device count=0
Invista logical device count=0
HP xp logical device count=0
Ess logical device count=0
HP HSx logical device count=0
==============================================================================
----- Host Bus Adapters --------- ------ I/O Paths ----- Stats
### HW Path                       Summary   Total   Dead Q-IOs Mode
==============================================================================
   3 0/4/0/0/0/1                   optimal     424      0     0 Enabled
   5 0/5/0/0/0/1                   optimal     424      0     0 Enabled

7.powermt display paths ===> Display available I/O Paths.

This displays all available path for your SAN device.

8.powermt displays port_mode ===>Display Port Status

Displays the status of the individual ports on the HBA. i.e Whether the port is enabled or not.

9.powermt version ====> Display EMC PowerPath Version

How to identify the version number of EMC PowerPath software?

10.powermt check ===>Check the I/O Paths

If we made changes to the HBA’s, or I/O paths, then run the powermt check, to take appropriate action. For example,
if you manually removed an I/O path, check command will detect a dead path and remove it from the EMC path list.

Most commonly used XSCF commands- solaris

The eXtended System Control Facility Unit (XSCFU) is a service processor that operates and administrates both midrange servers. The XSCFU diagnoses and starts the entire server, configures domains, offers dynamic reconfiguration, as well as detects and notifies various failures. The XSCFU enables standard control and monitoring function through network. Using this function enables starts, settings, and operation managements of the server from remote locations.

Connecting a domain

Check for the available domains on the servers :

XSCF> showdomainstatus -a
DID Domain    Status
00            Running
01            Running
02            Running
03              -

Now to connect to the domain with ID 00 :

XSCF> console -d 0

Power/reboot/reset/panic commands

Poweron all domains

XSCF> poweron -a

Poweron only domain 0

XSCF> poweron -d 0

Poweroff all domains

XSCF> poweroff -a

Poweroff domain 0

XSCF> poweroff -d 0

Reboot XSCF

XSCF> rebootxscf

The 3 modes to reset a domain are :

por: To reset the domain
panic: To panic the domain
xir: To reset the CPU in domain

XSCF> reset -d 0 por
XSCF> reset -d 0 panic
XSCF> reset -d 0 xir

Send a break signal to a domain (with ID 0)

XSCF> sendbreak -d 0

User Administration

Creating a New user

XSCF> adduser -u 345 admin

Delete a user

XSCF> deleteuser admin

Disable a user

XSCF> disableuser admin

Enable a user

XSCF> enableuser admin

Display user account information

XSCF> showuser -a

Set or change a User (admin) password

XSCF> password admin

Network related commands

Display complete network configuration

XSCF> shownetwork  -a

Set IP address for XSCF-LAN#0 in the XSCFU#0

XSCF> setnetwork xscf#0-lan#0 -m 255.255.255.0 192.168.1.10

We must apply the network settings and reboot the XSCF in order to the network settings to take effect :

XSCF> applynetwork
XSCF> rebootxscf

Setup 2 NTP servers with IP 192.168.1.10 and 192.168.1.20

XSCF> setntp 192.168.1.10 192.168.1.20
Please reset the XSCF by rebootxscf to reflect the ntp settings.

To delete a NTP server 192.168.1.20

XSCF> setntp -c del 192.168.1.20
  Please reset the XSCF by rebootxscf to reflect the ntp settings.

Hardware Related Commands

Show field replaceable units(FRU)

XSCF> showhardconf

Display degraded units

XSCF> showstatus

To display configured devices on XSBs

XSCF> showdevices

Fault Management configuration tool

To view fault management logs

XSCF> fmdump -v
TIME                    UUID                                    MSG-ID
Nov 30 20:44:55.1283    9f773e33-e46f-466c-be86-fd3fcc449935   FMD-8000-0W
   100%  defect.sunos.fmd.nosub
   .....

Display Very Verbose Event Detail for a UUID

XSCF> fmdump -e -V -u 5f88d7d5-a107-4435-99c9-7c59479d22ed TIME CLASS

Logs

show the logs

XSCF> showlogs -v
XSCF> showlogs error
XSCF> showlogs power

Snapshots

We can take a snapshot of M series servers XSCF either on a remote server or on a USB device locally connected. To take a snaphot on a remote system 192.168.1.10 (in /var/tmp directory) by using root user credentials :

XSCF> snapshot -L F -t root@192.168.1.10:/var/tmp

To take the snapshot on a Local USB device connected to the server :

XSCF> snapshot -L F -d usb0

Connect DVD Device to a domain

Run the following from the XSCF to connect the DAT & DVD to the needed port based on the domain(s) configuration.

XSCF> cfgdevice -q -y -c attach -p 0-0

Confirm that the device is attached to domain and is enabled :

XSCF> cfgdevice -l

From the OS prompt configure the newly added DVD device and restart the volmgmt service:

# cfgadm -c configure # c0
# /etc/init.d/volmgt start

Hostname

Display current hostanems of XSCF units

XSCF> showhostname -a
xscf#0: hostname01.example.com 
xscf#1: hostname02.example.com

To set hostname and DNS domain name for XSCF doamin respectively

XSCF> sethostname xscf#0 hostname01
XSCF> sethostname -d example.com

Shut Down or Reboot a Solaris System

Normally, the system reboots at power-up or after a system crash. You can reboot a system by using either the init command or the reboot command. The init 6 command asks for stop methods (either SMF or rc.d). Whereas, the reboot command does not, thereby making the reboot command a more reliable way of rebooting a system.

Solaris is usually used as a server operating system. Because of this, you want to make sure that you shut the system down as gracefully as possible to ensure there isn’t any data loss.

For every application that is installed on your server, you should make sure that you have the correct scripts in /etc/rc(x).d to gracefully shut down the service.

Shutdown

You have more than one command option that you can use. The best command is this, executed as root:

shutdown -y -i5 -g0

This will immediately shut the system down. You can also use the older command that still works:

sync;sync;init 5

You can even use:

poweroff

Reboot
If you are trying to reboot the system as opposed to turning it off, you could use:

shutdown -y -i6 -g0

Or:

sync;sync;init 6

Or even:

reboot

So many commands to do the same thing…

PowerHA/HACMP Moving Resource Group (RG) one node to other

In this post, you will be learning the steps for moving a resource group from one node to the other node. The steps as follows:

1) Extending PATH variable with cluster paths

Sometimes cluster paths are not included in default path ,run below command in case if you are not able to run commands directly.

export PATH=$PATH:/usr/es/sbin/cluster:/usr/es/sbin/cluster/utilities:/usr/es/sbin/cluster/sbin:/usr/es/sbin/cluster/cspoc

2) Check the cluster services are up or not in destination node

#clshowsrv -v
Status of the RSCT subsystems used by HACMP:
Subsystem         Group            PID          Status
 topsvcs          topsvcs          278684       active
 grpsvcs          grpsvcs          332026       active
 grpglsm          grpsvcs                       inoperative
 emsvcs           emsvcs           446712       active
 emaixos          emsvcs           294942       active
 ctrmc            rsct             131212       active

Status of the HACMP subsystems:
Subsystem         Group            PID          Status
 clcomdES         clcomdES         204984       active
 clstrmgrES       cluster          86080        active

Status of the optional HACMP subsystems:
Subsystem         Group            PID          Status
 clinfoES         cluster          360702       active

3) Check the availability of resource group

# clRGinfo

-----------------------------------------------------------------------------

Group Name     Type           State      Location

-----------------------------------------------------------------------------

UMRG1            non-concurrent OFFLINE    umhaserv1

                                ONLINE     umhaserv2

#

4) Move the resourcegroup by using below command

==>  clRGmove -g <RG> -n  <node> -m

# clRGmove -g UMRG1 -n umhaserv1 -m

Attempting to move group UMRG1 to node umhaserv1.

Waiting for cluster to process the resource group movement request....

Waiting for the cluster to stabilize..................

Resource group movement successful.

Resource group UMRG1 is online on node umhaserv1.

You can use smitty path also

smitty cl_admin => HACMP Resource Group and Application Management => Move a Resource Group to Another Node / Site

5) Verify the RG movement

# clRGinfo

-----------------------------------------------------------------------------

Group Name     Type           State      Location

-----------------------------------------------------------------------------

UMRG1          non-concurrent   ONLINE     umhaserv1

                                OFFLINE    umhaserv2

#

Manually Install or Upgrade VMware Tools in a Linux Virtual Machine

For Linux virtual machines, you manually install or upgrade VMware Tools by using the command line.

Install the latest version of VMware Tools to enhance the performance of the virtual machine's guest operating system and improve virtual machine management. When you power on a virtual machine, if a new version of VMware Tools is available, you see a notification in the status bar of the guest operating system.

Note
This procedure describes how to use the VMware Tools tar installer to install or upgrade VMware Tools. For virtual machines in a vSphere environment, you can alternatively use VMware Tools operating system specific packages (OSPs) to install and upgrade VMware Tools. With OSPs you can use the native update mechanisms of your operating system to download, install, and manage VMware Tools. For more information, see Operating System Specific Packages for Linux Guest Operating Systems.

Prerequisites

■ Power on the virtual machine.

■ Verify that the guest operating system is running.

■ Because the VMware Tools installer is written in Perl, verify that Perl is installed in the guest operating system.

■ To determine whether you have the latest version of VMware Tools, look on the Summary tab for the virtual machine.

Procedure

1) Select the menu command to mount the VMware Tools virtual disc on the guest operating system.

VMware Product

Menu Command

vSphere Client

Inventory > Virtual Machine > Guest > Install/Upgrade VMware Tools

2) If you are performing an upgrade or reinstallation, in the Install/Upgrade VMware Tools dialog box, select Interactive Tools Installation or Interactive Tools Upgrade and click OK.

The process starts by mounting the VMware Tools virtual disc on the guest operating system.

3) In the virtual machine, log in to the guest operating system as root and open a terminal window.

4) Run the mount command with no arguments to determine whether your Linux distribution automatically mounted the VMware Tools virtual CD-ROM image.

If the CD-ROM device is mounted, the CD-ROM device and its mount point are listed as something like this:

/dev/cdrom on /mnt/cdrom type iso9660 (ro,nosuid,nodev)

5) If the VMware Tools virtual CD-ROM image is not mounted, mount the CD-ROM drive.

a : If a mount point directory does not already exist, create it.

mkdir /mnt/cdrom

Some Linux distributions use different mount point names. For example, on some distributions the mount point is /media/VMware Tools rather than /mnt/cdrom. Modify the command to reflect the conventions that your distribution uses.

b : Mount the CD-ROM drive.

mount /dev/cdrom /mnt/cdrom

Some Linux distributions use different device names or organize the /dev directory differently. If your CD-ROM drive is not /dev/cdrom or if the mount point for a CD-ROM is not /mnt/cdrom, modify the command to reflect the conventions that your distribution uses.

6) Change to a working directory (for example, /tmp).

cd /tmp

7) Delete any previous vmware-tools-distrib directory before you install VMware Tools.

The location of this directory depends on where you placed it during the previous installation. Often this directory is placed in /tmp/vmware-tools-distrib.

8) List the contents of the mount point directory and note the filename of the VMware Tools tar installer.

ls mount-point

9) Uncompress the installer.

tar zxpf /mnt/cdrom/VMwareTools-x.x.x-yyyy.tar.gz
The value x.x.x is the product version number, and yyyy is the build number of the product release.

If you attempt to install a tar installation over an RPM installation, or the reverse, the installer detects the previous installation and must convert the installer database format before continuing.

10) If necessary, unmount the CD-ROM image.

umount /dev/cdrom

If your Linux distribution automatically mounted the CD-ROM, you do not need to unmount the image.

11) Run the installer and configure VMware Tools.

cd vmware-tools-distrib

./vmware-install.pl

Usually, the vmware-config-tools.pl configuration file runs after the installer file finishes running.

12) Respond to the prompts by pressing Enter to accept the default values, if appropriate for your configuration.

13) Follow the instructions at the end of the script.

Depending on the features you use, these instructions can include restarting the X session, restarting networking, logging in again, and starting the VMware User process. You can alternatively reboot the guest operating system to accomplish all these tasks.

The VMware Tools label on the Summary tab changes to OK.

NIC Channel Bonding in Linux

Today I had implemented NIC bonding (bind both NIC so that it works as a single device). Bonding is nothing but Linux kernel feature that allows to aggregate multiple like interfaces (such as eth0, eth1) into a single virtual link such as bond0. The idea is pretty simple get higher data rates and as well as link failover. NIC channel bonding allows multiple network cards to act as one, allowing increased bandwidth and redundancy.

Linux allows binding of multiple network interfaces into a single channel/NIC using special kernel module called bonding. According to official bonding documentation:

The Linux bonding driver provides a method for aggregating multiple network interfaces into a single logical "bonded" interface. The behavior of the bonded interfaces depends upon the mode; generally speaking, modes provide either hot standby or load balancing services. Additionally, link integrity monitoring may be performed.

Step #1: Create a Bond0 Configuration File

Red Hat Enterprise Linux (and its clone such as CentOS) stores network configuration in /etc/sysconfig/network-scripts/ directory. First, you need to create a bond0 config file as follows:

# vi /etc/sysconfig/network-scripts/ifcfg-bond0
Append the following linest:

 
DEVICE=bond0
IPADDR=192.168.1.20
NETWORK=192.168.1.0
NETMASK=255.255.255.0
USERCTL=no
BOOTPROTO=none
ONBOOT=yes

You need to replace IP address with your actual setup. Save and close the file.

Step #2: Modify eth0 and eth1 config files

Open both configuration using a text editor such as vi/vim, and make sure file read as follows for eth0 interface

# vi /etc/sysconfig/network-scripts/ifcfg-eth0
Modify/append directive as follows:

DEVICE=eth0
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none

Open eth1 configuration file using vi text editor, enter:

# vi /etc/sysconfig/network-scripts/ifcfg-eth1
Make sure file read as follows for eth1 interface:

DEVICE=eth1
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none

Save and close the file.

Step # 3: Load bond driver/module

Make sure bonding module is loaded when the channel-bonding interface (bond0) is brought up. You need to modify kernel modules configuration file:

# vi /etc/modprobe.conf
Append following two lines:

alias bond0 bonding
options bond0 mode=balance-alb miimon=100

Save file and exit to shell prompt. A description of the bonding options is available here.

Step # 4: Test configuration

First, load the bonding module, enter:

# modprobe bonding
Restart the networking service in order to bring up bond0 interface, enter:

# service network restart
Make sure everything is working. Type the following to query the current status of Linux kernel bounding driver, enter:

# cat /proc/net/bonding/bond0
Sample outputs:

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 200
Down Delay (ms): 200
Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:0c:29:c6:be:59
Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:0c:29:c6:be:63

To list all network interfaces, enter:

# ifconfig
Sample outputs:

bond0     Link encap:Ethernet  HWaddr 00:0C:29:C6:BE:59
 inet addr:192.168.1.20  Bcast:192.168.1.255  Mask:255.255.255.0
 inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
 UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
 RX packets:2804 errors:0 dropped:0 overruns:0 frame:0
 TX packets:1879 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:0
 RX bytes:250825 (244.9 KiB)  TX bytes:244683 (238.9 KiB)
eth0      Link encap:Ethernet  HWaddr 00:0C:29:C6:BE:59
 inet addr:192.168.1.20  Bcast:192.168.1.255  Mask:255.255.255.0
 inet6 addr: fe80::20c:29ff:fec6:be59/64 Scope:Link
 UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
 RX packets:2809 errors:0 dropped:0 overruns:0 frame:0
 TX packets:1390 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:251161 (245.2 KiB)  TX bytes:180289 (176.0 KiB)
 Interrupt:11 Base address:0x1400
eth1      Link encap:Ethernet  HWaddr 00:0C:29:C6:BE:59
 inet addr:192.168.1.20  Bcast:192.168.1.255  Mask:255.255.255.0
 inet6 addr: fe80::20c:29ff:fec6:be59/64 Scope:Link
 UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
 RX packets:4 errors:0 dropped:0 overruns:0 frame:0
 TX packets:502 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:258 (258.0 b)  TX bytes:66516 (64.9 KiB)
 Interrupt:10 Base address:0x1480

Once the bond is configured it acts like any other Ethernet device. For example, you can configure alias interfaces to handle multiple IP addresses, as shown below.

Create the "ifcfg-bond0:1" and "ifcfg-bond0:2" files in the "/etc/sysconfig/network-scripts" directory with the following contents.

# ifcfg-bond0:1 file contents
DEVICE=bond0:1
BOOTPROTO=none
ONBOOT=yes
NETWORK=192.168.0.0
NETMASK=255.255.255.0
IPADDR=192.168.0.172
USERCTL=no
BONDING_OPTS="mode=1 miimon=100"

# ifcfg-bond0:2 file contents
DEVICE=bond0:2
BOOTPROTO=none
ONBOOT=yes
NETWORK=192.168.0.0
NETMASK=255.255.255.0
IPADDR=192.168.0.173
USERCTL=no
BONDING_OPTS="mode=1 miimon=100"

Notice, the device names and IP addresses differ from the original "ifcfg-bond0" file.

Restart the network service for the changes to take effect.

# service network restart
Shutting down interface bond0:                             [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface bond0:                               [  OK  ]
#

The ifconfig command shows the three IP addresses being handled by the bond.

[root@wls11g-1 network-scripts]# ifconfig
bond0     Link encap:Ethernet  HWaddr 08:00:27:FC:F5:B7  
          inet addr:192.168.0.171  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:14635 errors:0 dropped:306 overruns:0 frame:0
          TX packets:7310 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:17571270 (16.7 MiB)  TX bytes:554475 (541.4 KiB)

bond0:1   Link encap:Ethernet  HWaddr 08:00:27:FC:F5:B7  
          inet addr:192.168.0.172  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1

bond0:2   Link encap:Ethernet  HWaddr 08:00:27:FC:F5:B7  
          inet addr:192.168.0.173  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1

eth0      Link encap:Ethernet  HWaddr 08:00:27:FC:F5:B7  
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:1835 errors:0 dropped:0 overruns:0 frame:0
          TX packets:961 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:189616 (185.1 KiB)  TX bytes:129841 (126.7 KiB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:FC:F5:B7  
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:12800 errors:0 dropped:306 overruns:0 frame:0
          TX packets:6349 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:17381654 (16.5 MiB)  TX bytes:424634 (414.6 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:1541 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1541 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3612733 (3.4 MiB)  TX bytes:3612733 (3.4 MiB)

#

Tcpdump command

Tcpdump is a really great tool for network security analyst, you can dump packets that flows within your networks into file for further analysis. With some filters you can capture only the interested packets, which it reduce the size of saved dump and further reduce loading and processing time of packets analysis.

This post will only covers the fundamental of tcpdump usage, bare in mind tcpdump can do much much more than what I illustrate here.

Lets start with capturing packets based on network interface, ports and protocols. Let assume I wanna capture tcp packets that flow over eth1, port 6881. The dump file with be save as test.pcap.

tcpdump -w test.pcap -i eth1 tcp port 6881

Simple right? What if at the same time I am interested on getting packets on udp port 33210 and 33220?

tcpdump -w test.pcap -i eth1 tcp port 6881 or udp \( 33210 or 33220 \)

‘\’ is an escape symbol for ‘(‘ and ‘)’. Logic OR implies PLUS (+). In plain text is I want to capture tcp packets flows over port 6881 plus udp ports 33210 and 33220.

Careful with ‘and’ in tcpdump filter expression, it means intersection. Thats why I put ‘or’ instead of and within udp port 33210 and 33220. The usage of ‘and’ in tcpdump will be illustrate later.

Ok, how about reading pcap that I saved previously?

tcpdump -nnr test.pcap

The -nn is to tell tcpdump not to resolve DNS on IP and Ports, where r is read.

Adding -tttt to makes the timestamp appears more readable format.

tcpdump -ttttnnr test.pcap

How about capture based on IP ?
You need to tell tcpdump which IP you are interested in? Destination IP? or Source IP ? Let say I wanna sniff on destination IP 10.168.28.22 tcp port 22, how should i write?

tcpdump -w test.pcap dst 10.168.28.22 and tcp port 22

So the ‘and’ makes the intersection of destination IP and port.

By default the sniff size of packets is 96 bytes, you somehow can overload that size by specified with -s.

tcpdump -w test.pcap -s 1550 dst 10.168.28.22 and tcp port 22

Some version of tcpdump allows you to define port range. You can as bellow for capturing packets based on a range of tcp port.

tcpdump tcp portrange 20-24

Bare in mind, the line above I didn’t specified -w which it won’t write to a file but i will just print the captured packets on the screen.

Basic examples of linux netstat command

Netstat

Netstat is a command line utility that can be used to list out all the network (socket) connections on a system. It lists out all the tcp, udp socket connections and the unix socket connections.

Apart from connected sockets it can also list listening sockets that are waiting for incoming connections. So by verifying an open port 80 you can confirm if a web server is running on the system or not. This makes it a very useful tool for network and system administrators.

In this tutorial we shall be checking out few examples of how to use netstat to find information about network connections and open ports on a system.

Here is a quick intro to netstat from the man pages

netstat - Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships

1. List out all connections

The first and most simple command is to list out all the current connections. Simply run the netstat command with the a option.

$ netstat -a

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 enlightened:domain      *:*                     LISTEN     
tcp        0      0 localhost:ipp           *:*                     LISTEN     
tcp        0      0 enlightened.local:54750 li240-5.members.li:http ESTABLISHED
tcp        0      0 enlightened.local:49980 del01s07-in-f14.1:https ESTABLISHED
tcp6       0      0 ip6-localhost:ipp       [::]:*                  LISTEN     
udp        0      0 enlightened:domain      *:*                                
udp        0      0 *:bootpc                *:*                                
udp        0      0 enlightened.local:ntp   *:*                                
udp        0      0 localhost:ntp           *:*                                
udp        0      0 *:ntp                   *:*                                
udp        0      0 *:58570                 *:*                                
udp        0      0 *:mdns                  *:*                                
udp        0      0 *:49459                 *:*                                
udp6       0      0 fe80::216:36ff:fef8:ntp [::]:*                             
udp6       0      0 ip6-localhost:ntp       [::]:*                             
udp6       0      0 [::]:ntp                [::]:*                             
udp6       0      0 [::]:mdns               [::]:*                             
udp6       0      0 [::]:63811              [::]:*                             
udp6       0      0 [::]:54952              [::]:*                             
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  2      [ ACC ]     STREAM     LISTENING     12403    @/tmp/dbus-IDgfj3UGXX
unix  2      [ ACC ]     STREAM     LISTENING     40202    @/dbus-vfs-daemon/socket-6nUC6CCx

The above command shows all connections from different protocols like tcp, udp and unix sockets. However this is not quite useful. Administrators often want to pick out specific connections based on protocols or port numbers for example.

2. List only TCP or UDP connections

To list out only tcp connections use the t options.

$ netstat -at
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 enlightened:domain      *:*                     LISTEN     
tcp        0      0 localhost:ipp           *:*                     LISTEN     
tcp        0      0 enlightened.local:36310 del01s07-in-f24.1:https ESTABLISHED
tcp        0      0 enlightened.local:45038 a96-17-181-10.depl:http ESTABLISHED
tcp        0      0 enlightened.local:37892 ABTS-North-Static-:http ESTABLISHED
.....

Similarly to list out only udp connections use the u option.

$ netstat -au
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
udp        0      0 *:34660                 *:*                                
udp        0      0 enlightened:domain      *:*                                
udp        0      0 *:bootpc                *:*                                
udp        0      0 enlightened.local:ntp   *:*                                
udp        0      0 localhost:ntp           *:*                                
udp        0      0 *:ntp                   *:*                                
udp6       0      0 fe80::216:36ff:fef8:ntp [::]:*                             
udp6       0      0 ip6-localhost:ntp       [::]:*                             
udp6       0      0 [::]:ntp                [::]:*

The above output shows both ipv4 and ipv6 connections.

3. Disable reverse dns lookup for faster output

By default, the netstat command tries to find out the hostname of each ip address in the connection by doing a reverse dns lookup. This slows down the output. If you do not need to know the host name and just the ip address is sufficient then suppress the hostname lookup with the n option.

$ netstat -ant
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 127.0.1.1:53            0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN     
tcp        0      0 192.168.1.2:49058       173.255.230.5:80        ESTABLISHED
tcp        0      0 192.168.1.2:33324       173.194.36.117:443      ESTABLISHED
tcp6       0      0 ::1:631                 :::*                    LISTEN

The above command shows ALL TCP connections with NO dns resolution. Got it ? Good.

4. List out only listening connections

Any network daemon/service keeps an open port to listen for incoming connections. These too are like socket connections and are listed out by netstat. To view only listening ports use the l options.

$ netstat -tnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 127.0.1.1:53            0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN     
tcp6       0      0 ::1:631                 :::*                    LISTEN

Now we can see only listening tcp ports/connections. If you want to see all listening ports, remove the t option. If you want to see only listening udp ports use the u option instead of t.
Make sure to remove the 'a' option, otherwise all connections would get listed and not just the listening connections.

5. Get process name/pid and user id

When viewing the open/listening ports and connections, its often useful to know the process name/pid which has opened that port or connection. For example the Apache httpd server opens port 80. So if you want to check whether any http server is running or not, or which http server is running, apache or nginx, then track down the process name.

The process details are made available by the 'p' option.

~$ sudo netstat -nlpt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.1.1:53            0.0.0.0:*               LISTEN      1144/dnsmasq    
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      661/cupsd       
tcp6       0      0 ::1:631                 :::*                    LISTEN      661/cupsd

When using the p option, netstat must be run with root privileges, otherwise it cannot detect the pids of processes running with root privileges and most services like http and ftp often run with root privileges.

Along with process name/pid its even more useful to get the username/uid owning that particular process. Use the e option along with the p option to get the username too.

$ sudo netstat -ltpe
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
tcp        0      0 enlightened:domain      *:*                     LISTEN      root       11090       1144/dnsmasq    
tcp        0      0 localhost:ipp           *:*                     LISTEN      root       9755        661/cupsd       
tcp6       0      0 ip6-localhost:ipp       [::]:*                  LISTEN      root       9754        661/cupsd

The above example lists out Listening connections of Tcp type with Process information and Extended information.
The extended information contains the username and inode of the process. This is a useful command for network administrators.

Note - If you use the n option with the e option, the uid would be listed and not the username.

6. Print statistics

The netstat command can also print out network statistics like total number of packets received and transmitted by protocol type and so on.

To list out statistics of all packet types

$ netstat -s
Ip:
    32797 total packets received
    0 forwarded
    0 incoming packets discarded
    32795 incoming packets delivered
    29115 requests sent out
    60 outgoing packets dropped
Icmp:
    125 ICMP messages received
    0 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 125
    125 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 125
... OUTPUT TRUNCATED ...

To print out statistics of only select protocols like TCP or UDP use the corresponding options like t and u along with the s option. Simple!

7. Display kernel routing information

The kernel routing information can be printed with the r option. It is the same output as given by the route command. We also use the n option to disable the hostname lookup.

$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0

8. Print network interfaces

The netstat command can also print out the information about the network interfaces. The i option does the task.

$ netstat -i
Kernel Interface table
Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0       1500 0     31611      0      0 0         27503      0      0      0 BMRU
lo        65536 0      2913      0      0 0          2913      0      0      0 LRU

The above output contains information in a very raw format. To get a more human friendly version of the output use the e option along with i.

$ netstat -ie
Kernel Interface table
eth0      Link encap:Ethernet  HWaddr 00:16:36:f8:b2:64  
          inet addr:192.168.1.2  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::216:36ff:fef8:b264/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:31682 errors:0 dropped:0 overruns:0 frame:0
          TX packets:27573 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:29637117 (29.6 MB)  TX bytes:4590583 (4.5 MB)
          Interrupt:18 Memory:da000000-da020000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:2921 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2921 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:305297 (305.2 KB)  TX bytes:305297 (305.2 KB)

The above output is similar to the output shown by the ifconfig command.

9. Get netstat output continuously

Netstat can output connection information continuously with the c option.

$ netstat -ct

The above command will output tcp connections continuously.

10. Display multicast group information

The g option will display the multicast group information for IPv4 and IPv6 protocols.

$ netstat -g
IPv6/IPv4 Group Memberships
Interface       RefCnt Group
--------------- ------ ---------------------
lo              1      all-systems.mcast.net
eth0            1      224.0.0.251
eth0            1      all-systems.mcast.net
lo              1      ip6-allnodes
lo              1      ff01::1
eth0            1      ff02::fb
eth0            1      ff02::1:fff8:b264
eth0            1      ip6-allnodes
eth0            1      ff01::1
wlan0           1      ip6-allnodes
wlan0           1      ff01::1

More examples of netstat command

Okay, we covered the basic examples of netstat command above. Now its time to do some geek stuff with style.

Print active connections

Active socket connections are in "ESTABLISHED" state. So to get all current active connections use netstat with grep as follows

$ netstat -atnp | grep ESTA
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 192.168.1.2:49156       173.255.230.5:80        ESTABLISHED 1691/chrome     
tcp        0      0 192.168.1.2:33324       173.194.36.117:443      ESTABLISHED 1691/chrome

To watch a continous list of active connections, use the watch command along with netstat and grep

$ watch -d -n0 "netstat -atnp | grep ESTA"

Check if a service is running

If you want to check if a server like http,smtp or ntp is running or not, use grep again.

$ sudo netstat -aple | grep ntp
udp        0      0 enlightened.local:ntp   *:*                                 root       17430       1789/ntpd       
udp        0      0 localhost:ntp           *:*                                 root       17429       1789/ntpd       
udp        0      0 *:ntp                   *:*                                 root       17422       1789/ntpd       
udp6       0      0 fe80::216:36ff:fef8:ntp [::]:*                              root       17432       1789/ntpd       
udp6       0      0 ip6-localhost:ntp       [::]:*                              root       17431       1789/ntpd       
udp6       0      0 [::]:ntp                [::]:*                              root       17423       1789/ntpd       
unix  2      [ ]         DGRAM                    17418    1789/ntpd

So we found that ntp server is running. Grep for http or smtp or whatever you are looking for.

Well, that was most of what netstat is used for. If you are looking for more advanced information or want to dig deeper, read up the netstat manual (man netstat).