Cloud DevOps Admin Guide

Most commonly used XSCF commands- solaris

The eXtended System Control Facility Unit (XSCFU) is a service processor that operates and administrates both midrange servers. The XSCFU diagnoses and starts the entire server, configures domains, offers dynamic reconfiguration, as well as detects and notifies various failures. The XSCFU enables standard control and monitoring function through network. Using this function enables starts, settings, and operation managements of the server from remote locations.

Connecting a domain

Check for the available domains on the servers :

XSCF> showdomainstatus -a
DID Domain    Status
00            Running
01            Running
02            Running
03              -

Now to connect to the domain with ID 00 :

XSCF> console -d 0

Power/reboot/reset/panic commands

Poweron all domains

XSCF> poweron -a

Poweron only domain 0

XSCF> poweron -d 0

Poweroff all domains

XSCF> poweroff -a

Poweroff domain 0

XSCF> poweroff -d 0

Reboot XSCF

XSCF> rebootxscf

The 3 modes to reset a domain are :

por: To reset the domain
panic: To panic the domain
xir: To reset the CPU in domain

XSCF> reset -d 0 por
XSCF> reset -d 0 panic
XSCF> reset -d 0 xir

Send a break signal to a domain (with ID 0)

XSCF> sendbreak -d 0

User Administration

Creating a New user

XSCF> adduser -u 345 admin

Delete a user

XSCF> deleteuser admin

Disable a user

XSCF> disableuser admin

Enable a user

XSCF> enableuser admin

Display user account information

XSCF> showuser -a

Set or change a User (admin) password

XSCF> password admin

Network related commands

Display complete network configuration

XSCF> shownetwork  -a

Set IP address for XSCF-LAN#0 in the XSCFU#0

XSCF> setnetwork xscf#0-lan#0 -m 255.255.255.0 192.168.1.10

We must apply the network settings and reboot the XSCF in order to the network settings to take effect :

XSCF> applynetwork
XSCF> rebootxscf

Setup 2 NTP servers with IP 192.168.1.10 and 192.168.1.20

XSCF> setntp 192.168.1.10 192.168.1.20
Please reset the XSCF by rebootxscf to reflect the ntp settings.

To delete a NTP server 192.168.1.20

XSCF> setntp -c del 192.168.1.20
  Please reset the XSCF by rebootxscf to reflect the ntp settings.

Hardware Related Commands

Show field replaceable units(FRU)

XSCF> showhardconf

Display degraded units

XSCF> showstatus

To display configured devices on XSBs

XSCF> showdevices

Fault Management configuration tool

To view fault management logs

XSCF> fmdump -v
TIME                    UUID                                    MSG-ID
Nov 30 20:44:55.1283    9f773e33-e46f-466c-be86-fd3fcc449935   FMD-8000-0W
   100%  defect.sunos.fmd.nosub
   .....

Display Very Verbose Event Detail for a UUID

XSCF> fmdump -e -V -u 5f88d7d5-a107-4435-99c9-7c59479d22ed TIME CLASS

Logs

show the logs

XSCF> showlogs -v
XSCF> showlogs error
XSCF> showlogs power

Snapshots

We can take a snapshot of M series servers XSCF either on a remote server or on a USB device locally connected. To take a snaphot on a remote system 192.168.1.10 (in /var/tmp directory) by using root user credentials :

XSCF> snapshot -L F -t root@192.168.1.10:/var/tmp

To take the snapshot on a Local USB device connected to the server :

XSCF> snapshot -L F -d usb0

Connect DVD Device to a domain

Run the following from the XSCF to connect the DAT & DVD to the needed port based on the domain(s) configuration.

XSCF> cfgdevice -q -y -c attach -p 0-0

Confirm that the device is attached to domain and is enabled :

XSCF> cfgdevice -l

From the OS prompt configure the newly added DVD device and restart the volmgmt service:

# cfgadm -c configure # c0
# /etc/init.d/volmgt start

Hostname

Display current hostanems of XSCF units

XSCF> showhostname -a
xscf#0: hostname01.example.com 
xscf#1: hostname02.example.com

To set hostname and DNS domain name for XSCF doamin respectively

XSCF> sethostname xscf#0 hostname01
XSCF> sethostname -d example.com

Shut Down or Reboot a Solaris System

Normally, the system reboots at power-up or after a system crash. You can reboot a system by using either the init command or the reboot command. The init 6 command asks for stop methods (either SMF or rc.d). Whereas, the reboot command does not, thereby making the reboot command a more reliable way of rebooting a system.

Solaris is usually used as a server operating system. Because of this, you want to make sure that you shut the system down as gracefully as possible to ensure there isn’t any data loss.

For every application that is installed on your server, you should make sure that you have the correct scripts in /etc/rc(x).d to gracefully shut down the service.

Shutdown

You have more than one command option that you can use. The best command is this, executed as root:

shutdown -y -i5 -g0

This will immediately shut the system down. You can also use the older command that still works:

sync;sync;init 5

You can even use:

poweroff

Reboot
If you are trying to reboot the system as opposed to turning it off, you could use:

shutdown -y -i6 -g0

Or:

sync;sync;init 6

Or even:

reboot

So many commands to do the same thing…

PowerHA/HACMP Moving Resource Group (RG) one node to other

In this post, you will be learning the steps for moving a resource group from one node to the other node. The steps as follows:

1) Extending PATH variable with cluster paths

Sometimes cluster paths are not included in default path ,run below command in case if you are not able to run commands directly.

export PATH=$PATH:/usr/es/sbin/cluster:/usr/es/sbin/cluster/utilities:/usr/es/sbin/cluster/sbin:/usr/es/sbin/cluster/cspoc

2) Check the cluster services are up or not in destination node

#clshowsrv -v
Status of the RSCT subsystems used by HACMP:
Subsystem         Group            PID          Status
 topsvcs          topsvcs          278684       active
 grpsvcs          grpsvcs          332026       active
 grpglsm          grpsvcs                       inoperative
 emsvcs           emsvcs           446712       active
 emaixos          emsvcs           294942       active
 ctrmc            rsct             131212       active

Status of the HACMP subsystems:
Subsystem         Group            PID          Status
 clcomdES         clcomdES         204984       active
 clstrmgrES       cluster          86080        active

Status of the optional HACMP subsystems:
Subsystem         Group            PID          Status
 clinfoES         cluster          360702       active

3) Check the availability of resource group

# clRGinfo

-----------------------------------------------------------------------------

Group Name     Type           State      Location

-----------------------------------------------------------------------------

UMRG1            non-concurrent OFFLINE    umhaserv1

                                ONLINE     umhaserv2

#

4) Move the resourcegroup by using below command

==>  clRGmove -g <RG> -n  <node> -m

# clRGmove -g UMRG1 -n umhaserv1 -m

Attempting to move group UMRG1 to node umhaserv1.

Waiting for cluster to process the resource group movement request....

Waiting for the cluster to stabilize..................

Resource group movement successful.

Resource group UMRG1 is online on node umhaserv1.

You can use smitty path also

smitty cl_admin => HACMP Resource Group and Application Management => Move a Resource Group to Another Node / Site

5) Verify the RG movement

# clRGinfo

-----------------------------------------------------------------------------

Group Name     Type           State      Location

-----------------------------------------------------------------------------

UMRG1          non-concurrent   ONLINE     umhaserv1

                                OFFLINE    umhaserv2

#

Manually Install or Upgrade VMware Tools in a Linux Virtual Machine

For Linux virtual machines, you manually install or upgrade VMware Tools by using the command line.

Install the latest version of VMware Tools to enhance the performance of the virtual machine's guest operating system and improve virtual machine management. When you power on a virtual machine, if a new version of VMware Tools is available, you see a notification in the status bar of the guest operating system.

Note
This procedure describes how to use the VMware Tools tar installer to install or upgrade VMware Tools. For virtual machines in a vSphere environment, you can alternatively use VMware Tools operating system specific packages (OSPs) to install and upgrade VMware Tools. With OSPs you can use the native update mechanisms of your operating system to download, install, and manage VMware Tools. For more information, see Operating System Specific Packages for Linux Guest Operating Systems.

Prerequisites

■ Power on the virtual machine.

■ Verify that the guest operating system is running.

■ Because the VMware Tools installer is written in Perl, verify that Perl is installed in the guest operating system.

■ To determine whether you have the latest version of VMware Tools, look on the Summary tab for the virtual machine.

Procedure

1) Select the menu command to mount the VMware Tools virtual disc on the guest operating system.

VMware Product

Menu Command

vSphere Client

Inventory > Virtual Machine > Guest > Install/Upgrade VMware Tools

2) If you are performing an upgrade or reinstallation, in the Install/Upgrade VMware Tools dialog box, select Interactive Tools Installation or Interactive Tools Upgrade and click OK.

The process starts by mounting the VMware Tools virtual disc on the guest operating system.

3) In the virtual machine, log in to the guest operating system as root and open a terminal window.

4) Run the mount command with no arguments to determine whether your Linux distribution automatically mounted the VMware Tools virtual CD-ROM image.

If the CD-ROM device is mounted, the CD-ROM device and its mount point are listed as something like this:

/dev/cdrom on /mnt/cdrom type iso9660 (ro,nosuid,nodev)

5) If the VMware Tools virtual CD-ROM image is not mounted, mount the CD-ROM drive.

a : If a mount point directory does not already exist, create it.

mkdir /mnt/cdrom

Some Linux distributions use different mount point names. For example, on some distributions the mount point is /media/VMware Tools rather than /mnt/cdrom. Modify the command to reflect the conventions that your distribution uses.

b : Mount the CD-ROM drive.

mount /dev/cdrom /mnt/cdrom

Some Linux distributions use different device names or organize the /dev directory differently. If your CD-ROM drive is not /dev/cdrom or if the mount point for a CD-ROM is not /mnt/cdrom, modify the command to reflect the conventions that your distribution uses.

6) Change to a working directory (for example, /tmp).

cd /tmp

7) Delete any previous vmware-tools-distrib directory before you install VMware Tools.

The location of this directory depends on where you placed it during the previous installation. Often this directory is placed in /tmp/vmware-tools-distrib.

8) List the contents of the mount point directory and note the filename of the VMware Tools tar installer.

ls mount-point

9) Uncompress the installer.

tar zxpf /mnt/cdrom/VMwareTools-x.x.x-yyyy.tar.gz
The value x.x.x is the product version number, and yyyy is the build number of the product release.

If you attempt to install a tar installation over an RPM installation, or the reverse, the installer detects the previous installation and must convert the installer database format before continuing.

10) If necessary, unmount the CD-ROM image.

umount /dev/cdrom

If your Linux distribution automatically mounted the CD-ROM, you do not need to unmount the image.

11) Run the installer and configure VMware Tools.

cd vmware-tools-distrib

./vmware-install.pl

Usually, the vmware-config-tools.pl configuration file runs after the installer file finishes running.

12) Respond to the prompts by pressing Enter to accept the default values, if appropriate for your configuration.

13) Follow the instructions at the end of the script.

Depending on the features you use, these instructions can include restarting the X session, restarting networking, logging in again, and starting the VMware User process. You can alternatively reboot the guest operating system to accomplish all these tasks.

The VMware Tools label on the Summary tab changes to OK.

NIC Channel Bonding in Linux

Today I had implemented NIC bonding (bind both NIC so that it works as a single device). Bonding is nothing but Linux kernel feature that allows to aggregate multiple like interfaces (such as eth0, eth1) into a single virtual link such as bond0. The idea is pretty simple get higher data rates and as well as link failover. NIC channel bonding allows multiple network cards to act as one, allowing increased bandwidth and redundancy.

Linux allows binding of multiple network interfaces into a single channel/NIC using special kernel module called bonding. According to official bonding documentation:

The Linux bonding driver provides a method for aggregating multiple network interfaces into a single logical "bonded" interface. The behavior of the bonded interfaces depends upon the mode; generally speaking, modes provide either hot standby or load balancing services. Additionally, link integrity monitoring may be performed.

Step #1: Create a Bond0 Configuration File

Red Hat Enterprise Linux (and its clone such as CentOS) stores network configuration in /etc/sysconfig/network-scripts/ directory. First, you need to create a bond0 config file as follows:

# vi /etc/sysconfig/network-scripts/ifcfg-bond0
Append the following linest:

 
DEVICE=bond0
IPADDR=192.168.1.20
NETWORK=192.168.1.0
NETMASK=255.255.255.0
USERCTL=no
BOOTPROTO=none
ONBOOT=yes

You need to replace IP address with your actual setup. Save and close the file.

Step #2: Modify eth0 and eth1 config files

Open both configuration using a text editor such as vi/vim, and make sure file read as follows for eth0 interface

# vi /etc/sysconfig/network-scripts/ifcfg-eth0
Modify/append directive as follows:

DEVICE=eth0
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none

Open eth1 configuration file using vi text editor, enter:

# vi /etc/sysconfig/network-scripts/ifcfg-eth1
Make sure file read as follows for eth1 interface:

DEVICE=eth1
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none

Save and close the file.

Step # 3: Load bond driver/module

Make sure bonding module is loaded when the channel-bonding interface (bond0) is brought up. You need to modify kernel modules configuration file:

# vi /etc/modprobe.conf
Append following two lines:

alias bond0 bonding
options bond0 mode=balance-alb miimon=100

Save file and exit to shell prompt. A description of the bonding options is available here.

Step # 4: Test configuration

First, load the bonding module, enter:

# modprobe bonding
Restart the networking service in order to bring up bond0 interface, enter:

# service network restart
Make sure everything is working. Type the following to query the current status of Linux kernel bounding driver, enter:

# cat /proc/net/bonding/bond0
Sample outputs:

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 200
Down Delay (ms): 200
Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:0c:29:c6:be:59
Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:0c:29:c6:be:63

To list all network interfaces, enter:

# ifconfig
Sample outputs:

bond0     Link encap:Ethernet  HWaddr 00:0C:29:C6:BE:59
 inet addr:192.168.1.20  Bcast:192.168.1.255  Mask:255.255.255.0
 inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
 UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
 RX packets:2804 errors:0 dropped:0 overruns:0 frame:0
 TX packets:1879 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:0
 RX bytes:250825 (244.9 KiB)  TX bytes:244683 (238.9 KiB)
eth0      Link encap:Ethernet  HWaddr 00:0C:29:C6:BE:59
 inet addr:192.168.1.20  Bcast:192.168.1.255  Mask:255.255.255.0
 inet6 addr: fe80::20c:29ff:fec6:be59/64 Scope:Link
 UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
 RX packets:2809 errors:0 dropped:0 overruns:0 frame:0
 TX packets:1390 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:251161 (245.2 KiB)  TX bytes:180289 (176.0 KiB)
 Interrupt:11 Base address:0x1400
eth1      Link encap:Ethernet  HWaddr 00:0C:29:C6:BE:59
 inet addr:192.168.1.20  Bcast:192.168.1.255  Mask:255.255.255.0
 inet6 addr: fe80::20c:29ff:fec6:be59/64 Scope:Link
 UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
 RX packets:4 errors:0 dropped:0 overruns:0 frame:0
 TX packets:502 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:258 (258.0 b)  TX bytes:66516 (64.9 KiB)
 Interrupt:10 Base address:0x1480

Once the bond is configured it acts like any other Ethernet device. For example, you can configure alias interfaces to handle multiple IP addresses, as shown below.

Create the "ifcfg-bond0:1" and "ifcfg-bond0:2" files in the "/etc/sysconfig/network-scripts" directory with the following contents.

# ifcfg-bond0:1 file contents
DEVICE=bond0:1
BOOTPROTO=none
ONBOOT=yes
NETWORK=192.168.0.0
NETMASK=255.255.255.0
IPADDR=192.168.0.172
USERCTL=no
BONDING_OPTS="mode=1 miimon=100"

# ifcfg-bond0:2 file contents
DEVICE=bond0:2
BOOTPROTO=none
ONBOOT=yes
NETWORK=192.168.0.0
NETMASK=255.255.255.0
IPADDR=192.168.0.173
USERCTL=no
BONDING_OPTS="mode=1 miimon=100"

Notice, the device names and IP addresses differ from the original "ifcfg-bond0" file.

Restart the network service for the changes to take effect.

# service network restart
Shutting down interface bond0:                             [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface bond0:                               [  OK  ]
#

The ifconfig command shows the three IP addresses being handled by the bond.

[root@wls11g-1 network-scripts]# ifconfig
bond0     Link encap:Ethernet  HWaddr 08:00:27:FC:F5:B7  
          inet addr:192.168.0.171  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:14635 errors:0 dropped:306 overruns:0 frame:0
          TX packets:7310 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:17571270 (16.7 MiB)  TX bytes:554475 (541.4 KiB)

bond0:1   Link encap:Ethernet  HWaddr 08:00:27:FC:F5:B7  
          inet addr:192.168.0.172  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1

bond0:2   Link encap:Ethernet  HWaddr 08:00:27:FC:F5:B7  
          inet addr:192.168.0.173  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1

eth0      Link encap:Ethernet  HWaddr 08:00:27:FC:F5:B7  
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:1835 errors:0 dropped:0 overruns:0 frame:0
          TX packets:961 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:189616 (185.1 KiB)  TX bytes:129841 (126.7 KiB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:FC:F5:B7  
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:12800 errors:0 dropped:306 overruns:0 frame:0
          TX packets:6349 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:17381654 (16.5 MiB)  TX bytes:424634 (414.6 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:1541 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1541 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3612733 (3.4 MiB)  TX bytes:3612733 (3.4 MiB)

#

Tcpdump command

Tcpdump is a really great tool for network security analyst, you can dump packets that flows within your networks into file for further analysis. With some filters you can capture only the interested packets, which it reduce the size of saved dump and further reduce loading and processing time of packets analysis.

This post will only covers the fundamental of tcpdump usage, bare in mind tcpdump can do much much more than what I illustrate here.

Lets start with capturing packets based on network interface, ports and protocols. Let assume I wanna capture tcp packets that flow over eth1, port 6881. The dump file with be save as test.pcap.

tcpdump -w test.pcap -i eth1 tcp port 6881

Simple right? What if at the same time I am interested on getting packets on udp port 33210 and 33220?

tcpdump -w test.pcap -i eth1 tcp port 6881 or udp \( 33210 or 33220 \)

‘\’ is an escape symbol for ‘(‘ and ‘)’. Logic OR implies PLUS (+). In plain text is I want to capture tcp packets flows over port 6881 plus udp ports 33210 and 33220.

Careful with ‘and’ in tcpdump filter expression, it means intersection. Thats why I put ‘or’ instead of and within udp port 33210 and 33220. The usage of ‘and’ in tcpdump will be illustrate later.

Ok, how about reading pcap that I saved previously?

tcpdump -nnr test.pcap

The -nn is to tell tcpdump not to resolve DNS on IP and Ports, where r is read.

Adding -tttt to makes the timestamp appears more readable format.

tcpdump -ttttnnr test.pcap

How about capture based on IP ?
You need to tell tcpdump which IP you are interested in? Destination IP? or Source IP ? Let say I wanna sniff on destination IP 10.168.28.22 tcp port 22, how should i write?

tcpdump -w test.pcap dst 10.168.28.22 and tcp port 22

So the ‘and’ makes the intersection of destination IP and port.

By default the sniff size of packets is 96 bytes, you somehow can overload that size by specified with -s.

tcpdump -w test.pcap -s 1550 dst 10.168.28.22 and tcp port 22

Some version of tcpdump allows you to define port range. You can as bellow for capturing packets based on a range of tcp port.

tcpdump tcp portrange 20-24

Bare in mind, the line above I didn’t specified -w which it won’t write to a file but i will just print the captured packets on the screen.

Basic examples of linux netstat command

Netstat

Netstat is a command line utility that can be used to list out all the network (socket) connections on a system. It lists out all the tcp, udp socket connections and the unix socket connections.

Apart from connected sockets it can also list listening sockets that are waiting for incoming connections. So by verifying an open port 80 you can confirm if a web server is running on the system or not. This makes it a very useful tool for network and system administrators.

In this tutorial we shall be checking out few examples of how to use netstat to find information about network connections and open ports on a system.

Here is a quick intro to netstat from the man pages

netstat - Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships

1. List out all connections

The first and most simple command is to list out all the current connections. Simply run the netstat command with the a option.

$ netstat -a

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 enlightened:domain      *:*                     LISTEN     
tcp        0      0 localhost:ipp           *:*                     LISTEN     
tcp        0      0 enlightened.local:54750 li240-5.members.li:http ESTABLISHED
tcp        0      0 enlightened.local:49980 del01s07-in-f14.1:https ESTABLISHED
tcp6       0      0 ip6-localhost:ipp       [::]:*                  LISTEN     
udp        0      0 enlightened:domain      *:*                                
udp        0      0 *:bootpc                *:*                                
udp        0      0 enlightened.local:ntp   *:*                                
udp        0      0 localhost:ntp           *:*                                
udp        0      0 *:ntp                   *:*                                
udp        0      0 *:58570                 *:*                                
udp        0      0 *:mdns                  *:*                                
udp        0      0 *:49459                 *:*                                
udp6       0      0 fe80::216:36ff:fef8:ntp [::]:*                             
udp6       0      0 ip6-localhost:ntp       [::]:*                             
udp6       0      0 [::]:ntp                [::]:*                             
udp6       0      0 [::]:mdns               [::]:*                             
udp6       0      0 [::]:63811              [::]:*                             
udp6       0      0 [::]:54952              [::]:*                             
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  2      [ ACC ]     STREAM     LISTENING     12403    @/tmp/dbus-IDgfj3UGXX
unix  2      [ ACC ]     STREAM     LISTENING     40202    @/dbus-vfs-daemon/socket-6nUC6CCx

The above command shows all connections from different protocols like tcp, udp and unix sockets. However this is not quite useful. Administrators often want to pick out specific connections based on protocols or port numbers for example.

2. List only TCP or UDP connections

To list out only tcp connections use the t options.

$ netstat -at
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 enlightened:domain      *:*                     LISTEN     
tcp        0      0 localhost:ipp           *:*                     LISTEN     
tcp        0      0 enlightened.local:36310 del01s07-in-f24.1:https ESTABLISHED
tcp        0      0 enlightened.local:45038 a96-17-181-10.depl:http ESTABLISHED
tcp        0      0 enlightened.local:37892 ABTS-North-Static-:http ESTABLISHED
.....

Similarly to list out only udp connections use the u option.

$ netstat -au
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
udp        0      0 *:34660                 *:*                                
udp        0      0 enlightened:domain      *:*                                
udp        0      0 *:bootpc                *:*                                
udp        0      0 enlightened.local:ntp   *:*                                
udp        0      0 localhost:ntp           *:*                                
udp        0      0 *:ntp                   *:*                                
udp6       0      0 fe80::216:36ff:fef8:ntp [::]:*                             
udp6       0      0 ip6-localhost:ntp       [::]:*                             
udp6       0      0 [::]:ntp                [::]:*

The above output shows both ipv4 and ipv6 connections.

3. Disable reverse dns lookup for faster output

By default, the netstat command tries to find out the hostname of each ip address in the connection by doing a reverse dns lookup. This slows down the output. If you do not need to know the host name and just the ip address is sufficient then suppress the hostname lookup with the n option.

$ netstat -ant
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 127.0.1.1:53            0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN     
tcp        0      0 192.168.1.2:49058       173.255.230.5:80        ESTABLISHED
tcp        0      0 192.168.1.2:33324       173.194.36.117:443      ESTABLISHED
tcp6       0      0 ::1:631                 :::*                    LISTEN

The above command shows ALL TCP connections with NO dns resolution. Got it ? Good.

4. List out only listening connections

Any network daemon/service keeps an open port to listen for incoming connections. These too are like socket connections and are listed out by netstat. To view only listening ports use the l options.

$ netstat -tnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 127.0.1.1:53            0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN     
tcp6       0      0 ::1:631                 :::*                    LISTEN

Now we can see only listening tcp ports/connections. If you want to see all listening ports, remove the t option. If you want to see only listening udp ports use the u option instead of t.
Make sure to remove the 'a' option, otherwise all connections would get listed and not just the listening connections.

5. Get process name/pid and user id

When viewing the open/listening ports and connections, its often useful to know the process name/pid which has opened that port or connection. For example the Apache httpd server opens port 80. So if you want to check whether any http server is running or not, or which http server is running, apache or nginx, then track down the process name.

The process details are made available by the 'p' option.

~$ sudo netstat -nlpt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.1.1:53            0.0.0.0:*               LISTEN      1144/dnsmasq    
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      661/cupsd       
tcp6       0      0 ::1:631                 :::*                    LISTEN      661/cupsd

When using the p option, netstat must be run with root privileges, otherwise it cannot detect the pids of processes running with root privileges and most services like http and ftp often run with root privileges.

Along with process name/pid its even more useful to get the username/uid owning that particular process. Use the e option along with the p option to get the username too.

$ sudo netstat -ltpe
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
tcp        0      0 enlightened:domain      *:*                     LISTEN      root       11090       1144/dnsmasq    
tcp        0      0 localhost:ipp           *:*                     LISTEN      root       9755        661/cupsd       
tcp6       0      0 ip6-localhost:ipp       [::]:*                  LISTEN      root       9754        661/cupsd

The above example lists out Listening connections of Tcp type with Process information and Extended information.
The extended information contains the username and inode of the process. This is a useful command for network administrators.

Note - If you use the n option with the e option, the uid would be listed and not the username.

6. Print statistics

The netstat command can also print out network statistics like total number of packets received and transmitted by protocol type and so on.

To list out statistics of all packet types

$ netstat -s
Ip:
    32797 total packets received
    0 forwarded
    0 incoming packets discarded
    32795 incoming packets delivered
    29115 requests sent out
    60 outgoing packets dropped
Icmp:
    125 ICMP messages received
    0 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 125
    125 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 125
... OUTPUT TRUNCATED ...

To print out statistics of only select protocols like TCP or UDP use the corresponding options like t and u along with the s option. Simple!

7. Display kernel routing information

The kernel routing information can be printed with the r option. It is the same output as given by the route command. We also use the n option to disable the hostname lookup.

$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0

8. Print network interfaces

The netstat command can also print out the information about the network interfaces. The i option does the task.

$ netstat -i
Kernel Interface table
Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0       1500 0     31611      0      0 0         27503      0      0      0 BMRU
lo        65536 0      2913      0      0 0          2913      0      0      0 LRU

The above output contains information in a very raw format. To get a more human friendly version of the output use the e option along with i.

$ netstat -ie
Kernel Interface table
eth0      Link encap:Ethernet  HWaddr 00:16:36:f8:b2:64  
          inet addr:192.168.1.2  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::216:36ff:fef8:b264/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:31682 errors:0 dropped:0 overruns:0 frame:0
          TX packets:27573 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:29637117 (29.6 MB)  TX bytes:4590583 (4.5 MB)
          Interrupt:18 Memory:da000000-da020000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:2921 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2921 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:305297 (305.2 KB)  TX bytes:305297 (305.2 KB)

The above output is similar to the output shown by the ifconfig command.

9. Get netstat output continuously

Netstat can output connection information continuously with the c option.

$ netstat -ct

The above command will output tcp connections continuously.

10. Display multicast group information

The g option will display the multicast group information for IPv4 and IPv6 protocols.

$ netstat -g
IPv6/IPv4 Group Memberships
Interface       RefCnt Group
--------------- ------ ---------------------
lo              1      all-systems.mcast.net
eth0            1      224.0.0.251
eth0            1      all-systems.mcast.net
lo              1      ip6-allnodes
lo              1      ff01::1
eth0            1      ff02::fb
eth0            1      ff02::1:fff8:b264
eth0            1      ip6-allnodes
eth0            1      ff01::1
wlan0           1      ip6-allnodes
wlan0           1      ff01::1

More examples of netstat command

Okay, we covered the basic examples of netstat command above. Now its time to do some geek stuff with style.

Print active connections

Active socket connections are in "ESTABLISHED" state. So to get all current active connections use netstat with grep as follows

$ netstat -atnp | grep ESTA
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 192.168.1.2:49156       173.255.230.5:80        ESTABLISHED 1691/chrome     
tcp        0      0 192.168.1.2:33324       173.194.36.117:443      ESTABLISHED 1691/chrome

To watch a continous list of active connections, use the watch command along with netstat and grep

$ watch -d -n0 "netstat -atnp | grep ESTA"

Check if a service is running

If you want to check if a server like http,smtp or ntp is running or not, use grep again.

$ sudo netstat -aple | grep ntp
udp        0      0 enlightened.local:ntp   *:*                                 root       17430       1789/ntpd       
udp        0      0 localhost:ntp           *:*                                 root       17429       1789/ntpd       
udp        0      0 *:ntp                   *:*                                 root       17422       1789/ntpd       
udp6       0      0 fe80::216:36ff:fef8:ntp [::]:*                              root       17432       1789/ntpd       
udp6       0      0 ip6-localhost:ntp       [::]:*                              root       17431       1789/ntpd       
udp6       0      0 [::]:ntp                [::]:*                              root       17423       1789/ntpd       
unix  2      [ ]         DGRAM                    17418    1789/ntpd

So we found that ntp server is running. Grep for http or smtp or whatever you are looking for.

Well, that was most of what netstat is used for. If you are looking for more advanced information or want to dig deeper, read up the netstat manual (man netstat).

LVM in Linux step by step

LVM stands for Logical Volume Manager.

With LVM, we can create logical partitions that can span across one or more physical hard drives. First, the hard drives are divided into physical volumes, then those physical volumes are combined together to create the volume group and finally the logical volumes are created from volume group.

The LVM commands listed in this article are used under Ubuntu Distribution. But, it is the same for other Linux distributions.

Before we start, install the lvm2 package as shown below.

$ sudo apt-get intall lvm2

To create a LVM, we need to run through the following steps.

Select the physical storage devices for LVM
Create the Volume Group from Physical Volumes
Create Logical Volumes from Volume Group

Select the Physical Storage Devices for LVM – Use pvcreate, pvscan, pvdisplay Commands

In this step, we need to choose the physical volumes that will be used to create the LVM. We can create the physical volumes using pvcreate command as shown below.

$ sudo pvcreate /dev/sda6 /dev/sda7 
Physical volume "/dev/sda6" successfully created                                                 
Physical volume "/dev/sda7" successfully created

As shown above two physical volumes are created – /dev/sda6 and /dev/sda7.

If the physical volumes are already created, you can view them using the pvscan command as shown below.

$ sudo pvscan                                                                   
  PV /dev/sda6                      lvm2 [1.86 GB]                                                 
  PV /dev/sda7                      lvm2 [1.86 GB]                                                 
  Total: 2 [3.72 GB] / in use: 0 [0   ] / in no VG: 2 [3.72 GB]

You can view the list of physical volumes with attributes like size, physical extent size, total physical extent size, the free space, etc., using pvdisplay command as shown below.

$ sudo pvdisplay 
--- Physical volume --- 
  PV Name             /dev/sda6 
  VG Name             
  PV Size               1.86 GB / not usable 2.12 MB 
  Allocatable           yes 
  PE Size (KByte)    4096 
  Total PE              476 
  Free PE               456 
  Allocated PE          20 
  PV UUID               m67TXf-EY6w-6LuX-NNB6-kU4L-wnk8-NjjZfv 
 
  --- Physical volume --- 
  PV Name             /dev/sda7 
  VG Name             
  PV Size               1.86 GB / not usable 2.12 MB 
  Allocatable           yes 
  PE Size (KByte)    4096 
  Total PE              476 
  Free PE               476 
  Allocated PE          0 
  PV UUID               b031x0-6rej-BcBu-bE2C-eCXG-jObu-0Boo0x

Note : PE – Physical Extents are nothing but equal-sized chunks. The default size of extent is 4MB.

Create the Volume Group – Use vgcreate, vgdisplay Commands

Volume groups are nothing but a pool of storage that consists of one or more physical volumes. Once you create the physical volume, you can create the volume group (VG) from these physical volumes (PV).

In this example, the volume group vol_grp1 is created from the two physical volumes as shown below.

$ sudo vgcreate vol_grp1 /dev/sda6 /dev/sda7                                  
  Volume  group "vol_grp1" successfully created

LVM processes the storage in terms of extents. We can also change the extent size (from the default size 4MB) using -s flag.

vgdisplay command lists the created volume groups.

$ sudo vgdisplay 
  --- Volume group ---              
  VG Name                     vol_grp1  
  System ID                         
  Format                        lvm2        
  Metadata Areas            2           
  Metadata Sequence No  1           
  VG Access                   read/write  
  VG Status                    resizable   
  MAX LV                       0           
  Cur LV                        0           
  Open LV                      0           
  Max PV                       0           
  Cur PV                        2           
  Act PV                       2           
  VG Size                      3.72 GB     
  PE Size                      4.00 MB     
  Total PE                     952         
  Alloc PE / Size             0 / 0       
  Free  PE / Size            952 / 3.72 GB 
  VG UUID                     Kk1ufB-rT15-bSWe-5270-KDfZ-shUX-FUYBvR

LVM Create: Create Logical Volumes – Use lvcreate, lvdisplay command

Now, everything is ready to create the logical volumes from the volume groups. lvcreate command creates the logical volume with the size of 80MB.

$ sudo lvcreate -l 20 -n logical_vol1 vol_grp1 
  Logical volume "logical_vol1" created

Use lvdisplay command as shown below, to view the available logical volumes with its attributes.

 
$ sudo lvdisplay                                  
  --- Logical volume ---                                             
  LV Name                /dev/vol_grp1/logical_vol1              
  VG Name                vol_grp1                                  
  LV UUID                 ap8sZ2-WqE1-6401-Kupm-DbnO-2P7g-x1HwtQ      
  LV Write Access      read/write                                  
  LV Status              available                                   
  # open                  0                                           
  LV Size                  80.00 MB                                    
  Current LE              20                                          
  Segments               1                                           
  Allocation               inherit                                     
  Read ahead sectors  auto                                        
  - currently set to     256                                         
  Block device            252:0

After creating the appropriate filesystem on the logical volumes, it becomes ready to use for the storage purpose.

$ sudo  mkfs.ext3 /dev/vol_grp1/logical_vol1

LVM resize: Change the size of the logical volumes – Use lvextend Command

We can extend the size of the logical volumes after creating it by using lvextend utility as shown below. The changes the size of the logical volume from 80MB to 100MB.

$ sudo lvextend -L100 /dev/vol_grp1/logical_vol1 
  Extending logical volume logical_vol1 to 100.00 MB 
  Logical volume logical_vol1 successfully resized

We can also add additional size to a specific logical volume as shown below.

$ sudo lvextend -L+100 /dev/vol_grp1/logical_vol1 
  Extending logical volume logical_vol1 to 200.00 MB 
  Logical volume logical_vol1 successfully resized

AIX as DNS client

nslookup is the command used to query DNS servers. Normally nslookup looks up the hostname for a ip address or IP address for a hostname.

DNS server IP address/hostnames are defined in /etc/resolv.conf in AIX servers.

Here is an example of /etc/resolv.conf

nameserver 192.168.2.12
nameserver 192.168.2.13
nameserver 192.168.2.14
search india.cope.com usa.cope.com uk.cope.com

Let us see few tips and tricks on using nslookup.

1. To look up address in non interactive way,

$ nslookup webserv
Server: dnserver1.india.cope.com
Address: 192.168.2.12

Name:    webserv.india.cope.com
Address: 192.168.2.211
$

2. To look up address in interactive way,

$nslookup
Default Server: dnserver1.india.cope.com
Address: 192.168.2.12

> websrv
Server: dnserver1.india.cope.com
Address: 192.168.2.12

Name:    webserv.india.cope.com
Address: 192.168.2.211

> exit
$

3. To look up hostname in non interactive way,

$ nslookup 192.168.2.211
Server: dnserver1.india.cope.com
Address: 192.168.2.12

Name:    webserv.india.cope.com
Address: 192.168.2.211
$

4. To look up hostname in interactive way,

$ nslookup
Default Server: dnserver1.india.cope.com
Address: 192.168.2.12

> 192.168.2.211
Server: dnserver1.india.cope.com
Address: 192.168.2.12

Name:    webserv.india.cope.com
Address: 192.168.2.211

> exit
$

5. To look up MX data,

$ nslookup
Default Server: dnserver1.india.cope.com
Address: 192.168.2.12

> set q=mx
> rajs
Server: dnserver1.india.cope.com
Address: 192.168.2.12

Name:    rajs.india.cope.in
Address: 0.0.0.0
> exit
$

6. How to query a specific DNS server for an address ?

We can do in both interactive and non-interactive ways.
Below example will query for the IP address of the host websrv using the DNS serer "192.168.2.15" which is not specified in the /etc/resolv.conf file.

Interactive Way:

$nslookup
Default Server: dnserver1.india.cope.com
Address: 192.168.2.12

> server 192.168.2.15
Default Server: dnserver4.india.cope.com
Address: 192.168.2.15

> websrv
Server: dnserver4.india.cope.com
Address: 192.168.2.15

Name:    webserv.india.cope.com
Address: 192.168.2.211

> exit
$

Non-Interactive Way:

$nslookup - websrv 192.168.2.15

Server: dnserver4.india.cope.com
Address: 192.168.2.15

Name:    webserv.india.cope.com
Address: 192.168.2.211

7.What is the difference between Authoritave and Non-authoritative answers ?

When you query for something for the first time, we get the answer from the DNS server and it will be displayed as well as stored in the local cache. This is called as authoritative answer. ie., getting the answer directly from the DNS server. This answer will be kept in cache for certain time.

But when you do the same query for the second time, we get the answer from the cache instead of the DNS server. This is called as non-authoritative answer.

8. How will you specify an alternate DNS server when using nslookup ?

For using 192.168.2.24 as an alternate DNS server,
$ nslookup - 192.168.2.24

This will query the alternate server instead of the DNS servers configured in /etc/resolv.conf file.

9. How will you query a MX record in an alternate server ?

$ nslookup - type=mx bashi.usa.cope.com 192.168.2.24

10. How will you debug while querying a DNS server ?

$ nslookup
Default Server: dnserver1.india.cope.com
Address: 192.168.2.12
> set debug
> webserv

Server: dnserver1.india.cope.com
Address: 192.168.2.12

;; res_nmkquery(QUERY, websrv.india.cope.com, IN, A)
------------
Got answer:
    HEADER:
        opcode = QUERY, id = 54305, rcode = NOERROR
        header flags: response, authoritive answer, want recursion, recursion available
        questions = 1, answers = 1, authority records = 0, additional = 0

    QUESTIONS:
        websrv.india.cope.com, type = A, class = IN
    ANSWERS:
    -> webserv.india.cope.com
        internet address = 192.168.2.211
        ttl = 3600 (1H)

------------
Name:    webserv.india.cope.com
Address: 192.168.2.211

Each DNS packet is composed of 5 sections as given below

Header Section
Question Section
Answer Section
Authority Section
Additional Section

11. You can use options with the nslookup command using 'set' sub-command.
Here are few options ...

port=53        By default, DNS service uses port 53. If you have a DNS service on a different port, you can sue the port option to specify the prot number.

timeout=10    It is used to specify the timeout value. If the name server doesn't respond in 10 seconds, nslookup will send the query again.

debug      To turn on debug mode

nodebug         To turn off debug mode

querytype=A By default, nslookup looks for A record. If you type the IP address, it will look for PTR record.. You can change the querytype to MX or SOA.

12. How will you come out of the interactive nslookup session.

You can use exit command or type ^D (control+D) to come out of the session.

0301-168 bosboot: The current boot logical volume, /dev/ does not exist on /dev/hdisk0

Follow the below steps when you get this error.

root@yyxxxx4:/dev
# bosboot -ad /dev/ipldevice
0516-602 lslv: Logical volume name not entered.
Usage: lslv [-L] [-l | -m] [-n DescriptorPV] LVname
lslv: [-L] [-n DescriptorPV] -p PVname [LVname]
Lists the characteristics of a logical volume.

0301-168 bosboot: The current boot logical volume, /dev/, does not exist on /dev/hdisk0.

Solution:

lsvg -p rootvg - hdisk0

lslv -m hd5 - hdisk0 on PV1 1st partition

savebase -v - successful

will remove/recreate hd5

rmlv hd5

# mklv -y hd5 -t boot -a e rootvg 1 hdisk0

cd /dev

rm ipldevice

ln /dev/rhdisk0 /dev/ipldevice

bosboot -ad /dev/ipldevice - same error

bootinfo -B hdisk0 - 1

# ln /dev/rhd5 /dev/ipl_blv

cd /dev - same maj/min numbers

bosboot -ad /dev/ipldevice - works!

alt_disk_install was able to proceed now

EMC ODM definations cleanup

Before making any changes, collect host logs to document the current configuration. At a minimum, save the following: inq, lsdev -Cc disk, lsdev -Cc adapter, lspv, and lsvg
Shutdown the application(s), unmount the file system(s), and varyoff all volume groups except for rootvg. Do not export the volume groups.

# varyoffvg

Check with lsvg -o (confirm that only rootvg is varied on)

If no PowerPath, skip all steps with power names.

For CLARiiON configuration, if Navisphere Agent is running, stop it:
# /etc/rc.agent stop

Remove paths from Powerpath configuration:
# powermt remove hba=all

Delete all hdiskpower devices:
# lsdev -Cc disk -Fname grep power xargs -n1 rmdev -dl

Remove the PowerPath driver instance:
# rmdev -dl powerpath0

Delete all hdisk devices:For Symmetrix devices, use this command:
# lsdev -CtSYMM* -Fname xargs -n1 rmdev -dl

For CLARiiON devices, use this command:
# lsdev -CtCLAR* -Fname xargs -n1 rmdev -dl

Confirm with lsdev -Cc disk that there are no EMC hdisks or hdiskpowers.
Remove all Fiber driver instances:
# rmdev -Rdl fscsiX(X being driver instance number, i.e. 0,1,2, etc.)

Verify through lsdev -Cc driver that there are no more fiber driver instances (fscsi).
Change the adapter instances in Defined state
# rmdev -l fcsX(X being adapter instance number, i.e. 0,1,2, etc.)

Create the hdisk entries for all EMC devices:
# emc_cfgmgror
# cfgmgr -vl fcsx(x being each adapter instance which was rebuilt).

Skip this part if no PowerPath.
Configure all EMC devices into PowerPath:
# powermt config

Check the system to see if it now displays correctly:
# powermt display
# powermt display dev=all
# lsdev -Cc disk
# /etc/rc.agent start

Recovering emc dead path

# powermt display dev=all
And you notice that there are "dead" paths, then these are the commands to run in order to set these paths back to "alive" again, of course, AFTER ensuring that any SAN related issues are resolved.

To have PowerPath scan all devices and mark any dead devices as alive, if it finds that a device is in fact capable of doing I/O commands, run:

# powermt restore To delete any dead paths, and to reconfigure them again:
# powermt reset
# powermt config

Or you could run:
# powermt check