DISCLAIMER : Please note that blog owner takes no responsibility of any kind for any type of data loss or damage by trying any of the command/method mentioned in this blog. You may use the commands/method/scripts on your own responsibility.If you find something useful, a comment would be appreciated to let other viewers also know that the solution/method work(ed) for you.


AIX as DNS client - Tips & Tricks

nslookup is the command used to query DNS servers. Normally nslookup looks up the hostname for a ip address or IP address for a hostname.

DNS server IP address/hostnames are defined in /etc/resolv.conf in AIX servers.

Here is an example of /etc/resolv.conf

nameserver 192.168.2.12
nameserver 192.168.2.13
nameserver 192.168.2.14
search india.cope.com usa.cope.com uk.cope.com

Let us see few tips and tricks on using nslookup.

1. To look up address in non interactive way,

$ nslookup webserv
Server:  dnserver1.india.cope.com
Address:  192.168.2.12

Name:    webserv.india.cope.com
Address:  192.168.2.211
$

2. To look up address in interactive way,

$nslookup
Default Server:  dnserver1.india.cope.com
Address:  192.168.2.12

> websrv
Server:  dnserver1.india.cope.com
Address:  192.168.2.12

Name:    webserv.india.cope.com
Address:  192.168.2.211

> exit
$

3. To look up hostname in non interactive way,

$ nslookup 192.168.2.211
Server:  dnserver1.india.cope.com
Address:  192.168.2.12

Name:    webserv.india.cope.com
Address:  192.168.2.211
$

4. To look up hostname in interactive way,

$ nslookup
Default Server:  dnserver1.india.cope.com
Address:  192.168.2.12

> 192.168.2.211
Server:  dnserver1.india.cope.com
Address:  192.168.2.12

Name:    webserv.india.cope.com
Address:  192.168.2.211

> exit
$

4. To look up MX data,

$ nslookup
Default Server:  dnserver1.india.cope.com
Address:  192.168.2.12

> set q=mx
> rajs
Server:  dnserver1.india.cope.com
Address:  192.168.2.12

Name:    rajs.india.cope.in
Address:  0.0.0.0
> exit
$

5. How to query a specific DNS server for an address ?

We can do in both interactive and non-interactive ways.
Below example will query for the IP address of the host websrv using the DNS serer "192.168.2.15" which is not specified in the /etc/resolv.conf file.

Interactive Way:

$nslookup
Default Server:  dnserver1.india.cope.com
Address:  192.168.2.12

> server 192.168.2.15
Default Server:  dnserver4.india.cope.com
Address:  192.168.2.15

> websrv
Server:  dnserver4.india.cope.com
Address:  192.168.2.15

Name:    webserv.india.cope.com
Address:  192.168.2.211

> exit
$

Non-Interactive Way:

$nslookup - websrv 192.168.2.15

Server:  dnserver4.india.cope.com
Address:  192.168.2.15

Name:    webserv.india.cope.com
Address:  192.168.2.211

6.What is the difference between Authoritave and Non-authoritative answers ?

When you query for something for the first time, we get the answer from the DNS server and it will be displayed as well as stored in the local cache. This is called as authoritative answer. ie., getting the answer directly from the DNS server. This answer will be kept in cache for certain time.

But when you do the same query for the second time, we get the answer from the cache instead of the DNS server. This is called as non-authoritative answer.

7. How will you specify an alternate DNS server when using nslookup ?

For using 192.168.2.24 as an alternate DNS server,
$ nslookup - 192.168.2.24

This will query the alternate server instead of the DNS servers configured in /etc/resolv.conf file.

8. How will you query a MX record in an alternate server ?

$ nslookup - type=mx bashi.usa.cope.com 192.168.2.24

9. How will you debug while querying a DNS server ?


$ nslookup
Default Server:  dnserver1.india.cope.com
Address:  192.168.2.12
> set debug
> webserv

Server:  dnserver1.india.cope.com
Address:  192.168.2.12

;; res_nmkquery(QUERY, websrv.india.cope.com, IN, A)
------------
Got answer:
    HEADER:
        opcode = QUERY, id = 54305, rcode = NOERROR
        header flags:  response, authoritive answer, want recursion, recursion available
        questions = 1,  answers = 1,  authority records = 0,  additional = 0

    QUESTIONS:
        websrv.india.cope.com, type = A, class = IN
    ANSWERS:
    ->  webserv.india.cope.com
        internet address = 192.168.2.211
        ttl = 3600 (1H)

------------
Name:    webserv.india.cope.com
Address:  192.168.2.211

10.
Each DNS packet is composed of 5 sections as given below
  1. Header Section
  2. Question Section
  3. Answer Section
  4. Authority Section
  5. Additional Section

11. You can use options with the nslookup command using 'set' sub-command.
Here are few options ...

port=53          By default, DNS service uses port 53. If you have a DNS service on a different port, you can sue the port option to specify the prot number.

timeout=10    It is used to specify the timeout value. If the name server doesn't respond in 10 seconds, nslookup will send the query again.

debug              To turn on debug mode

nodebug         To turn off debug mode

querytype=A  By default, nslookup looks for A record. If you type the IP address, it will look for PTR record.. You can change the querytype to MX or SOA.

12. How will you come out of the interactive nslookup session.

You can use exit command or type ^D (control+D) to come out of the session.

Paging space commands in AIX


Below are the commands regarding the paging space in AIX.

To monitor paging space utilization:

lsps –a or lsps -s


To create an additional paging space:

mkps -s <#LPs> <vgname> <disk> or smit mkps


To activate a paging space:

swapon <device file name>


To deactivate a paging space:

swapoff <device file name>


To remove a paging space (must be inactive):

rmps <device file name> or smit mkps


To increase the size of a paging space:

chps –s <#LPs> <paging space name> or smit chps


To decrease the size of a paging space:

chps –d <#LPs> <paging space name> or smit chps


Activate paging space at restart:

chps -a -y <device file name>

Prevent and Detect Orphaned Mksysb NIM resources

In order to have a working mksysb resource in a NIM environment you need to have 2 items:   a NIM mksysb resource which points to a mksysb file in the filesystem. 
A good NIM mksysb resource looks like the image below.   The NIM mksysb resource is stored in the ODM and has a "location" attribute that points to a file in the filesystem.
image

However, if something (or someone) deletes the mksysb file from the filesystem, but doesn't delete the NIM mksysb resource, you are left with an orphaned mksysb NIM resource.   The mksysb resource will still show up in NIM and still appear to be useable, however any operations that try to use it will fail since its backing mksysb file isn't present.   An oprhaned mksysb resource looks like the image below:
image


How to prevent orphaned NIM mksysb resources:

The best way to prevent a orphaned NIM mksysb resource is to never delete mksysb files from the filesystem using "rm".   Instead, if you no longer need a NIM mksysb, use the "nim" command to delete it and also specify that the backing mksysb file should be deleted as well.   This can be done with a command such as this:
nim -o remove -a rm_image=yes aix3_mksysb
Substitute "aix3_mksysb" for the name of the mksysb that you want to delete.  The "-a rm_image=yes" tells NIM to not only delete the NIM resource from the ODM, but to also delete the backing mksysb file from the filesystem.

Detect orphaned NIM mksysb resources


Here is a handy one-line script that will check all your NIM mksysb resources and tell you if you have any orphaned mksysb resources that don't have a backing file present:

for mksysb in `lsnim -t mksysb | awk '{print $1}'`; do printf "%-20s " $mksysb; location=`lsnim -l $mksysb | grep location | awk '{print $3}'`; [ -e "$location" ] && echo " OK         $location" || echo " Not Found  $location"; done

The output looks like this:
aix1_mksysb           OK         /tmp/aix1_mksysb
aix2_mksysb           OK         /tmp/aix2_mksysb
aix3_mksysb           OK         /tmp/aix3_mksysb
aix4_mksysb           OK         /tmp/aix4_mksysb
aix5_mksysb           OK         /tmp/aix5_mksysb
aix6_mksysb           Not Found  /tmp/aix6_mksysb
aix7_mksysb           OK         /tmp/aix7_mksysb
aix8_mksysb           OK         /tmp/aix8_mksysb

Based on the output we can clearly see all of the mksysb's are good except for aix6_mksysb which doesn't have a a backing mksysb file present in the filesystem.

Using savevg on AIX to save time creating filesystems, LV's and volume groups

If you ever need to build multiple servers that will all have the same volume groups, logical volumes, and filesystems you can use "savevg" and "restvg" to save yourself a bunch of time and duplicated work. 
This also works if you are ever asked to build a new server that should be setup with the same VG/LV/FS's as an older server. 
You start by setting up one of the servers with the volume groups, logical volumes, and filesystems that you will need.  Next you can use the "savevg -r" command to backup just the volume group/LV/Filesystem structure information.  With the "-r" flag it doesn't backup any data in the filesystems, which makes it quick and the backup file very small. 
In this example we want to duplicate the "appvg" structure on to another server:
# lsvg -l appvg
appvg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
loglv02             jfs2log    1       1       1    open/syncd    N/A
applv05             jfs2       52      52      4    open/syncd    /app5
fslv05              jfs2       25      50      2    open/syncd    /app2
loglv03             jfslog     1       1       1    closed/syncd  N/A
#
# savevg -r -f /appvg.savevg appvg

Creating information file for volume group appvg.................................................................

Backing up user Volume Group information files only.
Creating list of files to back up.
Backing up 6 files

6 of 6 files (100%)
0512-038 savevg: Backup Completed Successfully.

 
 
Next you copy the "/appvg.savevg" file on to all the servers that you want to setup the VG/LV/FS's on.  You can use something like "scp" or "sftp" or another protocol to transfer the file. 
On the other servers, you run "restvg" to restore the VG/LV/FS structures from the file:
 
# restvg -r -f /appvg.savevg hdisk1 hdisk2

Will create the Volume Group:   appvg
Target Disks:   hdisk1 hdisk2
Allocation Policy:
        Shrink Filesystems:     no
        Preserve Physical Partitions for each Logical Volume:   no

Enter y to continue: y
0516-1254 /usr/sbin/mkvg: Changing the PVID in the ODM.
appvg
loglv02
applv05
fslv05
loglv03
#
# lsvg -l appvg
appvg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
loglv02             jfs2log    1       1       1    open/syncd    N/A
applv05             jfs2       52      52      2    open/syncd    /app5
fslv05              jfs2       25      50      2    open/syncd    /app2
loglv03             jfslog     1       1       1    closed/syncd  N/A
 
As you can see the "restvg -r" command restored the volume group, logical volumes, and filesystems, and even mounts the filesystems for you!  Note that the fslv05 LV was mirrored before on the original server and when restored it is still mirrored.   The original volume group on the source server was on 4 hdisks, but on the destination server only had 2 hdisks.   But this isn't a problem, restvg is able to take care of it as long as you have enough disks to accommodate the mirroring and enough total space for all the logical volumes.  You can even use the "-s" flag on restvg to attempt to shrink the filesystems if the destination hdisks aren't large enough to hold all the original LV's. 

The Shell Scripts that make up AIX

Over the years I've noticed that a lot of the core utilities on AIX are actually shell scripts. 
Here are some examples of these utilities on AIX that are either shell scripts (ksh/csh) or in some cases Perl scripts:
 
mksysb
oslevel
mkcd / mkdvd
useradd
userdel
usermod
prtconf
bosboot
mklv
shutdown
snap
lsconf
dsh
lsmksysb
savevg
which
chpv
chvg
cplv
exportvg
extendlv
migratelp
migratepv
mirrorvg
mktcpip
mkwpar
multibos
reducevg
reorgvg
replacepv
rmlv
rmlvcopy
splitlvcopy
splitvg
unmirrorvg
varyoffvg
 
As you can see, there are some pretty important commands in this list.  And this is just a small sample of them.  On my AIX server I found that there are over 400 scripts included as part of base AIX!  You can see a full list of all the scripts that make up your system by running a command like this:
 
for dir in `echo $PATH | tr ":" " "`; do for file in `ls -1 "$dir" 2>/dev/null`; do [ -x "$dir/$file" ] && file "$dir/$file"; done; done | grep -i script
 
It is pretty cool that so many of the core commands/utilities on AIX are made up of shell scripts.  For one, it shows that shell scripts can take on very important and critical tasks.  It can also be extremely helpful to be able to review the scripts if you are having any issues with any of these commands.  And these scripts can be an excellent learning tool.  These are extremely well written and robust scripts many of which have been used for decades on thousands and thousands of servers. 

Display the contents of gzip text files without unzipping

To save the disk space on my unix server, I have compressed thousands of reports[0-1000].txt files using gunzip program. However, I need to cat .gz file (or open in vim text editor) for reference purposes. I have a log files stores on my server in compressed format using gzip command. How do I display compressed Apache log file without using cat command? How to display the contents of gzip text files in screen without unzipping?

You can easily display compressed files on Linux or Unix without using using the cat command, less, or more command. In this example, show the contents of a text file called resume.txt.gz that has been compressed using gzip and friends. Open the Terminal and then type the following commands.

Syntax

 

Display resume.txt.gz on screen using cat command like syntax:

zcat resume.txt.gz

Display access_log_1.gz on screen at a time:

 zmore access_log_1.gz

Or try zless (less command):

zless access_log_1.gz

Search access_log_1.gz for 1.2.3.4 IP address using grep command like syntax:

zgrep '1.2.3.4' access_log_1.gz

You can use egrep command like syntax:

egrep 'regex' access_log_1.gz
egrep 'regex1|regex2' access_log_1.gz

 

Monitoring Filesystem Events with incron on RHEL 6

Have you ever wanted to know when a file is changed or accessed by the system or user? There is a program that does just that task called Inotify cron (incron).

Incron is for monitoring filesystem activity. It consists of a daemon and a table manipulator. You can use it a similar way as the regular cron. The difference is that the inotify cron handles filesystem events rather than time periods

incron provides a simple way how to solve many and many various situations. Every time when something depends on file system events, it’s a job for incron.

Here you can see a few examples where incron is a good solution:
  1.     Notifying programs (e.g. server daemons) about changes in configuration
  2.     Guarding changes in critical files (with their eventual recovery)
  3.     File usage monitoring, statistics.

First we will need to install incron:
$ sudo yum install incron

Make sure we set it to start on reboot:
$ sudo chkconfig incrond on

And now to start incron:
$ sudo service incrond start

incrond uses inotify. So to use it effectively we need to have it act on inotify events which are:
 IN_ACCESS         File was accessed (read) (*).
 IN_ATTRIB         Metadata  changed,  e.g.,  permissions,  timestamps, 
                   extended attributes, link count (since Linux 2.6.25), 
                   UID, GID, etc. (*).
 IN_CLOSE_WRITE    File opened for writing was closed (*).
 IN_CLOSE_NOWRITE  File not opened for writing was closed (*).
 IN_CREATE         File/directory created in watched directory (*).
 IN_DELETE         File/directory deleted from watched directory (*).
 IN_DELETE_SELF    Watched file/directory was itself deleted.
 IN_MODIFY         File was modified (*).
 IN_MOVE_SELF      Watched file/directory was itself moved.
 IN_MOVED_FROM     File moved out of watched directory (*).
 IN_MOVED_TO       File moved into watched directory (*).
 IN_OPEN           File was opened (*).
 
The incron table manipulator may be run under any regular user since it SUIDs. For manipulation with the tables use basically the same syntax as for the crontab program. You can import a table, remove and edit the current table.

The user table rows have the following syntax (use one or more spaces between elements):
<path> <mask> <command>
Where?
<path> is a filesystem path (each whitespace must be prepended by a backslash)
<mask> is a symbolic or numeric mask for events (see man inotify for more details)
<command> is an application or script to run on the events

The command may contain these wildcards:
 $$ - a dollar sign
 $@ - the watched filesystem path (see above)
 $# - the event-related file name
 $% - the event flags (textually)
 $& - the event flags (numerically)
Now with all that information, what can I do? Say you want to be notified each time /etc/hosts is modified and email us. Open incrontab make sure you are root for this example:
# incrontab -e

/etc/hosts IN_MODIFY mailx -s "Hosts file Has Been modified" mymail@mymail.com
Save the changes and open /etc/hosts and make a change and you should receive an email in your inbox.
At this point we have covered just the basics of what is possible with incron. Experiment with incron and see what other items you can monitor and what other commands you can execute on filesystem actions.

How to disable fsck on reboot in linux

In Linux, when a filesystem is mounted for certain times, or its last fsck was more than certain days ago, system will perform fsck on it when server reboot. The fsck process can take a few minutes to hours to finish, depending on the filesystem size.

If we want fast reboot, we can disable the fsck check, although it's not recommended to do so.
There’s several ways of accomplishing this. I will list all the methods beneath, just pick the one that fits the situation/you.
  1. Filesystem tunable
  2. Grub boot parameter
  3. Placing command files on your root device
  4. Update /etc/fstab
  5. Active reboot without FSCK

Filesystem tunable


Use the tune2fs command to tell your filesystem to have a max count of mounts before a check to 0 to disable it.
# tune2fs -c 0 /dev/sda2 
To list the current settings:
$ tune2fs /dev/sda2 | egrep -i 'mount count|check'
Mount count:                        1
Maximum mount count:        21
Last checked:                     Sat Mar 24 16:15:33 2012
Check interval:                    15552000 (6 months)
Next check after:                 Thu Sep 20 16:15:33 2012

the output is self-explained, for my system, /dev/sda2 will be checked after it's mounted for 21 times, or after Sep 20 16:15:33 2012.

To disable fsck check on /dev/sda2
$ tune2fs -c 0 -i 0 /dev/sda2
tune2fs 1.41.12 (17-May-2010)
Setting maximal mount count to -1
Setting interval between checks to 0 seconds.
check it again:
$ tune2fs /dev/sda2 | egrep -i 'mount count|check'
Mount count:                        1
Maximum mount count:        -1
Last checked:                     Sat Mar 24 16:15:33 2012
Check interval:                    0 (<none>)

Grub boot parameter


Add the following at the end of your grub boot linux line.

fastboot

This can be done by editing “grub.conf” or by editing the boot command via the grub menu at boot.

Placing command files on your root device 

 

To disable the filesystem check on boot.

# touch /fastboot

To enable a filesystem check on boot.

# touch /forcefsck

update /etc/fstab

in /etc/fstab, the last column is used by fsck to determine the order of performing file system check at reboot time. For root file system /, it should be 1, for other file systems, it should be 2. If we want to disable the fsck check for certain file system, we can specify 0 in the last column.

$ grep nofsck /etc/fstab
/dev/sda2        /mnt/nofsck        ext4        defaults        0  0

Active reboot without FSCK

 

  # shutdown -rf

Parameter reference: 

-r     Reboot after shutdown.
-f     Skip fsck on reboot.

the -f flags tells system to skip fsck for all filesystems during the reboot. Unlike the fstab and tune2fs methods, it only takes effect during current reboot, will not disable fsck permanently.


AIX Tips & tricks

Below are few of the AIX commands which will be useful for AIX admins.

1. To list machines configured in a NIM Server,
# lsnim -c machines

2. To list networks configured in a NIM Server,
# lsnim -c networks

3. To reset a machine (return to ready state)
# nim -Fo reset MachineName

4. To list core file settings for a user,
# lscore user1

The output will look like:
compression: on
path specification: default
corefile location: default
naming specification: off

5. To list the default settings for the system,

# lscore -d

The output will look like:
compression: off
path specification: on
corefile location: /corefiles
naming specification: off

6. To make any process run by root dump compressed core files and restore the location of the core files to the system default,

# chcore -c on -p default root
Note: If no default is specified, cores will dump in the current directory.

7. To enable a default core path for the system, type:

# chcore -p on -l /corefiles -d

8. To scan logical volume lv01, report the status of each partition, and have every block of each partition read to determine whether it is capableof performing I/O operations, type:

# mirscan -l lv01

9. To do the above operation in a PV,

# mirscan -p hdisk1

10. To do the above operation in a VG,

# mirscan -v vg01

11. To determine if the 64-bit kernel extension is loaded,

# genkex grep 64

12. To list all JFS file systems,

# lsjfs

13. To list all JFS2 file systems

# lsjfs2

14. To mirror a terminal1 on terminal2
a. Open terminal 1 and find the pts value (ps -ef grep pts)

b. Open terminal 2 and enter 'portmir -t pts/1'
c. Now you will see commands and outputs from terminal 1 in terminal 2.
This is basically monitor a terminal.
d. Say "portmir -o" to end the mirroring after the use

15. To identify the current run level,

# cat /etc/.init.state

16. To list the available CD ROM drives,

# lsdev -Cc cdrom

17. To find out the speed of your network adapter,

# entstat -d ent0 grep "Media Speed"

18. To find out when your system was last installed/updated

# lslpp -f bos.rte

19. To list the status of your tape drive,

# tctl -f /dev/rmt0 status

20. How to setup anonymous ftp in AIX

Run the below script to setup anon ftp,
# /usr/lpp/tcpip/samples/anon.ftp

21. If telnet takes more time to produce a prompt, do the below checks

a. do nslookup of the client ip from the aix serverb.
b. Check the nameservers in /etc/resolv.confc.
c. Check the 'hosts' entry in /etc/netsvc.conf or NSORDER variable

This issue might be due to the DNS configuration issue. Pointing to a good nameserver should solve the problem.

22. How to shutdown the system to maintenance mode ?

# shutdown -Fm

23. How to log ftp accesses to a file

a. Add the below line in /etc/syslog.confdaemon.debug /tmp/daemon.log
b. # touch /tmp/daemon.log
c. # refresh syslogd
d. Modify your inetd.conf so that ftpd is called with the "-l" flag.

24. How to find a file name from inode number ?

# ncheck -i xxxx /mountpoint
where xxxx -> inode number of the file

25. How to redirect the system console to a file or tty temporarily

# swcons /tmp/console.out

or

# swcons /dev/tty5

26. How to recreate a deleted /dev/null file ?

# /bin/mknod /dev/null c 2 2

27. How to add commands that should get executed during every system shutdown ?

Add them to /etc/rc.shutdown

28. How to reduce the size or do cleanup of /var/adm/wtmp ?

# > /var/adm/wtmp

29. How to find out the fileset a file belongs to ?

# which_fileset command_name

30. In which file, the mapping of file Vs fileset stored ?

# /usr/lpp/bos/AIX_file_list

31. How to set maximum logins for a user in a system ?

Change the value of "maxlogins" under "usw" stanza in /etc/security/login.cfg

32. How to change the initial message that prints while logging in ?

Change the value of "herald" in /etc/security/login.cfg

33. How to set the # of seconds the user is given to enter their password ?

Change the value of "logintimeout" under "usw" stanza in /etc/security/login.cfg

Unmirroring rootvg in AIX

Root Volume Group (rootvg) is a volume group containing the Base Operating System (BOS). Logical volume (lv) in rootvg may be doubled or more in copies with 2 or more physical volume (hard disk) for availability and reliability of the AIX system. The following steps are to unmirror a rootvg, if for whatever reason the rootvg needs to run on single logical volume (lv) on single physical volume (pv) only.

Check and Determine if rootvg is Mirrored

 

In mirror mode, each logical volume in rootvg such as filesystems “/”, “/usr”, “/var”, “/tmp”, “/home”, “/opt” and default boot, paging and jfslog LVs should be mirrored. In AIX, mirrorvg will create additional copy of image for all logical volumes in the volume group.

Use the following command to check if a rootvg is mirrored:

# lsvg -l rootvg

If the output shows that for each LP there are 2 PPs then its mirrored.

Check and Determine Which Disks the Mirrored rootvg is Located

For each logical volume (LV) name listed in output of “lsvg -l rootvg” command, run the following command:

lslv -m

The output data will tell you on which disk each copy of each logical partitions for each LV.

Unmirror rootvg

 

Important: The following instructions have the risk of making your AIX system unbootable or corrupting the data. So make you have advanced system administration experience before running the process of unmirroring.

To unmirror the root volume group (rootvg), follow the steps below (scenario: rootvg is contained on hdisk01 and mirrored onto hdisk11, and the steps will remove the mirror on hdisk11 (regardless of the disk from which you previously booted)):

   1. To unmirror the rootvg from hdisk11, enter the following command:

      unmirrorvg rootvg hdisk11

      This command turns quorum back on for rootvg. When unmirrorvg is executed, the default COPIES value for each logical volume becomes 1.

   2. To update the booted disk link, enter the following command:

      ln -f /dev/rhdisk01 /dev/ipldevice
  
 3. To reduce the disk out of rootvg, type the following command:

      reducevg rootvg hdisk11
  
  4. To initilize the boot record of the remaining disk again, enter the following command:

      bosboot -a -d /dev/hdisk01

      bosboot command is a must to initialize the boot record on the remaining disk hdisk01 again.
  
 5. To modify the boot list to remove the unmirrored disk, type the following command:

      bootlist -m normal hdisk01

      bootlist command is a must so that the system only boots to the disk remaining (hdisk01) in rootvg.

 6. Restart AIX machine, as unmirroring turns quorum back on for rootvg, a reboot is required for this to take effect.

Note: The reducevg command in step 3 will fail if there are non-mirrored logical volumes such as raw logical volumes and system dump devices on the disk.

UNIX one-liners continued

These one liners may be of help to identify user or group info...

1. To extract only the userid from the /etc/passwd file based on a string sequence in the GECOS field.

    cat /etc/passwd | awk -F":" '/\[I]/ { print $1 }'
                        OR
    cat /etc/passwd | awk -F":" '/GB\/I/ { print $1 }'


2.  To list records in /etc/passwd of users who are members of  the staff group.

    grep <groupname>  /etc/group | awk -F":" '{ print$4}'  | sed 's/\,/ /g'  | xargs -n1 echo  > patFile ; grep -f  patFile  /etc/passwd ; rm patFile;

    grep staff /etc/group | awk -F":" '{ print$4}'  | sed 's/\,/ /g'  | xargs -n1 echo  > patFile ; grep -f  patFile  /etc/passwd ; rm patFile;

3. To list userids and home directories of users of the staff group.

    grep <groupname> /etc/group | awk -F":" '{ print$4}'  | sed 's/\,/ /g'  | xargs -n1 echo  > patFile ; grep -f  patFile  /etc/passwd  | awk -F ":" '{ print $1, $6}'; rm patFile;

    grep staff /etc/group | awk -F":" '{ print$4}'  | sed 's/\,/ /g'  | xargs -n1 echo  > patFile ; grep -f  patFile  /etc/passwd  | awk -F ":" '{ print $1, $6}'; rm patFile;


4. To list all the groups of which a userid is a member of

    grep <userid>  /etc/group | awk -F":" '{ print $1 }'

    grep sabari /etc/group | awk -F":" '{ print $1 }'

5. To list all users of a group and further grep them by a department signifier in the GECOS field.

    grep <groupname> /etc/group | awk -F":" '{ print$4}' | sed 's/\,/ /g'  | xargs -n1 echo  > patFile ;  grep -f  patFile  /etc/passwd | egrep \/I\/ | awk -F":" '{ print $1," ", $5 }'; rm patFile;

    grep sys group.file | awk -F":" '{ print$4}' | sed 's/\,/ /g'  | xargs -n1 echo  > patFile ;  grep -f  patFile  /etc/passwd | egrep \/I\/ | awk -F":" '{ print $1," ", $5 }'; rm patFile;


6. To print only one particular field from a file

        cat <filename> |awk '{ print $5 }'

UNIX one liners - Very helpful in daily activities

In our day to day activities, we will be doing most of the routine tasks which needs many commands to run for completeing the task.
Below are few of the one-liner commands that can achieve our tasks.


These one liners may be of help to identify files to be acted upon by xargs.

1. To identify files in dir /tmp dated March with  psx2 anywhere in the name :and remove them

    find /tmp -type f -name *psx2* -ls | awk '$8=="Mar"' | awk -F" " ' { print  $11 }' | xargs rm -rf


2. To identify multiple name formats dated Apr 10 and gzip them ...

    cd /logs/of/surpriseme

    find `pwd` -type f \( -name user1_log..txt -o -name system3_log.txt \) -ls | grep "Apr 10" | awk -F" " ' { print  $11 }' | xargs gzip



3. To identify files dated March 20-31 and remove them.

    find `pwd` -type f  -ls |  awk '$6=="Mar" && $7>19 && $7<32' | awk -F" " ' { print  $11 }' | xargs rm -rf
       
        find `pwd` -type f  -ls |  awk '$8=="Aug" && $9>19 && $9<22'


4. To find the total filespace used by User willy on a system.

    find / -user willy -type f -ls 2> /dev/null | awk '{ sum += $7 } END { printf " %7.3f MB \n", (sum / (1024 * 1024)) }'

5. To find 20 biggest files on the /var filesystem showing their owner, group,  size in MB and path:

    find  /var -type f   -ls | sort -rn +6 | head -20 | awk -F " " '{ printf " %s \t %s \t %7.3f MB \t %s \n" , $5, $6,  ( $2/1024), $11}'

6. To find  files older than 12-Jan-2007  and move them to the /tmp dir. First identify a file with that date stamp in the file system. Then

    find `pwd` -type f  !-newer /absolute_path_to/file_with_cutoffpoint/datestamp - ls | awk -F " " '{ print $11 }' | xargs -I { } mv { } /tmp
7. To rename files names with extension .txt to .log

    find $(pwd) -type f -name *.txt | xargs -i mv {} {}.log


Converting ext2 filesystem to ext3 in Linux

The tune2fs allows you to convert an ext2 filesystem to ext3.

Note : Always use the e2fsck utility to check your filesystem before and after using tune2fs.

A default installation of Red Hat Enterprise Linux uses ext3 for all file systems.
To convert an ext2 filesystem to ext3, log in as root and type the following command in a terminal:

/sbin/tune2fs -j <block_device>

where <block_device> contains the ext2 filesystem you wish to convert.

A valid block device could be one of two types of entries:
• A mapped device — A logical volume in a volume group, for example, /dev/mapper/VolGroup00-LogVol02.

• A static device — A traditional storage volume, for example, /dev/hdbX, where hdb is a storage
device name and X is the partition number.

Issue the df command to display mounted file systems.

You must recreate the initrd image so that it will contain the ext3 kernel module. To create this,
run the mkinitrd program. For information on using the mkinitrd command, type man mkinitrd.
Also, make sure your GRUB configuration loads the initrd.

If you fail to make this change, the system still boots, but the file system is mounted as ext2 instead
of ext3.

Converting back Linux ext3 file system to ext2 file system

If you wish to revert a partition from ext3 to ext2 for any reason, you must first unmount the partition
by logging in as root and typing,

umount /dev/mapper/VolGroup00-LogVol02

Next, change the file system type to ext2 by typing the following command as root:

/sbin/tune2fs -O ^has_journal /dev/mapper/VolGroup00-LogVol02

Check the partition for errors by typing the following command as root:

/sbin/e2fsck -y /dev/mapper/VolGroup00-LogVol02

Then mount the partition again as ext2 file system by typing:

mount -t ext2 /dev/mapper/VolGroup00-LogVol02/mount/point

In the above command, replace /mount/point with the mount point of the partition.

Next, remove the .journal file at the root level of the partition by changing to the directory
where it is mounted and typing:

rm -f .journal

You now have an ext2 partition.

If you want to permanently change the partition to ext2, remember to update the /etc/fstab file.

Disabling IPv6 support in Red Hat Enterprise Linux 6


Create a file /etc/modprobe.d/ipv6.conf with the following contents:

options ipv6 disable=1

For completeness, it is a good idea to configure the ip6tables service not to start at boot by issuing the following command:

# chkconfig ip6tables off

disable ipv6 support in the kernel through /etc/sysctl.conf :

# ipv6 support in the kernel, set to 0 by default
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1


Reboot the system to disable IPv6 support.


Note: There is a special case where this might not work, please see The "ipv6 disable=1" option does not seem to work on Red Hat Enterprise Linux 6 system. Disabling ipv6 in the sysctl.conf will ensure ipv6 isn't used even if the ipv6 module is loaded and can work as a short term solution (until a full reboot).

"vmount: operation not permitted" Error From Linux NFS Server

While trying to mount a Linux NFS share in an AIX server, we get the below error.
localhost:root:/#mount 10.1.1.1:/nfs_data /mnt/nfsShare
mount: 1831-008 giving up on:
10.1.1.1:/nfs_data
vmount: Operation not permitted.

Sometimes Linux NFS servers will do port checking and require that the NFS client use a reserved port.

nfso -o nfs_use_reserved_ports=1

If the mount is going to be permanent, then the change needs to survive across a reboot. The nfs option must be changed permanently. On AIX 4.x and 5.1, the command above should be added to the startup scripts (possibly /etc/rc.nfs). On AIX 5.2 and above, the change can be made permanent by adding the -p flag.

 nfso -p -o nfs_use_reserved_ports=1

Top Netstat commands with examples

Netstat command displays various network related information such as network connections, routing tables, interface statistics, masquerade connections, multicast memberships etc.,
In this article, let us review 10 practical unix netstat command examples.

1. List All Ports (both listening and non listening ports)

List all ports using netstat -a

# netstat -a | more
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 localhost:30037         *:*                     LISTEN
udp        0      0 *:bootpc                *:*                                

Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  2      [ ACC ]     STREAM     LISTENING     6135     /tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     5140     /var/run/acpid.socket

List all tcp ports using netstat -at

# netstat -at
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 localhost:30037         *:*                     LISTEN
tcp        0      0 localhost:ipp           *:*                     LISTEN
tcp        0      0 *:smtp                  *:*                     LISTEN
tcp6       0      0 localhost:ipp           [::]:*                  LISTEN

List all udp ports using netstat -au

# netstat -au
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
udp        0      0 *:bootpc                *:*
udp        0      0 *:49119                 *:*
udp        0      0 *:mdns                  *:*

2. List Sockets which are in Listening State

List only listening ports using netstat -l

# netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 localhost:ipp           *:*                     LISTEN
tcp6       0      0 localhost:ipp           [::]:*                  LISTEN
udp        0      0 *:49119                 *:*

List only listening TCP Ports using netstat -lt

# netstat -lt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 localhost:30037         *:*                     LISTEN
tcp        0      0 *:smtp                  *:*                     LISTEN
tcp6       0      0 localhost:ipp           [::]:*                  LISTEN

List only listening UDP Ports using netstat -lu

# netstat -lu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
udp        0      0 *:49119                 *:*
udp        0      0 *:mdns                  *:*

List only the listening UNIX Ports using netstat -lx

# netstat -lx
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  2      [ ACC ]     STREAM     LISTENING     6294     private/maildrop
unix  2      [ ACC ]     STREAM     LISTENING     6203     public/cleanup
unix  2      [ ACC ]     STREAM     LISTENING     6302     private/ifmail
unix  2      [ ACC ]     STREAM     LISTENING     6306     private/bsmtp

3. Show the statistics for each protocol

Show statistics for all ports using netstat -s

# netstat -s
Ip:
    11150 total packets received
    1 with invalid addresses
    0 forwarded
    0 incoming packets discarded
    11149 incoming packets delivered
    11635 requests sent out
Icmp:
    0 ICMP messages received
    0 input ICMP message failed.
Tcp:
    582 active connections openings
    2 failed connection attempts
    25 connection resets received
Udp:
    1183 packets received
    4 packets to unknown port received.
.....

Show statistics for TCP (or) UDP ports using netstat -st (or) -su

# netstat -st

# netstat -su

4. Display PID and program names in netstat output using netstat -p

netstat -p option can be combined with any other netstat option. This will add the “PID/Program Name” to the netstat output. This is very useful while debugging to identify which program is running on a particular port.
# netstat -pt
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        1      0 ramesh-laptop.loc:47212 192.168.185.75:www        CLOSE_WAIT  2109/firefox
tcp        0      0 ramesh-laptop.loc:52750 lax:www ESTABLISHED 2109/firefox

5. Don’t resolve host, port and user name in netstat output

When you don’t want the name of the host, port or user to be displayed, use netstat -n option. This will display in numbers, instead of resolving the host name, port name, user name.
This also speeds up the output, as netstat is not performing any look-up.
# netstat -an
If you don’t want only any one of those three items ( ports, or hosts, or users ) to be resolved, use following commands.
# netsat -a --numeric-ports

# netsat -a --numeric-hosts

# netsat -a --numeric-users

6. Print netstat information continuously

netstat will print information continuously every few seconds.
# netstat -c
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 ramesh-laptop.loc:36130 101-101-181-225.ama:www ESTABLISHED
tcp        1      1 ramesh-laptop.loc:52564 101.11.169.230:www      CLOSING
tcp        0      0 ramesh-laptop.loc:43758 server-101-101-43-2:www ESTABLISHED
tcp        1      1 ramesh-laptop.loc:42367 101.101.34.101:www      CLOSING
^C

7. Find the non supportive Address families in your system

netstat --verbose
At the end, you will have something like this.
 netstat: no support for `AF IPX' on this system.
 netstat: no support for `AF AX25' on this system.
 netstat: no support for `AF X25' on this system.
 netstat: no support for `AF NETROM' on this system.

8. Display the kernel routing information using netstat -r

# netstat -r
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.1.0     *               255.255.255.0   U         0 0          0 eth2
link-local      *               255.255.0.0     U         0 0          0 eth2
default         192.168.1.1     0.0.0.0         UG        0 0          0 eth2
Note: Use netstat -rn to display routes in numeric format without resolving for host-names.

9. Find out on which port a program is running

# netstat -ap | grep ssh
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        1      0 dev-db:ssh           101.174.100.22:39213        CLOSE_WAIT  -
tcp        1      0 dev-db:ssh           101.174.100.22:57643        CLOSE_WAIT  -
Find out which process is using a particular port:
# netstat -an | grep ':80'

10. Show the list of network interfaces

# netstat -i
Kernel Interface table
Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0       1500 0         0      0      0 0             0      0      0      0 BMU
eth2       1500 0     26196      0      0 0         26883      6      0      0 BMRU
lo        16436 0         4      0      0 0             4      0      0      0 LRU
Display extended information on the interfaces (similar to ifconfig) using netstat -ie:
# netstat -ie
Kernel Interface table
eth0      Link encap:Ethernet  HWaddr 00:10:40:11:11:11
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Memory:f6ae0000-f6b00000
 
 

Uniq command in Unix

Uniq command in unix or linux system is used to suppress the duplicate lines from a file. It discards all the successive identical lines except one from the input and writes the output.

The syntax of uniq command is
uniq [option] filename

The options of uniq command are:
  • c : Count of occurrence of each line.
  • d : Prints only duplicate lines.
  • D : Print all duplicate lines
  • f : Avoid comparing first N fields.
  • i : Ignore case when comparing.
  • s : Avoid comparing first N characters.
  • u : Prints only unique lines.
  • w : Compare no more than N characters in lines
Uniq Command Examples:

First create the following example.txt file in your unix or linux operating system.
# cat example.txt
Unix operating system
unix operating system
unix dedicated server
linux dedicated server

1. Suppress duplicate lines

The default behavior of the uniq command is to suppress the duplicate line. Note that, you have to pass sorted input to the uniq, as it compares only successive lines.
# uniq example.txt
unix operating system
unix dedicated server
linux dedicated server

If the lines in the file are not in sorted order, then use the sort command and then pipe the output to the uniq command.
# sort example.txt | uniq

2. Count of lines.

The -c option is used to find how many times each line occurs in the file. It prefixes each line with the count.
# uniq -c example.txt
      2 unix operating system
      1 unix dedicated server
      1 linux dedicated server

3. Display only duplicate lines.

You can print only the lines that occur more than once in a file using the -d option.
# uniq -d example.txt
unix operating system

# uniq -D example.txt
unix operating system
unix operating system

The -D option prints all the duplicate lines.

4. Skip first N fields in comparison.

The -f option is used to skip the first N columns in comparison. Here the fields are delimited by the space character.
# uniq -f2 example.txt
unix operating system
unix dedicated server

In the above example the uniq command, just compares the last fields. For the first two lines, the last field contains the string "system". Uniq prints the first line and skips the second. Similarly it prints the third line and skips the fourth line.

5. Print only unique lines.

You can skip the duplicate lines and print only unique lines using the -u option
# uniq -u example.txt
unix dedicated server
linux dedicated server
 

Using iptrace command in AIX

Creating, formatting, and reading packet traces is sometimes required to resolve problems. However, the most appropriate tool varies, depending on operating system.
The iptrace command can be very useful to find out what network traffic flows to and from an AIX system.

You can use any combination of these options, but you do not need to use them all:

-a   Do NOT print out ARP packets.

-s   [source IP] Limit trace to source/client IP address, if known.

-d   [destination IP] Limit trace to destination IP, if known.

-b   Capture bidirectional network traffic (send and receive packets).

-p   [port] Specify the port to be traced.

-i   [interface] Only trace for network traffic on a specific interface.

Examples:

1. Run iptrace on AIX interface en1 to capture port 80 traffic to file trace.out from a single client IP to a server IP:

    # iptrace -a -i en1 -s clientip -b -d serverip -p 80 trace.out

This trace will capture both directions of the port 80 traffic on interface en1 between the clientip and serverip and sends this to the raw file of trace.out.

2. To stop the trace:

    # ps -ef|grep iptrace
    # kill <PID>


3. The ipreport command can be used to transform the trace file generated by iptrace to human readable format:

    # ipreport trace.out > trace.report

4. To start the iptrace daemon with the System Resource Controller (SRC), enter:

startsrc -s iptrace -a "/tmp/nettrace"

To stop the iptrace daemon with SRC enter the following:

stopsrc -s iptrace


5. To record packets coming in and going out to any host on every interface, enter the command in the following format:

iptrace /tmp/nettrace

The recorded packets are received on and sent from the local host. All packet flow between the local host and all other hosts on any interface is recorded. The trace information is placed into the /tmp/nettrace file.

6. To record packets received on an interface from a specific remote host, enter the command in the following format:

iptrace -i en0 -p telnet -s airmail /tmp/telnet.trace

The packets to be recorded are received on the en0 interface, from remote host airmail, over the telnet port. The trace information is placed into the /tmp/telnet.trace file.

7. To record packets coming in and going out from a specific remote host, enter the command in the following format:

 iptrace -i en0 -s airmail -b /tmp/telnet.trace

The packets to be recorded are received on the en0 interface, from remote host airmail. The trace information is placed into the /tmp/telnet.trace file.



YUM in RHEL

 Yum or Yellow dog Update, Modified is a package manager that was developed by Duke University to improve the installation of RPMs.

Yum searches numerous repositories for packages and their dependencies so they may be installed together in an effort to alleviate dependency issues. Red Hat Enterprise Linux 5 uses Yum to fetch packages and install RPMs.

Yum uses a configuration file at /etc/yum.conf. Also refer yum(8) man page for more information.

There are multiple ways by which you can install a repository on the system and install/update packages :
  1. Add an existing repository.
  2. Setup a new repository having packages populated from ISO's downloaded from RHN.
  3. Register the system on RHN and subscribe to the channels depending on the subscription you have.
To automatically install software from the DVD I would like to use YUM, but I have to set it up first. There is a main configuration file - yum.conf which resides in /etc. This is the default content of it:

# cat /etc/yum.conf
[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=0
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
installonly_limit=3


Next we need the Media ID from the DVD. It can be obtained from a file call media.repo from the DVD. First mount the DVD:

# mkdir -p /mnt/dvd
# mount /dev/sr0 /mnt/dvd


Or mount the ISO file:

# mkdir -p /mnt/dvd
# mount /root/rhel6.iso /mnt/dvd -o loop


Get Media ID from DVD:

# cat /mnt/dvd/media.repo
[InstallMedia]
name=Red Hat Enterprise Linux 6.0
mediaid=0123456789.012345
...


Now configure /etc/yum.repos.d/file.repo and add the DVD (or the ISO file) as a repository and add /mnt/dvd as baseurl:

# cat /etc/yum.repos.d/file.repo
[dvd]
mediaid=0123456789.012345
name=DVD
baseurl=file:///mnt/dvd
enabled=1
gpgcheck=0


Check if it accept the repository:

# yum repolist
Loaded plugins: rhnplugin
This system is not registered with RHN.
RHN support will be disabled.
dvd                          | 3.7 kB     00:00 ...
dvd/primary_db               | 2.9 MB     00:00 ...


Install a package from the repository (e.g. tree):

# yum install tree
Loaded plugins: rhnplugin
This system is not registered with RHN.
RHN support will be disabled.
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package tree.x86_64 0:1.5.3-2.el6 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved
Package   Arch       Version         Repository     Size
tree      x86_64     1.5.3-2.el6     dvd            36 k

Transaction Summary
Install       1 Package(s)
Upgrade       0 Package(s)

Total download size: 36 k
Installed size: 65 k
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
  Installing     : tree-1.5.3-2.el6.x86_64

1/1
Installed:
  tree.x86_64 0:1.5.3-2.el6


Complete! Just keep in mind that you always have to mount the DVD or the ISO file before you can install any software via YUM.

Yum must be running as root. Here are some useful commands:

1) Install a package:
yum install package
Example:
yum install httpd
2) Remove a package:
yum remove package
Example:
yum remove httpd
3) Update a package:
yum update package
Example:
yum update httpd
4) Search for a package:
yum search package
Example:
yum search httpd
5) Find information about a package:
yum info package
Example:
yum info httpd
6) List packages containing a certain term:
yum list term
Example:
yum list httpd
7) Find what package provides a particular file:
yum whatprovides 'path/filename'
Example:
yum whatprovides 'etc/httpd.conf'
yum whatprovides '*/libXp.so.6'
8) Update all installed packages with kernel package :
yum update
Example:
yum update
9) To update a specific package:
yum update <package-name>
Example:
yum update openssh-server 
 
  

Longer login names in AIX

User names can only be eight characters or fewer in AIX version 5.2 and earlier. Starting with AIX version 5.3, IBM increased the maximum number of characters to 255. To verify the setting in AIX 5.3 and later, you can extract the value from getconf:
# getconf LOGIN_NAME_MAX
9
Or use lsattr:
# lsattr -El sys0 -a max_logname
max_logname 9 Maximum login name length at boot time True
To change the value, simply adjust the v_max_logname parameter (shown as max_logname in lsattr) using chdev to the maximum number of characters desired plus one to accommodate the terminating character. For example, if you want to have user names that are 128 characters long, you would adjust the v_max_logname parameter to 129:
# chdev -l sys0 -a max_logname=129
sys0 changed
Please note that this change will not go into effect until you have rebooted the operating system. Once the server has been rebooted, you can verify that the change has taken effect:
# getconf LOGIN_NAME_MAX
128
Keep in mind, however, that if your environment includes IBM RS/6000 servers prior to AIX version 5.3 or operating systems that cannot handle user names longer than eight characters and you rely on NIS or other authentication measures, it would be wise to continue with the eight-character user names.
 

Difference between major and minor number in AIX


A major number refers to a type of device, and a minor number specifies a particular device of that type or sometimes the operation mode of that device type.

Example:
# lsdev -Cc tape
rmt0 Available 3F-08-02 IBM 3580 Ultrium Tape Drive (FCP)
rmt1 Available 3F-08-02 IBM 3592 Tape Drive (FCP)
smc0 Available 3F-08-02 IBM 3576 Library Medium Changer (FCP)
In the list above:

rmt1 is a standalone IBM 3592 tape drive;
rmt0 is an LTO4 drive of a library;
smc0 is the medium changer (or robotic part) of above tape library.

Now look at their major and minor numbers:
# ls -l /dev/rmt* /dev/smc*
crw-rw-rwT 1 root system 38, 0 Nov 13 17:40 /dev/rmt0
crw-rw-rwT 1 root system 38,128 Nov 13 17:40 /dev/rmt1
crw-rw-rwT 1 root system 38, 1 Nov 13 17:40 /dev/rmt0.1
crw-rw-rwT 1 root system 38, 66 Nov 13 17:40 /dev/smc0
All use IBM tape device driver (and so have the same major number of 38), but actually they are different entities (with minor number of 0, 128 and 66 respectively). Also, compare rmt0 and rmt0.1. It's the same device, but with different mode of operation.