Cloud DevOps Admin Guide

DNS server IP address/hostnames are defined in /etc/resolv.conf in AIX servers.

Here is an example of /etc/resolv.conf

nameserver 192.168.2.12
nameserver 192.168.2.13
nameserver 192.168.2.14
search india.cope.com usa.cope.com uk.cope.com

Let us see few tips and tricks on using nslookup.

1. To look up address in non interactive way,

$ nslookup webserv
Server:  dnserver1.india.cope.com
Address:  192.168.2.12

Name:    webserv.india.cope.com
Address:  192.168.2.211
$

2. To look up address in interactive way,

$nslookup
Default Server:  dnserver1.india.cope.com
Address:  192.168.2.12

> websrv
Server:  dnserver1.india.cope.com
Address:  192.168.2.12

Name:    webserv.india.cope.com
Address:  192.168.2.211

> exit
$

3. To look up hostname in non interactive way,

$ nslookup 192.168.2.211
Server:  dnserver1.india.cope.com
Address:  192.168.2.12

Name:    webserv.india.cope.com
Address:  192.168.2.211
$

4. To look up hostname in interactive way,

$ nslookup
Default Server:  dnserver1.india.cope.com
Address:  192.168.2.12

> 192.168.2.211
Server:  dnserver1.india.cope.com
Address:  192.168.2.12

Name:    webserv.india.cope.com
Address:  192.168.2.211

> exit
$

4. To look up MX data,

$ nslookup
Default Server:  dnserver1.india.cope.com
Address:  192.168.2.12

> set q=mx
> rajs
Server:  dnserver1.india.cope.com
Address:  192.168.2.12

Name:    rajs.india.cope.in
Address:  0.0.0.0
> exit
$

5. How to query a specific DNS server for an address ?

We can do in both interactive and non-interactive ways.
Below example will query for the IP address of the host websrv using the DNS serer "192.168.2.15" which is not specified in the /etc/resolv.conf file.

Interactive Way:

$nslookup
Default Server:  dnserver1.india.cope.com
Address:  192.168.2.12

> server 192.168.2.15
Default Server:  dnserver4.india.cope.com
Address:  192.168.2.15

> websrv
Server:  dnserver4.india.cope.com
Address:  192.168.2.15

Name:    webserv.india.cope.com
Address:  192.168.2.211

> exit
$

Non-Interactive Way:

$nslookup - websrv 192.168.2.15

Server:  dnserver4.india.cope.com
Address:  192.168.2.15

Name:    webserv.india.cope.com
Address:  192.168.2.211

6.What is the difference between Authoritave and Non-authoritative answers ?

When you query for something for the first time, we get the answer from the DNS server and it will be displayed as well as stored in the local cache. This is called as authoritative answer. ie., getting the answer directly from the DNS server. This answer will be kept in cache for certain time.

But when you do the same query for the second time, we get the answer from the cache instead of the DNS server. This is called as non-authoritative answer.

7. How will you specify an alternate DNS server when using nslookup ?

For using 192.168.2.24 as an alternate DNS server,
$ nslookup - 192.168.2.24

This will query the alternate server instead of the DNS servers configured in /etc/resolv.conf file.

8. How will you query a MX record in an alternate server ?

$ nslookup - type=mx bashi.usa.cope.com 192.168.2.24

9. How will you debug while querying a DNS server ?


$ nslookup
Default Server:  dnserver1.india.cope.com
Address:  192.168.2.12
> set debug
> webserv

Server:  dnserver1.india.cope.com
Address:  192.168.2.12

;; res_nmkquery(QUERY, websrv.india.cope.com, IN, A)
------------
Got answer:
    HEADER:
        opcode = QUERY, id = 54305, rcode = NOERROR
        header flags:  response, authoritive answer, want recursion, recursion available
        questions = 1,  answers = 1,  authority records = 0,  additional = 0

    QUESTIONS:
        websrv.india.cope.com, type = A, class = IN
    ANSWERS:
    ->  webserv.india.cope.com
        internet address = 192.168.2.211
        ttl = 3600 (1H)

------------
Name:    webserv.india.cope.com
Address:  192.168.2.211

10.
Each DNS packet is composed of 5 sections as given below

1. Header Section
2. Question Section
3. Answer Section
4. Authority Section
5. Additional Section

11. You can use options with the nslookup command using 'set' sub-command.
Here are few options ...

port=53          By default, DNS service uses port 53. If you have a DNS service on a different port, you can sue the port option to specify the prot number.

timeout=10    It is used to specify the timeout value. If the name server doesn't respond in 10 seconds, nslookup will send the query again.

debug              To turn on debug mode

nodebug         To turn off debug mode

querytype=A  By default, nslookup looks for A record. If you type the IP address, it will look for PTR record.. You can change the querytype to MX or SOA.

12. How will you come out of the interactive nslookup session.

You can use exit command or type ^D (control+D) to come out of the session.

In order to have a working mksysb resource in a NIM environment you need to have 2 items:   a NIM mksysb resource which points to a mksysb file in the filesystem. 

A good NIM mksysb resource looks like the image below.   The NIM mksysb resource is stored in the ODM and has a "location" attribute that points to a file in the filesystem.

However, if something (or someone) deletes the mksysb file from the filesystem, but doesn't delete the NIM mksysb resource, you are left with an orphaned mksysb NIM resource.   The mksysb resource will still show up in NIM and still appear to be useable, however any operations that try to use it will fail since its backing mksysb file isn't present.   An oprhaned mksysb resource looks like the image below:

How to prevent orphaned NIM mksysb resources:

The best way to prevent a orphaned NIM mksysb resource is to never delete mksysb files from the filesystem using "rm".   Instead, if you no longer need a NIM mksysb, use the "nim" command to delete it and also specify that the backing mksysb file should be deleted as well.   This can be done with a command such as this:

nim -o remove -a rm_image=yes aix3_mksysb

Substitute "aix3_mksysb" for the name of the mksysb that you want to delete.  The "-a rm_image=yes" tells NIM to not only delete the NIM resource from the ODM, but to also delete the backing mksysb file from the filesystem.

for mksysb in `lsnim -t mksysb | awk '{print $1}'`; do printf "%-20s " $mksysb; location=`lsnim -l $mksysb | grep location | awk '{print $3}'`; [ -e "$location" ] && echo " OK         $location" || echo " Not Found  $location"; done

The output looks like this:

aix1_mksysb           OK         /tmp/aix1_mksysb
aix2_mksysb           OK         /tmp/aix2_mksysb
aix3_mksysb           OK         /tmp/aix3_mksysb
aix4_mksysb           OK         /tmp/aix4_mksysb
aix5_mksysb           OK         /tmp/aix5_mksysb
aix6_mksysb           Not Found  /tmp/aix6_mksysb
aix7_mksysb           OK         /tmp/aix7_mksysb
aix8_mksysb           OK         /tmp/aix8_mksysb

Based on the output we can clearly see all of the mksysb's are good except for aix6_mksysb which doesn't have a a backing mksysb file present in the filesystem.

If you ever need to build multiple servers that will all have the same volume groups, logical volumes, and filesystems you can use "savevg" and "restvg" to save yourself a bunch of time and duplicated work. 

This also works if you are ever asked to build a new server that should be setup with the same VG/LV/FS's as an older server. 

You start by setting up one of the servers with the volume groups, logical volumes, and filesystems that you will need.  Next you can use the "savevg -r" command to backup just the volume group/LV/Filesystem structure information.  With the "-r" flag it doesn't backup any data in the filesystems, which makes it quick and the backup file very small. 

In this example we want to duplicate the "appvg" structure on to another server:

# lsvg -l appvg
appvg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
loglv02             jfs2log    1       1       1    open/syncd    N/A
applv05             jfs2       52      52      4    open/syncd    /app5
fslv05              jfs2       25      50      2    open/syncd    /app2
loglv03             jfslog     1       1       1    closed/syncd  N/A
#
# savevg -r -f /appvg.savevg appvg

Creating information file for volume group appvg.................................................................

Backing up user Volume Group information files only.
Creating list of files to back up.
Backing up 6 files

6 of 6 files (100%)
0512-038 savevg: Backup Completed Successfully.
 

 

Next you copy the "/appvg.savevg" file on to all the servers that you want to setup the VG/LV/FS's on.  You can use something like "scp" or "sftp" or another protocol to transfer the file. 

On the other servers, you run "restvg" to restore the VG/LV/FS structures from the file:

 

# restvg -r -f /appvg.savevg hdisk1 hdisk2

Will create the Volume Group:   appvg
Target Disks:   hdisk1 hdisk2
Allocation Policy:
        Shrink Filesystems:     no
        Preserve Physical Partitions for each Logical Volume:   no

Enter y to continue: y
0516-1254 /usr/sbin/mkvg: Changing the PVID in the ODM.
appvg
loglv02
applv05
fslv05
loglv03
#
# lsvg -l appvg
appvg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
loglv02             jfs2log    1       1       1    open/syncd    N/A
applv05             jfs2       52      52      2    open/syncd    /app5
fslv05              jfs2       25      50      2    open/syncd    /app2
loglv03             jfslog     1       1       1    closed/syncd  N/A

 

As you can see the "restvg -r" command restored the volume group, logical volumes, and filesystems, and even mounts the filesystems for you!  Note that the fslv05 LV was mirrored before on the original server and when restored it is still mirrored.   The original volume group on the source server was on 4 hdisks, but on the destination server only had 2 hdisks.   But this isn't a problem, restvg is able to take care of it as long as you have enough disks to accommodate the mirroring and enough total space for all the logical volumes.  You can even use the "-s" flag on restvg to attempt to shrink the filesystems if the destination hdisks aren't large enough to hold all the original LV's. 

Over the years I've noticed that a lot of the core utilities on AIX are actually shell scripts. 

Here are some examples of these utilities on AIX that are either shell scripts (ksh/csh) or in some cases Perl scripts:

 

mksysb
oslevel
mkcd / mkdvd
useradd
userdel
usermod
prtconf
bosboot
mklv
shutdown
snap
lsconf
dsh
lsmksysb
savevg
which
chpv
chvg
cplv
exportvg
extendlv
migratelp
migratepv
mirrorvg
mktcpip
mkwpar
multibos
reducevg
reorgvg
replacepv
rmlv
rmlvcopy
splitlvcopy
splitvg
unmirrorvg
varyoffvg

 

As you can see, there are some pretty important commands in this list.  And this is just a small sample of them.  On my AIX server I found that there are over 400 scripts included as part of base AIX!  You can see a full list of all the scripts that make up your system by running a command like this:

 

for dir in `echo $PATH | tr ":" " "`; do for file in `ls -1 "$dir" 2>/dev/null`; do [ -x "$dir/$file" ] && file "$dir/$file"; done; done | grep -i script

 

It is pretty cool that so many of the core commands/utilities on AIX are made up of shell scripts.  For one, it shows that shell scripts can take on very important and critical tasks.  It can also be extremely helpful to be able to review the scripts if you are having any issues with any of these commands.  And these scripts can be an excellent learning tool.  These are extremely well written and robust scripts many of which have been used for decades on thousands and thousands of servers. 

Display resume.txt.gz on screen using cat command like syntax:

zcat resume.txt.gz

Display access_log_1.gz on screen at a time:

 zmore access_log_1.gz

Or try zless (less command):

zless access_log_1.gz

Search access_log_1.gz for 1.2.3.4 IP address using grep command like syntax:

zgrep '1.2.3.4' access_log_1.gz

You can use egrep command like syntax:

egrep 'regex' access_log_1.gz
egrep 'regex1|regex2' access_log_1.gz

 

Here you can see a few examples where incron is a good solution:

1.     Notifying programs (e.g. server daemons) about changes in configuration
2.     Guarding changes in critical files (with their eventual recovery)
3.     File usage monitoring, statistics.

First we will need to install incron:

$ sudo yum install incron

Make sure we set it to start on reboot:

$ sudo chkconfig incrond on

And now to start incron:

$ sudo service incrond start

incrond uses inotify. So to use it effectively we need to have it act on inotify events which are:

 IN_ACCESS         File was accessed (read) (*).
 IN_ATTRIB         Metadata  changed,  e.g.,  permissions,  timestamps, 

                   extended attributes, link count (since Linux 2.6.25), 

                   UID, GID, etc. (*).
 IN_CLOSE_WRITE    File opened for writing was closed (*).
 IN_CLOSE_NOWRITE  File not opened for writing was closed (*).
 IN_CREATE         File/directory created in watched directory (*).
 IN_DELETE         File/directory deleted from watched directory (*).
 IN_DELETE_SELF    Watched file/directory was itself deleted.
 IN_MODIFY         File was modified (*).
 IN_MOVE_SELF      Watched file/directory was itself moved.
 IN_MOVED_FROM     File moved out of watched directory (*).
 IN_MOVED_TO       File moved into watched directory (*).
 IN_OPEN           File was opened (*).

The incron table manipulator may be run under any regular user since it SUIDs. For manipulation with the tables use basically the same syntax as for the crontab program. You can import a table, remove and edit the current table.

The user table rows have the following syntax (use one or more spaces between elements):

<path> <mask> <command>

Where?
<path> is a filesystem path (each whitespace must be prepended by a backslash)
<mask> is a symbolic or numeric mask for events (see man inotify for more details)
<command> is an application or script to run on the events

The command may contain these wildcards:

 $$ - a dollar sign
 $@ - the watched filesystem path (see above)
 $# - the event-related file name
 $% - the event flags (textually)
 $& - the event flags (numerically)

Now with all that information, what can I do? Say you want to be notified each time /etc/hosts is modified and email us. Open incrontab make sure you are root for this example:

# incrontab -e

/etc/hosts IN_MODIFY mailx -s "Hosts file Has Been modified" mymail@mymail.com

Save the changes and open /etc/hosts and make a change and you should receive an email in your inbox.
At this point we have covered just the basics of what is possible with incron. Experiment with incron and see what other items you can monitor and what other commands you can execute on filesystem actions.

# tune2fs -c 0 /dev/sda2 

in /etc/fstab, the last column is used by fsck to determine the order of performing file system check at reboot time. For root file system /, it should be 1, for other file systems, it should be 2. If we want to disable the fsck check for certain file system, we can specify 0 in the last column.