Cloud DevOps Admin Guide

1. Copy Sendmail.cf
2. Tar Perl Mondule if you use it in 5.3, this will be upgraded when going 6.1 and you might want to fall back to older version
3. Make sure you note the Number of Allowed Envrioment Process, this will fall back to default which 128 allowed process. **THIS IS CRITICAL**
4. Copy MOTD “Message of the Day”
5. Please Make Sure you turn of RSH since you use NIMSH/RSH to upgrade
6. Make sure you not the NIC Tunning TCP Send/Rcv you will have to put it back afterwards
7. Memory Tunning (no –a) please review after upgrade, upgrading to 6.1 will always take the best tunning parameters but please review to make sure its good
8. After upgrade please make sure you run the following command to make sure your OS 6.1 is consistent “lppchk –v” and “lppchk –c”
9. Copy "/etc/security/ulimits" and make sure after upgrade they are the same
10. Please upddate the NMON script and cron job to reference new AIX 6.1

Please Note: If you do your copy/tars make sure you do it before your nimadm migration so when you reboot to 6.1 disk the copy files are there. Also make sure have your replace back the files you take note of the permission owner and group of the files.

1) create on NIM server command:

nim -o define -t mksysb -a server=master -a source=<server name> -a mk_image=yes -a location=<location of the store image> <mksysb image name>

This will create the mksysb image of the client server and define it on the NIM server.

Example:
nim -o define -t mksysb -a server=master -a source=edppbuslvd01 -a mk_image=yes -a location=/nim/mksysb/edppbuslvd01_6100-04-03-05112010 edppbuslvd01_6100-04-03-05112010

server=master: server to store image, in this case is master
source=edppbuslvd01: the source of the image, which is client
location: the location of the stored mksysb image


2) create on client machine and then copy to NIM server and define on NIM server, or NFS mount the filesystem from NIM server on the client server.

let say you successfully NFS mount nim server filesystem on the client machine as /mnt.

mksysb -ieX /mnt/edppbuslvd01_6100-04-03-05112010


-e: exclude the filesystem/dir that defined on /etc/exclude.rootvg
-i: call the mkszfile command to generate the /image.data file
The /image.data file contains information on volume groups, logical volumes, file systems, paging space, and physical volumes.
 This information is included in the backup for future use by the installation process.
-X: set to automatically expand the /tmp if necessary

After the mksysb image created, you need to define it on NIM server.

nim -o define -t mksysb -a server=master -a location=<image location> <image name>

Steps to remove PowerPath software, cleanup ODM and reinstall PowerPath

varyoff Volume Group (varyoffvg <VGNAME>)

/etc/rc.agent stop (if you have clariion devices)

Remove paths from Powerpath configuration  
powermt remove hba=all

Delete all hdiskpower devices
lsdev -Cc disk -Fname | grep power | xargs -n1 rmdev -dl

Remove the PowerPath driver instance
rmdev -dl powerpath0

Delete all hdisk devices

- for Symmetrix devices, use this command:

lsdev -CtSYMM* -Fname | xargs -n1 rmdev -dl

- for CLARiiON devices, use this command:

lsdev -CtCLAR* -Fname | xargs -n1 rmdev -dl

Confirm with lsdev -Cc disk that there are no EMC hdisks or hdiskpowers

***If needed:

odmdelete -q name=powerpath0 -o CuDv

odmdelete -q name=powerpath0 -o CuAt

rm /dev/powerpath0


odmget CuDv |grep hdisk

odmdelete -q name=xxxxx -o CuDv (value you get from above NOT ROOTVG DISK)


odmget CuAt |grep hdisk

odmdelete -q name=xxxxx -o CuAt (value you get from above NOT ROOTVG DISK)


odmget CuDvDr|grep hdisk

odmdelete –q value3=xxxxxxxx -o CuDvDr (value you get from above NOT ROOTVG DISK)



odmget CuVPD|grep hdisk

odmdelete –q name=xxxxxxx -o CuVPD (value you get from above NOT ROOTVG DISK)



odmget CuDvDr|grep hdisk

odmdelete –q value3 =xxxxxxx -o CuDvDr (value you get from above NOT ROOTVG DISK)



cd /dev

rm hdiskxxxx (NOT ROOTVG DISK)

rm rhdiskxxxx (NOT ROOTVG DISK)

rm hdiskpower*

rm rhdiskpower*

savebase -v

***

Remove all Fiber driver instances rmdev -Rdl fscsiX ---> X being driver instance number i.e. 0,1,2, etc.

Verify through lsdev -Cc driver that there are no more fiber driver instances (fscsi)

Change the adapter instances in Defined state rmdev -l fcsX ---> X being adapterr instance number i.e. 0,1,2, etc.


Create the hdisk entries for all EMC devices

--> remove Clarrays definition here.

--> install ODM definition. (if you are reinstall ODM)


emc_cfgmgr or cfgmgr -vl fcsx ---> x being each adapter instance which was rebuilt Skip this part if no PowerPath.

Configure all EMC devices into PowerPath powermt config
Check the system to see if it now displays correctly powermt display
powermt display dev=all lsdev -Cc disk

A filesystem is a logical collection of files on a partition or disk. A partition is a container for information and can span an entire hard drive if desired.
Everything in Unix is considered to be a file, including physical devices such as DVD-ROMs, USB devices, floppy drives, and so forth.
Unix uses a hierarchical file system structure, much like an upside-down tree, with root (/) at the base of the file system and all other directories spreading from there.
A UNIX filesystem is a collection of files and directories that has the following properties:

• It has a root directory (/) that contains other files and directories.
• Each file or directory is uniquely identified by its name, the directory in which it resides, and a unique identifier, typically called an inode.
• By convention, the root directory has an inode number of 2 and the lost+found directory has an inode number of 3. Inode numbers 0 and 1 are not used. File inode numbers can be seen by specifying the -i option to ls command.
• It is self contained. There are no dependencies between one filesystem and any other.

The directories have specific purposes and generally hold the same types of information for easily locating files. Following are the directories that exist on the major versions of Unix:

Directory	Description
/	This is the root directory which should contain only the directories needed at the top level of the file structure.
/bin	This is where the executable files are located. They are available to all user.
/dev	These are device drivers.
/etc	Supervisor directory commands, configuration files, disk configuration files, valid user lists, groups, ethernet, hosts, where to send critical messages.
/lib	Contains shared library files and sometimes other kernel-related files.
/boot	Contains files for booting the system.
/home	Contains the home directory for users and other accounts.
/mnt	Used to mount other temporary file systems, such as cdrom and floppy for the CD-ROM drive and floppy diskette drive, respectively
/proc	Contains all processes marked as a file by process number or other information that is dynamic to the system.
/tmp	Holds temporary files used between system boots
/usr	Used for miscellaneous purposes, or can be used by many users. Includes administrative commands, shared files, library files, and others
/var	Typically contains variable-length files such as log and print files and any other type of file that may contain a variable amount of data
/sbin	Contains binary (executable) files, usually for system administration. For example fdisk and ifconfig utlities.
/kernel	Contains kernel files

 The table below summarizes some of the common UNIX variants and clones. While the table lists about forty different variants, the UNIX world isn't nearly as diverse as it used to be. Some of them are defunct and are listed for historical purposes. Others are on their way out. In some cases, vendors have defected to Microsoft technology. In others, mergers and acquisitions have led to the consolidation of different UNIX implementations. A list of "dead" UNIX implementations would be substantial indeed, consisting of hundreds of variations on the letters "U," "I," and "X" (CLIX, CX/UX, MV/UX, SINIX, VENIX, etc.).

UNIX Variants and Clones

 

UNIX Variant	Company/Org.	For More Info
A/UX	Apple Computer, Inc.	defunct
AIX	IBM	http://www.rs6000.ibm.com/ software/
AT&T System V	AT&T	defunct
BS2000/OSD-BC	Siemens AG	http://www.siemens.com/ servers/bs2osd/
BSD/OS	Berkeley Software Design, Inc.	http://www.bsdi.com
CLIX	Intergraph Corp.	http://www.intergraph.com
Debian GNU/Hurd	Software in the Public Interest, Inc.	http://www.gnu.org/ software/hurd/debian- gnu-hurd.html
Debian GNU/Linux	Software in the Public Interest, Inc.	http://www.debian.org
DG/UX	Data General Corp.	http://www.dg.com/ products/html/dg_ux.html
Digital Unix	Compaq Computer Corporation	http://www.unix.digital.com/
DYNIX/ptx	Sequent Computer Systems, Inc.	http://www.sequent.com/ products/software/ operatingsys/dynix.html
Esix UNIX	Esix Systems	http://www.esix.com/
FreeBSD	FreeBSD group	http://www.freebsd.org
GNU Herd	GNU organization	http://www.gnu.org
HAL SPARC64/OS	HAL Computer Systems, Inc.	http://www.hal.com
HP-UX	Hewlett-Packard Company	http://www.hp.com/ unixwork/hpux/
Irix	Silicon Graphics, Inc.	http://www.sgi.com/ software/irix6.5/
Linux	several	http://www.linux.org
LynxOS	Lynx Real-Time Systems, Inc.	http://www.lynx.com/ products/lynxos.html
MachTen	Tenon Intersystems	http://www.tenon.com/ products/machten/
MacOS X Server	Apple Computer, Inc.	http://www.apple.com/macosx/
Minix	none	http://www.cs.vu.nl/~ast/ minix.html
MkLinux	Apple Computer, Inc.	http://www.mklinux.apple.com
NCR UNIX SVR4 MP-RAS	NCR Corporation	http://www3.ncr.com/ product/integrated/ software/p2.unix.html
NetBSD	NetBSD group	http://www.netbsd.org
NeXTSTEP	NeXT Computer Inc.	defunct, see http://www.apple.com/ enterprise/
NonStop-UX	Compaq Computer Corporation	http://www.tandem.com
OpenBSD	OpenBSD group	http://www.openbsd.org
OpenLinux	Caldera Systems, Inc.	http://www.calderasystems.com
Openstep	Apple Computer, Inc.	http://www.apple.com/ enterprise/
QNX Realtime OS	QNX Software Systems Ltd.	http://www.qnx.com/ products/os/qnxrtos.html
Red Hat Linux	Red Hat Software, Inc.	http://www.redhat.com/
Reliant UNIX	Siemens AG	http://www.siemens.com/ servers/rm/
Solaris	Sun Microsystems	http://www.sun.com/ software/solaris/
SunOS	Sun Microsystems	defunct
SuSE	S.u.S.E., Inc.	http://www.suse.com
UNICOS	Silicon Graphics, Inc.	http://www.sgi.com/software/ unicos/
UnixWare	SCO -- The Santa Cruz Operation Inc.	http://www.sco.com/unix/
UTS	Amdahl Corporation	http://www.amdahl.com/uts/

AIX provides 'mkramdisk' command for producing a disk that resides in the RAM for very high I/O intensive applications like database.
Here is a simple set of commands to create a ramdisk and a filesystem on top of it:


1.create a RAM disk specifying the size

# mkramdisk 5G

The system will assign the available RAM disk. Since this is the first one, it will be called as ramdisk0

2.Check for the new disk

# ls -l /dev | grep -i ram

If there isn't sufficient available memory, the mkramdisk command will warn about the same during the creation.


3.Create and mount a filesystem on top of the ram disk

/sbin/helpers/jfs2/mkfs -V jfs2 /dev/ramdiskx

mount -V jfs2 -o log=NULL /dev/ramdiskx /ramdiskx

The new filesystem will now be available like any other FS.

To remove a ram disk, unmount/remove the filesystem and use 'rmramdisk' command to remove the ram disk.

Do the following on a server that is having problem exporting NFS mounts
------------------------------------------------------------------------------------

1) Move the currents exports file to another name
       mv /etc/exports /etc/exports.old

2) Create a new exports file
       touch /etc/exports

3)  Unexport everything
       exportfs -ua

4)  Stop NFS
       stopsrc -g nfs

5)  Stop portmapper
       stopsrc -s portmap

6)  Change directory to /etc and remove or rename the following files if they exist.
       rm -rf xtab state sm sm.bak rmtab   
   
7)  change directory to /var/statman and remote the status monitoring files.
       rm -rf state sm sm.bak

8)  start the portmapper
       startsrc -s portmap

9)   start nfs
       startsrc -g nfs

10)  re-export what is left in /etc/exports
       exportfs -va

11)  refresh the inetd daemon subsystem
       refresh -s inetd

12)  Move the /etc/exports file that you backed up back in place.
       mv /etc/exports.old /etc/exports

13)  export all directories in /etc/exports
       exportfs -a

1)  Show what is being exported on the source server

     showmount -e

    Note:  If the command above does not show the correct mount points
    that needs to be exported.  You can run the following command to attempt
    to export the filesystems.

     exportfs -a

2)  To unmount the filesystem on the source server that is being NFS on other systems.

     a)  unmount the NFS mount points on the target server.

          umount (filesystems)   target servers
         
     b)  umount the filesystem on the source server once the target servers
         are unmounted.

           umount (filesystems)
          
3)  Mounting NFS mount points on target server.

     a) mount (IP):(mount point)  (mount point)

The following procedure should be used to replace a failed internal (boot) disk on AIX 5 or higher, with software mirroring.
(Note: in these examples, hdisk0 and hdisk1 are doubly-mirrored internal disks and members of rootvg; hdisk1 has failed)

1. Identify the failed disk by analyzing the errpt logs. Confirm the failure using lspv by checking if "PV State" is "Missing".

2. Break the mirror and remove the device from AIX:

# unmirrorvg rootvg hdisk1
# reducevg rootvg hdisk1
# rmdev -l hdisk1 -d

3. Confirm that the device is no longer present using lspv.

4. Replace the disk drive, letting the new device take the same device name (hdisk1).

5. Add the new device into rootvg:

# extendvg rootvg hdisk1

6. Re-mirror the volume group. No additional arguments are required to doubly-mirror the two internal disks.

# mirrorvg rootvg

7. Re-add the boot image to the new internal disk:

# bosboot -ad hdisk1

8. Re-add the new disk to the bootlist and confirm it is present:

# bootlist -m normal hdisk0 hdisk1
# bootlist -m normal -o
hdisk0 blv=hd5
hdisk1 blv=hd5

In this topic we will discuss indepth of Linux Boot Sequence.How a linux system boots?
This will help unix administrators in troubleshooting some bootup problem.
Before discussing about it I will notedown the major component we need to know which are responsible for the booting process.

        1.BIOS(Basic Input/Output System)
        2.MBR(Master Boot Record)
        3.LILO or GRUB
             LILO:-LInux LOader
             GRUB:-GRand Unified Bootloader
        4.Kernel
        5.init
        6.Run Levels


1.BIOS:
      i.When we power on BIOS performs a Power-On Self-Test (POST) for all of the different hardware components in the system to make sure everything is working properly
     ii.Also it checks for whether the computer is being started from an off position (cold boot) or from a restart (warm boot) is
stored at this location.
     iii.Retrieves information from CMOS (Complementary Metal-Oxide Semiconductor) a battery operated memory chip on the motherboard that stores time, date, and critical system information.
     iv.Once BIOS sees everything is fine it will begin searching for an operating system Boot Sector on a valid master boot sector
on all available drives like hard disks,CD-ROM drive etc.
     v.Once BIOS finds a valid MBR it will give the instructions to boot and executes the first 512-byte boot sector that is the first
sector (“Sector 0″) of a partitioned data storage device such as hard disk or CD-ROM etc .
2.MBR
     i. Normally we use multi-level boot loader.Here MBR means I am referencing to DOS MBR.
     ii.Afer BIOS executes a valid DOS MBR,the DOS MBR will search for a valid primary partition marked as bootable on the hard disk.
     iii.If MBR finds a valid bootable primary partition then it executes the first 512-bytes of that partition which is second level MBR.
     iv. In linux we have two types of the above mentioned second level MBR known as LILO and GRUB
3.LILO
     i.LILO is a linux boot loader which is too big to fit into single sector of 512-bytes.
     ii.So it is divided into two parts :an installer and a runtime module.
     iii.The installer module places the runtime module on MBR.The runtime module has the info about all operating systems installed.
     iv.When the runtime module is executed it selects the operating system to load and transfers the control to kernel.
     v.LILO does not understand filesystems and boot images to be loaded and treats them as raw disk offsets
GRUB
     i.GRUB MBR consists of 446 bytes of primary bootloader code and 64 bytes of the partition table.
     ii.GRUB locates all the operating systems installed and gives a GUI to select the operating system need to be loaded.
     iii.Once user selects the operating system GRUB will pass control to the karnel of that operating system.
see below what is the difference between LILO and GRUB
4.Kernel
     i.Once GRUB or LILO transfers the control to Kernel,the Kernels does the following tasks

• Intitialises devices and loads initrd module
• mounts root filesystem

5.Init
     i.The kernel, once it is loaded, finds init in sbin(/sbin/init) and executes it.
     ii.Hence the first process which is started in linux is init process.
     iii.This init process reads /etc/inittab file and sets the path, starts swapping, checks the file systems, and so on.
     iv.It runs all the boot scripts(/etc/rc.d/*,/etc/rc.boot/*)
     v.starts the system on specified run level in the file /etc/inittab

6.Runlevel
     i.There are 7 run levels in which the linux OS runs and different run levels serves for different purpose.The descriptions are
given below.

• 0  – halt
• 1  – Single user mode
• 2  – Multiuser, without NFS (The same as 3, if you don’t have networking)
• 3  – Full multiuser mode
• 4  – unused
• 5  – X11
• 6  – Reboot

     ii.We can set in which runlevel we want to run our operating system by defining it on /etc/inittab file.
Now as per our setting in /etc/inittab the Operating System the operating system boots up and finishes the bootup process.
Below are given some few  important differences about LILO and GRUB

LILO	GRUB
LILO has no interactive command interface	GRUB has interactive command interface
LILO does not support booting from a network	GRUB does support booting from a network
If you change your LILO config file, you have to rewrite the LILO stage one boot loader to the MBR	GRUB automatically detects any change in config file and auto loads the OS
LILO supports only linux operating system	GRUB supports large number of OS

Three phases available in BOOT Process

1. Ros kernel init phase
2. Base Device Configuration
3. System boot phase

1. Ros Kernel init phase (PHASE1)

A. Post (power on self test)

In this post it will do basic hardware checking

B. Then it will go to NVRAM and check the boot list for last boot device (hdisk0 or hdisk1).

C. Then it will check the BLV (hd5) in boot device.

D. Then it will check the boot image

E. Then boot image is moved to memory.

F. Then kernel will execute.

2. Base Device configuration (PHASE2)

A. Here cfgmgr will run for device configuration.

3. System Boot Phase (PHASE3)

A. Kernel will execute.
B. The paging space (hd6) will get started.
C. Then following file system will be mounted /, /var. /usr, /home. /tmp
D. Kernel start the init process, it will read the /etc/inittab file and execute the following process.

/etc/rc.boot,
srcmstr
/etc/rc.tcpip
/etc/rc.net

The above network related files /etc/rc.tcpip, /etc/rc.net, used to configure the ip address and routing.

E. Then it will start the system by default run level 2.

NOTE:

Run level 2: It contains all of the terminal process and daemons that are run in the multi user environment. This is default run level.

/etc/inittab file contains four fields, 1. Identifier, 2. Command, 3. Action, 4. Runlevel

These are the steps for modifying /etc/inittab without using Vi editor in AIX.

Before editing  take a copy the inittab file to a file named inittab.old