Cloud DevOps Admin Guide

Traditionally the default dump device for system dumps was: /dev/hd6 (paging space) and still is on a lot of systems. If there is not enough space to copy over the dump file after a crash, then the system administrator is prompted upon restart to copy the dump file over to some removable media , like a tape or DVD. This can be time consuming and it is sometimes the case that you want to get your system back up quickly. I can sympathise with system administrators who just ignore the prompt to get the system back up due to business pressure, thus deleting the dump, so then one does not know why it crashed in the first place. If you do not have enough space on your dump device to copy the dump, then during the start-up process, the copydumpmenu menu utility is invoked to give the system administrator the opportunity to copy the dump to a removable media, for example to a tape device if present. The copydumpmenu utility can also be called from the command line when the system is up. The copy directory by default is /var/adm/ras with the file-name:vmcore.<X>.BZ , where X is a sequence number. The dump file is a BZ (BZIP) and not a Z compressed file format.

The snap command can be used to gather information about the dump file, be-sure to include the -D flag, it gathers the information from the primary dump device.

With systems now having more memory available, this has provided more flexibility as to where the primary dump device could be placed. Typically, for systems with over 4 GB of memory there is now a dedicated dump device, called: lg_dumplv

# lsvg -l rootvg |grep sysdump

Using the sysdumpdev command, one can determine what devices are used for the system dumps.

The following output shows a system using AIX 7.1 having the lg_dumplv as its primary dump device:
# sysdumpdev -l 

Looking more closely at the above output fields. Notice that an extra field is now present for AIX 6.1 onwards: type of dump. Currently set to traditional, here you can have it set at (firmware) fw-assisted, if your hardware supports it. For the secondary field, there is no dump device. This is denoted by using the sysdumpnull device. This means all system dumps are lost if it goes to that device. The copy directory is /var/adm/ras, this is where the system dump will be copied to , for either further examination, or to be copied off to go to IBM support. Note that 'always allow dump' is set to true, this must be the case if a dump is to be successfully initiated. Dump compression is on by default.

Common settings using sysdumpdev are:
To change the primary device use: sysdumpdev -P -p <device_name>
To change the secondary device use: sysdumpdev -P -s <device_name>
To change the copy directory use: sysdumpdev -D <path_name>
To change the always dump condition use: sysdumpdev -k for false, sysdumpdev -K for true
To change the type of dump use: sysdumpdev -t <fw-assisted | traditional>


Few Commands:

1. To view the current dump configuration :

# sysdumpdev -l

primary /dev/hd6
secondary /dev/sysdumpnull
copy directory /var/adm/ras
forced copy flag TRUE
always allow dump FALSE
dump compression OFF

2. To change the primary dump device temporarily :

# sysdumpdev -p /dev/dumplv

3. To change the primary dump device permanently :

# sysdumpdev -P -p /dev/dumplv

4. To change the secondary dump device temporarily :

# sysdumpdev -s /dev/dumplv

5. To change the secondary dump device permanently :

# sysdumpdev -P -s /dev/dumplv

6. To set the copy flag :

# sysdumpdev -K

7. To unset the copy flag :

# sysdumpdev -k

8. To estimate the dump size :

# sysdumpdev -e

9. To list the last dump information :

# sysdumpdev -L
Device name: /dev/lg_dumplv
Major device number: 12
Minor device number: 4
Size: 42123543 bytes
Date/Time: Wed Jan 01 12:03:00 CDT 2009
Dump status: 0
dump completed successfully
Dump copy filename: /var/adm/ras/vmcore.1

10. To copy the saved vmcoren file to tape :

# snap -gfkD -o /dev/rmt0

11. To read the dump file :

# crash dump unix
>

12. To change the dump file location and if the copy fails it should ask external media to copy the dump file:

# sysdumpdev -D /opt/dumpfiles

13. To change the dump file location and if the copy fails it should ignore the system dump:

# sysdumpdev -d /opt/dumpfiles

14. To specify the dumps should not be compressed :

# sysdumpdev -c

15. To specify the dumps should be always compress :

# sysdmpdev -C

16. To find out whether a new systemp dump has occured before the last reboot :

# sysdumpdev -z

The compressed dump is now on the LV lg_dumplv. The dump was not copied across to the copy directory when issuing a user initiated dump. To copy the most recent system dump from a system dump device to a directory, use the savecore command. For example, to copy the dump to the directory /var/adm/ras. I could use:

# savecore -d /var/adm/ras vmcore.0.BZ

If you need to uncompress the file use the dmpuncompress utility. The format of the command is:

dmpuncompress < filename>

After uncompressing, the dump file is now ready for further investigation using kdb or for transfer to IBM support.

# dmpuncompress vmcore.0.BZ
replaced with vmcore.0

Alternatively you can use the smit dump menu option and select,Copy a system dump. The following screen displays:

Copy dump image to: Type or select values in entry fields. Press Enter after making all desired changes. [Entry Fields] * Copy dump image from: [/dev/lg_dumplv] / * Copy dump image to: [/var/adm/ras/dump_fil> * Input and output file blocksize for copy [4096] # Size in bytes of dump image 63894528 Date of last dump Thu Oct 27 18-02-28 B>

The fields are populated with the current dump that is on the primary dump device. This is the default setting, after the copy, the dump file is present in: /var/adm/ras:

# ls -l dump_file_copy.BZ -rw-r--r-- 1 root system 63894528 Oct 27 18:15 dump_file_copy.BZ

After a dump has occurred there may well be a minidump generated as a well. Contained in the errorlog output listing earlier in the article, there was an entry for:

F48137AC 1027180411 U O minidump COMPRESSED MINIMAL DUMP

The minidump is a small compress dump that will be present in: /var/adm/ras. This file contains a snapshot of the system when the system was dumped or crashed. This file can be used for diagnosing if the main dump is not present, due to the dump being removed or not captured.