AIX Knowledge Base

Monday, May 2, 2011

AIX Logical Volume Manager (LVM)

Examples	What it means
PHYSICAL VOLUME (PV)	Represents a hard disk (hdisk0).
PHYSICAL PARTITION (PP)	The smallest allocation unit in the LVM. All PPs within a VG are the same size, usually 4 or 8 MB.
VOLUME GROUP (VG)	A set of one or more PVs which form a single storage pool. You can define multiple VGs on each AIX system.
LOGICAL VOLUME (LV)	One or more PPs. A file system resides on top of an LV. Only one LV is mapped to a file system. A LV can't span across a VG. Up to 255 LVs in a VG
LOGICAL PARITITION (LP)	One or more PPs. LP represents a mirrored copy of a PP. Up to two copies of a PP can be mirrored resulting in a LP count of three (2 mirrors plus original).
Volume Group Descriptor Area (VGDA)	Information about all the LVs and PVs within a VG. The first 64K of a PV is reserved for this area - defined in <sys/bootrecord.h>. The VGDA consists of BOOTRECORD: - first 512 bytes. Allows the Read Only System (ROS) to boot system BAD BLK DIRECTORY - found in <sys/bddir.h> LVM RECORD - found in <lvmrec.h>
Volume Group Status Area (VGSA)	Information about which PPs that are stale and which PVs are missing within a VG. The LVM and SCSI driver reserves somewhere between 7-10% of the available disk space for LVM maps, etc.
Physical Volume ID (PVID)	The PVID is an amalgamation of the machine’s serial number (from the systems EPROMs) and the date that the PVID is being generated. This combination insures theextremely low chance of two disks being created with the same PVID. Finally, when a system is booted, the disk conﬁgurator goes and looks at the PVID sitting on each disk platter and then compares that to an entry in ODM. If the entry is found, then the disk is given the hdiskX name that is associated with the ODM entry for the PVID.
Quorum	Quorum is a sort of “sanity” check that LVM uses to resolve possible data conﬂiction and prevent data corruption. Quorum is a method by which 51% or more quorum votes must be available to a volume group before LVM actions can continue. Quorum is issued to a disk in a volume group according to how the disk was created within the volume group. When a volume group consists of one disk, there are two VGDA’s on that disk. Thus, this single disk volume group has a quorum vote of 2. When another disk is added to the volume group with an “extendvg”, then this new disk gets one VGDA, but the original, ﬁrst disk still retains the two VGDA’s. When the volume group has been extended to three disks, the third disk gets the spare VGDA sitting on the ﬁrst disk and then each disk has a quorum vote of 1. Every disk after the third disk is automatically given one VGDA, and thus one vote.
Volume Group ID (VGID)	Just as the PVID is a soft serial number for a disk, the VGID is the soft serial number for the volume group. It is this serial number, not the volume group’s ascii name, which all low level LVM commands reference. Additionally, it is the basis for the LVIDs created on that VGID.
Logical Volume Control Block (LVCB)	The logical volume control block (lvcb) consists of the ﬁrst 512 bytes of a logical volume. This area holds important information such as the creation date of the logical volume, information about mirrored copies, and possible mount points in a journaled ﬁlesystem.
Logical Volume ID (LVID)	The LVID is the soft serial number used to represent the logical volume to the LVM libraries and low level commands. The LVID is created from the VGID of the volume group, a decimal point, and a number which represents the order which the logical volume was created on the volume group.

Now for the cheet sheet

Directory and Files
Directories and Files
Tools
diagnostic	diag - used to hot swap the disk cfgmgr - used mak sure the new disk is seen # to add new disk from the scsi0 controller cfgmgr -l scsi0
Create/Remove hard disk	cfgmgr -l scsi0 mkdev -c disk -l <pv> rmdev -dl <pv>
Physical Volumes
display	lspv lspv <pv> (detailed) lspv -l <pv> (list logical volumes) lspv -p <pv> (physical partition usage)
PVID	chdev -l <pv> -a pv=yes chdev -l <pv> -a pv=clear Note: PVID's are automatically added when the disk is placed into a vg
adding	chdev -l <pv> -a pv=yes (new) chpv -v a <pv> (adds back the removed disk)
removing	chpv -v r <pv>
change physical attributes	chpv -a y <pv> (changes allocatable state to YES) chpv -a n <pv> (changes allocatable state to NO)
moving	migratepv <old pv> <new pv>
Volume Groups
display	lsvg lsvg <vg> (detailed) lsvg -l <vg> (list all logical volumes in goup) lsvg -p <vg> (list all physical volumes in group) lsvg -o (lists all varied on) lsvg -M <vg> (lists assicated disks and state) ## Details volume group info for the hard disk lqueryvg -Atp <pv> lqueryvg -p <disk> -v (Determine the VG ID# on disk) lqueryvg -p <disk> -L (Show all the LV ID#/names in the VG on disk) lqueryvg -p <disk> -P (Show all the PV ID# that reside in the VG on disk)
varyon	varyonvg <vg> varyonvg -f <vg> (force) varyonvg -s <vg> (maintenance mode can use VG commands but lv 's cannot be opened for i/o access) varyoffvg <vg> Note: the varyon command activiates the volume goup which means it is available for use
ODM related	## Determine if the ODM and VGDA are correct (in sync) getlvodm -u <vg> ## tries to resync VGDA, LV control blocks and ODM synclvodm <vg> ## If the message 0516-366 lsvg: Volume group <vg> is locked is ever seen putlvodm -K `gtlvodm -v <vg>`
creating	mkvg -y <vg> -s <PP size> <pv> mkvg -y datavg -s 4 hdisk1 Note: the PP size will be the size of the physical partition size you want 4MB, 8MB
extending	extendvg <vg> <pv>
reducing	reducevg -d <vg> <pv> ## removes the PVID from the VGDA when a disk has vanished without using the reducevg command reducevg <vg> <PVID>
removing	varyoffvg <vg> exportvg <vg> Note: the export command nukes everything regardingthe volume goup in the ODM and /etc/filesystems
checking	## check to see if underlying disk has grown in size chvg -g <vg> Note: use this command if you are using SAN LUN's that have increased in size
change volume attributes	## auto vary on a volume at system start chvg -a y # Turns on/off quorum checking on a volume group chvg -Q [y\|n] <vg>
renaming	varyoffvg <old vg name> lsvg -p <old vg name> (obtain disk names) exportvg <old vg name> import -y <new vg name> <pv> varyonvg <new vg name> mount -a
importing	importvg -y <vg> <pv> importvg <pv> (will use rootvg as default vg)
exporting	varyoffvg <vg> exportvg <vg> Note: if the volume has an active paging space this must be turned off before
Logical Volumes
display	lslv <lv> lslv -l <lv> (list all physical volumes in logical volume) lslv -m <lv> (list ppartition mapping) ## Display lv control block information getlvcb -AT <lv>
creating	mklv <vg> <# of PP's> <pv> mklv -y <lv name> <vg> <# of PP's> <pv> ## Create a mirrored named logical volume mklv -y <lv> -c <copies 2 or 3> <vg> <# of PP's> <pv> ## create a JFSlog logical Volume mklv -y <lv name> -t jfslog <vg> <# of PP's> <pv>
extending	extendlv <lv> <additonal # of PP's> extendlv <lv> <size of volume in B\|\|M\|G>
reducing/resizing	see filesystem below
removing	rmlv <lv>
moving	migratepv -l <lv> <old pv> <new pv>
adding a mirror to a non-mirrored volume	mklvcopy -s n <lv> <copies 2 or 3> <pv>
removing a mirror copy from a mirrored volume	rmlvcopy <lv> <copies 1 or 2> rmlvcopy <lv> <copies 1 or 2> <pv> (specified pv) unmirrorvg <vg> <pv>
synchronize logical volume	syncvg -p <pv> syncvg -v <vg> syncvg -l <lv>
mirror any unmirrored volumes	mirrorvg <vg> <pv>
change volume attributes	## Enable the bad-block relocation policy chlv -b [y\|n] <lv>
renaming	chlv -n <new lv name> <old lv name>
Miscellaneous	## Initialises an LV for use as an JFSlog logform </dev/lv>
Filesystems
display	lsfs lsfs -q <fs> (detailed) Note: use the '-q' to see if the logical volume size is bigger than the filesystem size
create	## create new filesystem, -A means to mount after restart crfs -v jfs -d <lv> -m <mountpoint> -A yes ## Create logical volume, filesystem, mountpoint, add entry to /etc/filesystems at the specified size crfs -v jfs2 -g <vg> -m <mountpoint> -a size=<size in 512k blocks\|M\|G) -A yes Note: there are two types of filesystems jfs and jfs2, jfs2 allows you to decrease the filesystem size , you cannot reduce a jfs filesystem
remove	rmfs <fs> Note: if all filesystems have been removed from a logical volume then the logical volume is removed as well.
resize	chfs -a size=<new size> <fs> chfs -a size=1G /var (specific size, can be used to increase and decrease) chfs -a size=+1G /var (increase by 1GB) chfs -a size=-1GB /var (reduce by 1GB) Note: this will automatically increase or decrease the underlying logical volume as well
freeze/unfreeze	chfs -a freeze=<time in seconds> <fs> chfs -a freeze=off <fs>
split mirrored copy	chfs -a splitcopy=<split copy mountpoint>-a copy=2 <fs> chfs -a splitcopy=/backup -a copy=2 /testfs
change	## Change the mountpoint chfs -m <new mountpoint> <fs> ## Do not mount after a restart chfs -A no <fs> ## Mount read-only chfs -p ro <fs>
mount	mount mount [<fs>\|<lv>] mount -a mount all
defrag	defragfs -q <fs> (report defrag status) defragfs -r <fs> (runs in report only mode - no action) defragfs <fs> (actually defrags the filesystem)
checking and repairing	fsck [-y\|-n] <fs> (check a filesystem) fsck -p <fs> (restores primary superblock from backup copy if corrupt)
Miscellaneous
Complete VG, LV and FS with mirroring example	## Create the volume group mkvg -s 256 -y datavg hdisk2 ## Create the jfs2 log logical volume and initialize it this for the volume group mklv -t jfs2log -y dataloglv datavg 1 logform /dev/dataloglv ## Create the logical volume mklv -t jfs2 -y data01lv datavg 8 ## Create the filesystems that will use the logical volume crfs -v jfs -d data01lv -m /data01 -A yes ## Add an additional hard disk to the volume group extendvg datavg hdisk3 ## Now mirror both the volume group log logical volume and the logical volume mklvcopy dataloglv 2 mklvcopy data01lv 2 ## Make sure everything is sync'ed both the log and the logical volume syncvg -v datavg ## Make sure everything is OK lsvg -l datavg ## a quick way to perform the above in two steps mklv -c 2 -t jfs2 -y data02lv datavg 8 crfs -v jfs -d data02lv -m /data02 -A yes ## mount everything and check mount -a
Replaced failed mirror drive	## break the mirror (two ways to do this) rmlvcopy <lv name> 1 <broken disk> unmirrorvg <lv> <broken pv > ## remove the disk from the vg reducevg <vgname> <broken pv > ## remove the hdisk from ODM rmdev -dl <broken pv> ## physically replace the disk diag -> function select -> task selection -> hot plug task -> scsi and scsi raid hot plug manager -> replace/remove a device attached to an scsi hot swap enclosure device -> select disk and follow instructions ## configure new disk an check the new number (hopefully the same) cfgmgr -v lsdev -Cc <pv> ## add back to volume group extendvg <vg> <pv> ## create mirror (two ways to do this) mklvcopy <lv> 2 <pv> mirrorvg <lv> ## sync mirror syncvg -l <lv> ## If this is the rootvg there are additonal steps to take bosboot -ad /dev/<pv> bootlist -m normal <pv> <pv> bootlist -m normal -o
Accidently remove a mirrored disk or SAN LUN disappeared off the network	## This procedure places back a mirror disk that you have accidently pulled or that a SAN LUN disappeared off the network ## and its states is classed as "missing" ## see that the disk is in a missing state (see PV state column), also see stale volumes lsvg -p <vg> lsvg -M <vg> ## To make the disk active again we use the varyonvg command varyonvg <vg> ## see that the disk is in a active state (see PV state column) lsvg -p <vg> ## Now re-sync the volumes in that volume group syncvg -v <vg> ## Make sure that no volumes are stale lsvg -M <vg> ## Determine if the ODM and VGDA are correct (in sync) getlvodm -u <vg>

Diagnostic LED numbers and codes RS6K - AIX

----------------------------------------------------------------------------

Built-In Self-Test (BIST) Indicators
------------------------------------

100 BIST completed successfully; control was passed to IPL ROS.
101 BIST started following reset.
102 BIST started, following the system unit's power-on reset.
103 BIST could not determine the system model number.
104 Equipment conflict; BIST could not find the CBA.
105 BIST could not read from the OCS EPROM.
106 BIST failed: CBA not found
111 OCS stopped; BIST detected a module error.
112 A checkstop occurred during BIST; checkstop results could not be logged out.
113 Three checkstops have occurred.
120 BIST starting a CRC check on the 8752 EPROM.
121 BIST detected a bad CRC in the first 32K bytes of the OCS EPROM.
122 BIST started a CRC check on the first 32K bytes of the OCS EPROM.
123 BIST detected a bad CRC on the OCS area of NVRAM.
124 BIST started a CRC check on the OCS area of NVRAM.
125 BIST detected a bad CRC on the time-of-day area of NVRAM.
126 BIST started a CRC check on the time-of-day area of NVRAM.
127 BIST detected a bad CRC on the 8752 EPROM.
130 BIST presence test started.
140 Running BIST. (Box Manufacturing Mode Only)
142 Box manufacturing mode operation.
143 Invalid memory configuration.
144 Manufacturing test failure.
151 BIST started AIPGM test code.
152 BIST started DCLST test code.
153 BIST started ACLST test code.
154 BIST started AST test code.
160 Bad EPOW Signal/Power status signal.
161 BIST being conducted on BUMP I/O.
162 BIST being conducted on JTAG.
163 BIST being conducted on Direct I/O.
164 BIST being conducted on CPU.
165 BIST being conducted on DCB and Memory.
166 BIST being conducted on Interrupts.
170 BIST being conducted on Multi-Processors.
180 Logout in progress.
182 BIST COP bus not responding.
185 A checkstop condition occurred during the BIST.
186 System logic-generated checkstop (Model 250 only).
187 Graphics-generated checkstop (Model 250).
195 Checkstop logout complete
199 Generic SCSI backplane
888 BIST did not start.

Power-On Self-Test (POST) Indicators
------------------------------------

200 IPL attempted with keylock in the Secure position.
201 IPL ROM test failed or checkstop occurred (irrecoverable).
202 Unexpected machine check interrupt.
203 Unexpected data storage interrupt.
204 Unexpected instruction storage interrupt.
205 Unexpected external interrupt.
206 Unexpected alignment interrupt.
207 Unexpected program interrupt.
208 Unexpected floating point unavailable interrupt.
209 Unexpected SVC interrupt.
20c L2 cache POST error. (The display shows a solid 20c for 5 seconds.)
210 Unexpected SVC interrupt.
211 IPL ROM CRC comparison error (irrecoverable).
212 RAM POST memory configuration error or no memory found (irrecoverable).
213 RAM POST failure (irrecoverable).
214 Power status register failed (irrecoverable).
215 A low voltage condition is present (irrecoverable).
216 IPL ROM code being uncompressed into memory.
217 End of boot list encountered.
218 RAM POST is looking for good memory.
219 RAM POST bit map is being generated.
21c L2 cache is not detected. (The display shows a solid 21c for 2 seconds.)
220 IPL control block is being initialized.
221 NVRAM CRC comparison error during AIX IPL(key mode switch in Normal mode).
Reset NVRAM by reaccomplishing IPL in Service mode. For systems with an
internal, direct-bus-attached (DBA) disk, IPL ROM attempted to perform an IPL from
that disk before halting with this operator panel display value.
222 Attempting a Normal mode IPL from Standard I/O planar-attached devices specified
in NVRAM IPL Devices List.
223 Attempting a Normal mode IPL from SCSI-attached devices specified in NVRAM IPL
Devices List.
224 Attempting a Normal mode IPL from 9333 subsystem device specified in NVRAM IPL
Devices List.
225 Attempting a Normal mode IPL from 7012 DBA disk-attached devices specified in
NVRAM IPL Devices List.
226 Attempting a Normal mode IPL from Ethernet specified in NVRAM IPL Devices List.
227 Attempting a Normal mode IPL from Token-Ring specified in NVRAM IPL Devices List.
228 Attempting a Normal mode IPL from NVRAM expansion code.
229 Attempting a Normal mode IPL from NVRAM IPL Devices List; cannot IPL from any
of the listed devices, or there are no valid entries in the Devices List.
22c Attempting a normal mode IPL from FDDI specified in NVRAM IPL device list.
230 Attempting a Normal mode IPL from adapter feature ROM specified in IPL ROM
Device List.
231 Attempting a Normal mode IPL from Ethernet specified in IPL ROM Device List.
232 Attempting a Normal mode IPL from Standard I/O planar-attached devices specified
in ROM Default Device List.
233 Attempting a Normal mode IPL from SCSI-attached devices specified in IPL ROM
Default Device List.
234 Attempting a Normal mode IPL from 9333 subsystem device specified in IPL ROM
Device List.
235 Attempting a Normal mode IPL from 7012 DBA disk-attached devices specified in
IPL ROM Default Device List.
236 Attempting a Normal mode IPL from Ethernet specified in IPL ROM Default Device
List.
237 Attempting a Normal mode IPL from Token-Ring specified in IPL ROM Default
Device List.
238 Attempting a Normal mode IPL from Token-Ring specified by the operator.
239 System failed to IPL from the device chosen by the operator.
23c Attempting a normal mode IPL from FDDI specified in IPL ROM device list.
240 Attempting a Service mode IPL from adapter feature ROM.
241 Attempting a normal boot from devices specified in the NVRAM boot list.
242 Attempting a Service mode IPL from Standard I/O planar-attached devices specified
in the NVRAM IPL Devices List.
243 Attempting a Service mode IPL from SCSI-attached devices specified in the
NVRAM IPL Devices List.
244 Attempting a Service mode IPL from 9333 subsystem device specified in the
NVRAM IPL Devices List.
245 Attempting a Service mode IPL from 7012 DBA disk-attached devices specified in
the NVRAM IPL Devices List.
246 Attempting a Service mode IPL from Ethernet specified in the NVRAM IPL Devices
List.
247 Attempting a Service mode IPL from Token-Ring specified in the NVRAM Device
List.
248 Attempting a Service mode IPL from NVRAM expansion code.
249 Attempting a Service mode IPL from the NVRAM IPL Devices List; cannot IPL from
any of the listed devices, or there are no valid entries in the Devices List.
24c Attempting a service mode IPL from FDDI specified in NVRAM IPL device list.
250 Attempting a Service mode IPL from adapter feature ROM specified in the IPL ROM
Device List.
251 Attempting a Service mode IPL from Ethernet specified in the IPL ROM Default
Device List.
252 Attempting a Service mode IPL from Standard I/O planar-attached devices specified
in the ROM Default Device List.
253 Attempting a Service mode IPL from SCSI-attached devices specified in the IPL
ROM Default Device List.
254 Attempting a Service mode IPL from 9333 subsystem device specified in the IPL
ROM Devices List.
255 Attempting a Service mode IPL from 7012 DBA disk-attached devices specified in
IPL ROM Default Device List.
256 Attempting a Service mode IPL from Ethernet specified in the IPL ROM Devices
List.
257 Attempting a Service mode IPL from Token-Ring specified in the IPL ROM Devices
List.
258 Attempting a Service mode IPL from Token-Ring specified by the operator.
259 Attempting a Service mode IPL from FDDI specified by the operator.
25c Attempting a service mode IPL from FDDI specified in IPL ROM device list.
260 Information is being displayed on the display console.
261 No supported local system display adapter was found.
262 Keyboard not detected as being connected to the system's keyboard port.
263 Attempting a Normal mode IPL from adapter feature ROM specified in the NVRAM
Device List.
269 Stalled state - the system is unable to IPL.
270 Low Cost Ethernet Adapter (LCE) POST executing
271 Mouse and Mouse port POST.
272 Tablet Port POST.
276 10/100Mbps MCA Ethernet Adapter POST executing
277 Auto Token-Ring LANstreamer MC 32 Adapter.
278 Video ROM scan POST.
279 FDDI POST.
280 3com Ethernet POST.
281 Keyboard POST executing.
282 Parallel port POST executing.
283 Serial port POST executing.
284 POWER Gt1 graphics adapter POST executing.
285 POWER Gt3 graphics adapter POST executing.
286 Token-Ring adapter POST executing.
287 Ethernet adapter POST executing.
288 Adapter card slots being queried.
289 POWER GT0 Display Adapter POST.
290 IOCC POST error (irrecoverable).
291 Standard I/O POST running.
292 SCSI POST running.
293 7012 DBA disk POST running.
294 IOCC bad TCW memory module in slot location J being tested.
295 Graphics Display adapter POST, color or grayscale.
296 ROM scan POST.
297 System model number does not compare between OCS and ROS (irrecoverable).
298 Attempting a software IPL.
299 IPL ROM passed control to the loaded program code.
301 Flash Utility ROM test failed or checkstop occurred (irrecoverable
302 Flash Utility ROM: User prompt, move the key to the service position in order to
perform an optional Flash Update. LED 3ð2 will only appear if the key switch is in
the secure position. This signals the user that a Flash Update may be initiated by
moving the key switch to the service position. If the key is moved to the service
position then LED 3ð3 will be displayed, this signals the user to press the Reset
button and select optional Flash Update.
303 Flash Utility ROM: User prompt, press the Reset button in order to perform an
optional Flash Update. LED 3ð2 will only appear if the key switch is the secure
position. This signals the user that a Flash Update may be initiated by moving the
key switch to the service position. If the key is moved to the service position LED
3ð3 will be displayed, this signals the user to press the Reset button and select
optional Flash Update.
304 Flash Utility ROM IOCC POST error (irrecoverable).
305 Flash Utility ROM standard I/O POST running.
306 Flash Utility ROM is attempting IPL from Flash Update media device.
307 Flash Utility ROM system model number does not compare between OCS and
ROM (irrecoverable).
308 Flash Utility ROM: IOCC TCW memory is being tested.
309 Flash Utility ROM passed control to a Flash Update Boot Image.
311 Flash Utility ROM CRC comparison error (irrecoverable).
312 Flash Utility ROM RAM POST memory configuration error or no memory found
(irrecoverable).
313 Flash Utility ROM RAM POST failure (irrecoverable).
314 Flash Utility ROM Power status register failed (irrecoverable).
315 Flash Utility ROM detected a low voltage condition.
318 Flash Utility ROM RAM POST is looking for good memory.
319 Flash Utility ROM RAM POST bit map is being generated.
322 CRC error on media Flash Image. No Flash Update performed.
323 Current Flash Image is being erased.
324 CRC error on new Flash Image after Update was performed. (Flash Image is cor-rupted.)
325 Flash Update successful and complete.

Configuration Program Indicators
--------------------------------

500 Querying Standard I/O slot.
501 Querying card in Slot 1.
502 Querying card in Slot 2.
503 Querying card in Slot 3.
504 Querying card in Slot 4.
505 Querying card in Slot 5.
506 Querying card in Slot 6.
507 Querying card in Slot 7.
508 Querying card in Slot 8.
510 Starting device configuration.
511 Device configuration completed.
512 Restoring device configuration files from media.
513 Restoring basic operating system installation files from media.
516 Contacting server during network boot.
517 Mounting client remote file system during network IPL.
518 Remote mount of the root and /usr file systems failed during network boot.
520 Bus configuration running.
521 /etc/init invoked cfgmgr with invalid options; /etc/init has been corrupted or incor-rectly
modified (irrecoverable error).
522 The configuration manager has been invoked with conflicting options (irrecoverable
error).
523 The configuration manager is unable to access the ODM database (irrecoverable
error).
524 The configuration manager is unable to access the config.rules object in the ODM
database (irrecoverable error).
525 The configuration manager is unable to get data from a customized device object in
the ODM database (irrecoverable error).
526 The configuration manager is unable to get data from a customized device driver
object in the ODM database ( irrecoverable error).
527 The configuration manager was invoked with the phase 1 flag; running phase 1 at
this point is not permitted (irrecoverable error).
528 The configuration manager cannot find sequence rule, or no program name was
specified in the ODM database (irrecoverable error).
529 The configuration manager is unable to update ODM data (irrecoverable error).
530 The program savebase returned an error.
531 The configuration manager is unable to access the PdAt object class (irrecoverable
error).
532 There is not enough memory to continue (malloc failure); irrecoverable error.
533 The configuration manager could not find a configure method for a device.
534 The configuration manager is unable to acquire database lock (irrecoverable error).
535 HIPPI diagnostics interface driver being configured.
536 The configuration manager encountered more than one sequence rule specified in
the same phase (irrecoverable error).
537 The configuration manager encountered an error when invoking the program in the
sequence rule.
538 The configuration manager is going to invoke a configuration method.
539 The configuration method has terminated, and control has returned to the configura-tion
manager.
551 IPL vary-on is running.
552 IPL varyon failed.
553 IPL phase 1 is complete.
554 The boot device could not be opened or read, or unable to define NFS swap device
during network boot.
555 An ODM error occurred when trying to varyon the rootvg, or unable to create an
NFS swap device during network boot.
556 Logical Volume Manager encountered error during IPL vary-on.
557 The root filesystem will not mount.
558 There is not enough memory to continue the system IPL.
559 Less than 2 M bytes of good memory are available to load the AIX kernel.
570 Virtual SCSI devices being configured.
571 HIPPI common function device driver being configured.
572 HIPPI IPI-3 master transport driver being configured.
573 HIPPI IPI-3 slave transport driver being configured.
574 HIPPI IPI-3 transport services user interface device driver being configured.
575 A 9570 disk-array driver is being configured.
576 Generic async device driver being configured.
577 Generic SCSI device driver being configured.
578 Generic commo device driver being configured.
579 Device driver being configured for a generic device.
580 HIPPI TCPIP network interface driver being configured.
581 Configuring TCP/IP.
582 Configuring Token-Ring data link control.
583 Configuring an Ethernet data link control.
584 Configuring an IEEE Ethernet data link control.
585 Configuring an SDLC MPQP data link control.
586 Configuring a QLLC X.25 data link control.
587 Configuring a NETBIOS.
588 Configuring a Bisync Read-Write (BSCRW).
589 SCSI target mode device being configured.
590 Diskless remote paging device being configured.
591 Configuring an LVM device driver.
592 Configuring an HFT device driver.
593 Configuring SNA device drivers.
594 Asynchronous I/O being defined or configured.
595 X.31 pseudo-device being configured.
596 SNA DLC/LAPE pseudo-device being configured.
597 OCS software being configured.
598 OCS hosts being configured during system reboot.
599 Configuring FDDI data link control.
5c0 Streams-based hardware drive being configured.
5c1 Streams-based X.25 protocol being configured.
5c2 Streams-based X.25 COMIO emulator driver being configured.
5c3 Streams-based X.25 TCP/IP interface driver being configured.
5c4 FCS adapter device driver being configured.
5c5 SCB network device driver for FCS is being configured.
5c6 AIX SNA channel being configured.
600 Starting network boot portion of /sbin/rc.boot
602 Configuring network parent devices.
603 /usr/lib/methods/defsys, /usr/lib/methods/cfgsys, or /usr/lib/methods/cfgbus
failed.
604 Configuring physical network boot device.
605 Configuration of physical network boot device failed.
606 Running /usr/sbin/ifconfig on logical network boot device.
607 /usr/sbin/ifconfig failed.
608 Attempting to retrieve the client.info file with tftp.Note that a flashing 608 indicates
multiple attempt(s) to retrieve the client_info file are occurring.
609 The client.info file does not exist or it is zero length.
610 Attempting remote mount of NFS file system.
611 Remote mount of the NFS file system failed.
612 Accessing remote files; unconfiguring network boot device.
614 Configuring local paging devices.
615 Configuration of a local paging device failed.
616 Converting from diskless to dataless configuration.
617 Diskless to dataless configuration failed.
618 Configuring remote (NFS) paging devices.
619 Configuration of a remote (NFS) paging device failed.
620 Updating special device files and ODM in permanent filesystem with data from boot
RAM filesystem.
622 Boot process configuring for operating system installation.
650 IBM SCSD disk drive being configured
668 25MB ATM MCA Adapter being configured
680 POWER GXT800M Graphics Adapter
689 4.5GB Ultra SCSI Single Ended Disk Drive being configured
690 9.1GB Ultra SCSI Single Ended Disk Drive being configured
694 Eicon ISDN DIVA MCA Adapter for PowerPC Systems
700 Progress indicator. A 1.1 GB 8-bit SCSI disk drive being identified or configured.
701 Progress indicator. A 1.1 GB 16-bit SCSI disk drive is being identified or configured.
702 Progress indicator. A 1.1 GB 16-bit differential SCSI disk drive is being identified or
configured.
703 Progress indicator. A 2.2 GB 8-bit SCSI disk drive is being identified or configured.
704 Progress indicator. A 2.2 GB 16-bit SCSI disk drive is being identified or configured.
705 The configuration method for the 2.2 GB 16-bit differential SCSI disk drive is being
run. If an irrecoverable error occurs, the system halts.
706 Progress indicator. A 4.5 GB 16-bit SCSI disk drive is being identified or configured.
707 Progress indicator. A 4.5 GB 16-bit differential SCSI disk drive is being identified or
configured.
708 Progress indicator. A L2 cache is being identified or configured.
710 POWER GXT150M graphics adapter being identified or configured.
711 Unknown adapter being identified or configured.
712 Graphics slot bus configuration is executing.
713 The IBM ARTIC960 device is being configured.
714 A video capture adapter is being configured.
715 The Ultimedia Services audio adapter is being configured. This LED displays briefly
on the panel.
717 TP Ethernet Adapter being configured.
718 GXT500 Graphics Adapter being configured.
720 Unknown read/write optical drive type being configured.
721 Unknown disk or SCSI device being identified or configured.
722 Unknown disk being identified or configured.
723 Unknown CD-ROM being identified or configured.
724 Unknown tape drive being identified or configured.
725 Unknown display adapter being identified or configured.
726 Unknown input device being identified or configured.
727 Unknown async device being identified or configured.
728 Parallel printer being identified or configured.
729 Unknown parallel device being identified or configured.
730 Unknown diskette drive being identified or configured.
731 PTY being identified or configured.
732 Unknown SCSI initiator type being configured.
733 7GB 8mm tape drive being configured.
734 4x SCSI-2 640MB CD-ROM Drive
741 1080MB SCSI Disk Drive
745 16GB 4mm Tape Auto Loader
748 MCA keyboard/mouse adapter being configured.
749 7331 Model 205 Tape Library
754 1.1GB 16-bit SCSI disk drive being configured.
755 2.2GB 16-bit SCSI disk drive being configured.
756 4.5GB 16-bit SCSI disk drive being configured.
757 External 13GB 1.5M/s 1/4 inch tape being configured.
772 4.5GB SCSI F/W Disk Drive
773 9.1GB SCSI F/W Disk Drive
774 9.1GB External SCSI Disk Drive
77c Progress indicator. A 1.0 GB 16-bit SCSI disk drive being identified or configured.
783 4mm DDS-2 Tape Autoloader
789 2.6GB External Optical Drive
794 10/100MB Ethernet PX MC Adapter
797 Turboways 155 UTP/STP ATM Adapter being identified or configured.
798 Video streamer adapter being identified or configured.
800 Turboways 155 MMF ATM Adapter being identified or configured.
803 7336 Tape Library Robotics being configured
804 8x Speed SCSI-2 CD ROM drive being configured
807 SCSI Device Enclosure being configured
808 System Interface Full (SIF) configuration process
80c SSA 4-Port Adapter being identified or configured.
811 Processor complex being identified or configured.
812 Memory being identified or configured.
813 Battery for time-of-day, NVRAM, and so on being identified or configured, or system
I/O control logic being identified or configured.
814 NVRAM being identified or configured.
815 Floating-point processor test
816 Operator panel logic being identified or configured.
817 Time-of-day logic being identified or configured.
819 Graphics input device adapter being identified or configured.
821 Standard keyboard adapter being identified or configured.
823 Standard mouse adapter being identified or configured.
824 Standard tablet adapter being identified or configured.
825 Standard speaker adapter being identified or configured.
826 Serial Port 1 adapter being identified or configured.
827 Parallel port adapter being identified or configured.
828 Standard diskette adapter being identified or configured.
831 3151 adapter being identified or configured, or Serial Port 2 being identified or con-figured.
834 64-port async controller being identified or configured.
835 16-port async concentrator being identified or configured.
836 128-port async controller being identified or configured.
837 16-port remote async node being identified or configured.
838 Network Terminal Accelerator Adapter being identified or configured.
839 7318 Serial Communications Server being configured.
841 8-port async adapter (EIA-232) being identified or configured.
842 8-port async adapter (EIA-422A) being identified or configured.
843 8-port async adapter (MIL-STD 188) being identified or configured.
844 7135 RAIDiant Array disk drive subsystem controller being identified or configured.
845 7135 RAIDiant Array disk drive subsystem drawer being identified or configured.
846 RAIDiant Array SCSI 1.3GB Disk Drive
847 16-port serial adapter (EIA-232) being identified or configured.
848 16-port serial adapter (EIA-422) being identified or configured.
849 X.25 Interface Co-Processor/2 adapter being identified or configured.
850 Token-Ring network adapter being identified or configured.
851 T1/J1 Portmaster adapter being identified or configured.
852 Ethernet adapter being identified or configured.
854 3270 Host Connection Program/6000 connection being identified or configured.
855 Portmaster Adapter/A being identified or configured.
857 FSLA adapter being identified or configured.
858 5085/5086/5088 adapter being identified or configured.
859 FDDI adapter being identified or configured.
85c Progress indicator. Token-Ring High-Performance LAN adapter is being identified or
configured.
861 Optical adapter being identified or configured.
862 Block Multiplexer Channel Adapter being identified or configured.
865 ESCON Channel Adapter or emulator being identified or configured.
866 SCSI adapter being identified or configured.
867 Async expansion adapter being identified or configured.
868 SCSI adapter being identified or configured.
869 SCSI adapter being identified or configured.
870 Serial disk drive adapter being identified or configured.
871 Graphics subsystem adapter being identified or configured.
872 Grayscale graphics adapter being identified or configured.
874 Color graphics adapter being identified or configured.
875 Vendor generic communication adapter being configured.
876 8-bit color graphics processor being identified or configured.
877 POWER Gt3/POWER Gt4 being identified or configured.
878 POWER Gt4 graphics processor card being configured.
879 24-bit color graphics card, MEV2
880 POWER Gt1 adapter being identified or configured.
887 Integrated Ethernet adapter being identified or configured.
889 SCSI adapter being identified or configured.
890 SCSI-2 Differential Fast/Wide and Single-Ended Fast/Wide Adapter/A.
891 Vendor SCSI adapter being identified or configured.
892 Vendor display adapter being identified or configured.
893 Vendor LAN adapter being identified or configured.
894 Vendor async/communications adapter being identified or configured.
895 Vendor IEEE 488 adapter being identified or configured.
896 Vendor VME bus adapter being identified or configured.
897 S/370 Channel Emulator adapter being identified or configured.
898 POWER Gt1x graphics adapter being identified or configured.
899 3490 attached tape drive being identified or configured.
89c Progress indicator. A multimedia SCSI CD-ROM is being identified or configured.
901 Vendor SCSI device being identified or configured.
902 Vendor display device being identified or configured.
903 Vendor async device being identified or configured.
904 Vendor parallel device being identified or configured.
905 Vendor other device being identified or configured.
908 POWER GXT1000 Graphics subsystem being identified or configured.
910 1/4GB Fibre Channel/266 Standard Adapter being identified or configured.
911 Fibre Channel/1063 Adapter Short Wave
912 2.0GB SCSI-2 differential disk drive being identified or configured.
913 1.0GB differential disk drive being identified or configured.
914 5GB 8 mm differential tape drive being identified or configured.
915 4GB 4 mm tape drive being identified or configured.
916 Non-SCSI vendor tape adapter being identified or configured.
917 Progress indicator. 2.0GB 16-bit differential SCSI disk drive is being identified or
configured.
918 Progress indicator. 2GB 16-bit single-ended SCSI disk drive is being identified or
configured.
920 Bridge Box being identified or configured.
921 101 keyboard being identified or configured.
922 102 keyboard being identified or configured.
923 Kanji keyboard being identified or configured.
924 Two-button mouse being identified or configured.
925 Three-button mouse being identified or configured.
926 5083 tablet being identified or configured.
927 5083 tablet being identified or configured.
928 Standard speaker being identified or configured.
929 Dials being identified or configured.
930 Lighted program function keys (LPFK) being identified or configured.
931 IP router being identified or configured.
933 Async planar being identified or configured.
934 Async expansion drawer being identified or configured.
935 3.5-inch diskette drive being identified or configured.
936 5.25-inch diskette drive being identified or configured.
937 An HIPPI adapter is being configured.
942 POWER GXT 100 graphics adapter being identified or configured.
943 Progress indicator. 3480 and 3490 control units attached to a System/370 Channel
Emulator/A adapter are being identified or configured.
944 100MB ATM adapter being identified or configured
945 1.0GB SCSI differential disk drive being identified or configured.
946 Serial port 3 adapter is being identified or configured.
947 Progress indicator. A 730MB SCSI disk drive is being configured.
948 Portable disk drive being identified or configured.
949 Unknown direct bus-attach device being identified or configured.
950 Missing SCSI device being identified or configured.
951 670MB SCSI disk drive being identified or configured.
952 355MB SCSI disk drive being identified or configured.
953 320MB SCSI disk drive being identified or configured.
954 400MB SCSI disk drive being identified or configured.
955 857MB SCSI disk drive being identified or configured.
956 670MB SCSI disk drive electronics card being identified or configured.
957 120MB DBA disk drive being identified or configured.
958 160 MB DBA disk drive being identified or configured.
959 160MB SCSI disk drive being identified or configured.
960 1.37GB SCSI disk drive being identified or configured.
964 Internal 20GB 8mm tape drive identified or configured.
968 1.0GB SCSI disk drive being identified or configured.
970 Half-inch, 9-track tape drive being identified or configured.
971 150MB 1/4-inch tape drive being identified or configured.
972 2.3GB 8 mm SCSI tape drive being identified or configured.
973 Other SCSI tape drive being identified or configured.
974 CD-ROM drive being identified or configured.
975 Progress indicator. An optical disk drive is being identified or configured.
977 M-Audio Capture and Playback Adapter being identified or configured.
981 540MB SCSI-2 single-ended disk drive being identified or configured.
984 1GB 8-bit disk drive being identified or configured.
985 M-Video Capture Adapter being identified or configured.
986 2.4GB SCSI disk drive being identified or configured.
987 Progress indicator. Enhanced SCSI CD-ROM drive is being identified or configured.
989 200MB SCSI disk drive being identified or configured.
990 2.0GB SCSI-2 single-ended disk drive being identified or configured.
991 525MB 1/4-inch cartridge tape drive being identified or configured.
994 5GB 8 mm tape drive being identified or configured.
995 1.2GB 1/4 inch cartridge tape drive being identified or configured.
996 Progress indicator. Single-port, multi-protocol communications adapter is being
identified or configured.
997 FDDI adapter being identified or configured.
998 2.0GB4 mm tape drive being identified or configured.
999 7137 or 3514 Disk Array Subsystem being configured.
D81 T2 Ethernet Adapter being configured.

Diagnostic Load Progress Indicators
-----------------------------------

Note: When a lowercase c is listed, it displays in the lower half of the seven-segment
character position.

c00 AIX Install/Maintenance loaded successfully.
c01 Insert the first diagnostic diskette.
c02 Diskettes inserted out of sequence.
c03 The wrong diskette is in diskette drive.
c04 The loading stopped with a nonrecoverable error.
c05 A diskette error occurred.
c06 The rc.boot configuration shell script is unable to determine type of boot.
c07 Insert the next diagnostic diskette.
c08 RAM file system started incorrectly.
c09 The diskette drive is reading or writing a diskette.
c20 An unexpected halt occurred, and the system is configured to enter the kernel
debug program instead of entering a system dump.
c21 The ifconfig command was unable to configure the network for the client network
host.
c22 The tftp command was unable to read client's ClientHostName info file during a
client network boot.
c24 Unable to read client's ClientHostName.info file during a client network boot.
c25 Client did not mount remote miniroot during network install.
c26 Client did not mount the /usr file system during the network boot.
c29 The system was unable to configure the network device.
c31 Select the console display for the diagnostics. To select No console display, set the
key mode switch to Normal then to Service. The diagnostic programs will then load
and run the diagnostics automatically.
c32 A direct-attached display (HFT) was selected.
c33 A tty terminal attached to serial ports S1 or S2 was selected.
c34 A file was selected. The console messages store in a file.
c40 Configuration files are being restored.
c41 Could not determine the boot type or device.
c42 Extracting data files from diskette.
c43 Cannot access the boot/install tape.
c44 Initializing installation database with target disk information.
c45 Cannot configure the console.
c46 Normal installation processing.
c47 Could not create a physical volume identifier (PVID) on disk.
c48 Prompting you for input.
c49 Could not create or form the JFS log.
c50 Creating root volume group on target disks.
c51 No paging devices were found.
c52 Changing from RAM environment to disk environment.
c53 Not enough space in the /tmp directory to do a preservation installation.
c54 Installing either BOS or additional packages.
c55 Could not remove the specified logical volume in a preservation installation.
c56 Running user-defined customization.
c57 Failure to restore BOS.
c58 Displaying message to turn the key.
c59 Could not copy either device special files, device ODM, or volume group information
from RAM to disk.
c61 Failed to create the boot image.
c62 Loading platform dependent debug files
c63 Loading platform dependent data files
c64 Failed to load platform dependent data files
c70 Problem Mounting diagnostic CDROM disc
c99 Diagnostics have completed. This code is only used when there is no console.

Sunday, April 24, 2011

Recovering from 552, 554 and 556 in AIX

Causes of an LED 552, 554, or 556

An LED code of 552, 554, or 556 during a standard disk based boot indicates a failure occurred during the varyon of the rootvg volume group.
Some known causes of an LED 552, 554, or 556 are:

a corrupted file system
a corrupted Journaled File System (JFS) log device
a bad IPL-device record or bad IPL-device magic number; the magic number indicates the device type
a corrupted copy of the Object Data Manager (ODM) database on the boot logical volume
a fixed disk (hard disk) in the inactive state in the root volume group

Recovery procedure

To diagnose and fix the problem, boot to a Service mode shell and run the fsck command (file system check) on each file system. If the file system check fails, you may need to perform other steps.
WARNING: Do not use this document if the system is a /usr client, diskless client, or dataless client.

Boot your system into a limited function maintenance shell (Service or Maintenance mode) from bootable AIX media to use this recovery procedure. Refer to your system user's or installation and service guide for specific IPL procedures related to your type and model of hardware. You can also refer to the document titled "Booting in Service Mode", available at http://techsupport.services.ibm.com/server/aix.srchBroker for more information.
With bootable media of the same version and level as the system, boot the system into Service mode. The bootable media can be any ONE of the following:
- Bootable CD-ROM
- mksysb
- Bootable Install Tape
Follow the screen prompts to the Welcome to Base OS menu.
Choose Start Maintenance Mode for System Recovery (Option 3). The next screen displays prompts for the Maintenance menu.
- Choose Access a Root Volume Group (Option 1). The next screen displays a warning that indicates you will not be able to return to the Base OS menu without rebooting.
- Choose 0 continue. The next screen displays information about all volume groups on the system.
- Select the root volume group by number. The logical volumes in rootvg will be displayed with two options below.
- Choose Access this volume group and start a shell before mounting the file systems (Option 2).
If you receive errors from the preceding option, do not continue with the rest of this procedure. Correct the problem causing the error. If you need assistance correcting the problem causing the error, contact one of the following:
- local branch office
- your point of sale
- your AIX support center
Run the following commands to check and repair file systems.
```
fsck -p /dev/hd4 
   fsck -p /dev/hd2 
   fsck -p /dev/hd9var 
   fsck -p /dev/hd3
   fsck -p /dev/hd1 
```
NOTE: The -y option gives the fsck command permission to repair file system corruption when necessary. This flag can be used to avoid having to manually answer multiple confirmation prompts, however, use of this flag can cause permanent data loss in some situations.
If any of the following conditions occur, proceed accordingly.
- If fsck indicates that block 8 could not be read, the file system is probably unrecoverable. See step 5 for information on unrecoverable file systems.
- If fsck indicates that a file system has an unknown log record type, or if fsck fails in the logredo process, then go to step 6.
- If the file system checks were successful, skip to step 8.
The easiest way to fix an unrecoverable file system is to recreate it. This involves deleting it from the system and restoring it from a very current system backup. Note that hd9var and hd3 can be recreated, but hd4 and hd2 cannot be recreated. If hd4 and/or hd2 is unrecoverable, AIX must be reinstalled or restored from system backup. For assistance with unrecoverable file systems, contact your local branch office, point of sale, or AIX support center. Do not follow the rest of the steps in this document.
A corruption of the JFS log logical volume has been detected. Use the logform command to reformat it.
```
/usr/sbin/logform /dev/hd8
```
Answer yes when asked if you want to destroy the log.
Repeat step 4 for all file systems that did not successfully complete fsck the first time. If step 4 fails a second time, the file system is almost always unrecoverable. See step 5 for an explanation of the options at this point. In most cases, step 4 will be successful. If step 4 is successful, continue to step 8.
With the key in Normal position (for microchannel machines), run the following commands to reboot the system:
```
exit
   sync;sync;sync;reboot
```
As you reboot in Normal mode, notice how many times LED 551 appears. If LED 551 appears twice, fsck is probably failing because of a bad fshelper file. If this is the case and you are running AFS, see step 11.
The majority of instances of LED 552, 554, and 556 will be resolved at this point. If you still have an LED 552, 554, or 556, you may try the following steps.
ATTENTION: The following steps will overwrite your Object Data Manager (ODM) database files with a very primitive, minimal ODM database. Due to the potential loss of user configuration data caused by this procedure, it should only be used as a last resort effort to gain access to your system to attempt to back up any data that you can. It is NOT recommended to use the following procedure in lieu of restoring from a system backup.
Repeat step 1 through step 3.
Run the following commands, which remove much of the system's configuration and save it to a backup directory.
```
mount /dev/hd4 /mnt
   mount /dev/hd2 /mnt/usr
   mkdir /mnt/etc/objrepos/bak
   cp /mnt/etc/objrepos/Cu* /mnt/etc/objrepos/bak
   cp /etc/objrepos/Cu* /mnt/etc/objrepos
   umount /dev/hd2
   umount /dev/hd4
   exit
```
Determine which disk is the boot disk with the lslv command. The boot disk will be shown in the PV1 column of the lslv output.
```
lslv -m hd5
```
Save the clean ODM database to the boot logical volume. (# is the number of the fixed disk, determined with the previous command.)
```
savebase -d /dev/hdisk# 
```
If you are running AFS, go to step 11; otherwise, go to step 12.
If you are running the Andrew File System (AFS), use the following commands to find out whether you have more than one version of the v3fshelper file.
```
cd /sbin/helpers
   ls -l v3fshelper*
```
If you have only one version of the v3fshelper file (for example, v3fshelper), proceed to step 12.
If there is a version of v3fshelper marked as original (for example, v3fshelper.orig), run the following commands:
```
cp v3fshelper v3fshelper.afs
   cp v3fshelper.orig v3fshelper
```
WARNING: Do not proceed further if the system is a /usr client, diskless client, or dataless client. Make sure that hd5 is on the edge of the drive and if it is more than 1 partition that the partitions are contiguous. For systems of 5.1 and above, make sure that hd5 is greater than 12 MB:
```
lslv hd5 (Check to see what the PP Size: is equal to)
     lslv -m hd5
```
```
LP    PP1  PV1           PP2   PV2                    PP3   PV3
0001   0001 hdisk2
0002   0002 hdisk2
```
Recreate the boot image (hdisk# is the fixed disk determined in step 11):
```
bosboot -a -d /dev/hdisk# 
```
Make sure the bootlist is set correctly:
```
bootlist -m normal -o
```
Make changes, if necessary:
```
bootlist -m normal hdiskX cdX
```
(This can be edited to whatever you wish it to be.) NOTE: If you suspect an inactive or damaged disk device is causing the boot problem and the boot logical device, hd5, is mirrored onto another device, you may wish to list this other boot device first in the bootlist.

Make sure that the disk drive that you have chosen as your bootable device has a yes next to it:
```
ipl_varyon -i
```
Example:
```
PVNAME                     BOOT DEVICE       
PVID                                 VOLUME GROUP ID
hdisk1                     NO                
0007b53cbfd04a9000000000000000000007b53c00004c00
hdisk4                     NO                
0007b53c1244625d00000000000000000007b53c00004c00
hdisk2                     YES               
0007b53c8ffd631200000000000000000007b53c00004c00
```
From the above example, hdisk2 would be a bootable disk drive while hdisk1 and hdisk4 would not be.
If you copied files in step 11, copy the AFS file system helper back to v3fshelper:
```
cp v3fshelper.afs v3fshelper
```
Turn the key to the Normal position if dealing with microchannel machine and run
```
sync;sync;sync;reboot
```

If you followed all of the preceding steps and the system still stops at an LED 552, 554, or 556 during a reboot in Normal mode, you may want to consider reinstalling your system from a recent backup. Isolating the cause of the hang could be excessively time-consuming and may not be cost-effective in your operating environment. To isolate the possible cause of the hang, would require a debug boot of the system. Instructions for doing this are included in the document "Capturing Boot Debug", available at http://techsupport.services. ibm.com/server/aix.srchBroker . It is still possible, in the end, that isolation of the problem may indicate a restore or reinstall of AIX is necessary to correct it.
If you wish to pursue further system recovery, you may be able to obtain assistence from one of the following:

local branch office
your point of sale
your AIX support center

Causes of an LED 552, 554, or 556

An LED code of 552, 554, or 556 during a standard disk based boot indicates a failure occurred during the varyon of the rootvg volume group.
Some known causes of an LED 552, 554, or 556 are:

a corrupted file system
a corrupted Journaled File System (JFS) log device
a bad IPL-device record or bad IPL-device magic number; the magic number indicates the device type
a corrupted copy of the Object Data Manager (ODM) database on the boot logical volume
a fixed disk (hard disk) in the inactive state in the root volume group

Recovery procedure

Boot your system into a limited function maintenance shell (Service or Maintenance mode) from bootable AIX media to use this recovery procedure. Refer to your system user's or installation and service guide for specific IPL procedures related to your type and model of hardware. You can also refer to the document titled "Booting in Service Mode", available at http://techsupport.services.ibm.com/server/aix.srchBroker for more information.
With bootable media of the same version and level as the system, boot the system into Service mode. The bootable media can be any ONE of the following:
- Bootable CD-ROM
- mksysb
- Bootable Install Tape
Follow the screen prompts to the Welcome to Base OS menu.
Choose Start Maintenance Mode for System Recovery (Option 3). The next screen displays prompts for the Maintenance menu.
- Choose Access a Root Volume Group (Option 1). The next screen displays a warning that indicates you will not be able to return to the Base OS menu without rebooting.
- Choose 0 continue. The next screen displays information about all volume groups on the system.
- Select the root volume group by number. The logical volumes in rootvg will be displayed with two options below.
- Choose Access this volume group and start a shell before mounting the file systems (Option 2).
If you receive errors from the preceding option, do not continue with the rest of this procedure. Correct the problem causing the error. If you need assistance correcting the problem causing the error, contact one of the following:
- local branch office
- your point of sale
- your AIX support center
Run the following commands to check and repair file systems.
```
fsck -p /dev/hd4 
   fsck -p /dev/hd2 
   fsck -p /dev/hd9var 
   fsck -p /dev/hd3
   fsck -p /dev/hd1 
```
NOTE: The -y option gives the fsck command permission to repair file system corruption when necessary. This flag can be used to avoid having to manually answer multiple confirmation prompts, however, use of this flag can cause permanent data loss in some situations.
If any of the following conditions occur, proceed accordingly.
- If fsck indicates that block 8 could not be read, the file system is probably unrecoverable. See step 5 for information on unrecoverable file systems.
- If fsck indicates that a file system has an unknown log record type, or if fsck fails in the logredo process, then go to step 6.
- If the file system checks were successful, skip to step 8.
The easiest way to fix an unrecoverable file system is to recreate it. This involves deleting it from the system and restoring it from a very current system backup. Note that hd9var and hd3 can be recreated, but hd4 and hd2 cannot be recreated. If hd4 and/or hd2 is unrecoverable, AIX must be reinstalled or restored from system backup. For assistance with unrecoverable file systems, contact your local branch office, point of sale, or AIX support center. Do not follow the rest of the steps in this document.
A corruption of the JFS log logical volume has been detected. Use the logform command to reformat it.
```
/usr/sbin/logform /dev/hd8
```
Answer yes when asked if you want to destroy the log.
Repeat step 4 for all file systems that did not successfully complete fsck the first time. If step 4 fails a second time, the file system is almost always unrecoverable. See step 5 for an explanation of the options at this point. In most cases, step 4 will be successful. If step 4 is successful, continue to step 8.
With the key in Normal position (for microchannel machines), run the following commands to reboot the system:
```
exit
   sync;sync;sync;reboot
```
As you reboot in Normal mode, notice how many times LED 551 appears. If LED 551 appears twice, fsck is probably failing because of a bad fshelper file. If this is the case and you are running AFS, see step 11.
The majority of instances of LED 552, 554, and 556 will be resolved at this point. If you still have an LED 552, 554, or 556, you may try the following steps.
ATTENTION: The following steps will overwrite your Object Data Manager (ODM) database files with a very primitive, minimal ODM database. Due to the potential loss of user configuration data caused by this procedure, it should only be used as a last resort effort to gain access to your system to attempt to back up any data that you can. It is NOT recommended to use the following procedure in lieu of restoring from a system backup.
Repeat step 1 through step 3.
Run the following commands, which remove much of the system's configuration and save it to a backup directory.
```
mount /dev/hd4 /mnt
   mount /dev/hd2 /mnt/usr
   mkdir /mnt/etc/objrepos/bak
   cp /mnt/etc/objrepos/Cu* /mnt/etc/objrepos/bak
   cp /etc/objrepos/Cu* /mnt/etc/objrepos
   umount /dev/hd2
   umount /dev/hd4
   exit
```
Determine which disk is the boot disk with the lslv command. The boot disk will be shown in the PV1 column of the lslv output.
```
lslv -m hd5
```
Save the clean ODM database to the boot logical volume. (# is the number of the fixed disk, determined with the previous command.)
```
savebase -d /dev/hdisk# 
```
If you are running AFS, go to step 11; otherwise, go to step 12.
If you are running the Andrew File System (AFS), use the following commands to find out whether you have more than one version of the v3fshelper file.
```
cd /sbin/helpers
   ls -l v3fshelper*
```
If you have only one version of the v3fshelper file (for example, v3fshelper), proceed to step 12.
If there is a version of v3fshelper marked as original (for example, v3fshelper.orig), run the following commands:
```
cp v3fshelper v3fshelper.afs
   cp v3fshelper.orig v3fshelper
```
WARNING: Do not proceed further if the system is a /usr client, diskless client, or dataless client. Make sure that hd5 is on the edge of the drive and if it is more than 1 partition that the partitions are contiguous. For systems of 5.1 and above, make sure that hd5 is greater than 12 MB:
```
lslv hd5 (Check to see what the PP Size: is equal to)
     lslv -m hd5
```
```
LP    PP1  PV1           PP2   PV2                    PP3   PV3
0001   0001 hdisk2
0002   0002 hdisk2
```
Recreate the boot image (hdisk# is the fixed disk determined in step 11):
```
bosboot -a -d /dev/hdisk# 
```
Make sure the bootlist is set correctly:
```
bootlist -m normal -o
```
Make changes, if necessary:
```
bootlist -m normal hdiskX cdX
```
(This can be edited to whatever you wish it to be.) NOTE: If you suspect an inactive or damaged disk device is causing the boot problem and the boot logical device, hd5, is mirrored onto another device, you may wish to list this other boot device first in the bootlist.

Make sure that the disk drive that you have chosen as your bootable device has a yes next to it:
```
ipl_varyon -i
```
Example:
```
PVNAME                     BOOT DEVICE       
PVID                                 VOLUME GROUP ID
hdisk1                     NO                
0007b53cbfd04a9000000000000000000007b53c00004c00
hdisk4                     NO                
0007b53c1244625d00000000000000000007b53c00004c00
hdisk2                     YES               
0007b53c8ffd631200000000000000000007b53c00004c00
```
From the above example, hdisk2 would be a bootable disk drive while hdisk1 and hdisk4 would not be.
If you copied files in step 11, copy the AFS file system helper back to v3fshelper:
```
cp v3fshelper.afs v3fshelper
```
Turn the key to the Normal position if dealing with microchannel machine and run
```
sync;sync;sync;reboot
```

local branch office
your point of sale
your AIX support center

Causes of an LED 552, 554, or 556
An LED code of 552, 554, or 556 during a standard disk based boot indicates a failure occurred during the varyon of the rootvg volume group.
Some known causes of an LED 552, 554, or 556 are:

a corrupted file system
a corrupted Journaled File System (JFS) log device
a bad IPL-device record or bad IPL-device magic number; the magic number indicates the device type
a corrupted copy of the Object Data Manager (ODM) database on the boot logical volume
a fixed disk (hard disk) in the inactive state in the root volume group

Recovery procedure

Boot your system into a limited function maintenance shell (Service or Maintenance mode) from bootable AIX media to use this recovery procedure. Refer to your system user's or installation and service guide for specific IPL procedures related to your type and model of hardware. You can also refer to the document titled "Booting in Service Mode", available at http://techsupport.services.ibm.com/server/aix.srchBroker for more information.
With bootable media of the same version and level as the system, boot the system into Service mode. The bootable media can be any ONE of the following:
- Bootable CD-ROM
- mksysb
- Bootable Install Tape
Follow the screen prompts to the Welcome to Base OS menu.
Choose Start Maintenance Mode for System Recovery (Option 3). The next screen displays prompts for the Maintenance menu.
- Choose Access a Root Volume Group (Option 1). The next screen displays a warning that indicates you will not be able to return to the Base OS menu without rebooting.
- Choose 0 continue. The next screen displays information about all volume groups on the system.
- Select the root volume group by number. The logical volumes in rootvg will be displayed with two options below.
- Choose Access this volume group and start a shell before mounting the file systems (Option 2).
If you receive errors from the preceding option, do not continue with the rest of this procedure. Correct the problem causing the error. If you need assistance correcting the problem causing the error, contact one of the following:
- local branch office
- your point of sale
- your AIX support center
Run the following commands to check and repair file systems.
```
fsck -p /dev/hd4 
   fsck -p /dev/hd2 
   fsck -p /dev/hd9var 
   fsck -p /dev/hd3
   fsck -p /dev/hd1 
```
NOTE: The -y option gives the fsck command permission to repair file system corruption when necessary. This flag can be used to avoid having to manually answer multiple confirmation prompts, however, use of this flag can cause permanent data loss in some situations.
If any of the following conditions occur, proceed accordingly.
- If fsck indicates that block 8 could not be read, the file system is probably unrecoverable. See step 5 for information on unrecoverable file systems.
- If fsck indicates that a file system has an unknown log record type, or if fsck fails in the logredo process, then go to step 6.
- If the file system checks were successful, skip to step 8.
The easiest way to fix an unrecoverable file system is to recreate it. This involves deleting it from the system and restoring it from a very current system backup. Note that hd9var and hd3 can be recreated, but hd4 and hd2 cannot be recreated. If hd4 and/or hd2 is unrecoverable, AIX must be reinstalled or restored from system backup. For assistance with unrecoverable file systems, contact your local branch office, point of sale, or AIX support center. Do not follow the rest of the steps in this document.
A corruption of the JFS log logical volume has been detected. Use the logform command to reformat it.
```
/usr/sbin/logform /dev/hd8
```
Answer yes when asked if you want to destroy the log.
Repeat step 4 for all file systems that did not successfully complete fsck the first time. If step 4 fails a second time, the file system is almost always unrecoverable. See step 5 for an explanation of the options at this point. In most cases, step 4 will be successful. If step 4 is successful, continue to step 8.
With the key in Normal position (for microchannel machines), run the following commands to reboot the system:
```
exit
   sync;sync;sync;reboot
```
As you reboot in Normal mode, notice how many times LED 551 appears. If LED 551 appears twice, fsck is probably failing because of a bad fshelper file. If this is the case and you are running AFS, see step 11.
The majority of instances of LED 552, 554, and 556 will be resolved at this point. If you still have an LED 552, 554, or 556, you may try the following steps.
ATTENTION: The following steps will overwrite your Object Data Manager (ODM) database files with a very primitive, minimal ODM database. Due to the potential loss of user configuration data caused by this procedure, it should only be used as a last resort effort to gain access to your system to attempt to back up any data that you can. It is NOT recommended to use the following procedure in lieu of restoring from a system backup.
Repeat step 1 through step 3.
Run the following commands, which remove much of the system's configuration and save it to a backup directory.
```
mount /dev/hd4 /mnt
   mount /dev/hd2 /mnt/usr
   mkdir /mnt/etc/objrepos/bak
   cp /mnt/etc/objrepos/Cu* /mnt/etc/objrepos/bak
   cp /etc/objrepos/Cu* /mnt/etc/objrepos
   umount /dev/hd2
   umount /dev/hd4
   exit
```
Determine which disk is the boot disk with the lslv command. The boot disk will be shown in the PV1 column of the lslv output.
```
lslv -m hd5
```
Save the clean ODM database to the boot logical volume. (# is the number of the fixed disk, determined with the previous command.)
```
savebase -d /dev/hdisk# 
```
If you are running AFS, go to step 11; otherwise, go to step 12.
If you are running the Andrew File System (AFS), use the following commands to find out whether you have more than one version of the v3fshelper file.
```
cd /sbin/helpers
   ls -l v3fshelper*
```
If you have only one version of the v3fshelper file (for example, v3fshelper), proceed to step 12.
If there is a version of v3fshelper marked as original (for example, v3fshelper.orig), run the following commands:
```
cp v3fshelper v3fshelper.afs
   cp v3fshelper.orig v3fshelper
```
WARNING: Do not proceed further if the system is a /usr client, diskless client, or dataless client. Make sure that hd5 is on the edge of the drive and if it is more than 1 partition that the partitions are contiguous. For systems of 5.1 and above, make sure that hd5 is greater than 12 MB:
```
lslv hd5 (Check to see what the PP Size: is equal to)
     lslv -m hd5
```
```
LP    PP1  PV1           PP2   PV2                    PP3   PV3
0001   0001 hdisk2
0002   0002 hdisk2
```
Recreate the boot image (hdisk# is the fixed disk determined in step 11):
```
bosboot -a -d /dev/hdisk# 
```
Make sure the bootlist is set correctly:
```
bootlist -m normal -o
```
Make changes, if necessary:
```
bootlist -m normal hdiskX cdX
```
(This can be edited to whatever you wish it to be.) NOTE: If you suspect an inactive or damaged disk device is causing the boot problem and the boot logical device, hd5, is mirrored onto another device, you may wish to list this other boot device first in the bootlist.

Make sure that the disk drive that you have chosen as your bootable device has a yes next to it:
```
ipl_varyon -i
```
Example:
```
PVNAME                     BOOT DEVICE       
PVID                                 VOLUME GROUP ID
hdisk1                     NO                
0007b53cbfd04a9000000000000000000007b53c00004c00
hdisk4                     NO                
0007b53c1244625d00000000000000000007b53c00004c00
hdisk2                     YES               
0007b53c8ffd631200000000000000000007b53c00004c00
```
From the above example, hdisk2 would be a bootable disk drive while hdisk1 and hdisk4 would not be.
If you copied files in step 11, copy the AFS file system helper back to v3fshelper:
```
cp v3fshelper.afs v3fshelper
```
Turn the key to the Normal position if dealing with microchannel machine and run
```
sync;sync;sync;reboot
```

local branch office
your point of sale
your AIX support center

VIO basics

The Virtual I/O Server is part of the IBM System p Advanced Power Virtualization hardware feature. Virtual I/O Server allows sharing of physical resources between LPARs including virtual SCSI and virtual networking. This allows more efficient utilization of physical resources through sharing between LPARs and facilitates server consolidation.

The Virtual I/O Server is software that is located in a logical partition. This software facilitates the sharing of physical I/O resources between AIX® and Linux® client logical partitions within the server. The Virtual I/O Server provides virtual SCSI target and Shared Ethernet Adapter capability to client logical partitions within the system, allowing the client logical partitions to share SCSI devices and Ethernet adapters. The Virtual I/O Server software requires that the logical partition be dedicated solely for its use.
The Virtual I/O Server is available as part of the Advanced POWER™ Virtualization hardware feature.
Using the Virtual I/O Server facilitates the following functions:
-->Sharing of physical resources between logical partitions on the system
-->Creating logical partitions without requiring additional physical I/O resources
-->Creating more logical partitions than there are I/O slots or physical devices available with the ability for partitions to have dedicated I/O, virtual I/O, or both
-->Maximizing use of physical resources on the system
-->Helping to reduce the Storage Area Network (SAN) infrastructure
The Virtual I/O Server supports client logical partitions running the following operating systems:
-->AIX 5.3 or later
-->SUSE Linux Enterprise Server 9 for POWER (or later)
-->Red Hat® Enterprise Linux AS for POWER Version 3 (update 2 or later)
-->Red Hat Enterprise Linux AS for POWER Version 4 (or later)
For the most recent information about devices that are supported on the Virtual I/O Server, to download Virtual I/O Server fixes and updates, and to find additional information about the Virtual I/O Server, see the Virtual I/O Server Web site.
The Virtual I/O Server comprises the following primary components:
-->Virtual SCSI
-->Virtual Networking
-->Integrated Virtualization Manager
The following sections provide a brief overview of each of these components.

Virtual SCSIPhysical adapters with attached disks or optical devices on the Virtual I/O Server logical partition can be shared by one or more client logical partitions. The Virtual I/O Server offers a local storage subsystem that provides standard SCSI-compliant logical unit numbers (LUNs). The Virtual I/O Server can export a pool of heterogeneous physical storage as an homogeneous pool of block storage in the form of SCSI disks.
Unlike typical storage subsystems that are physically located in the SAN, the SCSI devices exported by the Virtual I/O Server are limited to the domain within the server. Although the SCSI LUNs are SCSI compliant, they might not meet the needs of all applications, particularly those that exist in a distributed environment.
The following SCSI peripheral-device types are supported:
-->Disks backed by a logical volume
-->Disks backed by a physical volume
-->Optical devices (DVD-RAM and DVD-ROM)

Virtual networking
Shared Ethernet Adapter allows logical partitions on the virtual local area network (VLAN) to share access to a physical Ethernet adapter and to communicate with systems and partitions outside the server. This function enables logical partitions on the internal VLAN to share the VLAN with stand-alone servers.

Integrated Virtualization Manager
The Integrated Virtualization Manager provides a browser-based interface and a command-line interface that you can use to manage IBM® System p5™ and IBM eServer™ pSeries® servers that use the IBM Virtual I/O Server. On the managed system, you can create logical partitions, manage the virtual storage and virtual Ethernet, and view service information related to the server. The Integrated Virtualization Manager is packaged with the Virtual I/O Server, but it is activated and usable only on certain platforms and where no Hardware Management Console (HMC) is present.

Intorduction to VIO

Prior to the introduction of POWER5 systems, it was only possible to create as many separate logical partitions (LPARs) on an IBM system as there were physical processors. Given that the largest IBM eServer pSeries POWER4 server, the p690, had 32 processors, 32 partitions were the most anyone could create. A customer could order a system with enough physical disks and network adapter cards to so that each LPAR would have enough disks to contain operating systems and enough network cards to allow users to communicate with each partition.
The Advanced POWER Virtualization™ feature of POWER5 platforms1 makes it possible to allocate fractions of a physical CPU to a POWER5 LPAR. Using virtual CPU's and virtual I/O a user can create many more LPARs on a p5 system than there are CPU's or I/O slots. The Advanced POWER Virtualization feature accounts for this by allowing users to create shared network adapters and virtual SCSI disks. Custome

rs can use these virtual resources to provide disk space and network adapters for each LPAR they create on their POWER5 system
(see Figure ).

There are three components of the Advanced POWER Virtualization feature: Micro-Partitioning™, shared Ethernet adapters, and virtual SCSI. In addition, AIX 5L Version
5.3 allows users to define virtual Ethernet adapters permitting inter-LPAR communication. This paper provides an overview of how each of these components works and then shows the details of how to set up a simple three-partition system where one partition is a Virtual I/O Server and the other two partitions use virtual Ethernet and virtual SCSI to differing degrees. What follows is a practical guide to help a new POWER5 customer set up simple systems where high availability is not a concern, but becoming familiar with this new technology in a development environment is the primary goal.

Micro-Partitioning
An element of the IBM POWER Virtualization feature called Micro-Partitioning can divide a single processor into many different processors. In POWER4 systems, each physical processor is dedicated to an LPAR. This concept of dedicated processors is still present in POWER5 systems, but so is the concept of shared processors. A POWER5 system administrator can use the Hardware Management Console (HMC) to place processors in
a shared processor pool. Using the HMC, the administrator can assign fractions of a CPU to individual partitions. If one LPAR is defined to use processors in the shared processor pool, when those CPUs are idle, the POWER Hypervisor™ makes them available to other partitions. This ensures that these processing resources are not wasted. Also, the ability to assign fractions of a CPU to a partition means it is possible to partition POWER5 servers into many different partitions. Allocation of physical processor and memory resources on POWER5 systems is managed by a system firmware component called the POWER Hypervisor.

Virtual Networking
Virtual networking on POWER5 hardware consists of two main capabilities. One capability is provided by a software IEEE 802.1q (VLAN) switch that is implemented in the Hypervisor on POWER5 hardware. Users can use the HMC to add Virtual Ethernet adapters to their partition definitions. Once these are added and the partitions booted, the new adapters can be configured just like real physical adapters, and the partitions can communicate with each other without having to connect cables between the LPARs. Users can separate traffic from different VLANs by assigning different VLAN IDs to each virtual Ethernet adapter. Each AIX 5.3 partition can support up to 256 Virtual Ethernet adapters

In addition, a part of the Advanced POWER virtualization virtual networking feature allows users to share physical adapters between logical partitions. These shared adapters, called Shared Ethernet Adapters (SEAs), are managed by a Virtual I/O Server partition which maps physical adapters under its control to virtual adapters. It is possible to map many physical Ethernet adapters to a single virtual Ethernet adapter thereby eliminating a single physical adapter as a point of failure in the architecture.
There are a few things users of virtual networking need to consider before implementing it. First, virtual networking ultimately uses more CPU cycles on the POWER5 machine than when physical adapters are assigned to a partition. Users should consider assigning a physical adapter directly to a partition when heavy network traffic is predicted over a certain adapter. Secondly, users may want to take advantage of larger MTU sizes that virtual Ethernet allows if they know that their applications will benefit from the reduced fragmentation and better performance that larger MTU sizes offer. The MTU size limit for SEA is smaller than Virtual Ethernet adapters, so users will have to carefully choose an MTU size so that packets are sent to external networks with minimum fragmentation.

Virtual SCSI
The Advanced POWER Virtualization feature called virtual SCSI allows access to physical disk devices which are assigned to the Virtual I/O Server (VIOS). The system administrator uses VIOS logical volume manager commands to assign disks to volume groups. The administrator creates logical volumes in the Virtual I/O Server volume groups. Either these logical volumes or the physical disks themselves may ultimately appear as physical disks (hdisks) to the Virtual I/O Server’s client partitions once they are associated with virtual SCSI host adapters. While the Virtual I/O Server software is
packaged as an additional software bundle that a user purchases separately from the AIX 53 distribution, the virtual I/O client software is a part of the AIX 5.3 base installation media so an administrator does not need to install any additional filesets on a Virtual SCSI client partition. Srikrishnan provides more details on how the Virtual SCSI feature works