Friday, May 10, 2013

AIX 6.1 TL Upgrade using multibos


Starting with AIX* 5.3 Technology Level (TL) 3, the new multibos utility has been provided that allows an AIX administrator to create and maintain two separate, bootable instances of the AIX OS within the same root volume group (rootvg). This second instance of rootvg is known as a standby Base Operating System (BOS) and is an extremely handy tool for performing AIX TL and Service Pack (SP) updates.
Multibos lets you install, update and customize a standby instance of the AIX OS without impacting the running and active production instance of the AIX OS. This is valuable in environments with tight maintenance windows. Instead of requiring an outage window of several hours to apply a new TL or SP, you’ll only need a small outage at a convenient time to reboot the system. This may help reduce the size of the after-hours effort often required when performing AIX updates, as all the maintenance activities can be performed during business hours. After hours you could log in from home and reboot the system.
Backing out from an AIX TL update is also made easier with multibos. To go back to a previous TL, you reboot the system on the original AIX instance boot logical volume (BLV). It’s also possible to update several AIX systems at once using multibos, which again reduces the amount of after-hours effort required when performing AIX maintenance activities.
Multibos is similar to an alternate disk installation. However, there are several differences between the two methods, one of which is that there’s no need for an additional disk to clone the rootvg. Both utilities can be used to achieve the same goal. Choose the one that’s the best fit for your AIX environment.

Getting Started


Before attempting to use multibos, check that the prerequisites have been met. First, the system must be running AIX 5.3 with TL3 or higher. Next, ensure that there’s enough free space in rootvg for a copy of each BOS logical volume (LV). By default, the BOS file systems in rootvg(/, /usr, /var, and /opt) and the BLV are copied. All other file systems and LVs are shared between BOS instances. Check the number of free physical partitions in rootvg (i.e., # lsvg rootvg | grep FREE). If all of the requirements can’t be met, then a traditional update should be performed.
Ensure that you document the system and perform a mksysb before performing any maintenance activity.

Steps to Upgrade using Multibos

 
1. Create a separate filesystem for the updates
2. Create a backup of the system
3. Check current TL level
4. Preview/Create the multibos standby BOS instance.
5. Preview/Patch the OS in the standby instance
6. Verify the oslevel in the standby instance after the update
7. Reboot and check
8. Optionally return to the pre-update BOS image.
 
1. Create a separate FileSystem for the /usr/sys/inst.images folder 

Where I will put the (TL) updates.
 # rm -R /usr/sys/inst.images/
 # crfs -v jfs2 -A yes -g rootvg -m /usr/sys/inst.images -a size=6G
 # mount /usr/sys/inst.images/
 # df -g /usr/sys/inst.images
 Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
 /dev/fslv00        6.00      6.00    1%        3     1% /usr/sys/inst.images

==============================================================================
 -v      FS type
 -A yes  Append /etc/filesystems - automatic mount at boot
 -g      volume group
 -m      mount point
 -a size The expected size of the new FS
==============================================================================

2. Create a backup

If something should go wrong, it is good to have a backup. The following backup is preformed from a NIM master server, backing up the client (power2) that will be updated.
 #  nim -o define -t mksysb  -a server=master -a location=/export2/mksysb/P2_MK_PREU -a mk_image=yes -a source=power2 P2_MK_PREU
+---------------------------------------------------------------------+
                System Backup Image Space Information
              (Sizes are displayed in 1024-byte blocks.)
+---------------------------------------------------------------------+

Required = 2732693 (2669 MB)    Available = 5957884 (5819 MB)

Creating information file (/image.data) for rootvg.

Creating list of files to back up 

0512-038 savevg: Backup Completed Successfully.

==============================================================================
-o operation
-t type
-a attribute
   location=where the backup will reside
   mk_images= create mksysb image file
   source= NIM client name
P2_MK_PREU NIM object mksysb name
==============================================================================
Now copy the updates from wherever you have them to the folder /usr/sys/inst.images/
# cp -R -p 6100-07-00-1140 /usr/sys/inst.images/
Next create a .toc (table of content file) in the directory where the updates reside using the inutoc command
# iutoc /usr/sys/inst.images/6100-07-00-1140/

3. Check the current TL 

# oslevel -s 6100-04-04-1014 AIX Release 6.1 Technology Level 04 Service Pack 04 Year 2010 Week 14

4. Preview/Create the multibos standby BOS instance

Remove any pervious stanby BOS instances, so we can update a clean environment.
# multibos -R
...
Return Status = SUCCESS
Create a standby BOS (Base Operating System) instance using the multibos command.
Initially I will run a preview operation
# multibos -Xsp
Return Status = SUCCESS

==============================================================================
-X automatically extend FS
-s create standby BOS instance
-p preview mode
==============================================================================
Now we actually run the command without preview mode. This may take a while even two.
# multibos -Xs
Initializing multibos methods ...
Initializing log /etc/multibos/logs/op.alog ...
Gathering system information ...

+-----------------------------------------------------------------------------+
Setup Operation
+-----------------------------------------------------------------------------+
Verifying operation parameters ...
Creating image.data file ...

+-----------------------------------------------------------------------------+
Logical Volumes
+-----------------------------------------------------------------------------+
Creating standby BOS logical volume hd5
Creating standby BOS logical volume hd4
Creating standby BOS logical volume hd2
Creating standby BOS logical volume hd9var
Creating standby BOS logical volume hd10opt

+-----------------------------------------------------------------------------+
File Systems
+-----------------------------------------------------------------------------+
Creating all standby BOS file systems ...
Creating standby BOS file system /bos_inst on logical volume hd4
reating standby BOS file system /bos_inst on logical volume hd4
Creating standby BOS file system /bos_inst/usr on logical volume hd2
Creating standby BOS file system /bos_inst/var on logical volume hd9var
Creating standby BOS file system /bos_inst/opt on logical volume hd10opt

+-----------------------------------------------------------------------------+
Mount Processing
+-----------------------------------------------------------------------------+
Mounting all standby BOS file systems ...
Mounting /bos_inst
Mounting /bos_inst/usr
Mounting /bos_inst/var
Mounting /bos_inst/opt

+-----------------------------------------------------------------------------+
BOS Files
+-----------------------------------------------------------------------------+
Including files for file system /
Including files for file system /usr
Including files for file system /var
Including files for file system /opt

Copying files using backup/restore utilities ...
Percentage of files copied:   0.00%
Percentage of files copied:   1.35%
...
Percentage of files copied:  98.74%
Percentage of files copied: 100.00%

+-----------------------------------------------------------------------------+
Boot Partition Processing
+-----------------------------------------------------------------------------+
Active boot logical volume is bos_hd5.
Standby boot logical volume is hd5.
Creating standby BOS boot image on boot logical volume hd5
bosboot: Boot image is 43345 512 byte blocks.

+-----------------------------------------------------------------------------+
Mount Processing
+-----------------------------------------------------------------------------+
Unmounting all standby BOS file systems ...
Unmounting /bos_inst/opt
Unmounting /bos_inst/var
Unmounting /bos_inst/usr
Unmounting /bos_inst

+-----------------------------------------------------------------------------+
Bootlist Processing
+-----------------------------------------------------------------------------+
Verifying operation parameters ...
Setting bootlist to logical volume hd5 on hdisk0.
ATTENTION: firmware recovery string for standby BLV (hd5):
boot /pci@800000020000003/pci@2,4/pci1069,b166@1/scsi@0/sd@8:4
ATTENTION: firmware recovery string for active BLV (bos_hd5):
boot /pci@800000020000003/pci@2,4/pci1069,b166@1/scsi@0/sd@8:2

Log file is /etc/multibos/logs/op.alog
Return Status = SUCCESS

5. Preview/Patch the OS in the standby instance
Preview TL update
# multibos -Xacp -l /usr/sys/inst.images/6100-07-00-1140
...
Log file is /etc/multibos/logs/op.alog
Return Status = SUCCESS

==============================================================================
-X automatically extend FS if required
-a update_all install option
-c Performs a customized update of the software in standby BOS
-p preview
-l location
==============================================================================
Now we can run the update without the -p option
# multibos -Xac -l /usr/sys/inst.images/6100-07-00-1140 

Initializing multibos methods ...
Initializing log /etc/multibos/logs/op.alog ...
Gathering system information ...

+-----------------------------------------------------------------------------+
Customization Operation
+-----------------------------------------------------------------------------+
Verifying operation parameters ...
Validating install images location /usr/sys/inst.images/6100-07-00-1140

+-----------------------------------------------------------------------------+
Mount Processing
+-----------------------------------------------------------------------------+
Mounting all standby BOS file systems ...
Mounting /bos_inst
Mounting /bos_inst/usr
Mounting /bos_inst/var
Mounting /bos_inst/opt

+-----------------------------------------------------------------------------+
Software Update
+-----------------------------------------------------------------------------+
Installing software to standby BOS ...

install_all_updates: Initializing system parameters.
install_all_updates: Log file is /var/adm/ras/install_all_updates.log
install_all_updates: Checking for updated install utilities on media
......

installp:  * * * A T T E N T I O N ! ! !
        Software changes processed during this session require
        any diskless/dataless clients to which this SPOT is
        currently allocated to be rebooted. 

install_all_updates: Checking for recommended maintenance level 6100-07.
install_all_updates: Executing /usr/bin/oslevel -rf, Result = 6100-05
install_all_updates: ATTENTION, the system recommended maintenance level
does not correspond to the highest level known to install_all_updates.
For more details, execute /usr/bin/oslevel -rl 6100-07.

install_all_updates: Log file is /var/adm/ras/install_all_updates.log
install_all_updates: Result = SUCCESS

+-----------------------------------------------------------------------------+
Boot Partition Processing
+-----------------------------------------------------------------------------+
Active boot logical volume is bos_hd5.
Standby boot logical volume is hd5.
Creating standby BOS boot image on boot logical volume hd5
Could not load program ls:
Symbol resolution failed for ls because:
        Symbol time64 (number 40) is not exported from dependent
          module /usr/lib/multibos_chroot/usr/ccs/lib/libc.a(shr.o).
Examine .loader section symbols with the 'dump -Tv' command.
/usr/sbin/bosboot[16]: kernsize /= 1024: bad number
multibos: 0565-039 Error processing standby BOS boot image.
Return Status: FAILURE
The filesets where updated fine but multibos failed to create a bootimage on hd5
The “Symbol resolution failed for ls because:
Symbol time64 (number 40) is not exported from dependent
module /usr/lib/multibos_chroot/usr/ccs/lib/libc.a(shr.o).”
This error is addressed in APAR IV03737.
==============================================================================
IV03737 Abstract: symbol resolution failure creating standby BOS boot 

IV03737 Symptom Text:
 Doing multibos with updates can fail:
 Creating standby BOS boot image on boot logical volume
 bos_hd5
 Could not load program ls:
 Symbol resolution failed for ls because:
 Symbol time64 (number 40) is not exported from dependent
 module /usr/lib/multibos_chroot/usr/ccs/lib/libc.a(shr.o).
 Examine .loader section symbols with the 'dump -Tv' command.
 /usr/sbin/bosboot[16]: kernsize /= 1024: 0403-009 The
 specified number is not valid for this command.
 multibos: 0565-039 Error processing standby BOS boot image.
 multibos: 0565-035 Error setting up standby BOS.

==============================================================================
In order to fix this issue with multibos it is nacessary to apply the APAR fixes on the live system (not the standby BOS instance). Afterwards verify if the APAR fixes have been applied. A system reboot will be required after the APAR fixes have been installed.
# inutoc /usr/sys/inst.images/IV03737/

# install_all_updates -d /usr/sys/inst.images/IV03737/ -cxY
==============================================================================
-d Device
-c Commit all
-x Expand FS if necessary
-Y Agree to all SW licenses
==============================================================================
# instfix -ik IV03737
    All filesets for IV03737 were found.
Now to start on a fresh environment after the APAR’s fixes have been installed recreate the multibos stanby BOS.
# multibos -R

# multibos -Xs
Finally return to the initial task of updating the system in the standby BOS.
# multibos -Xac -l /usr/sys/inst.images/6100-07-00-1140/
Initializing multibos methods ...
Initializing log /etc/multibos/logs/op.alog ...
Gathering system information ...

+-----------------------------------------------------------------------------+
Customization Operation
+-----------------------------------------------------------------------------+
Verifying operation parameters ...
Validating install images location /usr/sys/inst.images/6100-07-00-1140/

+-----------------------------------------------------------------------------+
Mount Processing
+-----------------------------------------------------------------------------+
Mounting all standby BOS file systems ...
Mounting /bos_inst
Mounting /bos_inst/usr
Mounting /bos_inst/var
Mounting /bos_inst/opt

+-----------------------------------------------------------------------------+
Software Update
+-----------------------------------------------------------------------------+
Installing software to standby BOS ...

install_all_updates: Initializing system parameters.
install_all_updates: Log file is /var/adm/ras/install_all_updates.log
install_all_updates: Checking for updated install utilities on media.
install_all_updates: Processing media.
install_all_updates: Generating list of updatable installp filesets.

...
...
...

install_all_updates: Checking for recommended maintenance level 6100-07.
install_all_updates: Executing /usr/bin/oslevel -rf, Result = 6100-07
install_all_updates: Verification completed.
install_all_updates: Log file is /var/adm/ras/install_all_updates.log
install_all_updates: Result = SUCCESS

+-----------------------------------------------------------------------------+
Boot Partition Processing
+-----------------------------------------------------------------------------+
Active boot logical volume is bos_hd5.
Standby boot logical volume is hd5.
Creating standby BOS boot image on boot logical volume hd5
bosboot: Boot image is 49180 512 byte blocks.

+-----------------------------------------------------------------------------+
Mount Processing
+-----------------------------------------------------------------------------+
Unmounting all standby BOS file systems ...
Unmounting /bos_inst/opt
Unmounting /bos_inst/var
Unmounting /bos_inst/usr
Unmounting /bos_inst

Log file is /etc/multibos/logs/op.alog
Return Status = SUCCESS

6. Verify the oslevel in the standby instance after the update

We can actually now enter the secondary BOS instance like a chroot and check the oslevel
# multibos -S
Initializing multibos methods ...
Initializing log /etc/multibos/logs/op.alog ...
Gathering system information ...

+-----------------------------------------------------------------------------+
Multibos Shell Operation
+-----------------------------------------------------------------------------+
Verifying operation parameters ...

+-----------------------------------------------------------------------------+
Mount Processing
+-----------------------------------------------------------------------------+
Mounting all standby BOS file systems ...
Mounting /bos_inst
Mounting /bos_inst/usr
Mounting /bos_inst/var
Mounting /bos_inst/opt

+-----------------------------------------------------------------------------+
Multibos Root Shell
+-----------------------------------------------------------------------------+
Starting multibos root shell ...
Active boot logical volume is bos_hd5.
Script started, file is /etc/multibos/logs/scriptlog.120204105327.txt
MULTIBOS> oslevel -s
6100-07-01-1141
MULTIBOS>exit
Verify the bootlist order
# bootlist -m normal -o
hdisk0 blv=hd5 pathid=0
hdisk0 blv=bos_hd5 pathid=0

7. Reboot and check

# oslevel -s
6100-07-01-1141
# instfix -i | grep ML
    All filesets for 6100-00_AIX_ML were found.
    All filesets for 6100-01_AIX_ML were found.
    All filesets for 6100-02_AIX_ML were found.
    All filesets for 6100-03_AIX_ML were found.
    All filesets for 6100-04_AIX_ML were found.
    All filesets for 6100-05_AIX_ML were found.
    All filesets for 6100-06_AIX_ML were found.
    All filesets for 6100-07_AIX_ML were found.


8. Optionally return to the pre-update BOS image.

If necessary you can return to the initial TL, just change the boot order.
 # bootlist -m normal -o
 hdisk0 blv=hd5 pathid=0
 hdisk0 blv=bos_hd5 pathid=0

 # bootlist -m normal hdisk0 blv=bos_hd5 hdisk0 blv=hd5

 # bootlist -m normal -o
 hdisk0 blv=bos_hd5 pathid=0
 hdisk0 blv=hd5 pathid=0

 # oslevel -s
 6100-07-01-1141
Reboot the server and check with oslevel.
 # oslevel -s
 6100-04-11-1140
So we are now back on the initial TL.

No comments:

Post a Comment