Force refresh of remote device's ARP cache

Background

ARP.
This is short for Address Resolution Protocol.
This is a method for finding the hardware address (MAC) when only its IP is known.

On OSI model, ARP is a Data Link Layer Protocol. (Layer 2). IP is on Network Layer (layer 3).

There are times when you need to force the remote devices, such as the routers, switches or gateways to refresh their ARP cache when you perform some network changes on servers. Some of these changes includes

• Physical Box relocation (requiring a new network IP)
• Logical IP relocation (especially in cluster or DR)

ARP cache will be updated usually when either of the following happens

• Expiry of the entry in the cache
• Host updates the router about the change in IP
• ARP cache is manually cleared.

Without forcing through an immediate ARP cache refresh, sometimes the network will not know the changes until very much later and end result is service unaccessible, causing losses to busineseses and much headaches for the IT team.

Procedures (RHEL)

On RHEL, usually arping is available in the default installation. You need to execute the command in root.

# /sbin/arping -s -b

where

-b : keep broadcasting, don't go unicast
-s source : source ip address

Procedures (Windows XP, Vista, 7)

netsh interface IP delete arpcache

What is linux-gate.so.1?

Recently, was tracing some faults on a RHEL system and noticed that there is this "linux-gate.so.1" library that is pointed by many of the binaries.

# ldd /sbin/vxconfigd

linux-gate.so.1 => (0xffffe000)
libcrypt.so.1 => /lib/libcrypt.so.1 (0x0054e000)
libm.so.6 => /lib/tls/libm.so.6 (0x004de000)
libc.so.6 => /lib/tls/libc.so.6 (0x003aa000)
libdl.so.2 => /lib/libdl.so.2 (0x004d8000)
libvxscsi.so => /etc/vx/slib/libvxscsi.so (0xf7fcd000)
libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00515000)
/lib/ld-linux.so.2 (0x00390000)

Seems that this is similar even on RHEL 4.2 and 4.6.
Asked Mr. Google and this URL turned up. [http://www.trilithium.com/johan/2005/08/linux-gate/]

In summary, this file is a virtual DSO, dynamically shared objected exposed by the kernel at a fixed address in every process memory.
If you want more details, check out the above link. :)

What does clfileprop process do

May have seen the following process

myserver:/:>ps -aef | grep clfile
root 4063286 5570752 0 00:59:56 - 0:00 /usr/es/sbin/cluster/utilities/clfileprop -a

This process, in essense,
- belongs to powerHA.
- runs every 10 minutes on my servers. Think this is default.
- propagate changes of configuration files to all other nodes.
- caveat is, if you run manually on node A, this node A will propagate files from node A to other nodes regardless of last modified date. e.g. if you run verification, etc.

My understanding of RBAC in AIX

What is RBAC?

It stand for Role Based Access Control.
There is major differences between RBAC in AIX 5.3 and older AND RBAC in AIX 6.1/7.1. No value in discussing older RBAC. Will explain for "enhanced RBAC" instead.
Three primary rules are defined for RBAC:

• Role assignment: A subject can exercise a permission only if the subject has selected or been assigned a role.
• Role authorization: A subject's active role must be authorized for the subject.
  With rule 1 above, this rule ensures that users can take on only roles for which they are authorized.
• Permission authorization: A subject can exercise a permission only if the permission is authorized for the subject's active role.
  With rules 1 and 2, this rule ensures that users can exercise only permissions for which they are authorized.

The traditional DAC

Traditional access control, as we call it DAC (discretional access control) has been used for ages and taken for granted. The familiar string r-x------ is fundamental for all sys admin. DAC provides SUID, GUID, etc but the control scope deals with All, GROUP or OWNER access.

AIX RBAC

RBAC provides precise access control such that the target role can only be assumed by a particular user. The range of commands the role can access could be a subset of all the commands that a root or any other account actually have. 

Difference from SUDO

SUDO is another means to control the access to privilege commands. However, it can be tedious to configure each and every commands that you want to allow an account to access.

Difference from Solaris RBAC

In essence, both Solaris RBAC and AIX RBAC are similar. The main difference is the way to implement it.
In Solaris, we use mainly the following files to setup RBAC.

root:/ #ls -l /etc/user_attr /etc/security/exec_attr /etc/security/prof_attr /etc/security/auth_attr
-rw-r--r--   1 root     sys        11855 Mar 28  2012 /etc/security/auth_attr
-rw-r--r--   1 root     sys        20934 Aug 15 11:40 /etc/security/exec_attr
-rw-r--r--   1 root     sys         8433 Aug 15 11:42 /etc/security/prof_attr
-rw-r--r--   1 root     sys         1292 Aug 30 12:01 /etc/user_attr

Authorisation file for Solaris.

root:/ #tail -3 /etc/security/auth_attr
solaris.system.:::Machine Administration::help=SysHeader.html
solaris.system.date:::Set Date & Time::help=SysDate.html
solaris.system.shutdown:::Shutdown the System::help=SysShutdown.html

In AIX, this authorisation list is kept in a DB. You can create custom ones, especially for those not already in the DB. AIX don't provide the help HTML file. In reality, do we use them?

server:/: lsauth ALL | tail -3
wpar.mobility.appli id=10014
wpar.mobility.appli.other id=10016
wpar.mobility.appli.owner id=10015

The Solaris file that manage the effective privilege level to execute the command

root:/ #tail -3 /etc/security/exec_attr
Zone Management:solaris:cmd:::/usr/sbin/zoneadm:uid=0
Zone Management:solaris:cmd:::/usr/sbin/zonecfg:uid=0
DisasterRecovery Admin:suser:cmd:::/opt/sysadmin/Portnet_DR_Scripts/*:uid=root

Next, not much meaning in this profile file but only to maintain the profile and description of the role

root:/: #tail -5 /etc/security/prof_attr
ZFS Storage Management:::Create and Manage ZFS Storage Pools:help=RtZFSStorageMngmnt.html
Zone Management:::Zones Virtual Application Environment Administration:help=RtZoneMngmnt.html
dtwm:::Do not assign to users. Actions and commands required for the window manager (dtwm).:help=Rtdtwm.html
shutdown:::Do not assign to users. Contains actions requiring shutdown authorization.:auths=solaris.system.shutdown;help=Rtshutdown.html
DisasterRecovery Admin:::For running DisasterRecovery scripts:

In AIX, here it is though we can set much more information, like password control, access to smitty and all that.

server:/: lsrole -f appadmin
appadmin:
        authorizations=aix.system.cluster
        rolelist=
        groups=admingrp
        visibility=1
        screens=*
        dfltmsg=role to manage Application resources
        msgcat=
        auth_mode=NONE
        id=11

In Solaris, the file that assign who can assume the role.

root:/ #tail -3 /etc/user_attr
me::::type=normal;profiles=DNS Admin
you::::type=normal;profiles=DNS Admin
her::::type=normal;profiles=DNS Admin

AIX keep this information in the ODM too.

server:/: lsuser -f meuser | grep role
        default_roles=
        roles=appadmin

How to setup

Say for instance, powerHA can only be accessed by root. but to allow menu control of cluster resources, we need to have a means to start/stop/restart/suspend/resume/failover the resources without using root. It is a bad security idea to allow menu to manage the cluster resources via root account.

Hence, we authorise, say meuser to access powerHA administrative commands by giving it ibm.hacmp.admin authorisation. How do we do that?

Check that Enhanced RBAC is enabled.

# lsattr -El sys0 -a enhanced_RBAC

enhanced_RBAC true Enhanced RBAC Mode True 

Let's create the authorisations.

/:> mkauth dfltmsg='IBM custom' ibm
/:> mkauth dfltmsg='IBM custom hacmp' ibm.hacmp
/:> mkauth dfltmsg='IBM custom hacmp admin' ibm.hacmp.admin 

Then check out what privileges that the commands that you are using requires.

# tracepriv -ef /usr/es/sbin/cluster/utilities/clRGinfo

-----------------------------------------------------------------------------
Group Name     State                        Node
-----------------------------------------------------------------------------
apps_rg     ONLINE                       servera
               OFFLINE                      serverb

9568366: Used privileges for /usr/es/sbin/cluster/utilities/clRGinfo:
  PV_AU_ADMIN                        PV_NET_CNTL
  PV_NET_PORT
 

# tracepriv -ef /usr/es/sbin/cluster/events/utils/cl_RMupdate 

...

...

...

if you need to use your own shell script, you may need to add it into the privileged command database. Allow EUID to be equal to the owner of that script.

Now we add the commands into the privileged command database.

/:> setsecattr -c innateprivs=PV_AU_ADMIN,PV_NET_PORT,PV_NET_CNTL accessauths=ibm.hacmp.admin /usr/es/sbin/cluster/utilities/clRGinfo

/:> setsecattr -c innateprivs=PV_AU_ADMIN,PV_KER_ACCT,PV_PROC_PRIV accessauths=ibm.hacmp.admin euid=0 /usr/es/sbin/cluster/events/utils/cl_RMupdate 

/:> setsecattr -c innateprivs=PV_AU_ADMIN,PV_KER_ACCT,PV_PROC_PRIV accessauths=ibm.hacmp.admin euid=0 /admin.sh
/:> setsecattr -c innateprivs=PV_AU_ADMIN accessauths=ibm.hacmp.admin euid=0 /dlpar.sh

You can verify by using lssecattr.

/:> lssecattr -F -c /dlpar.sh
/dlpar.sh:
        euid=0
        accessauths=ibm.hacmp.admin
        innateprivs=PV_AU_ADMIN

Now, we create a role with the above authorisations.

# mkrole authorizations=ibm.hacmp.admin dfltmsg="Custom role to do admin with hacmp" appadmin

if its for automation, you may want to remove password access to the role by the following command. chrole auth_mode=NONE appadmin By default, auth_mode is INVOKER which means that is password control.

Next, allow meuser to be able to assume the role

 chuser roles=appadmin meuser

Before you try it out, you need to update the kernel for all these to take effect. As AIX kernel is RBAC aware for all the IBM system commands, without updating the kernel, any changes will not take effect.

setket

Try it out

swrole 

If you are not allow to assume the role you will receive the following error. In this example, thatuser should not assume appadmin role.

server:/HAapps: su - thatuser
-bash-3.2$ swrole appadmin
swrole: 1420-052 appadmin is not a valid role for thatuser.

It is authorised to assume meuser role instead.

server:/HAapps: su - meuser
-bash-3.2$ swrole appadmin
bash-3.2$ /usr/es/sbin/cluster/events/utils/cl_RMupdate suspend_appmon apps apps_rg
Suspend HA Monitoring for apps.
2012-10-22T15:58:03.289727
2012-10-22T15:58:03.309369
Oct 22 2012 15:58:03 cl_RMupdate: Completed request to suspend monitor(s) for application apps.
Oct 22 2012 15:58:03 cl_RMupdate: The following monitor(s) are in use for application apps:
apps_svr
apps_dm

Reference: http://aixhelp.blogspot.sg/2010/12/aix6-rbac.html

Good reference link of powerHA

This is a bookmark of powerHA links i find it useful. Keep them here just in case. :)

http://aix4admins.blogspot.sg/2011/10/commands.html

Allowing longer web session going through Apache to Websphere Application Server

Had a tough one last month when migrating the system to WAS. I'm still new to WAS, hit a few problems and take this chance to document down so that this form my reference and hopefully it help you too.

Users has been complaining that the web service keep getting time out, returning a 500 error.

What i found out was that i can actually, tune the "ServerIOTimeout" parameter in the WAS plugin for Apache beyond the default. i used 900, which is 15min in seconds.

In addition,a little performance fine tuning was done using "LoadBalanceWeight" to keep the application servers from being 'hit' random when they are just started up, especially when i have a cluster of them. As recommended by IBM specialist, i used some numbers, with one of the application server assigned to an odd number, different from the rest.

The idea is to make one of the application server the first one to serve, instead of randomising it.

      Server CloneID="179d3la" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="20" MaxConnections="-1" Name="Node1" ServerIOTimeout="900" WaitForContinue="false"

...
...
     Server CloneID="179d5sb" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="20" MaxConnections="-1" Name="Node2" ServerIOTimeout="900" WaitForContinue="false"

...
...
      Server CloneID="179d8gc" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="21" MaxConnections="-1" Name="Node3" ServerIOTimeout="900" WaitForContinue="false"

...
...



Do let me know if you have better ideas of solving it.

Recovering Websphere Application Server (WAS) after hitting JMS error

Recently, the following errors were logged in the SystemOut.log of WAS version 7. The whole cluster of WAS came to a halt with a flurry of angry calls asking why the web service is down.

- BMXAA1580E - A Java Message System (JMS) error occurred
- CWSIT0088E: There are currently no messaging engines in bus intjmsbus running.

Apparently, the messaging engine went down. All the application servers can communicate via JMS and sat there idling.

Used the following way to recover the system. even though i still don't the reason even after logging a PMR yet, here's how.

1) bring down the Application Server
2) bring down the node agent if you have.
3) to be safe, i bring down the Deployment manager too.
4) move the messagestore Log file so that WAS can recreate it upon start up.
5) remove the transaction/tranlog/log1 and log2 file so that WAS can recreate it upon start up.
6) start up WAS and node agent.

My one liner to extracting lines using sed, perl or awk

While working on extracting data from large amount of files, i have compiled some commands over the years to really helps a lot.

Most of the time, we use head, tail, grep. However, these commands are good at wholesale extracting or just by some keywords. For more complex extraction, we may use sed, perl or awk instead.

Using myfile as example,

myserver:/tmp/:>head -10 myfile
# IBM_PROLOG_BEGIN_TAG
# This is an automatically generated prolog.
#
# bos61D src/bos/usr/sbin/netstart/hosts 1.2
#
# Licensed Materials - Property of IBM
#
# COPYRIGHT International Business Machines Corp. 1985,1989
# All Rights Reserved
#

myserver:/tmp:>tail -2 myfile
10.1.1.123     host1
10.2.1.124     host2


myserver:/tmp/:>grep host1 myfile
10.1.1.123     host1

Say, for more complicated stuffs, like extracting 2nd line PLUS 5th to 7th line, i find it tough to code using the above commands.

h2. sed, perl or awk?

Do note that sed will transverse the entire file, hence if you have a very large file, this might take some time.

Say, we want to extract the 2nd line, we can use sed or awk

myserver:/tmp/:>sed 2p myfile
# IBM_PROLOG_BEGIN_TAG
# This is an automatically generated prolog.
# This is an automatically generated prolog.
#
# bos61D src/bos/usr/sbin/netstart/hosts 1.2
...
...

myserver:/tmp/:>awk 'NR==2' myfile
# This is an automatically generated prolog.





If you have try it out, you will see that for sed, the 2nd line is indeed extracted but the rest of the file is also printed out! Use the following to disable printing out the old file.

myserver:/tmp/:>sed -n 2p myfile
# This is an automatically generated prolog.

Alternatively, you might want to 'delete' whatever that you don't want by using the '!d' parameter.

myserver:/tmp/:>sed '2!d' myfile
# This is an automatically generated prolog.

I wouldn't want to use this method as i have difficulty converting the line to use variables. Do give me suggestions or advice if you think otherwise. I don't claim to be expert in writing scripts. :)

IMPORTANT: Note that the single quotes are required. Else '!d' will bring back the last command you have executed with the letter 'd'.



If you only want one and only line from the file, you can get awk to exit after getting that line, otherwise the awk will transverse through the whole file.

myserver:/tmp/:>awk 'NR==6 {print; exit}' myfile
# Licensed Materials - Property of IBM

If we try to extract line 5 to 7 using sed or awk

myserver:/tmp/:>sed -n 5,7p myfile
#
# Licensed Materials - Property of IBM
#

myserver:/tmp/:>awk 'NR==5,NR==7' myfile
#
# Licensed Materials - Property of IBM
#

Here's another trick that i read from Mr Google. If you want to extract every 5th line of a file starting from the top of the file, perl or awk does the job easily.


myserver:/tmp/:>perl -ne 'print unless (0 != $. % 5)' myfile
#
#
# IBM_PROLOG_END_TAG
#
# Licensed Materials - Property of IBM
#  /etc/hosts
#
#
...
...


myserver:/tmp/:>awk '0 == NR % 5'   myfile
#
#
# IBM_PROLOG_END_TAG
#
# Licensed Materials - Property of IBM
#  /etc/hosts
#
#
...
...


Tip: If you don't want to start from the top of the file, you can put (NR + 1), which means to start from line 1.

Thats all folks.

How to files in AIX from a rm -rf / command

What to do if someone accidentally remove some system critical files in rootvg?

# rm -rf ~

In this case, the stuffs in root's home directory will be removed. You will see /admin, /dev, /bin, etc being deleted. If you are quick to notice the mistake and halt the rm command,

IMPORTANT: Keep your existing SSH session alive at all cost. Otherwise, working on a terminal via HDMC or similar is going to be painful.

h2. So, its "oh shit" right?

Hopefully, /lib is not removed yet, else you are in bigger shit.

ssh, rsync, scp all will no longer work. Let's do a little self repair before recovering the rest of the files.

Is your tar command gone? What commands do i have left?

h2. Recover mkdir

Read from http://coding-journal.com/restoring-your-unix-system-after-rm-rf/ about this one. Try the following.

# echo "mkdir 'bin', 0777;" | perl

This is on the assumption that you lost your mkdir command but still have perl. Here, the /bin directory is created. The full permission is just for this emergency purpose, you can probably change it later.

h2. Recovering /dev

If /dev is lost, you may need to create some of the more critical ones to enable scp and ssh to bring in your backups (mksysb). The steps below are used on AIX 7.1 SP4

# cd /dev

# mknod random c 36 0
# mknod urandom c 36 1

# mknod null c 2 2
# chmod 644 random urandom

# chmod 666 null

Go ahead and try a ssh or sync. If cannot, you may need to restart sshd.

# stopsrc -s sshd

# startsrc -s sshd

if you have another server with a similar make, you can also try to recreate the disk structure but this is not critical if you have a backup which you can extract later. The convention should follow a standard since IBM name all the basic disk the same way on AIX 7.1

# mknod hd1 b 10 8
# mknod  hd2 b 10 5
# mknod  hd3 b 10 7
# mknod  hd4 b 10 4
# mknod  hd5 b 10 1
# mknod  hd6 b 10 2
# mknod  hd8 b 10 3
# mknod  hd9var b 10 6
# mknod  hd10opt b 10 9
# mknod  hd11admin b 10 10
# chmod 660 hd1 hd10opt hd11admin hd2 hd3 hd4 hd5 hd6hd7hd8 hd9var

# mknod hd1 c 10 8

# mknod  hd2 c 10 5

# mknod  hd3 c 10 7

# mknod  hd4 c 10 4

# mknod  hd5 c 10 1

# mknod  hd6 c 10 2

# mknod  hd8 c 10 3

# mknod  hd9var c 10 6

# mknod  hd10opt c 10 9

# mknod  hd11admin c 10 10

# chmod 660 hd1 hd10opt hd11admin hd2 hd3 hd4 hd5 hd6hd7hd8 hd9var

h2. Lets bring back the files. 

After you bring in your backup, you can the commence restoration. I will list example using mksysb file.

Say we need to recover /dev, /admin, bosinst.data, etc. We just double check if the file is usable and whether the original files are inside this archive.

# restore -alvTf mksysb_mysever_date > /tmp/mksysb.server.txt

 Then proceed to restore.

# restore -xvqf mksysb_mysever_date ./bosinst.data

# mv bosinst.data /

# restore -xvqf mksysb_mysever_date ./dev

# cd ./dev

# mv * /dev/

# restore -xvqf mksysb_mysever_date ./admin

# cd ./admin

# mv * /admin/

# restore -xvqf mksysb_mysever_date ./.ssh

# mv .ssh /

# restore -xvqf mksysb_mysever_date ./.profile

# mv .profile /

so on and forth.

If you have another server with the similar make or build, you may want to go the extra step to verify if there are anything else that is still missing. 

In addition, go for a reboot at the nearest opportunity to ensure all is working well. Nothing is confirmed until it is tested and proven working.

Rotating AIX audit log

Found that audit log grow too much on my new servers.

myserver:/:>audit query | head -2
auditing on
bin processing off

The audit will record audit events like, 'su', 'passwd', file changes, cron, mail, tcpip, lvm, etc. Since audit files are kept on a separate partition for my case, risk of widespread diskspace full is still not that great.

myserver:/:>df -k | grep audit
/dev/fslv00        262144    227972   84%        8     1% /audit

myserver:/:>ls -l /audit/
total 67608
-rw-------    1 root     system            0 Sep 14 16:43 auditb
-rw-rw----    1 root     system        10453248 Sep 14 16:43 bin1
-rw-rw----    1 root     system        11456 May 14 10:25 bin2
drwxr-xr-x    2 root     system          256 Jul 10 14:43 lost+found
-rw-r-----    1 root     system     34589752 May 14 10:24 trail

Although the binsize in /etc/security/audit/config is set to 10240, which is 10240 bytes but the bin1 and bin2 files did not stay within the 10kb limit.

Also, there is a cron that 'rotate' the trail log file but it does not compress the rotated file, hence disk space is still being hogged.

myserver:/:>crontab -l | grep audit
0 * * * * /etc/security/aixpert/bin/cronaudit

So, let me suggest a workaround.

For the cron script, we add in a line to gzip the rotated log file after shifting the old file.

mv /audit/trail /audit/trailOneLevelBack
gzip /audit/trailOneLevelBack


For the bin1 and bin2 files, stop audit, rotate the files and start audit.

# audit shutdown
# cp -p /audit/bin1 /audit/bin1.
# cp -p /audit/bin2 /audit/bin2.

# gzip /audit/bin1.
# gzip /audit/bin2.

# cp /dev/null /audit/bin1
# cp /dev/null /audit/bin2

# audit start

Be careful not to change the inode of the files. Otherwise, i read from Mr Google that audit might get 'confused' and does not write audit logs into the bin files anymore. you might then need to reboot the host for audit to recover.

Essential boot information in AIX

Here's some practical tips on boot information in AIX.


h4. uptime and when was it last rebooted.

In RHEL and Solaris, we can only find the uptime and when was it last rebooted.

# uptime
  10:06AM   up  19:09,  1 user,  load average: 0.35, 0.64, 0.65

# who -b
   .        system boot Aug 16 14:58


In AIX, we have this additional command to find history of reboot records. Power of ODM.

# last reboot
reboot    ~                                   Aug 16 14:58
reboot    ~                                   Aug 16 14:42
reboot    ~                                   Aug 15 13:59
reboot    ~                                   Aug 15 10:44
reboot    ~                                   Aug 14 15:14
reboot    ~                                   Jul 10 16:25
reboot    ~                                   May 25 12:14
reboot    ~                                   May 10 16:22
reboot    ~                                   May 07 17:02
reboot    ~                                   May 02 16:24
reboot    ~                                   May 02 15:58
reboot    ~                                   Apr 30 16:41
reboot    ~                                   Apr 25 15:19
reboot    ~                                   Apr 24 16:15
reboot    ~                                   Apr 24 15:35

wtmp begins     Apr 24 15:35


h4. State of the boot record

Here, we can spool and find out which disk has boot records for you to boot from.

# ipl_varyon -i
[S 8257680 9306164 08/17/12-10:07:37:395 ipl_varyon.c 1270] ipl_varyon -i


PVNAME          BOOT DEVICE     PVID                    VOLUME GROUP ID
hdisk0          YES             00f72ff5025fdaf30000000000000000        00f72ff500004c00
hdisk1          YES             00f72ff5025fdb3c0000000000000000        00f72ff500004c00
hdisk2          NO              00f72ff32bcd79f10000000000000000        00f72ff500004c00
hdisk3          NO              00f72ff32b5332f20000000000000000        00f72ff300004c00
hdisk4          NO              00f72ff32b5334930000000000000000        00f72ff300004c00
hdisk5          NO              00f72ff32b5336370000000000000000        00f72ff300004c00
[E 8257680 0:274 ipl_varyon.c 1410] ipl_varyon: exited with rc=0


h4. Creation of boot record

In the firmware (SMS), we can set the boot devices, e.g. disk, cd-rom, network. For disk, we would also need to create the boot record so that the server know HOW to load up AIX.

we create the boot record like this.

# bosboot -ad /dev/hdisk1

If you want to remove the boot record, you can try the following.

# chpv -c hdisk1

h4. Creation of boot list

Here, we create the boot list so that the server know WHERE to load up AIX.

Below, we see that there are 2 devices that we can boot up from. And it corresponds to the above ipl_varyon command. You can compare with what is set in SMS, they should match.

# bootlist -m normal -ov
'ibm,max-boot-devices' = 0x5
NVRAM variable: (boot-device=/pci@800000020000101/pci1014,0339@0/sas/disk@40600:2 /pci@800000020000101/pci1014,0339@0/sas/disk@40700:2)
Path name: (/pci@800000020000101/pci1014,0339@0/sas/disk@40600:2)
match_specific_info: ut=disk/sas/scsd
hdisk0 blv=hd5 pathid=0
Path name: (/pci@800000020000101/pci1014,0339@0/sas/disk@40700:2)
match_specific_info: ut=disk/sas/scsd
hdisk1 blv=hd5 pathid=0

If you need just the list of what we can boot up from, just drop the 'v' to reduce verbosity.

# bootlist -m normal -o
hdisk0 blv=hd5 pathid=0
hdisk1 blv=hd5 pathid=0

How to create or add the boot list, you may ask. Here's the command and example to create for the above.

# bootlist -m normal hdisk0 hdisk1

If you need to do a network boot all the time, you can set like the following.

bootlist -m normal en0 bserver=10.106.101.1 gateway=10.106.101.250 client=10.106.101.5

where
'bs' means boot server
'client' means the machine what we reboot


h4. Where is the boot image?

The boot image is usually found in hd5 (Boot PV). Below example shows that the server is booted from hd5 in hdisk0.

# bootinfo -v
hd5

# bootinfo -b
hdisk0

Attention: Never reboot the system when you suspect the boot image is corrupted.

h4. Recreation of boot image (Boot LV).

In the event you suspect the boot image is corrupted, you can recreate it using the following.

# bosboot -a -d /dev/hdisk0

** If the command fails and you receive the following message:

    0301-165 bosboot: WARNING! bosboot failed - do not attempt to boot device.

    Try to resolve the problem using one of the following options, and then run the bosboot command again until you have successfully created a boot image:

        Delete the default boot logical volume (hd5) and then create a new hd5.

    Or
        Run diagnostics on the hard disk. Repair or replace, as necessary.{info}

* If the bosboot command continues to fail, contact your customer support representative.
* Attention: If the bosboot command fails while creating a boot image, do not reboot your machine.
* When the bosboot command is successful, reboot your system to confirm.

Once done, update 'mini-ODM' in boot LV.

# savebase -v

Quote from Reference:
The bootrec (also known as bootstrap) is read by a special part of the firmware called System ROS (- the Read Only Storage is responsibe for the initial preparation of the machine -), and it (bootrec)  tells the ROS that it needs to jump X bytes into the disk platter, to read the boot logical volume, hd5.

During reading the blv, there is a mini-ODM read into the RAM. (Later, when the real rootvg fs comes online, AIX merges the data in mini-ODM with the real ODM held in /etc/objrepos.)

When an LVM commands changes the mini-ODM, the command 'savebase' needed to run as well. Savebase takes a snapshot of the ODM and compresses it

h4. How all these gel together. (My understanding and value adding just in case, people complain i copy too much) :P

After powering on, the server will POST.

Then it will use the boot list to find which disk, cdrom, network to load the Boot LV (inside hd5) from. Boot LV contains the AIX kernel, rc.boot file and commands required during the boot process and the mini-ODM.

next, kernel will take over the boot process. 

The kernel will then loads up the file system before executing the init process (from Boot LV) which will execute the rc.boot. The rootvg will be activated and then init process from the disk will be executed to replace the init process from Boot LV and become PID 1.

The kernel will move through the rc states and get the system ready.


Thanks to the reference that i understand more. <http://aix4admins.blogspot.sg/2011/08/mkitab-adds-record-to-etcinittab-file.html>