Monday, August 20, 2012

Essential boot information in AIX

Here's some practical tips on boot information in AIX.


h4. uptime and when was it last rebooted.

In RHEL and Solaris, we can only find the uptime and when was it last rebooted.

# uptime
  10:06AM   up  19:09,  1 user,  load average: 0.35, 0.64, 0.65

# who -b
   .        system boot Aug 16 14:58


In AIX, we have this additional command to find history of reboot records. Power of ODM.

# last reboot
reboot    ~                                   Aug 16 14:58
reboot    ~                                   Aug 16 14:42
reboot    ~                                   Aug 15 13:59
reboot    ~                                   Aug 15 10:44
reboot    ~                                   Aug 14 15:14
reboot    ~                                   Jul 10 16:25
reboot    ~                                   May 25 12:14
reboot    ~                                   May 10 16:22
reboot    ~                                   May 07 17:02
reboot    ~                                   May 02 16:24
reboot    ~                                   May 02 15:58
reboot    ~                                   Apr 30 16:41
reboot    ~                                   Apr 25 15:19
reboot    ~                                   Apr 24 16:15
reboot    ~                                   Apr 24 15:35

wtmp begins     Apr 24 15:35


h4. State of the boot record

Here, we can spool and find out which disk has boot records for you to boot from.

# ipl_varyon -i
[S 8257680 9306164 08/17/12-10:07:37:395 ipl_varyon.c 1270] ipl_varyon -i


PVNAME          BOOT DEVICE     PVID                    VOLUME GROUP ID
hdisk0          YES             00f72ff5025fdaf30000000000000000        00f72ff500004c00
hdisk1          YES             00f72ff5025fdb3c0000000000000000        00f72ff500004c00
hdisk2          NO              00f72ff32bcd79f10000000000000000        00f72ff500004c00
hdisk3          NO              00f72ff32b5332f20000000000000000        00f72ff300004c00
hdisk4          NO              00f72ff32b5334930000000000000000        00f72ff300004c00
hdisk5          NO              00f72ff32b5336370000000000000000        00f72ff300004c00
[E 8257680 0:274 ipl_varyon.c 1410] ipl_varyon: exited with rc=0


h4. Creation of boot record

In the firmware (SMS), we can set the boot devices, e.g. disk, cd-rom, network. For disk, we would also need to create the boot record so that the server know HOW to load up AIX.

we create the boot record like this.

# bosboot -ad /dev/hdisk1

If you want to remove the boot record, you can try the following.

# chpv -c hdisk1

h4. Creation of boot list

Here, we create the boot list so that the server know WHERE to load up AIX.

Below, we see that there are 2 devices that we can boot up from. And it corresponds to the above ipl_varyon command. You can compare with what is set in SMS, they should match.

# bootlist -m normal -ov
'ibm,max-boot-devices' = 0x5
NVRAM variable: (boot-device=/pci@800000020000101/pci1014,0339@0/sas/disk@40600:2 /pci@800000020000101/pci1014,0339@0/sas/disk@40700:2)
Path name: (/pci@800000020000101/pci1014,0339@0/sas/disk@40600:2)
match_specific_info: ut=disk/sas/scsd
hdisk0 blv=hd5 pathid=0
Path name: (/pci@800000020000101/pci1014,0339@0/sas/disk@40700:2)
match_specific_info: ut=disk/sas/scsd
hdisk1 blv=hd5 pathid=0

If you need just the list of what we can boot up from, just drop the 'v' to reduce verbosity.

# bootlist -m normal -o
hdisk0 blv=hd5 pathid=0
hdisk1 blv=hd5 pathid=0

How to create or add the boot list, you may ask. Here's the command and example to create for the above.

# bootlist -m normal hdisk0 hdisk1

If you need to do a network boot all the time, you can set like the following.

bootlist -m normal en0 bserver=10.106.101.1 gateway=10.106.101.250 client=10.106.101.5

where
'bs' means boot server
'client' means the machine what we reboot


h4. Where is the boot image?

The boot image is usually found in hd5 (Boot PV). Below example shows that the server is booted from hd5 in hdisk0.

# bootinfo -v
hd5

# bootinfo -b
hdisk0

Attention: Never reboot the system when you suspect the boot image is corrupted.

h4. Recreation of boot image (Boot LV).

In the event you suspect the boot image is corrupted, you can recreate it using the following.

# bosboot -a -d /dev/hdisk0

** If the command fails and you receive the following message:

    0301-165 bosboot: WARNING! bosboot failed - do not attempt to boot device.

    Try to resolve the problem using one of the following options, and then run the bosboot command again until you have successfully created a boot image:

        Delete the default boot logical volume (hd5) and then create a new hd5.

    Or
        Run diagnostics on the hard disk. Repair or replace, as necessary.{info}

* If the bosboot command continues to fail, contact your customer support representative.
* Attention: If the bosboot command fails while creating a boot image, do not reboot your machine.
* When the bosboot command is successful, reboot your system to confirm.

Once done, update 'mini-ODM' in boot LV.

# savebase -v

Quote from Reference:
The bootrec (also known as bootstrap) is read by a special part of the firmware called System ROS (- the Read Only Storage is responsibe for the initial preparation of the machine -), and it (bootrec)  tells the ROS that it needs to jump X bytes into the disk platter, to read the boot logical volume, hd5.

During reading the blv, there is a mini-ODM read into the RAM. (Later, when the real rootvg fs comes online, AIX merges the data in mini-ODM with the real ODM held in /etc/objrepos.)

When an LVM commands changes the mini-ODM, the command 'savebase' needed to run as well. Savebase takes a snapshot of the ODM and compresses it

h4. How all these gel together. (My understanding and value adding just in case, people complain i copy too much) :P

After powering on, the server will POST.

Then it will use the boot list to find which disk, cdrom, network to load the Boot LV (inside hd5) from. Boot LV contains the AIX kernel, rc.boot file and commands required during the boot process and the mini-ODM.

next, kernel will take over the boot process. 

The kernel will then loads up the file system before executing the init process (from Boot LV) which will execute the rc.boot. The rootvg will be activated and then init process from the disk will be executed to replace the init process from Boot LV and become PID 1.

The kernel will move through the rc states and get the system ready.


Thanks to the reference that i understand more. <http://aix4admins.blogspot.sg/2011/08/mkitab-adds-record-to-etcinittab-file.html>

Friday, August 17, 2012

How to create empty file with a fixed size

In Linux, we use mktemp or mkfile.
In Solaris, we use touch.

In AIX, we try the following

# /usr/sbin/lmktemp filename filesize

e.g.


# /:> lmktemp Log 104857600
Log

# /:>chown myusr:mygrp Log

# /:>ls -ltr
total 204808
-rw-r--r--    1 myusr   mygrp    104857600 Aug 15 17:12 Log

# /:>file Log
Log: commands text


Thats all folks!

Wednesday, August 15, 2012

How to release file lock for Microsoft Office document which is Locked for editing by another user

Just to share, if you ever open up an Excel or Winword file (esp shared ones) and being prompted as titled although no one seems to be opening the file.

Try the following,

Approach the guys who is the ‘another user’, and if he/she is on Windows 7,

  • Open up windows explorer
  • Go to Menu -> Tools -> Folder Options
  • Select the View tab.
  • Uncheck the “Show preview handlers in preview pane” and click on “Apply”
  • Check the “Show preview handlers in preview pane” and click on “Apply”
Suspect the preview handler might have some kind of a deadlock, thus unable to release the lock on the file.
Hope it helps.

Friday, August 10, 2012

How to extract PowerHA configuration from ODM for quick recovery

In AIX, ODM holds a lot of information and configuration.

In the event that ODM goes kaput, all hell will break loose. Especially for powerHA, having a backup of the configuration will aid in the recovery of powerHA configuration issues.

As i'm still learning to use AIX and powerHA, do let me know if my method is good enought. :)

I have written the script to extract powerHA configurations.

#/bin/ksh
#
# Script Name : spool_HA_config.sh
# Written     : 08 Aug 2012
# Author      : Victor Kwan At gmail
#
# Description : This is to spool the powerHA 7 configuration on a
#               AIX 7.1 machine.
#               This script should be cron to run regularly for
#               quick recovery if powerHA configuration gets corrupted
#               in AIX ODM.
#
# Updates     : 08 Aug 2012 : First version
#             : 10 Aug 2012 : spooled files now uses DDMMYYYY_HHmmSS format
#


# Declarations
#
DATE=`date +'%d%m%Y_%H%M%S'`

# Safety Measure
#
WHO=`/usr/bin/whoami`

if [ ${WHO} != root ]
then
        echo "You shouldn't be running this using ${WHO}! Script will now terminate."
fi

#
# Spool the HA configuration from ODM
/usr/es/sbin/cluster/utilities/clsnapshot -c -i -n HA_snap_`hostname`_${DATE} -d "HA snapshot on ${DATE}" >/dev/null 2>&1

# Ends
# ~


The main star in this script is the clsnapshot command. By default, the output of clsnapshot command will be saved at /usr/es/sbin/cluster/snapshots.

Below is the sample of the files spooled. There are 2 files, one *.odm and one *.info. I think both are required to be imported to powerHA if we need to recover from configuration issues.

-rw-r--r--    1 root     system        57482 Aug 10 01:00 HA_snap_serverA_10082012_010000.odm
-rw-r--r--    1 root     system        86579 Aug 10 01:00 HA_snap_serverA_10082012_010000.info



Of course, there are many things we need to keep watch on, we wouldn't want to run this script manually. Hence, put it in root cron to be run daily.

# PowerHA configuration daily spool
0 1 * * * /myscript_folder/spool_HA_config.sh >/myscript_folder/spool_HA_config.output 2>&1


 and we are done.

Wednesday, August 01, 2012

Enabling Host based SSL vhost using SNI in Apache 2.2

Multiple virtual host on the same IP


It is not unusual to host multiple website using the same IP address over HTTP (port 80). In fact, it is very easy to build a family of websites based on *.your_web_site.com, for instance

- product1.mycompany.com
- product2.mycompanycom
- product3.mycompany.com
- product4.mycompany.com

and so on and so forth.

The problem.

 

It is when the system needs to move to HTTPS (port 443), problem arises. Upon migrating all the configuration to ssl.conf in Apache, you may hit the following

[Mon Jul 30 18:18:43 2012] [warn] Init: SSL server IP/port conflict: product2.mycompany.com:443 (/usr/local/apache2.2.22/conf/second_vhost.conf:1) vs. product1.mycompany.com:443 (/usr/local/apache2.2.22/conf/first_vhost.conf:1)
[Mon Jul 30 18:18:43 2012] [warn] Init: SSL server IP/port conflict: product3.mycompany.com:443 (/usr/local/apache2.2.22/conf/third_vhost.conf:1) vs. product1.mycompany.com:443 (/usr/local/apache2.2.22/conf/first_vhost.conf:1)
[Mon Jul 30 18:18:43 2012] [warn] Init: SSL server IP/port conflict: product4.mycompany.com:443 

and
 
[Mon Jul 30 18:18:43 2012] [warn] Init: You should not use name-based virtual hosts in conjunction with SSL!!

The reason is that the SSL protocol is a separate layer which encapsulates the HTTP protocol.

As SSL seesion is separate transaction that take place before HTTP session begins, it is impossible for the browser or the client to determine the SSL vhost it is supposed to access. Only IP and port is accessible at this stage and Apache will just return the first vhost if found which matches the port and iP address specified.

workaround?


The usual advise is to use a separate IP for each SSL vhost but this is merely a workaround. In addition, with the public IPs (v4) becoming scarce, soon we will run out of IP (v4).

Solution


With Apache 2.2.12 and newer PLUS OpenSSL 0.9.8j, it is possible to build what we have done on HTTP onto HTTPS.

We will need SNI (subject name indication). This allows you to host multiple SSL websites on the same IP address. This is effectively providing host headers for SSL.

Not much information (many questions but not much answer with good explanation) has been found on the internet to get SNI support, hence the below steps to help folks out.

background

The procedures is done on RHEL 4.6 64bit. I believe this should work for the variants of *NIX. We will do the following
  • Upgrade OpenSSL to v0.9.8x
  • Upgrade Apache to 2.2.22 with SNI support.

Upgrade Openssl with SNI support

Unpack openssl-0.9.8x.tar.gz then try the following

# ./config enable-tlsext shared
# make
# make install
 


Openssl 0.9.x series uses 'enable-tlsext' while Openssl 1.x series uses '--enable-tlsext' switch
ensure that you use the 'shared' command, else the config will fail

After its done, check that openssl is installed with the correct version

[root@server bin]# /usr/local/ssl/bin/openssl version
OpenSSL 0.9.8x 10 May 2012

Install Apache with openssl 0.9.8x support + SNI

Unpack httpd-2.2.22.tar.gz and then try the following

# LDFLAGS=-L/usr/local/ssl/lib/ CPPFLAGS=-I/usr/local/ssl/include/ ./configure --with-included-apr --prefix=/usr/local/apache2222_pntest/ --enable-so --with-ssl=/usr/local/ssl/lib/ --enable-ssl=static --enable-mods-shared='all proxy rewrite'
# make
# make install
 

LDFLAGS is to include any library path that is not default within the httpd binary directory.
CPPFLAGS is for including the header files for compilation that is not default within the httpd binary directory.

Configure the Apache

Now that Apache is installed, configure your httpd.conf and ssl.conf.

To prevent non SNI browsers from having issues connecting, add in below in ssl.conf

# Go ahead and accept connections for these vhosts
# from non-SNI clients
SSLStrictSNIVHostCheck off

Setup the environment

Lastly, add in LD_LIBRARY_PATH into /etc/profile so that the libraries can be found.

LD_LIBRARY_PATH="/usr/local/ssl/lib/:LD_LIBRARY_PATH"
export LD_LIBRARY_PATH

Check that now Apache can link to the new openssl library

[root@server bin]# ldd httpd | grep ssl
        libssl.so.0.9.8 => /usr/local/ssl/lib/libssl.so.0.9.8 (0x00002b7019d09000)
        libcrypto.so.0.9.8 => /usr/local/ssl/lib/libcrypto.so.0.9.8 (0x00002b7019f58000)
 
If environment is not setup properly, ldd will report that the library is "missing" Also you may need to re-apply your profile by either logout/login or ". /etc/profile".


Result


With SNI support, you should see the following line only in the error log.
[warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)

Client supportability

SNI support in browsers is as follows
  • FF 2.0 and newer
  • IE7 + vista and newer
  • Chrome + vista
  • Opera 8
  • Safari 8.2.1
My reference: <http://wiki.apache.org/httpd/NameBasedSSLVHostsWithSNI>

~Done.