Monday, September 12, 2011

Simple AIX LVM concept and Disk structure

The LVM controls everything here through logical volume device drive. It takes a complex structure of PP, including mirroring, and presents a simple LP to the application.















Physical Volume (hdiskX)

Each physical disk is called a physical volume (PV).

Every PV in use belongs to a VG unless the PV is used as raw storage device or a hot spare.
Each PV contains 1 or more disks (or platters) stacked on top of each other.

PV is also known as direct access storage devices (DASDs). It can be fixed or removable.
A block (sector) is a continuous 512-byte region on PV that corresponds in size to a DASD sector.
A Partition is a set of blocks (with sequential cylinder, head, and sector numbers) contained within a single PV.

Physical Parition

Each PV is divided into PP of a fixed size (e.g. 4Mb partition size).
This is the smallest unit a PV can have.

Volume Group (rootvg, datavg)

Each VG is a collection of 1 or more PV.

A VG can consist of any mixture of physical disk types, though performance is most consistent if devices of the same size are used in a single VG.

PV cannot be shared between VG, therefore entire PV becomes part of the VG.

Each system can have up to 32 VG only.
Information about all LVs and PVs within a VG are stored in VGDA area.
first 64K of a PV is reserved for this area as defined in .
The VGDA consists of bootrecord (first 512 bytes), bad block directory () and LVM record ()

Informations about which PP are stale and which PV are missing within a VG is stored in VGSA area.
The LVM adn SCSI driver reserves somewhere between 7 - 10% of the available disk space for LVM maps, etc.

Lets go deeper.

With each VG, 1 or more LV are defined.

Logical Volume (hd1, hd2var, etc)

Each LV group together 1 or more Logical Partition (LP).

LV is an area of disk used to store data, appearing contiguous to the application but may not be contiguous on the actual PV.
A file system resides on top of an LV, there only 1 LV is mapped to a file system.
LV reside only within VG.
LV can span on multiple PV within the VG.
LV cab also have mirrored copies on different PV within the VG.
Each VG can contain up to 255 LV.

All system LV should reside in the VG named rootvg, under 1 PV if possible. This will allow complete re-installation of the system from backup without affecting the application data that resides on other VG.

There is no difference in response time between a single or multiple VG. When a LV is accessed, the ODM db (/etc/objrepos/*) is searched to determine which VG the LV belongs to, then the VGDA for the VG is read to determine the physical placement of the LV on that PV and ultimately the physical track and sector where the data resides.

Logical Partition

Each LP contains 1 or more physical partition (PP).

If LV is mirrored, then additional PP are allocated.
These mirrors usually reside on different PV (for availability) but may reside on same PV (for performance).

Up to 2 copies of PP can be mirrored, where you have LP count of 3 (2 mirrors plus original)
Space Region in physical Volume










For space allocation purposes, each PV is divided into 5 regions, namely the outer_edge, out_middle, center, inner_middle, and inner_edge. These can be viewed as cylindrical segments cut perpendicularly through the disk platters. The number of physical partitions in each region depends on the total capacity of the disk drive.

AIX Useful commands

For my own references

Commands for memory

shows how much RAM does my machine has (as root)
bootinfo –r 
shows how much RAM does my machine have (as non root)
lsattr –E –l sys0 –a realmem

Simulates a system with various sizes of memory for performance testing of applications.

To set the memory size to 512 MB
rmss -c 512
To reset the memory size to the original one
rmss -r 

Commands for Disk / file system

Shows all disks
lsdev -Ccdisk
Gives info about the disk manufacture type
lscfg -vp -l hdisk0 | grep Machine
Displays disk size (even if no volume group is assigned)
bootinfo –s hdisk0
Displays information about a physical volume within a volume group (Display PP's used, location on disk, mount point)
lspv -p hdisk0
List of Filesystem items of jfs2 file system type.
lsfs -v jfs2  
List the parameter the filesystem of jfs2 file system type
lsfs -q -v jfs2

From here, we can see if e.g. /backup was or is a big_filesystem_enabled one. Important for the 2GB File limit.

Displays all exported volumes
showmount –e    
Show who's got my file systems mounted over IP !
showmount -a
Shows all mounted filesystems (nfs+local)
mount
Synchronize the device configuration database (ODM) with the logical volume manager information for rootvg
synclvodm -vP rootvg
Redfined VG definition in ODM
redefinevg rootvg
Reads logical volumes info from disk
lqueryvg -p hdisk0 –Avt  
Increases /var FS by about 100MB (each unit is 512 bytes)
chfs -a size=+200000 /var
Mounts all FS stated in /etc/filesystems
mount all
Unmounts a FS
umount /mnt/
Identifies processes using a file or file structure
fuser –c /dataVolumes/test
Releases a CD that will not unmount by sending SIGKILL signal to each local process. Must be root.
fuser -k /dev/cd0
Shows when the file was last created/modified/accessed
istat 
Reports input/output statistics for logical partitions, logical volumes and volume groups every 2 seconds
lvmstat –v hd2 2
Enable / Disable LVM statistic collection
lvmstat -e hd2

lvmstat -d hd2

Commands for Network

Displays en0 driver params
lsattr  -El  en0
Displays ent0 Hardware params
lsattr  -El  ent0
Display specific device specified by logical name ent0
lscfg  -v -l ent0
Display firmware device tree for the corresponding node to the ent0 device
lscfg -vp -lent0
Displays en0 driver params
ifconfig en0
Displays network interfaces setting
netstat  -i
Display all interface information - host:addr:mask:_rawname:nameserv:domain:gateway:cost:activedgd:type:start
for i in `lsdev | grep en | grep "Standard Ethernet" | awk '{print $1}'`; do echo Checking $i;mktcpip -S $i; done
Displays all IP oriented processes status
lssrc –g tcpip
Display any communication errors on ent2
entstat -drt ent2 |grep –i error
Resets all the statistics back to their initial values.
entstat -r
Shows a local arp cache
arp -a
Shows IP statistics
netstat  -ptcp
Shows UDP statistics
netstat  -pudp
Show network buffer cache statistic
netstat  -c
Show network statistic for each protocol
netstat  -s
Shows statistics recorded by the memory management routines
netstat -m
Trace en2 every 3 seconds
netstat -I en2 3
Ping with displaying the routing info
ping -R -c 1 myserver
Display routing info
netstat -rn
Shows the state of all configured interfaces
netstat -in
Display routing info with full hostnames
netstat -r
Trace all hobs (interconnections=routers) to the destination IP
traceroute 149.115.39.1

Monitors activity and reports statistics on network I/O and network-related CPU usage.

To start
netpmon -o netpmon.out
To Stop
trcstop
Displays a fully qualified domain name of a host
namerslv -s | grep domain | awk '{ print $2 }'
Shows the status of a remote host on the local network
rup myserver

Commands for Tape Device

Displays tape params
lsattr -El rmt0
all information about a tape drive
lscfg -vp -l rmt0
Shows all tapes lsdev -Cctape

Commands for NFS

Resets NFS stats without reboot
nfsstat  –z
To stop / start NFS services on a client
stopsrc –g NFS

startsrc –g NFS
Mount an NFS filesystem
mount hostname:/filesystem /mount-point
Exports a directory to NFS clients
mknfsexp -d /directory
Mounts a directory from an NFS server
mknfsmnt
Changes the configuration of the system to stop running NFS daemons
rmnfs 
Configures the system to run NFS
mknfs
Un-exports a filesystem
exportfs -u (filesystem)
Lists all exported filesystems
exportfs
Exports all directories listed in /etc/exports file
exportfs -a

Commands for Devices

Displays system type, firmware, etc driver params
lsattr  -El  sys0
Lists all system HW config (NVRAM)
lscfg –v
List all scsi devices
lsdev –Csscsi
List all pci devices
lsdev –Cspci
List all scsi adapters
lsparent –Ck scsi
List fiberchannel devices
lsdevfc
Configures devices
cfgmgr
This is similar to devfsadm in Solaris Specifies verbose output. The cfgmgr command writes information about what it is doing to standard output.
cfgmgr -v -l device –v
Name Specifies the named device to configure along with its children.
cfgmgr -v -l device
If you only turned on a disk tower at e.g. scsi2 cfgmgr -v -l scsi2 will only configure this with detailed output.

Commands for Graphic adapter

To check which graphic adapter is installed.
lsdisp
List all information about a adapter
lscfg -vp -l mga0

Commands for boot

Change the default bootlist
bootlist -m normal cd0  rmt0 hdisk0

Commands for Firmwares

Display the system firmware level and service processor
lsmcode -c
Display the adapter microcode levels for a RAID adapter scraid0
lsmcode -r -d scraid0
Display the microcode level for all supported devices
lsmcode -A

Commands for System Information

Get machine ID
/usr/bin/uname -m
Get platform type
/usr/bin/uname -M
Determines the system serial number
lsattr -El sys0 -a systemid | awk '{print $2}' | awk -F, '{print $2}'
Displays current AIX level
oslevel
Displays current AIX maintenance level
oslevel -r
List filesets at levels later than maintenance level !!!
oslevel -g
List all information about a processors
lscfg -vp -l proc0  (1,2,3)   
List all information about memory modules installed
lscfg -vp -l mem0 |pg
Determines the system Firmware level
lscfg –vp|grep ROM|grep -v CD  
Check last system dump status
sysdumpdev -L
Check system dump device settings
sysdumpdev -l
List contents of the package
lslpp -f Upd_Timna_DTM.obj
Shows create/modify/access file info
istat 
Display system boot log
alog -o -t boot | more
Identifies the users currently logged in
who
Shows recent 5 lines
last –5
Shows username ‘root’ login/logout record
last root
Displays local system statistics in interactive mode and records system statistics in recording mode
nmon
The nmon command provides the following views in interactive mode:
    * System resource view (using the r key)
    * Process view (using the t and u keys)
    * AIO processes view (using the A key)
    * Processor usage small view (using the c key)
    * Processor usage large view (using the C key)
    * Shared-processor logical partition view (using the p key)
    * NFS panel (using the N key)
    * Network interface view (using the n key)
    * WLM view (using the W key)
    * Disk busy map (using the o key)
    * Disk groups (using the g key)
    * ESS vpath statistics view (using the e key)
    * JFS view (using the j key)
    * Kernel statistics (using the k key)
    * Long term processor averages view (using the l key)
    * Large page analysis (using the L key)
    * Paging space (using the P key)
    * Volume group statistics (using the V key)
    * Disk statistics (using the D key)
    * Disk statistics with graph (using the d key)
    * Memory and paging statistics (using the m key)
    * Adapter I/O statistics (using the a key)
    * Shared Ethernet adapter statistics (using the O key)
    * Verbose checks OK/Warn/Danger view (using the v key)
    * Detailed Page Statistics (using the M key)
    * Fibre channel adapter statistics (using the ^ key)

Reports selected local and remote system statistics

topas
Monitors system 10 top processes with 2 seconds
monitor -top 10 -s 2
Displays disks activity every 2 seconds refresh interval
iostat 2
Displays the adapter and disk throughput report every 2 seconds
iostat –a 2
Monitors virtual memory statistics every 2 seconds
vmstat 2
Show all CPU’s activity on an SMP machine 2 times separated by 2 seconds.
Each row is for 1 logical processor.
'U' row is for system-wide Unused capacity.
Last line is for system-wide statistics.
sar –P ALL 2 2
Monitors real and virtual memory
svmon –i 2
Shows top 10 memory usage by process
ps auxw | sort –r +3 |head –10  
Shows top 10 CPU usage by process
ps auxw | sort –r +2 |head –10
Traces FS,LV,disks,files activity of a “find” command into a logfile (filemon.out). Must be preceded by a trcstop command.
filemon –O all –o filemon.out ; find / -name core ; trcstop
Traces CPU activity of a “find” command Several logfile are created. Must be preceded by a trcstop command.
tprof –x find / -name core ; trcstop
Shows the DNS server name and address
nslookup hostname
Trace CPU activity for next 30 seconds. Results in file sleep.tprof
tprof -ske -x "sleep 30"

Commands for paging.

Paging space settings.
lsps -a

Commands for user accounts

Environment setings - show user ulimit
env ulimit
Shows all user parameters (max .file size,etc)
lsuser –f root
Lists login users and their programs.
w

Commands to browse errlog

Generates a report of logged errors. Default command
errpt
All details
errpt -a
Moderate details
errpt -A
Eliminates double entries
errpt –D
Browse the errlog in detail for all errors within a timeframe
errpt -a  -s 0604090601  -e 0605090901
where date format is mmddhhmmyy Generates a report of resource names specified by the ResourceNameList variable (SYSPROC)
errpt -a  -N SYSPROC |more
Browse the errlog by the identifier
errpt -j 5DFED6F1
>h4>Misc Find all files containing "10.10.10.13" IP address
ksh
find / -type f|xargs grep "10.10.10.13" 2> /dev/null
Compresses the files while keeps the original
compress -c file > file.Z
Returns full path of program (Similar to 'which' command in Solaris / Linux. Also available in AIX)
whereis  

Finding out which HMC your server is attached to

Find out which HMC your server is attached to (in AIX 6.1 / 7.1)

bash-3.2# lsrsrc IBM.MCP
Resource Persistent Attributes for IBM.MCP
resource 1:
        MNName           = "10.10.10.15"
        NodeID           = 4935249694248815870
        KeyToken         = "10.10.10.10"
        IPAddresses      = {"10.0.3.20","10.10.10.15","10.10.10.16","fe80::e62f:13ff:f231:6218","fe80::e62f:13ff:fe33:24b0","fe80::e62f:13ff:fe33:24b2"}
        ActivePeerDomain = ""
        NodeNameList     = {"myserver"}
In this example, we see that myserver is connected to HMC via 10.10.10.15 and 10.10.10.16.

How a small difference in NFS can cause sleepless night

Setting up NFS to share to non-AIX hosts

I wanted to setup NFS between AIX and non-AIX systems using NFS. being more familiar with NFS setup in Solaris and Linux, i thought setup would be a breeze.

what went wrong

AIX is very strict on who can access the partition, i.e. root. Otherwise, even though the NFS share is created but you will not be able to mount the partition from the other hosts.

Also ensure that both NFS server and client can use the same security protocol, otherwise the result is same as above. As the NFSv3 is the more commonly used protocol, it is recommended for the current setup.

Lastly, put in explicitly who can read-write or read-only to control access.

Sunday, September 11, 2011

How NPIV can save on fibre cabling for SAN

What is NPIV

NPIV, which stands for N-Port ID Virtualisation is a fibre channel facility allowing multiple N-Port IDs to share a singale physic N-Port. Hence this allows multiple fibre channel to occupy a single physical port, easing hardware requirements in the SAN design.

It is noted that NPIV is an extension to a standard already defined in the fibre channel protocols that allow one to get past single initiator/single target design limitations.

In order to take advantage of this, both the HBA card (from the host and SAN array) and the switch must support NPIV to generate and publish an additional WWPN in a virtual fashion.

Why is it good

Traditionally, we provide at least 2 fibre link for each host, 1 link on 1 controller which is connected to 1 SAN switch in the production environment. With 4 LPARs in the p7 server requiring SAN connection, potentially, we need at least 8 fibre links with 8 SAN ports allocation. Additional links and SAN ports are needed to connect to the SAN arrays.

With the NPIV protocol, we can use just 2 fibre links to connect between the SAN switch and the p7 server. This is a savings of 75%!

Why it may be bad

In the event that there are lots of host sharing the same fibre link, the nightmare of link failure will be catastrophic. It may be mitigated by having 2 HBA controllers with 2 links each, and distributed connection to 2 different SAN switches.

How Disk Expansion on AIX can fail

IBM has been boasting about the ease of disk space management on its AIX hosts. Sure, we are able to expand the file system on AIX for many occasions. i.e. /opt, /home, /tmp and etc. However, trying on /opt/oracle/oradata failed. Let see why.

The story.

my oravg was full due to rapid expansion of business, hence needed more space. So no impact is expected as disk space expansion should be on the fly. I have personally tried on /tmp, / and /var, so i am pretty confident that this is OK.

Expansion was completed but from "df" command, we cant see the expected changes. Then the phone started ringing. Oracle DB has crashed.

Binary installation was still intact but the DB is gone.

Troubleshooting and recovery

fsck showed that the "bit allocation map" is corrupted. Tried to creat another VG and try to copy the LV over to the new VG, trying to see if the inodes, files, links and bit allocation map can be recreated. The map couldn't be recreated.

Searching in google did not yield any useful information other than creating snapshot to backup the system or apply snapshot to recover the system. So this is a provision for quick recovery rather than what i need.

It seems that oracle DB or IBM DB2 are "aware" of the disk physical boundaries and when the boundaries are changed, they do not know how to react and hence crashed. In addition, there was no space for the OS to process the file system expansion since it was 100% full. The next recovery step is to wipe the oravg clean and rebuild the DB.

The painful thing was that the building of DB took more than 3 days and this rebuild took another weekend. ouch.

The take away

Before doing any disk expansion for DB or on any other disk where the applications running off the disk are "aware" of the disk setup, please shut it down before doing any disk changes. In addition, should clear up some space so that the OS has some leeway to process the file system expansion on the fly.

I'm lucky that this is a trial setup.

So do not believe blindly that disk space management is on the fly without strings attached. There are bound to be some obscure conditions that will break it.

Murphy law - "Anything that can go wrong, will go wrong"

My thoughts on how can 802.1q can save on network cablings and potential problems.

IEEE 802.1Q or commonly known as VLAN tagging is a networking standard for sharing of physical Ethernet network link by multiple independent logical networks.

The protocol works with the MAC layer and Spanning Tree Protocol (802.1D) to allow nodes / hosts on different VLAN to communicate with each other through network switch or router on Network Layer.

VLAN tagging

If i have 2 different environment where servers are members of different network segments, in order to allow the different hosts which are in the p750 server to share the same physical ethernet link, the switch or router need to understand and route the network traffic for both the 10.10.10.* and 10.10.20.* to the p750 machine.

Within the p750 machine, the NIC is capable of deciphering the VLAN ID and route the traffic to the designated hosts or LPARs for inward traffic. For outward traffic, the NIC would tag the VLAN ID to the traffic and route it out to the gateway.

This is also similar to the VTP or ISL protocol that is proprietary to Cisco.

Why is it good.

With 2 VIO servers, the different LPARs in the same IBM p750 machine would traditionally need more than 20 UTP cables. With 802.1Q, we need only 2 cables for all the LPARs and 2 cables for HMC. This is more than 80% savings!

Why it may be bad

In the event that there are lots of host sharing the same UTP link, the nightmare of link failure will be catastrophic. It may be mitigated by having 2 NIC controllers with 2 links each, and distributed connection to 2 different switches.

i guess you cant have your cake and eat it!

How to automatically redirect HTTP to HTTPS in Apache

Redirecting HTTP to HTTPS is one common and popular way to protect user privacy and sensitive information without making user typing 'https' manually to access your site.

First, we verify that Apache is configured for HTTPS connection and necessary SSL certificates are already in placed.

Then, either we use redirect or mod_rewrite.

  • Using mod_rewrite. Add these directives to your configuration file:
  •           RewriteEngine On
              RewriteCond %{SERVER_PORT} !^443$
              RewriteRule ^/(.*) https://%{SERVER_NAME}/$1 [L,R]
    Make sure you have loaded mod_rewrite module into Apache.

  • Using redirect. Add these directives to your configuration file:
  •           SSLRequireSSL
              Redirect permanent /secure https://www.domain.com/secure


The 2nd method which uses redirect uses one less module, so security wise could be better. In addition, you don't need to worry about re-writing on logs and etc. Now we restart Apache and go test it out.

How to check LVM health in AIX

What to check for LVM

We can check the state, the utilisation of the PV and VG, whether is the state open, closed, stale, syncd or whether the utilisation is max out already.

As LVM consist of many components like, PP, PV, LV, VG and so on, the best way is to script the things to check. For details of what is the acronyms, please refer to AIX LVM concept and Disk structure.

My newbie demo script for checking LVM health. Gurus out there can really help me along if you think i can do better with some suggestions. :D

#!/bin/ksh
# FILENAME   : checkLVM.ksh
# AUTHOR     : Victor Kwan
# EMAIL     : victorkk [AT] gmail [DOT] com
# PURPOSE    : To check the health of PV, VG and LV
#            : and alert sys admin if threshold is breached.
# DATE       : Feb 2011
#

#
# Parameter setup
OUTPUTFILE="checkLVM.PV.`hostname`.`date '+%d%b%Y'`.output"
NOTIFICATION_MSG="checkLVM.PV.`hostname`.`date '+%d%b%Y'`.message"
isFOUND=0
isERROR=0

if [ $# -ne 2 ]
then
        printf "Usage: \n\t$0  \n\n"
        exit
fi

PV_THRESHOLD=$1
EMAIL="$2"

#
# Extract PV Information from ODM
lspv | while read PV; do
        printf "\n$PV\n" >> $OUTPUTFILE
        printf "--------------------------\n" >> $OUTPUTFILE
        lspv $PV >> $OUTPUTFILE
done

#
# Check for PV Errors
grep -n "PV STATE" $OUTPUTFILE > $OUTPUTFILE.PV

printf "\n\n\n------------------------------\n" > $NOTIFICATION_MSG
printf " Check for PV errors\n" >> $NOTIFICATION_MSG
printf "------------------------------\n" >> $NOTIFICATION_MSG

cat $OUTPUTFILE.PV | while read PVLINE
do
        isLOGICAL_CHECK=`echo $PVLINE | grep "PV STATE" | awk -F: '{print $3}' | grep -v "active" | wc -l`
        #printf "[DEBUG]isLOGICAL_CHECK is %d.\n" $isLOGICAL_CHECK

        if [ $isLOGICAL_CHECK == 1 ]
        then
                PV_STATUS=`echo $PVLINE | grep "PV STATE" | awk -F: '{print $3}' | grep -v "active"`
                PV_LINE=`echo $PVLINE | grep "PV STATE" | awk -F: '{print $1}'`
                PV=`head -$PV_LINE $OUTPUTFILE | tail -3 | head -1 | awk '{print $3}'`
                VG=`head -$PV_LINE $OUTPUTFILE | tail -3 | head -1 | awk '{print $6}'`
                PV_LINE_TOTALPP=`echo $PV_LINE + 3 | bc`
                PV_SIZE=`head -$PV_LINE_TOTALPP $OUTPUTFILE | tail -1 | awk '{print $4}' | awk -F\( '{print $2}'`
                PV_LINE_USEDPP=`echo $PV_LINE + 5 | bc`
                PV_USED=`head -$PV_LINE_USEDPP $OUTPUTFILE | tail -1 | awk '{print $4}' | awk -F\( '{print $2}'`

                #printf "[DEBUG]The line is $PVLINE\n"
                printf "Volume Group: %s\n" $VG >> $NOTIFICATION_MSG
                printf "Physical Volume: %s\n" $PV >> $NOTIFICATION_MSG
                printf "Status     | Size (Mb) | Used (Mb)\n" >> $NOTIFICATION_MSG
                printf "%-10s | %-9d | %-8d\n\n" $PV_STATUS $PV_SIZE $PV_USED >> $NOTIFICATION_MSG
                isFOUND=1
                isERROR=1
        fi
done

if [ $isERROR == 0 ]
then
        printf "All Physical Volumes are clean.\n" >> $NOTIFICATION_MSG
fi

# Check for PV full
grep -n "PV STATE" $OUTPUTFILE > $OUTPUTFILE.PV
isERROR=0

printf "\n\n\n------------------------------\n" >> $NOTIFICATION_MSG
printf " Check for PV utilisation\n" >> $NOTIFICATION_MSG
printf " PV Threshold: $PV_THRESHOLD \n" >> $NOTIFICATION_MSG
printf "------------------------------\n" >> $NOTIFICATION_MSG

cat $OUTPUTFILE.PV | while read PVLINE
do
        isLOGICAL_CHECK=`echo $PVLINE | grep "PV STATE" | awk -F: '{print $3}' | wc -l`
        #printf "[DEBUG]isLOGICAL_CHECK is %d.\n" $isLOGICAL_CHECK

        if [ $isLOGICAL_CHECK == 1 ]
        then

                PV_STATUS=`echo $PVLINE | grep "PV STATE" | awk -F: '{print $3}'`
                PV_LINE=`echo $PVLINE | grep "PV STATE" | awk -F: '{print $1}'`
                PV=`head -$PV_LINE $OUTPUTFILE | tail -3 | head -1 | awk '{print $3}'`
                VG=`head -$PV_LINE $OUTPUTFILE | tail -3 | head -1 | awk '{print $6}'`
                PV_LINE_TOTALPP=`echo $PV_LINE + 3 | bc`
                PV_SIZE=`head -$PV_LINE_TOTALPP $OUTPUTFILE | tail -1 | awk '{print $4}' | awk -F\( '{print $2}'`
                PV_LINE_USEDPP=`echo $PV_LINE + 5 | bc`
                PV_USED=`head -$PV_LINE_USEDPP $OUTPUTFILE | tail -1 | awk '{print $4}' | awk -F\( '{print $2}'`
                PV_PERCENTAGE=$(echo "scale=8; $PV_USED / $PV_SIZE * 100" | bc)

                if [ $PV_PERCENTAGE -ge $PV_THRESHOLD ]
                then
                        #printf "[DEBUG]The line is $PVLINE\n" >> $NOTIFICATION_MSG
                        printf "Volume Group: %s\n" $VG >> $NOTIFICATION_MSG
                        printf "Physical Volume: %s\n" $PV >> $NOTIFICATION_MSG
                        printf "Status     | Size (Mb) | Used (%%)\n" >> $NOTIFICATION_MSG
                        printf "%-10s | %-9d | %-5.2f\n\n" $PV_STATUS $PV_SIZE $PV_PERCENTAGE >> $NOTIFICATION_MSG
                        isFOUND=1
                        isERROR=1
                fi
        fi
done

if [ $isERROR == 0 ]
then
        printf "All Physical Volume within threshold.\n" >> $NOTIFICATION_MSG
fi

rm $OUTPUTFILE
rm $OUTPUTFILE.PV

# Extract VG Information from ODM
lsvg | while read VG
do
        print "\nListing $VG:\n" >> $OUTPUTFILE
        lsvg $VG >> $OUTPUTFILE
        lsvg -l $VG >> $OUTPUTFILE
        #lsvg -l $VG | egrep -v "^$VG:" | egrep -v "^LV NAME" | while read LV JUNK
        #do
        #       lslv $LV >> $OUTPUTFILE
        #done
done

# Check for VG errors in ODM
grep -n "VG STATE" $OUTPUTFILE > $OUTPUTFILE.VG
isERROR=0

printf "\n\n\n------------------------------\n" >> $NOTIFICATION_MSG
printf " Check for VG errors\n" >> $NOTIFICATION_MSG
printf "------------------------------\n" >> $NOTIFICATION_MSG

cat $OUTPUTFILE.VG | while read VGLINE
do
        isVG_CHECK=`echo $VGLINE | grep "VG STATE" | awk -F: '{print $3}' | wc -l`
        #printf "[DEBUG]isVG_CHECK is %d.\n" $isVG_CHECK

        if [ $isVG_CHECK == 1 ]
        then
                VG_STATUS=`echo $VGLINE | grep "VG STATE" | awk '{print $3}'`
                VG_LINE=`echo $VGLINE | grep "VG STATE" | awk -F: '{print $1}'`
                VG=`head -$VG_LINE $OUTPUTFILE | tail -2 | head -1 | awk '{print $3}'`
                VG_LINE_TOTALPP=`echo $VG_LINE + 1 | bc`
                VG_TOTALPP=`head -$VG_LINE_TOTALPP $OUTPUTFILE | tail -1 | awk '{print $7}' | awk -F\( '{print $2}'`
                VG_LINE_USEDPP=`echo $VG_LINE + 3 | bc`
                VG_USEDPP=`head -$VG_LINE_USEDPP $OUTPUTFILE | tail -1 | awk '{print $6}' | awk -F\( '{print $2}'`
                VG_LINE_TOTALPV=`echo $VG_LINE + 5 | bc`
                VG_TOTALPV=`head -$VG_LINE_TOTALPV $OUTPUTFILE | tail -1 | awk '{print $3}'`
                VG_LINE_STALEPV=`echo $VG_LINE + 6 | bc`
                VG_STALEPV=`head -$VG_LINE_STALEPV $OUTPUTFILE | tail -1 | awk '{print $3}'`
                VG_STALEPP=`head -$VG_LINE_STALEPV $OUTPUTFILE | tail -1 | awk '{print $6}'`
                VG_LINE_ACTIVEPV=`echo $VG_LINE + 7 | bc`
                VG_ACTIVEPV=`head -$VG_LINE_ACTIVEPV $OUTPUTFILE | tail -1 | awk '{print $3}'`

                PV_LINE=`lsvg -p $VG | wc -l`
                PV_NUMOFMEMBERS=`echo $PV_LINE - 2 | bc`
                PV_NAME="`lsvg -p $VG | tail -$PV_NUMOFMEMBERS | awk '{print $1}' | xargs`"
                PV_STALENAME=`lsvg -p $VG | tail -$PV_NUMOFMEMBERS | grep -v active | awk '{print $1}' | xargs`

                if [ -z "$PV_NAME" ]
                then
                        $PV_NAME="NA"
                fi

                LV_LINE=`lsvg -l $VG | wc -l`
                LV_NUMOFMEMBERS=`echo $LV_LINE - 2 | bc`
                LV_NUMOFPROBLEM=`lsvg -l $VG | tail -$LV_NUMOFMEMBERS | grep -v "open/syncd" |  wc -l`
                LV_NUMOFOPEN=`echo $LV_NUMOFMEMBERS - $LV_NUMOFPROBLEM | bc`
                LV_PROBLEMNAME=`lsvg -l $VG | tail -$LV_NUMOFMEMBERS | grep -v "open/syncd" | awk '{print $1}' | xargs`
                LV_NAME="`lsvg -l $VG | tail -$LV_NUMOFMEMBERS | awk '{print $1}' | xargs`"

                if [ -z "$LV_NAME" ]
                then
                        $LV_NAME="NA"
                fi

                #printf "[DEBUG]The line is $VGLINE\n"

                if [ $VG_STALEPP -ge 1 -o $VG_STALEPV -ge 1 -o $LV_NUMOFPROBLEM -ge 1 ]
                then
                        printf "Volume Group: %s\nVolume Group Status: %s\n\n" $VG $VG_STATUS >> $NOTIFICATION_MSG

                        printf "Total PP Size (Mb) | Used PP Size (Mb) | Stale PP\n" >> $NOTIFICATION_MSG
                        printf "%-18d | %-17d | %-5d \n\n" $VG_TOTALPP $VG_USEDPP $VG_STALEPP >> $NOTIFICATION_MSG

                        printf "Total PV | Active PV | All PV members\n" >> $NOTIFICATION_MSG
                        printf "%-8d | %-9d | %-s\n\n" $VG_TOTALPV $VG_ACTIVEPV "$PV_NAME" >> $NOTIFICATION_MSG

                        printf "Total LV | Open LV | All LV members\n" >> $NOTIFICATION_MSG
                        printf "%-8d | %-7d | %-s\n\n" $LV_NUMOFMEMBERS $LV_NUMOFOPEN "$LV_NAME" >> $NOTIFICATION_MSG

                        if [ $VG_STALEPV -ge 1 ]
                        then
                                printf "Status of PV with problems:\n" >> $NOTIFICATION_MSG
                                for i in $PV_STALENAME
                                do
                                        THIS_PV=`lspv $i | grep "PV STATE" | awk '{print $1,$2,$3}'`
                                        printf "$i ($THIS_PV) \n" >> $NOTIFICATION_MSG
                                done
                        fi
                        printf "\n" >> $NOTIFICATION_MSG

                        if [ $LV_NUMOFPROBLEM -ge 1 ]
                        then
                                LV_FLAG=0
                                for i in $LV_PROBLEMNAME
                                do
                                        THIS_LV=`lslv $i | grep "LV STATE" | awk '{print $4,$5,$6}'`
                                        THIS_STATE=`lslv $i | grep "LV STATE" | awk '{print $6}'`
                                        THIS_BOOT=`lslv $i | grep "TYPE" | awk '{print $2}' | grep "boot" | wc -l`
                                        THIS_DUMP=`lslv $i | grep "TYPE" | awk '{print $2}' | grep "sysdump" | wc -l`
                                        if [ "$THIS_STATE" != "closed/syncd" ]
                                        then
                                                if [ $LV_FLAG = 0 ]
                                                then
                                                        printf "Status of LV with problems:\n" >> $NOTIFICATION_MSG
                                                        LV_FLAG=1
                                                fi
                                                printf "$i ($THIS_LV) \n" >> $NOTIFICATION_MSG
                                        elif [ "$THIS_STATE" != "open/syncd" -a $THIS_DUMP = "1" ]
                                        then
                                                if [ $LV_FLAG = 0 ]
                                                then
                                                        printf "Status of LV with problems:\n" >> $NOTIFICATION_MSG
                                                        LV_FLAG=1
                                                fi
                                                printf "$i ($THIS_LV) \n" >> $NOTIFICATION_MSG
                                        elif [ "$THIS_STATE" != "closed/syncd" -a $THIS_BOOT = "1" ]
                                        then
                                                if [ $LV_FLAG = 0 ]
                                                then
                                                        printf "Status of LV with problems:\n" >> $NOTIFICATION_MSG
                                                        LV_FLAG=1
                                                fi
                                                printf "$i ($THIS_LV) \n" >> $NOTIFICATION_MSG
                                        fi
                                done
                        fi
                        printf "\n\n" >> $NOTIFICATION_MSG
                        isFOUND=1
                        isERROR=1
                fi
        fi
done

if [ $isERROR == 0 ]
then
        printf "All Volume Groups are clean.\n" >> $NOTIFICATION_MSG
fi

if [ $isFOUND == 1 ]
then
        cat $NOTIFICATION_MSG | mailx -s "[`hostname`] LVM Errors" $EMAIL
fi

rm $OUTPUTFILE
rm $OUTPUTFILE.VG
rm $NOTIFICATION_MSG

How to check status of subgroup TCPIP in AIX

To get the status of the subsystem group TCPIP

# lssrc -g tcpip
Subsystem         Group            PID          Status
 inetd            tcpip            3997848      active
 hostmibd         tcpip            2752750      active
 aixmibd          tcpip            3539102      active
 xntpd            tcpip            9633868      active
 muxatmd          tcpip                         inoperative
 rwhod            tcpip                         inoperative
 snmpd            tcpip                         inoperative
 snmpmibd         tcpip                         inoperative
 dpid2            tcpip                         inoperative
 dhcpcd           tcpip                         inoperative
 dhcpcd6          tcpip                         inoperative
 ndpd-host        tcpip                         inoperative
 ndpd-router      tcpip                         inoperative
 tftpd            tcpip                         inoperative
 gated            tcpip                         inoperative
 named            tcpip                         inoperative
 routed           tcpip                         inoperative
 iptrace          tcpip                         inoperative
 timed            tcpip                         inoperative
 dhcpsd           tcpip                         inoperative
 dhcpsdv6         tcpip                         inoperative
 dhcprd           tcpip                         inoperative
 mrouted          tcpip                         inoperative
 pxed             tcpip                         inoperative
 binld            tcpip                         inoperative
 dfpd             tcpip                         inoperative
Easy?! :)

How to configure link aggregation in AIX

Link aggregation means you can give one IP address to two network cards and connect to two different switches for redundancy purpose. In this only one network card will be active in one time, and when it got failed the other network card goes active and let us continue our work.

It is better to use through SMIT. (For newbies like me :D )

# smit
then goto Devices > Communication > EtherChannel / IEEE 802.3ad Link Aggregation > Add An EtherChannel / Link Aggregation

Here select the network card that you want to use, ie active.

Eg: select ent0

IMP : then select Mode as 8023ad

then select backup adapter for redundancy.(press F4 to show N/W adapters.)

Eg: ent1

press enter.

now ent0 and ent1 got bonded.

then automatically a virtual adapter will be created named ent2.

then put IP address and all to this virtual adapter.
# smit
Communications Applications and Services > TCP/IP > Minimum Configuration & Startup

here select ent2 ( new bonded virtual adapter )

put IP Address and all,

give start now option.

Now you are successfully completed Link aggregation and check whether it works or not by removing the 2nd cable to the network card and check ping, then put the 2nd cable and remove 1st cable. 2 - 3 drops normally occurs.

How to configure NTP in AIX 7.1

First, verify that you have a server suitable for synchronisation. Do note that the offset must be less than 1000 seconds for xntpd to sync. If the offset is more than 1000 seconds, change the time manually on the client and try again. Ensure that firewall is opened for port 123.

# ntpq -d 10.10.10.254
ntpq> ^C

# ntpdate -d 10.10.10.254
23 Feb 14:41:24 ntpdate[4653300]: 3.4y
transmit(10.10.10.254)
receive(10.10.10.254)
transmit(10.10.10.254)
receive(10.10.10.254)
transmit(10.10.10.254)
receive(10.10.10.254)
transmit(10.10.10.254)
receive(10.10.10.254)
transmit(10.10.10.254)
server 10.10.10.254, port 123
stratum 2, precision -18, leap 00, trust 000
refid [10.10.10.250], delay 0.02632, dispersion 0.00002
transmitted 4, in filter 4
reference time:      d10f27e7.15e9bbb7  Wed, Feb 23 2011 14:29:59.085
originate timestamp: d10f2a94.f62731bd  Wed, Feb 23 2011 14:41:24.961
transmit timestamp:  d10f2a94.f62e7000  Wed, Feb 23 2011 14:41:24.961
filter delay:  0.02638  0.02632  0.02637  0.02637
               0.00000  0.00000  0.00000  0.00000
filter offset: -0.00046 -0.00047 -0.00050 -0.00051
               0.000000 0.000000 0.000000 0.000000
delay 0.02632, dispersion 0.00002
offset -0.000479

23 Feb 14:41:24 ntpdate[4653300]: adjust time server 10.10.10.254 offset -0.000479
Now specify xntp server in /etc/ntp.conf. Add in the line "server ip-address-of-the-server prefer". Leave the driftfile and tracefile at their defaults.
vi /etc/ntp.conf
Now, ensure xntpd is started after every reboot.
vi /etc/rc.tcpip
Uncomment the following line
start /usr/sbin/xntpd "$src_running"
Then, start the client and verify.
# lssrc -s xntpd
Subsystem         Group            PID          Status
 xntpd            tcpip                         inoperative
startsrc -s xntpd
0513-059 The xntpd Subsystem has been started. Subsystem PID is 4391134.
#lssrc -s xntpd
Subsystem         Group            PID          Status
 xntpd            tcpip            4391134      active
# ntpq -p
     remote           refid      st t when poll reach   delay   offset    disp
==============================================================================
*10.10.10.254   10.10.10.250      2 u   25   64  377     0.87    0.266    0.02
For AIX NTP server, Firstly, verify that you have a suitable NTP server. "Sys peer" should show a valid server or 127.127.1.0. Otherwise, if the server is "insane", you will need to ad a server line to /etc/ntp.conf and restart xntpd.
# lssrc -ls xntpd
 Program name:    /usr/sbin/xntpd
 Version:         3
 Leap indicator:  00 (No leap second today.)
 Sys peer:        10.10.10.254
 Sys stratum:     3
 Sys precision:   -18
 Debug/Tracing:   DISABLED
 Root distance:   0.002701
 Root dispersion: 0.001129
 Reference ID:    10.10.10.254
 Reference time:  d10f3206.cca44000  Wed, Feb 23 2011 15:13:10.799
 Broadcast delay: 0.003906 (sec)
 Auth delay:      0.000122 (sec)
 System flags:    bclient pll monitor filegen
 System uptime:   2653 (sec)
 Clock stability: 1.338257 (sec)
 Clock frequency: 0.000000 (sec)
 Peer: 10.10.10.254
      flags: (configured)(sys peer)(preferred)
      stratum:  2, version: 3
      our mode: client, his mode: server
Subsystem         Group            PID          Status
 xntpd            tcpip            4391134      active
To add server line into /etc/ntp.conf
vi /etc/ntp.conf
Add the following ilne and ensure "broadcast client" is commented out.
server 127.127.1.0
Restart xntpd
stopsrc -s xntpd
startsrc -s xntpd
If the server runs databases, use the -x flag to prevent the clock from changing in a negative direction. Add into /etc/rc.tcpip if necessary. Remember to use the double quote as in "-x". The whole process may take up to 12 minutes.

How to configure RSA 2FA Authentication for AIX

This guide will record the way i have installed RSA's PAM Agent v7.0.0.484.10_12_10_05_06_01 on AIX 6.1 and AIX 7.1

Prepare the System

The RSA's PAM Agent requires the following
  • at least AIX 6.1 TL5 (SP2)
  • RSA Authentication v6.1.2, 7.1 SP2 or 7.1 SP3
  • sdconf.rec file from the RSA Authentication Manager and store it at /var/ace on the server.
The following tools are supported
  • telnet
  • login
  • rlogin
  • su
  • ssh, sftp, scp
  • sudo (at least v1.7.3)
You may have a 64bit OS, but as only 32-bit PAM agent binaries are available, therefore only 32-bit tools are supported. Configuration of Login control Enable PAM Authentication in AIX. Change the authentication method to PAM in /etc/security/login.cfg
usw:
        shells = ...  ...
        maxlogins = 32767
        logintimeout = 60
        maxroles = 8
        *auth_type = STD_AUTH
        auth_type = PAM_AUTH
The symbol * is used to comment the whole line as opposed to the usual # symbol.

Installation of PAM Agent

Go to the path where the PAM agent installer resides.
# tar -xvf PAM-Agent_v7.0.0.484.10_12_10_05_06_01.tar
# cd PAM-Agent_v7.0.0.484.10_12_10_05_06_01
# ./install_pam.sh
Provide the correct path to sdconf.rec and press For subsequent installation prompts, press to accept the default value, or enter appropriate value. Do check the "VAR_ACE" variable in /etc/sd_pam.conf file that it points to the correct location for sdconf.rec. Permission for sdconf.rec should be 600 and ownership root:root.

Configuration of PAM Control

Configure PAM to authenticate using BOTH the local PAM and RSA.
bash-3.2# grep sshd /etc/pam.conf 
sshd    auth    sufficient        pam_securid.so 
sshd    auth    required        pam_aix 
sshd    account sufficient        pam_securid.so 
sshd    account required        pam_aix 
sshd    password  sufficient      pam_securid.so 
sshd    password  required      pam_aix
sshd    session sufficient        pam_securid.so
sshd    session required        pam_aix

Configuration of RSA PAM Agent

Configure the RSA PAM Agent using group control and enable logging of the authentication at /etc/sd_pam.conf
Set "RSATRACELEVEL=1" for logging.
Set "RSATRACEDEST=/var/log/rsa_authlog" for the file to log to.
Set "ENABLE_GROUP_SUPPORT=1" to enable group support
Set "PAM_IGNORE_SUPPORT_FOR_USERS=0" to authenticate by UNIX if a user is not securid authenticated due to user exclusion support.
Set "INCL_EXCL_GROUPS=1" to prompt for securid authentication for the listed group
Set "LIST_OF_GROUPS=other:wheel:staff" for list of group
Set "PAM_IGNORE_SUPPORT=1" to authenticate by UNIX if a user is not securid authenticated due to their group membership.
Set "AUTH_CHALLENGE_PASSWORD_STR=Enter your UNIX PASSWORD :" to be clearer in asking for account password.
Enable logging of all RSA login via syslogd at /etc/syslog.conf
# AUTHENTICATION LOG
auth.info               /var/log/authlog rotate files 12 time 30d compress
Refresh syslogd to take effect
 # refresh -s syslogd

Configuration of SSHD

Edit the sshd configuration file at /etc/ssh/sshd_config.
Set "UsePAM yes" to use PAM authentication
Set "PasswordAuthentication no" to disable password authentication. We have set in pam.conf to authenticate.
Set "ChallengeResponseAuthentication yes"
Set "UsePrivilegeSeparation no"
Refresh the SSHD by restarting it.
# stopsrc -s sshd; startsrc -s sshd
# lssrc -s sshd
Please make sure you back up before changing any configuration files. You have been warned. If you messed up your AIX host, login through HMC and open up a terminal console.

Test

Communication with RSA server test
bash-3.2# /opt/pam/bin/32bit/acestatus

RSA ACE/Server Limits
---------------------
        Configuration Version : 14      Client Retries : 5
        Client Timeout : 5              DES Enabled : Yes

RSA ACE/Static Information
--------------------------
        Service : securid       Protocol : udp  Port Number : 5500

RSA ACE/Dynamic Information
---------------------------
        Server Release : 7.1.2.0        Communication : 5

RSA ACE/Server List
-------------------
        Server Name :           sec-server.com
        Server Address :        10.10.10.22
        Server Active Address : 10.10.10.22
        Master : Yes    Slave : No      Primary : Yes
        Usage : Available for Authentications
------------------------------------------------------------------------------
        Server Name :           sec-server2.com
        Server Address :        10.10.10.23
        Server Active Address : 10.10.10.23
        Master : No     Slave : No      Primary : No
        Usage : Available for Authentications
Basic RSA test using foo account
bash-3.2# /opt/pam/bin/32bit/acetest
Enter USERNAME: foo \
Enter PASSCODE:
Authentication successful.
RSA test using ssh protocol for account with membership to groups "wheel" or "staff"
~$ssh foo@myserver
Enter PASSCODE:
Last unsuccessful login: Tue May 10 14:09:32 SGT 2011 on ssh from 10.10.10.2
Last login: Tue May 10 14:17:54 SGT 2011 on /dev/pts/1 from 10.10.10.2
...
...
$ ^D
Connection to myserver closed.
Accounts that do not have membership to groups "wheel" or "staff" are authenticated "normally"
~$ssh appacct@myserver
Enter your UNIX PASSWORD:
Last unsuccessful login: Wed May 25 10:42:05 SGT 2011 on ssh from 10.10.10.2
Last login: Wed May 25 10:44:03 SGT 2011 on /dev/pts/2 from 10.10.10.2
…
…
When your PIN expires, RSA server will prompt you to change.
bash-3.2# ssh foo@myserver
Enter PASSCODE:
To continue you must enter a new PIN.
Are you ready to enter a new PIN? (y/n) [n]: y
Enter a new PIN of 8 alphanumeric characters:
Re-enter new PIN to confirm:
New PIN accepted, press enter to continue.
Enter PASSCODE:
Enter PASSCODE:
1 unsuccessful login attempt since last login.
Last unsuccessful login: Wed May 25 10:09:17 SGT 2011 on ssh
Last login: Wed May 25 10:07:58 SGT 2011 on /dev/pts/1 from 10.10.10.2
...
...

How to configure sendmail in AIX

This guide serve as a record on how i configured sendmail in AIX 7.1 to relay mails to my company's exchange.

Get sendmail running first

# lssrc -s sendmail
# ps -aef | grep sendmail
# startsrc -s sendmail -a "-bd -q30m"
# ps -aef | grep sendmail
You should see the following output
root  5704     1   0 11:08:42      -  0:00 sendmail: accepting connections on port 25
Ensure sendmail runs after a reboot. Edit the '/etc/rc.tcpip' file
# vi /etc/rc.tcpip
uncomment the following startup line in ‘/etc/rc.tcpip’
start /usr/lib/sendmail "$src_running" "-bd -q${qpi}"
Get sendmail to use local hosts for name resolution instead of DNS (default) Edit ‘/etc/netsvc.conf’ file
# vi /etc/netsvc.conf
Add the following:
hosts=local
For security purpose, change the permissions on the ‘/etc/netsvc.conf’ file to lock down root only access:
# chmod 600 /etc/netsvc.conf
Add into the ‘/etc/hosts’ file the IP address and hostname of the Exchange server. use telnet to verify network.
# telnet  25
Add the IP and hostname into ‘/etc/hosts’
# vi /etc/hosts
The email daemon requires FQDN, so say your host is "myserver", append "myserver." to your hostname in /etc/hosts. Backup the original ‘/etc/sendmail.cf’ file and edit.
# cp /etc/sendmail.cf /etc/sendmail.cf.
Make the following changes:
# vi /etc/sendmail.cf
For AIX 6.1, i changed the Dw as well. Don't need it in AIX 7.1. Change from:
#DwYourHostName
To:
Dw
Change from:
# "Smart" relay host (may be null)
# Relay host to forward outgoing mail not in the local domain to.
# To forward ALL mail to this relay host, uncomment the appropriate
# rule in ruleset 0, as indicated by the ruleset's comments.
#DSmailer:relayhostname
DS
To:
# "Smart" relay host (may be null)
# Relay host to forward outgoing mail not in the local domain to.
# To forward ALL mail to this relay host, uncomment the appropriate
# rule in ruleset 0, as indicated by the ruleset's comments.
#DSmailer:relayhostname
DS
Now refresh the ‘sendmail’ daemon with the new changes
# refresh –s sendmail
It will take a few minutes for the ‘ps’ process to return:
root  5704     1   0 11:08:42      -  0:00 sendmail: accepting connections on port 25
In the interim period the following will be displayed via ‘ps’:
root  5704     1   0 11:08:42      -  0:00 /usr/lib/sendmail –bd –q30
Test sendmail now.
# echo “test” |sendmail –v yourname@myinbox.com
Enable mail log by adding the following line in syslog.conf.
mail.debug     /var/log/maillog rotate files 6 time 30d compress
Now, create the log file and refresh syslogd.
# touch /var/log/maillog
# refresh -s syslogd

How to determine disk space in AIX

As i'm still pretty new to AIX, i need to learn as quickly as possible to manage the new babies in my care. :) Let say i want to determine disk size of physical volume The output below is in Mb.

# bootinfo -s hdisk2
51200
For free space left in volume group We can see that there are 36Gb of space left in VG unassigned.
# lsvg rootvg
VOLUME GROUP:       rootvg                   VG IDENTIFIER:  00f603d800004c000000012c441dd962
VG STATE:           active                   PP SIZE:        32 megabyte(s)
VG PERMISSION:      read/write               TOTAL PPs:      1918 (61376 megabytes)
MAX LVs:            256                      FREE PPs:       1140 (36480 megabytes)
LVs:                13                       USED PPs:       778 (24896 megabytes)
OPEN LVs:           12                       QUORUM:         1 (Disabled)
TOTAL PVs:          2                        VG DESCRIPTORS: 3
STALE PVs:          0                        STALE PPs:      0
ACTIVE PVs:         2                        AUTO ON:        yes
MAX PPs per VG:     32512
MAX PPs per PV:     1016                     MAX PVs:        32
LTG size (Dynamic): 256 kilobyte(s)          AUTO SYNC:      no
HOT SPARE:          no                       BB POLICY:      relocatable
PV RESTRICTION:     none
This is the demo script i have written to check both disk and inode utilisation. Please let me know if you find this useful or the gurus want to help me improve. :)
#!/bin/ksh
#
# FILENAME   : checkDiskSpace.ksh
# AUTHOR     : Victor Kwan
# EMAIL      : victorkk [AT] gmail [DOT] com
# PURPOSE    : To check disk and inode utilisation
#            : and alert sys admin if threshold is breached.
# DATE       : Feb 2011
#

# 
# Parameters setup
OUTPUTFILE_DF="checkDisk.DF.`hostname`.`date '+%d%b%Y'`.output"
NOTIFICATION_MSG="checkDisk.DF.`hostname`.`date '+%d%b%Y'`.message"
isFOUND=0

#
# Input Validation
if [ $# -ne 3 ]
then
        printf "Usage: \n\t$0   \n\n"
        exit
fi

DISK_THRESHOLD=$1
INODE_THRESHOLD=$2
EMAIL="$3"

#
# Check for Logical Disk Partition full
df -m > ./$OUTPUTFILE_DF

#
# Setup the messages
printf "----------------------------------------\n" > $NOTIFICATION_MSG
printf " Check for Logical Parition Utilisation\n\n" >> $NOTIFICATION_MSG
printf " Disk / Inode Threshold: $DISK_THRESHOLD | $INODE_THRESHOLD \n" >> $NOTIFICATION_MSG
printf "----------------------------------------\n\n" >> $NOTIFICATION_MSG


if [ `cat $OUTPUTFILE_DF | wc -l` == 1 ]
then
        printf "No Partition detected.\n"
else
        cat $OUTPUTFILE_DF | while read LINE
        do
                isFULL_CHECK=`echo $LINE | awk '{print $4}' | grep "%" | wc -l`
                #printf "[DEBUG]isFULL_CHECK is %d.\n" $isFULL_CHECK

                if [ $isFULL_CHECK == 1 ]
                then

                        LDISK=`echo $LINE | awk '{print $1}'`
                        LDISK_PARTITION=`echo $LINE | awk '{print $7}'`
                        LDISK_SIZE=`echo $LINE | awk '{print $2}'`
                        LDISK_USED=`echo $LINE | awk '{print $4}' | awk -F% '{print $1}'`
                        LDISK_INODE=`echo $LINE | awk '{print $6}' | awk -F% '{print $1}'`

                        if [ $LDISK_USED -ge $DISK_THRESHOLD -o $LDISK_INODE -ge $INODE_THRESHOLD ]
                        then
                                #printf "[DEBUG]The line is $LINE\n"
                                printf "Partition \"%s\" mounted on \"%s\"\n" $LDISK_PARTITION $LDISK >> $NOTIFICATION_MSG
                                printf "Size (Mb) | Disk (%%) | Inode (%%)\n"  >> $NOTIFICATION_MSG
                                printf "%-9.2f | %-8d | %-8d\n\n" $LDISK_SIZE $LDISK_USED $LDISK_INODE >> $NOTIFICATION_MSG
                                isFOUND=1
                        fi
                fi

        done
fi

if [ $isFOUND == 0 ]
then
        printf "All Partition within threshold.\n" >> $NOTIFICATION_MSG

elif [ $isFOUND == 1 ]
then
        cat $NOTIFICATION_MSG | mailx -s "[`hostname`] Disk Space Above Threshold" $EMAIL
fi

#
# Some Housekeeping
rm $OUTPUTFILE_DF
rm $NOTIFICATION_MSG

How to determine if the new oraerror.dat is loaded in OracleAgent

Recently, i need to change the behaviour of VCS how it managed my DB. To cut the story short, i wanted to check if the oraerror.dat file is loaded by the Oracle Agent so that monitoring behaviour is according to the oraerror.dat file. If the oraerror.dat is not loaded, the agent will assume the default behaviour after default monitoring (which is FAILOVER). Query the number of entries.

# egrep -v '^$|^#|}' /opt/VRTSagents/ha/bin/Oracle/oraerror.dat | wc -l
187
Check the gcore of the OracleAgent process, the data structure "token_data" stores the in-memory oraerror.dat.
# ps -ef |grep OracleAge
    root  5282     1  0   Aug 26 ?        5:10 /opt/VRTSagents/ha/bin/Oracle/OracleAgent -type Oracle -agdir /opt/VRTSagents/h
    root 23533 29154  0 10:33:32 pts/3    0:00 grep OracleAge
# gcore 5282
gcore: core.5282 dumped
Take a look at the 7th field, which is the number of entries in oraerror.dat plus 1.
# mdb /opt/VRTSagents/ha/bin/Oracle/OracleAgent core.5282
Loading modules: [ libthread.so.1 libc.so.1 ld.so.1 ]
> *token_data/20D
0x97000:        32              663408          0               690328          690424          629608
                188             3               1162824517      1330332928      97              1111573570
                1112492800      80              436440          2               262144          19800
                0               0
ctrl+D to exit. Both the 7th field and the number of entries of oraerror.dat matches. Also look at the permission of the file! If due to some reasons, the oraerror.dat file not loaded by the OracleAgent when it started, check that the file is readable and contains the valid entries, then restart the Oracle Agent.
# ls -l oraerror.dat
-rwxr--r--   1 root     sys         3611 Jul 15  2009 oraerror.dat
# haagent -stop Oracle -force -sys myserver
# haagent -start Oracle -sys myserver

# haagent -display | grep Oracle
Oracle        AgentDirectory /opt/VRTSagents/ha/bin/Oracle
Oracle        AgentFile
Oracle        Faults         0
Oracle        Running        Yes
Oracle        Started        Yes
Please noted that if oraerror.dat is modified, the agent has to be restarted in order to have the agent recognise the change. Reference: http://www.symantec.com/docs/TECH155872

How to disable weak ciphers in Apache

Rationale for disabling weak ciphers

From the whitepaper in the NLUUG autumn "security" conference in Nov 2010, Some Apache configuration updates are required to satisfy the following that needs to be disabled.
Note: This is from the NLUUG whitepaper, if the author or publisher do not agree with me posting the sreenshot, please let me know immediately and i will removed it. Thanks. The current apache default setting in my Apache ssl.conf is SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP . The older servers are even using the older openssl! The enabled ciphers are listed below.
# openssl ciphers -v 'SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP'
DHE-RSA-AES256-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA1
DHE-DSS-AES256-SHA      SSLv3 Kx=DH       Au=DSS  Enc=AES(256)  Mac=SHA1
AES256-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA1
KRB5-DES-CBC3-MD5       SSLv3 Kx=KRB5     Au=KRB5 Enc=3DES(168) Mac=MD5
KRB5-DES-CBC3-SHA       SSLv3 Kx=KRB5     Au=KRB5 Enc=3DES(168) Mac=SHA1
EDH-RSA-DES-CBC3-SHA    SSLv3 Kx=DH       Au=RSA  Enc=3DES(168) Mac=SHA1
EDH-DSS-DES-CBC3-SHA    SSLv3 Kx=DH       Au=DSS  Enc=3DES(168) Mac=SHA1
DES-CBC3-SHA            SSLv3 Kx=RSA      Au=RSA  Enc=3DES(168) Mac=SHA1
DHE-RSA-AES128-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA1
DHE-DSS-AES128-SHA      SSLv3 Kx=DH       Au=DSS  Enc=AES(128)  Mac=SHA1
AES128-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(128)  Mac=SHA1
DHE-DSS-RC4-SHA         SSLv3 Kx=DH       Au=DSS  Enc=RC4(128)  Mac=SHA1
KRB5-RC4-MD5            SSLv3 Kx=KRB5     Au=KRB5 Enc=RC4(128)  Mac=MD5
KRB5-RC4-SHA            SSLv3 Kx=KRB5     Au=KRB5 Enc=RC4(128)  Mac=SHA1
RC4-SHA                 SSLv3 Kx=RSA      Au=RSA  Enc=RC4(128)  Mac=SHA1
RC4-MD5                 SSLv3 Kx=RSA      Au=RSA  Enc=RC4(128)  Mac=MD5
KRB5-DES-CBC-MD5        SSLv3 Kx=KRB5     Au=KRB5 Enc=DES(56)   Mac=MD5
KRB5-DES-CBC-SHA        SSLv3 Kx=KRB5     Au=KRB5 Enc=DES(56)   Mac=SHA1
EDH-RSA-DES-CBC-SHA     SSLv3 Kx=DH       Au=RSA  Enc=DES(56)   Mac=SHA1
EDH-DSS-DES-CBC-SHA     SSLv3 Kx=DH       Au=DSS  Enc=DES(56)   Mac=SHA1
DES-CBC-SHA             SSLv3 Kx=RSA      Au=RSA  Enc=DES(56)   Mac=SHA1
DES-CBC3-MD5            SSLv2 Kx=RSA      Au=RSA  Enc=3DES(168) Mac=MD5
RC2-CBC-MD5             SSLv2 Kx=RSA      Au=RSA  Enc=RC2(128)  Mac=MD5
RC4-MD5                 SSLv2 Kx=RSA      Au=RSA  Enc=RC4(128)  Mac=MD5
RC4-64-MD5              SSLv2 Kx=RSA      Au=RSA  Enc=RC4(64)   Mac=MD5
DES-CBC-MD5             SSLv2 Kx=RSA      Au=RSA  Enc=DES(56)   Mac=MD5
EXP-KRB5-RC4-MD5        SSLv3 Kx=KRB5     Au=KRB5 Enc=RC4(40)   Mac=MD5  export
EXP-KRB5-RC2-CBC-MD5    SSLv3 Kx=KRB5     Au=KRB5 Enc=RC2(40)   Mac=MD5  export
EXP-KRB5-DES-CBC-MD5    SSLv3 Kx=KRB5     Au=KRB5 Enc=DES(40)   Mac=MD5  export
EXP-KRB5-RC4-SHA        SSLv3 Kx=KRB5     Au=KRB5 Enc=RC4(40)   Mac=SHA1 export
EXP-KRB5-RC2-CBC-SHA    SSLv3 Kx=KRB5     Au=KRB5 Enc=RC2(40)   Mac=SHA1 export
EXP-KRB5-DES-CBC-SHA    SSLv3 Kx=KRB5     Au=KRB5 Enc=DES(40)   Mac=SHA1 export
EXP-EDH-RSA-DES-CBC-SHA SSLv3 Kx=DH(512)  Au=RSA  Enc=DES(40)   Mac=SHA1 export
EXP-EDH-DSS-DES-CBC-SHA SSLv3 Kx=DH(512)  Au=DSS  Enc=DES(40)   Mac=SHA1 export
EXP-DES-CBC-SHA         SSLv3 Kx=RSA(512) Au=RSA  Enc=DES(40)   Mac=SHA1 export
EXP-RC2-CBC-MD5         SSLv3 Kx=RSA(512) Au=RSA  Enc=RC2(40)   Mac=MD5  export
EXP-RC4-MD5             SSLv3 Kx=RSA(512) Au=RSA  Enc=RC4(40)   Mac=MD5  export
EXP-RC2-CBC-MD5         SSLv2 Kx=RSA(512) Au=RSA  Enc=RC2(40)   Mac=MD5  export
EXP-RC4-MD5             SSLv2 Kx=RSA(512) Au=RSA  Enc=RC4(40)   Mac=MD5  export

Disabling of weak ciphers.

For the sake of better security but without compromising the user's experience, i will also use SSLv3 or TLSv1 instead of SSLv2. Futhermore, OptRenegotiate is disabled by default.

i changed the cipher suite to SSLCipherSuite ALL:!ADH:!SSLv2:!EXPORT56:!EXPORT40:!RC4:!DES:+HIGH:+MEDIUM:+EXP which does the following:

  • Disable all Anonymous DH key exchange
  • Disable all SSL v2 ciphers
  • Disable all 56-bit export ciphers
  • Disable all 40-bit export ciphers
  • Disable all RC4 ciphers
  • Disable all single DES ciphers
  • Enable all 3DES ciphers
  • Enable all 128 bit encryption
  • Enable all export ciphers
The final list of ciphers enabled now are listed below.
# openssl ciphers -v 'SSLCipherSuite ALL:!ADH:!SSLv2:!EXPORT56:!EXPORT40:!RC4:!DES:+HIGH:+MEDIUM:+EXP'
DHE-RSA-AES256-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA1
DHE-DSS-AES256-SHA      SSLv3 Kx=DH       Au=DSS  Enc=AES(256)  Mac=SHA1
AES256-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA1
KRB5-DES-CBC3-MD5       SSLv3 Kx=KRB5     Au=KRB5 Enc=3DES(168) Mac=MD5
KRB5-DES-CBC3-SHA       SSLv3 Kx=KRB5     Au=KRB5 Enc=3DES(168) Mac=SHA1
EDH-RSA-DES-CBC3-SHA    SSLv3 Kx=DH       Au=RSA  Enc=3DES(168) Mac=SHA1
EDH-DSS-DES-CBC3-SHA    SSLv3 Kx=DH       Au=DSS  Enc=3DES(168) Mac=SHA1
DES-CBC3-SHA            SSLv3 Kx=RSA      Au=RSA  Enc=3DES(168) Mac=SHA1
DHE-RSA-AES128-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA1
DHE-DSS-AES128-SHA      SSLv3 Kx=DH       Au=DSS  Enc=AES(128)  Mac=SHA1
AES128-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(128)  Mac=SHA1

After updating the configuration in Apache, it is recommended to test the configuration, i.e. at www.ssllabs.com. The top grade is A. Try testing your web configuration and see if you get A. :)

Reference: NLUUG Autumn Security Conference Whitepaper

How to disable first time password change in AIX

I'm new to AIX. Getting sick of AIX asking users for new password when you log in to the server for the first time ever after the setting the password using root. After some digging, this is due to the ADMCHD flag in /etc/security/password like below.

ahkow:
        password = Qpkm8APaNsdoI
        lastupdate = 1227154518
        flags = ADMCHG
Clear the ADMCHG flag of the user account with "pwdadm -c ".
> pwdadm -c ahkow
now the flag is gone.
ahkow:
        password = Qpkm8APaNsdoI
        lastupdate = 1227154518

Disabling TRACK / TRACE in Apache

Currently, Apache does not deny TRACE requests (per RFC2616) by default. Therefore, when an HTTP TRACE request is sent to a web server that supports it, that server will respond echoing the data that is passed to it, including any HTTP headers. By definition, HTTP TRACE method ask a web server to echo the contents of the request back to the client for debugging purposes. The complete request, including HTTP headers, is returned in the entity-body of a TRACE response. An example of the response of Apache when TRACE is enabled,

# telnet myserver.com 80
Trying 10.10.10.10...
Connected to myserver. (10.10.10.10).
Escape character is '^\]'.
TRACE / HTTP/1.0
Host: myserver.com
TestA: Hello
TestB: World\\

HTTP/1.1 200 OK
Date: Tue, 19 Jul 2011 10:31:38 GMT
Server: Apache
Connection: close
Content-Type: message/http

TRACE / HTTP/1.0
Host: myserver.com
TestA: Hello
TestB: World

Connection closed by foreign host.
The output in the 2nd and 3rd paragraph is actually the response from Apache with the exact data sent in the 1st paragraph. Status code of 200 indicate that TRACE request is allowed and hence this response. It is possible for the attackers prepare carefully crafted page to trick a browser on a user’s box to issue the TRACE request and after which pass on the cookies or authentication data to the attacker. TRACE requests can be disabled by making a change to the Apache server configuration. There are 2 methods to achieve this 1) Setting “TraceEnabled off” in httpd.conf This is only available for Apache 1.3.34, 2.0.55 and 2.2.x. 2) Using rewrite to deny TRACE request in all the vhost. This can be used universally in all the Apache versions. You can deny TRACE requests for both HTTP and HTTPS depending on your business system requirement and using either method depending on your Apache version. For example, using method 2 to disable TRACE support in Apache, here’s the example in the configuration file httpd.conf. For this purpose I have added additional lines to capture the log when the rules are invoked. After note: Apache version 1.3.34, 2.0.55 and 2.2.x and newer, please use option 1. The rest will use option 2 on all the pair.

    Servername myserver.com
    ErrorLog logs/myserver-error.log
    CustomLog logs/myserver-access.log common

    Block TRACE/TRACK XSS vector
    RewriteEngine On
    RewriteCond %{REQUEST_METHOD} \^TRAC(E\|K)*
    RewriteRule .\* - \[F\]*
    RewriteLogLevel 9
    RewriteLog logs/rewrite_log
 
After TRACE support is disabled, below is the rerun of the TRACE request.
# /usr/local/apache2/conf>telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^\]'.
TRACE / HTTP/1.0
Host: myserver.com
TestA: Hello
TestB: World

HTTP/1.1 403 Forbidden{*}
Date: Thu, 21 Jul 2011 08:06:02 GMT
Server: Apache
Content-Length: 202
Connection: close
Content-Type: text/html; charset=iso-8859-1

<\!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

403 Forbidden

Forbidden

You don't have permission to access / on this server. Connection to localhost closed by foreign host.
We can see that Apache now response with a 400 series status code which indicate client request was denied. Output of the log shown below when the rewrite rules are invoked.
10.10.10.10 - - [21/Jul/2011:17:09:34 +0800] [myserver.com/sid#552add28f8][rid#552b2f0bf8/initial] (2) init rewrite engine with requested uri /
10.10.10.10 - - [21/Jul/2011:17:09:34 +0800] [myserver.com/sid#552add28f8][rid#552b2f0bf8/initial] (3) applying pattern '.*' to uri '/'
10.10.10.10 - - [21/Jul/2011:17:09:34 +0800] [myserver.com/sid#552add28f8][rid#552b2f0bf8/initial] (4) RewriteCond: input='TRACE' pattern='^TRAC(E\|K)' => matched
10.10.10.10 - - [21/Jul/2011:17:09:34 +0800] [myserver.com/sid#552add28f8][rid#552b2f0bf8/initial] (2) forcing '/' to be forbidden
The examples so far shows only for HTTP, XST vulnerability can also be shown for HTTPS where we just use “openssl s_client --connect hostname:port” and using the same commands after the telnet command. Please note that this is not a vulnerability in TRACE, nor in Apache. This is more of a need to harden Apache not to divulge more information than it should.


Reference:
http://www.kb.cert.org/vuls/id/867593
http://www.apacheweek.com/issues/03-01-24

How To Allow Applications to connect to MQ6

On MQ v6, MCAUSER in SYSTEM.DEF.SVRCONN is blank by default, hence this would not allow anyone to connect to it.

This is an example of the error from the application when it couldn't connect even though LDAP setup is correct. Basic network connectivity test by telnet is OK.

2008-04-06 11:11:10,452 [main] ERROR - Hit exception at init JMSSubscriber, System reconnecting after [10] seconds .... com.psa.infra.messaging.PMException at com.psa.infra.messaging.PMSessionFactory.createConnection(PMSessionFactory.java:260) at com.psa.infra.messaging.PMSessionFactory.getConnection(PMSessionFactory.java:168) at com.psa.infra.messaging.PMSession.getPMJMSSession(PMSession.java:216) at com.psa.infra.messaging.PMSession.createHandler(PMSession.java:101) at com.psa.infra.messaging.PMSession.createHandler(PMSession.java:58) at com.psa.common.utility.jms.subscriber.JMSSubscriber.init(JMSSubscriber.java:95) at com.psa.common.utility.core.mesg.MessageTransfer.process(MessageTransfer.java:94) at com.psa.common.utility.core.mesg.MessageTransfer.main(MessageTransfer.java:160) Caused by: javax.jms.JMSException: MQJMS2005: failed to create MQQueueManager for 'png1comm2_ica:PSA.QUEUE.MANAGER' at com.ibm.mq.jms.services.ConfigEnvironment.newException(ConfigEnvironment.java:434) at com.ibm.mq.jms.MQQueueConnection.createQM(MQQueueConnection.java:478) at com.ibm.mq.jms.MQQueueConnection.(MQQueueConnection.java:182) at com.ibm.mq.jms.MQQueueConnectionFactory.createQueueConnection(MQQueueConnectionFactory.java:166) at com.psa.infra.messaging.PMSessionFactory.createConnection(PMSessionFactory.java:234) ... 7 more 2008-04-06 11:11:10,453 [main] INFO - -- Close JMS subscriber handler

To allow applications to connect, enter the following code after getting in 'runmqsc'.

> alter CHANNEL ('SYSTEM.DEF.SVRCONN') CHLTYPE(SVRCONN) MCAUSER('mqm')

Tuesday, December 29, 2009

Lighting Basics

I'm typing this for my own reference.
Still trying to get correct settings for white balance.
I hope that this will also help others who wish to have a quick reference on improving the quality of photo taking, especially if in indoor.

Tungsten (2500K - 3500K) - The bulb sign. For slightly bluish lighting.
Fluorescent (4000K - 5000K) - The tube sign. for slightly yellow lighting
Daylight (5000K - 6500K) - The sun sign. Used when outdoor w/o shade.
Using Add-on flash (5000K - 5500K) - The lighting sign. Used usually when indoor.
Cloudy (6500K - 8000K) - The cloud sign. For medium overcast sky.
Shade (9000K - 10000K)- The house sign. For heavy overcast sky or shade.

If the photo still appear too cool or warm on the LCF screen preview, can try to increase or decrease the colour temperature. This should balance it up a little.

Sometime the camera maybe 'off' in its auto WB mode. The camera might be more effective if it can detect something that is white or bright colorless object to use as a refernce. Or if we have multiple light source, the camera may do a 'averaging' of all lightings which may not be what we want.

We can however try the manual WB to capture a more realistic result.

Deciphering second field in shadow file in RHEL

The second field in shadow file contains encrypted password as well as other information. The file is readable by root only and the fields include the following:

1) username (usually up to 8 characters and case sensitive. Direct match to /etc/passwd)

2) encrypted password (13 characters)

3) The number of days since 1st Jan 1970 since the password has been last changed.

4) The number of days before password may be changed. If you see a '0', you can change any time.

5) The number of days after which the password must be changed. If you see 99999, no change is needed forever.

6) The number of days before to warn user of an imminent password expiry.

7) The number of days after password expiry that the account is disabled.

8) The number of days since 1st Jan 1970 that this account is disabled.

9) Last field is reserved. Not used at this moment.

More details on how to read the 2nd field?

- If you have blank entry, though is a bad idea but this mean that you do not need password to log in.

- If you have a '*' entry, this means the account is disabled. User cannot log in.

- If you have a '!' entry, this means the account is disabled as well. Use cannot log in.

- if you have a "!!" entry, this mean the account has never been used yet. User has not log in before and cannot log in.

Saturday, October 31, 2009

How to add a mirror disk back to VxVM

I have 3 disks which are part of the mirror group.

root@myserver # vxdisk list | grep ^c
c1t0d0s2 sliced rootdisk rootdg online
c1t1d0s2 sliced rootmirror1 rootdg online
c2t0d0s2 sliced - - online
c2t1d0s2 sliced - - online

Naming conventions:
- rootdisk
- rootmirror1
- rootmirror2
- rootmirror3

Now i want to add the mirror back. We are going to attach back device c2t0d0 into the rootdg disk group.

Always run the command vxdisk -ht to check what is the actual volume name to attach the disk to.

Setup the disk
root@myserver # vxdisksetup -if c2t0d0 (for VxVM 3.5)
root@myserver # vxdisksetup -if c2t0d0 format=sliced (for VxVM 4.0)
root@myserver # /usr/lib/vxvm/bin/vxdisksetup -if c2t0d0 format=sliced (for VxVM 4.1)

Add the disk into rootdg.
root@myserver # vxdg -g rootdg adddisk rootmirror2=c2t0d0

Mirror each of the partitions. Here i throttle the synchronisation speed to have less performance impact on the server.
root@myserver # vxassist -o slow=1500 -g rootdg mirror rootvol rootmirror2 &
root@myserver # vxassist -o slow=1500 -g rootdg mirror opt rootmirror2 &
root@myserver # vxassist -o slow=1500 -g rootdg mirror swapvol rootmirror2 &
root@myserver # vxassist -o slow=1500 -g rootdg mirror usr rootmirror2 &
root@myserver # vxassist -o slow=1500 -g rootdg mirror var rootmirror2 &

To monitor the progress of synchronisation.
root@myserver # vxtask list
TASKID PTID TYPE/STATE PCT PROGRESS
166 ATCOPY/R 07.88% 0/25166079/1982464 PLXATT opt opt-01
167 ATCOPY/R 25.29% 0/4194828/1060864 PLXATT swapvol swapvol-01
168 ATCOPY/R 15.13% 0/4194828/634880 PLXATT usr usr-01
169 ATCOPY/R 02.82% 0/16779312/473088 PLXATT var var-01

How to ensure VTOC is updated from VxVM in SunOS?

It is important to keep the VTOC in sync with the setup in VxVM else you risk not being able to boot up the disk after un-encapsulating.

SUN also advised that if the VxVM disk mirrors are not identical to the first disk, this would result in unusable mirrors.

Also after creation of VxVM rootdisk and rootmirrors, you may want to create the underlying partition on the rootdisk and rootmirror.

Here, i will show the steps for setting up new disk as RAID1 mirror

root@myserver # vxdisksetup -if c2t0d0
root@myserver # vxdg -g rootdg adddisk rootmirror2=c2t0d0
root@myserver # vxassist -g rootdg mirror rootvol rootmirror2 &
root@myserver # vxassist -g rootdg mirror usr rootmirror2 &
root@myserver # vxassist -g rootdg mirror var rootmirror2 &
root@myserver # vxassist -g rootdg mirror opt rootmirror2 &

Take a look at the VTOC

root@myserver # prtvtoc /dev/dsk/c1t0d0s2
* /dev/dsk/c1t0d0s2 partition map
*
* Dimensions:
* 512 bytes/sector
* 107 sectors/track
* 27 tracks/cylinder
* 2889 sectors/cylinder
* 24622 cylinders
* 24620 accessible cylinders
*
* Flags:
* 1: unmountable
* 10: read-only
*
* Unallocated space:
* First Sector Last
* Sector Count Sector
* 0 2889 2888
* 8395434 4286577640 5777
* 71127180 4232235550 8395433
* 58730481 12396699 71127179
*
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
0 2 00 5778 8389656 8395433
1 3 01 8395434 4194828 12590261
2 5 00 0 71127180 71127179
3 15 01 2889 2889 5777
4 14 01 5778 71121402 71127179
5 0 00 33564402 25166079 58730480
6 4 00 12590262 4194828 16785089
7 7 00 16785090 16779312 33564401

Now, the VTOC is not updated. we need to use VxVM command to update VTOC. not really advisable to update using "format" command manually since its more error prone.

Tag Description (2nd column from right)
0x00 UNASSIGNED
0x01 BOOT
0x02 ROOT
0x03 SWAP
0x04 USR
0x05 BACKUP
0x06 STAND
0x07 VAR
0x08 HOME

Flag Description (right-most column)
0x00 MOUNTABLE
0x01 UNMOUNTABLE
0x10 READ-ONLY

Note that Swap is NOT mountable.


root@myserver # vxprint -htg rootdg
DG NAME NCONFIG NLOG MINORS GROUP-ID
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
V NAME RVG KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg rootdg all all 0 1076053596.1025.myserver

dm rootdisk c1t0d0s2 sliced 2632 71121402 -
dm rootmirror1 c1t1d0s2 sliced 2632 71121402 -
dm rootmirror2 c2t0d0s2 sliced 2632 71121402 -

v opt - ENABLED ACTIVE 25166079 ROUND - fsgen
pl opt-01 opt ENABLED ACTIVE 25166079 CONCAT - RW
sd rootdisk-05 opt-01 rootdisk 33558624 25166079 0 c1t0d0 ENA
pl opt-02 opt ENABLED ACTIVE 25166079 CONCAT - RW
sd rootmirror1-05 opt-02 rootmirror1 12584484 25166079 0 c1t1d0 ENA
pl opt-03 opt ENABLED ACTIVE 25166079 CONCAT - RW
sd rootmirror2-05 opt-03 rootmirror2 33558624 25166079 0 c2t0d0 ENA

v rootvol - ENABLED ACTIVE 8389656 ROUND - root
pl rootvol-01 rootvol ENABLED ACTIVE 8389656 CONCAT - RW
sd rootdisk-01 rootvol-01 rootdisk 0 8389656 0 c1t0d0 ENA
pl rootvol-02 rootvol ENABLED ACTIVE 8389656 CONCAT - RW
sd rootmirror1-01 rootvol-02 rootmirror1 0 8389656 0 c1t1d0 ENA
pl rootvol-03 rootvol ENABLED ACTIVE 8389656 CONCAT - RW
sd rootmirror2-01 rootvol-03 rootmirror2 0 8389656 0 c2t0d0 ENA

v swapvol - ENABLED ACTIVE 4194828 ROUND - swap
pl swapvol-01 swapvol ENABLED ACTIVE 4194828 CONCAT - RW
sd rootdisk-02 swapvol-01 rootdisk 8389656 4194828 0 c1t0d0 ENA
pl swapvol-02 swapvol ENABLED ACTIVE 4194828 CONCAT - RW
sd rootmirror1-02 swapvol-02 rootmirror1 8389656 4194828 0 c1t1d0 ENA
pl swapvol-03 swapvol ENABLED ACTIVE 4194828 CONCAT - RW
sd rootmirror2-02 swapvol-03 rootmirror2 8389656 4194828 0 c2t0d0 ENA

v usr - ENABLED ACTIVE 4194828 ROUND - fsgen
pl usr-01 usr ENABLED ACTIVE 4194828 CONCAT - RW
sd rootdisk-03 usr-01 rootdisk 12584484 4194828 0 c1t0d0 ENA
pl usr-02 usr ENABLED ACTIVE 4194828 CONCAT - RW
sd rootmirror1-03 usr-02 rootmirror1 37750563 4194828 0 c1t1d0 ENA
pl usr-03 usr ENABLED ACTIVE 4194828 CONCAT - RW
sd rootmirror2-03 usr-03 rootmirror2 12584484 4194828 0 c2t0d0 ENA

v var - ENABLED ACTIVE 16779312 ROUND - fsgen
pl var-01 var ENABLED ACTIVE 16779312 CONCAT - RW
sd rootdisk-04 var-01 rootdisk 16779312 16779312 0 c1t0d0 ENA
pl var-02 var ENABLED ACTIVE 16779312 CONCAT - RW
sd rootmirror1-04 var-02 rootmirror1 41945391 16779312 0 c1t1d0 ENA
pl var-03 var ENABLED ACTIVE 16779312 CONCAT - RW
sd rootmirror2-04 var-03 rootmirror2 16779312 16779312 0 c2t0d0 ENA

For automated step, try this
# vxbootsetup

For manual step, try these
Notice that you did NOT create the disk partitions manually when you initialized the mirror disk. This is because VxVM automatically does its partitioning for you. This is also true for the root disk after it has been re-initialized and brought into rootdg. That is, the original rootdisk partition scheme has been replaced by Veritas's partition scheme. This means that the underlying partitions for each volume(swap, /opt, /var, /home) are not physically laid out on the drives corresponding to the veritas volumes.

In some situations though, you might want to be able to access the partitions corresponding to each volume by manually mounting the slice. In a recovery situation for example, you may want to boot from cdrom and access the /opt partition by manually mounting the slice without the veritas drivers loaded. This is not possible unless you create the underlying partition mapping directly to the opt volume using the "vxmksdpart" command.

All underlying partitions except / should be created. The / partition is automatically created and is mapped correctly to the corresponding veritas root volume.

One Caveat, you cannot use partition 3 or 4 because this is typically where Veritas stores is private and public region. The following is snipped from the "format" utility listing the Veritas partitions. This shows that the root partition exists but the swap, /var, /opt, home partitions are not mapped on the drive(Veritas knows where they exist on the drive because of its private and public regions):

Part      Tag    Flag     Cylinders        Size            Blocks

  0       root    wm       1 - 2282        3.91GB    (2282/0/0)  8194662        <=== "/" exists
  1 unassigned    wm       0               0         (0/0/0)           0        <=== Unassigned
  2     backup    wm       0 - 4923        8.43GB    (4924/0/0) 17682084
  3          -    wu       0 -    0        1.75MB    (1/0/0)        3591        <=== Private Region
  4          -    wu       1 - 4923        8.43GB    (4923/0/0) 17678493        <=== Public Region
  5 unassigned    wm       0               0         (0/0/0)           0        <=== Unassigned
  6 unassigned    wm       0               0         (0/0/0)           0        <=== Unassigned
  7 unassigned    wm       0               0         (0/0/0)           0        <=== Unassigned

You may use the unassigned partitions which are 1, 5, 6, and 7 to map to the veritas volumes. Be sure to document which partitions map to the Veritas volumes for future reference. The "vxmksdpart" arguments are the subdisk (which maps directly to the volume), slice, tag and flag. Be sure to use the correct tag and flag for your particular Solaris version. Reference the man page for "fmthard" for correct options. Use the "vxprint" command to get the correct subdisk names: Generic command # /usr/lib/vxvm/bin/vxmksdpart

In my case
Rootdisk:
# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootdisk-02 1 0x03 0x01 <=== swap
# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootdisk-05 5 0x00 0x00 <=== /unassigned
# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootdisk-03 6 0x04 0x00 <=== /usr
# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootdisk-04 7 0x07 0x00 <=== /var

RootMirror:

# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootmirror1-02 1 0x03 0x01 <=== swap
# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootmirror1-05 5 0x00 0x00 <=== /unassigned
# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootmirror1-03 6 0x04 0x00 <=== /usr
# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootmirror1-04 7 0x07 0x00 <=== /var

References:
http://sysunconfig.net/unixtips/jmr_Mirroring_Boot_Disks_Veritas_Volume_Manager.htm
http://unixway.com/vm/veritasvm/rootdg.html

How to unmount a online / mounted VXFS on 5.0 MP3

You may have encountered this

# umount -f /opt/app/oracle
UX:vxfs umount: ERROR: V-3-26360: file system /opt/app/oracle has been mount locked

Taken from the symantec support site - http://seer.entsupport.symantec.com/docs/313851.htm

Details:
Storage Foundation 5.0MP3 introduced a new feature called VxFS filesystem lock which disallows accidental unmounts when the file system resource is online. New umount option mntunlock is used to clear the lock and then unmount the filesystem. The offline script for the Mount resource will use this new option.

How to check if the filesystem is locked by VCS:
# mount -v | grep mntlock

Sometimes, it may be necessary to unmount a mount locked filesystem. This is for cases where VCS service groups having DiskGroup resources configured with UnMountVolumes attribute set and the volumes are mounted outside of VCS control (this is not very common). Use the VxFS umount command for Solaris and vxumount command for Linux to manually unmount the file system

Solaris:
# /opt/VRTS/bin/umount -o mntunlock=VCS

Linux:
# /usr/lib/fs/vxfs/umount -o mntunlock=VCS

Example:
# mount -v | grep mntlock
/dev/vx/dsk/oracledg/oracle on /opt/app/oracle type vxfs read/write/setuid/devices/mincache=direct/convosync=direct/delaylog/largefiles/qio/ioerror=mwdisable/mntlock=VCS/dev=46c2ee0 on Mon Sep 15 18:28:18 2008
# /opt/VRTS/bin/umount -o mntunlock=VCS /opt/app/oracle

HowTo setup Link-based IPMP with VCS for Solaris 10.

Thanks Bret Sanders for the knowledge. Have used this to setup and like to share.

Original: http://www.runningunix.com/2009/01/link-based-ipmp-setup-with-vcs/

We know that Solaris 10 allows us to setup link-based IPMP. Though a nice feature which the OS determines NIC availability solely on the NIC driver reporting the physical link status - UP or DOWN, setting link-based IPMP in VCS is a little different.

Previous versions used “probe-based” IPMP, where connectivity is tested by pinging something on the network from each interface.

While probe-based is actually a more thorough test (tests network layer 3 as well as 2), it is much more cumbersome to configure, and you need an extra IP address for each interface for “test” addresses. usually Link-based IPMP is sufficient for most applications.

For some reason, configuring link-based IPMP in VCS is somewhat tricky, and the documentation doesn’t seem to help much. It seems all the default values for VCS are for probe-based IPMP only.

To achieve link-based IPMP, There are values you need to change from the defaults in the MultiNICB resource:

Tells VCS to use mpathd for network link status
UseMpathd: 1

The default, /usr/sbin/in.mpathd is just incorrect - it doesn’t live there.
MpathCommand: /usr/lib/inet/in.mpathd -a

If you leave this at 1, it will overwrite your /etc/hostname.xxx files with probe-based IPMP configuration
ConfigCheck: 0

The “interface alias” for each device is not needed, leave them blank.
Device: (your IPMP interfaces here)

You want VCS to NOT ignore link status, since this is how link-based IPMP works.
IgnoreStatus: 0

Do not use your IPMP group name here, it’s not needed. VCS is not monitoring the group, mpathd is.
GroupName:

Here’s how it looks in main.cf:

MultiNICB csgmultinic (
UseMpathd = 1
MpathdCommand = “/usr/lib/inet/in.mpathd -a”
ConfigCheck = 0
Device = { ce0 = “”, ce4 = “” }
IgnoreLinkStatus = 0
)

How to Install another Veritas Demo License even though u have already two demo license extensions

Encountered this quite some time back, thought it would be nice to share this useful information.

i need to install an extension to the Veritas demo license even though i had already 2 Veritas demo licenses extensions.

While trying to install a Veritas demo license, encountered this error "ERROR V-21-1-19 Duplicate License key detected".

[root@myserver ~]# vxlicinst
Symantec License Manager vxlicinst utility version 3.02.16.0
Copyright (C) 1996-2006 Symantec Corporation. All rights reserved.
Enter your license key : XXXX-WDBK-ASG8-ST7O-ZZZZ-OCZW-YYY
vxlicinst ERROR V-21-1-19 Duplicate License key detected

Steps to resolve this.

Backup the current license
# cd /etc/vx/licenses/lic
# tar cfp vxlic.tar *.vxlic

Install the new license using vxlincinst again.

Do remember to verify that the demo license has been extended.

# vxlicrep

Restart VCS to verify that it can be started up using the new demo license.

Check how sendmail is configured

# /usr/lib/sendmail -d0.99 -bt < /dev/null

Using sendmail to check fully qualified hostname and permission

root@myserver:/>/usr/lib/mail/sh/check-hostname
Hostname myserver OK: fully qualified as myserver.

root@myserver:/>/usr/lib/mail/sh/check-permissions
No unsafe directories found.

How to resolve "Unable to qualify my own domain name" error

You may have received this error in /var/adm/messages

Symptom 1:
/var/adm/messages
Mar 13 11:15:16 myserver sendmail[8420]: [ID 702911 mail.crit] My unqualified host name (myserver.) unknown; sleeping for retry
Mar 13 11:15:16 myserver sendmail[8421]: [ID 702911 mail.crit] My unqualified host name (myserver.) unknown; sleeping for retry
Mar 13 11:16:16 myserver sendmail[8421]: [ID 702911 mail.alert] unable to qualify my own domain name (myserver.) -- using short name
Mar 13 11:16:16 myserver sendmail[8420]: [ID 702911 mail.alert] unable to qualify my own domain name (myserver.) -- using short name

Symptom 2:
when you try to manually telnet localhost 25 or mailx, either you can send to a particular domain or the respond is very slow after entering the "mail from" command.

Resolution:
update /etc/hosts ( and /etc/inet/ipnodes if you are on solaris 10) to use the following.

root@myserver:/>more /etc/hosts
# internet host table
#====================
127.0.0.1 localhost
10.106.127.105 myserver. myserver
10.106.63.32 myserver-rsc


#
# Internet host table
#
::1 localhost
127.0.0.1 localhost
10.106.127.105 myserver. myserver loghost

Description:
Usually i would add the "myserver." with the dot behind to /etc/hosts but in Solaris 10, things work a little differently.

It turn out that in solaris 10, the OS will go through /etc/inet/ipnodes for IPv4 address before going to /etc/hosts.

In this case, since ldap does not have the entry, OS will go straight to /etc/inet/ipnodes.

This also mean that if you change the host IP of the solaris 10 server, please change in /etc/inet/ipnodes as well otherwise you have conflict of IP addresses.

Note that this is Solaris specific.

# man ipnodes
...
...
NOTES
IPv4 addresses can be defined in the ipnodes file or in the
hosts file. See hosts(4). The ipnodes file will be searched
for IPv4 addresses when using the getipnodebyname(3SOCKET)
API. If no matching IPv4 addresses are found in the ipnodes
file, then the hosts file will be searched. To prevent
delays in name resolution and to keep /etc/inet/ipnodes and
/etc/inet/hosts synchronized, IPv4 addresses defined in the
hosts file should be copied to the ipnodes file.
...
..

# more /etc/nsswitch.conf
...
...
# consult /etc "files" only if ldap is down.
hosts: ldap [NOTFOUND=continue] files
...
...
# Note that IPv4 addresses are searched for in all of the ipnodes databases
# before searching the hosts databases.
...
...

How to Check the Product ID / Serial Number of the HP blade

This is similar to the sneep command in solaris.

To Check on the Product ID
root@myserver:~>#dmidecode |grep 'Product ID'
String 1: Product ID: 347957-B21

To Check on the Serial Number
root@myserver:~>#dmidecode |grep 'Serial Number'
Serial Number: XXXXXXXXXX (For the Blade Server)
Serial Number: YYYYYYYYYY (For the Blade Chasis/Enclosure)
Or

root@myserver:~>#hpasmcli -s "show server" | grep 'Serial'
Serial No. : XXXXXXXXXX

If not listed, then have to check the blade label physically... :)

Remove 169.254.0.0 routing in Linux

During the setup of Linux OS, you may see the special route even though you haven't added any,

[root@myserver ~]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.106.2.0 * 255.255.255.0 U 0 0 0 bond0
10.106.2.0 * 255.255.255.0 U 0 0 0 eth3
169.254.0.0 * 255.255.0.0 U 0 0 0 eth3
default 10.106.2.254 0.0.0.0 UG 0 0 0 bond0

This is the zeroconf route.

Though harmless, the less route you have, the more secure i suppose.

To disable the zeroconf route during system boot, edit the /etc/sysconfig/network file and add the following NOZEROCONF value to the end of the file.

NETWORKING=YES
HOSTNAME=localhost.localdomain
NOZEROCONF=yes

How to start HP Insight Diagnostics

Using HP Offline Insight Diagnostics using the SmartStart CD

Click on http://h18023.www1.hp.com/support/files/server/us/download/24733.html

Download the file to a temporary directory and extract the ISO image to that directory. To create a CD from this ISO image, you must use a software utility capable of writing an ISO file to a recordable media such as CD-R.

Note: Simply copying the ISO file to a CD-R will NOT result in a usable CD.

Then boot off the server using the SmartStart CD and then Click On Maintain Server
Then click on Diagnose Server
The Insight Diagnostics Window Opens Up
Change the 'view level' from the default 'Overview' to 'Advanced'.
Under 'categories' drop down box, select 'All'
Save the file as default on a floppy disk

Recently, i have encountered a few of the BL25p blades that cant even boot up. Then we won't be able to use this tool at all.

Do beware that HP support may insist that you patch, the firmwares, iLO, etc, which i think is a waste of time. Insist on a on-site engineer if you have the contract else likely its hardware issue. In my case, was a faulty mainboard.

How to find Temperature Range on Sun Blade [TM] and Sun Fire[TM] systems

To find out the temperature range and the current temperature on Sun Blade Desktop or a Sun Fire entry level server, we can try this

# prtpicl -v -c temperature-sensor

This will extract the information from FRU and display on screen. DO NOTE that the info in the FRU is set at manufacturing, and can varies from CPU speed to CPU type. It should not be assumed that there is a default temp for all CPUs.

e.g.

# prtpicl -v -c temperature-sensor
CPU0_DIE_TEMPERATURE_SENSOR (temperature-sensor, 3a00000579)
:PSVC_FRU (3a000006ebTBL)

| (3a0000058dH) |
:PSVC_PHYSICAL_DEVICE (3a00000678TBL)

| (3a000005a1H) |
:PSVC_PARENT (3a000005ebTBL)

| (3a0000049dH) |
:PSVC_TS_OVERTEMP_LED (3a000005b8TBL)

| (3a000004e1H) |
:HighShutdownThreshold 95
:HighWarningThreshold 93
:LowShutdownThreshold -10
:LowWarningThreshold 0
:Temperature 59
:FaultInformation NO_FAULT
:State OK
:_class temperature-sensor
:name CPU0_DIE_TEMPERATURE_SENSOR
CPU1_DIE_TEMPERATURE_SENSOR (temperature-sensor, 3a00000583)
:PSVC_FRU (3a000006eeTBL)

| (3a00000592H) |
:PSVC_PHYSICAL_DEVICE (3a0000067bTBL)

| (3a000005a6H) |
:PSVC_PARENT (3a000005eeTBL)

| (3a000004a2H) |
:PSVC_TS_OVERTEMP_LED (3a000005bbTBL)

| (3a000004e1H) |
:HighShutdownThreshold 95
:HighWarningThreshold 93
:LowShutdownThreshold -10
:LowWarningThreshold 0
:Temperature 57
:FaultInformation NO_FAULT
:State OK
:_class temperature-sensor
:name CPU1_DIE_TEMPERATURE_SENSOR

Here, system will send out alert upon 93 deg C, shutdown upon 95 deg C. Current temperature is 57 deg C.

How to Display IML logs without going to iLO

while logged into the blade, fire up your favourite terminal and try the following.

# /sbin/cpqimlview
# /sbin/hplog -v
# /sbin/hplog -V = view the entire log starting at entrynum