Oracle Database Listener not starting due to TNS-00525

Recently hit with this issue while starting the oracle db listener on AIX.

TNSLSNR for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production System parameter file is /opt/oracle/product/11.2/mydb/network/admin/listener.ora
Log messages written to /opt/oracle/diag/tnslsnr/myserver/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=myserver)(PORT=1234)))
Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC0)))
TNS-12555: TNS:permission denied
 TNS-12560: TNS:protocol adapter error
  TNS-00525: Insufficient privilege for operation
   IBM/AIX RISC System/6000 Error: 1: Not owner

Listener failed to start. See the error message(s) above...

Suspecting that the cause of this was because i was configuring the powerHA in AIX and trying to start the DB. Thanks to blauecorsa, this leads me to finding the root cause. Some where in the "truss -aedfo result.txt lsnrctl start" shows some ECONNREFUSED errors. I checked and found both the /var/tmp/.oracle and /tmp/.oracle are being owned by root:system as below.

root> ls -l /tmp/ | grep oracle
drwxrwxrwt    2 root   system             256 Sep 13 13:03 .oracle

root> ls -l /var/tmp/ | grep oracle
drwxrwxrwt    2 root   system             256 Sep 07 13:58 .oracle

so we can confirm that powerHA started the DB using the startup / shutdown script using root account. The startup / shutdown scripts were modified to su - oracle -c "" and we are back in business. last question, can powerHA assume oracle account to start / shutdown the database so that no change to the scripts is needed?

Listing and Verifying SSL Certificate using openssl

How to check the private key

Using openssl command,

# openssl rsa -in myserver.key -check -noout
RSA key ok

How to check SSL certificate (*.crt format) using openssl command

Using openssl command

 # openssl x509 -in myserver.crt -text -noout | more

To get only the validity dates

 # openssl x509 -in myserver.crt -text -noout | grep "Not"
            Not Before: Apr 24 14:35:54 2010 GMT
            Not After : Jul 27 04:17:47 2011 GMT

How to check SSL certificate (*.cer format)

Using openssl command,

 # openssl x509 -inform der -in myserver.cer -text | more 

To get only the validity dates,

 # openssl x509 -inform der -in myserver.cer -text | grep "Not"
            Not Before: Nov 14 05:44:26 2008 GMT
            Not After : Nov 14 05:54:26 2010 GMT

SSL Certificate Monitoring

Here's a guide on how to setup monitoring for SSL certificates

It is important to ensure that the SSL certificates used in services that are fronting users or for secure communication are valid otherwise, we risk service outage because of expired certificates.

The SSL monitoring script

Using the monitoring script "SSL Certificate Check" written by Matty, we can use it to monitor the SSL certificates either by verifying the certificate itself or querying the status through the application services.

Link to detailed documentation at SSL Certificate Check

How to set it up.

I am using the script v3.21 dated Oct 2010 in the example.

1) Download at SSL Certificate Check Script

2) Deploy it to a suitable location. Give it execute permission at least 0700.

I have configured the script to the following

# Who to page when an expired certificate is detected (cmdline: -e)
ADMIN="admin@myserver.com"

# Number of days in the warning threshhold  (cmdline: -x)
WARNDAYS=100

# If QUIET is set to TRUE, don't print anything on the console (cmdline: -q)
QUIET="FALSE"

# Don't send E-mail by default (cmdline: -a)
ALARM="TRUE"

# Don't run as a Nagios plugin by default (cmdline: -n)
NAGIOS="FALSE"

where the script will notify via email (default) when the certificate has less than 100 days of validity. It will print out to console. If you don't need it, change QUIET to "TRUE".

If you require to override the default settings in the script, you can use the following switches

#./sslcertcheck
Usage: ./sslcertcheck [ -e email address ] [ -x days ] [-q] [-a] [-b] [-h] [-i] [-n] [-v]
       { [ -s common_name ] && [ -p port] } || { [ -f cert_file ] } || { [ -c certificate file ] }

  -a                : Send a warning message through E-mail
  -b                : Will not print header
  -c cert file      : Print the expiration date for the PEM or PKCS12 formatted certificate in cert file
  -e E-mail address : E-mail address to send expiration notices
  -f cert file      : File with a list of FQDNs and ports
  -h                : Print this screen
  -i                : Print the issuer of the certificate
  -k password       : PKCS12 file password
  -n                : Run as a Nagios plugin
  -p port           : Port to connect to (interactive mode)
  -s commmon name   : Server to connect to (interactive mode)
  -q                : Don't print anything on the console
  -v                : Only print validation data
  -x days           : Certificate expiration interval (eg. if cert_date < days)

Requirements.

mktemp package needs to be available in the server.

Usage

1) Running the script against the certificate file.

$ sslcertcheck -c /etc/httpd/conf/ssl.crt/abc.pem
Host                                            Status       Expires      Days Left
----------------------------------------------- ------------ ------------ ----------
FILE:/etc/httpd/conf/ssl.crt/abc.pem            Valid        Jan 2 2010   807   

sslcertcheck will print the file or hostname in the first column, a value to indicate if the certifciate is valid in the second column, the date the certificate will expire in the third column, and the number of days remaining until the certificate expires in the fourth column.

2) If you do not have local access to the certificate files, you can use sslcertcheck's network connectivity option to extract the certificate expiration date from a live server. To check when the certificate used by the web server will expire, the server name or IP address and a port number can be passed to sslcertcheck's "-s" (server name) and "-p" (tcp port) options:

#./sslcertcheck -s 172.21.41.136 -p 443

Host                                            Status       Expires      Days
----------------------------------------------- ------------ ------------ ----
172.2.1.1:443                               Valid        Jul 24 2011  128

3) You may want to manage dozens of SSL-enabled servers, you can place the server names and port numbers in a file, and run sslcertcheck against that file:

The configuration file.

$ cat sslcertcheck.cfg
10.10.8.1 443
10.10.8.7 443
172.2.1.1 443

The output from the script with setting whom to email when any entry has validity less than threshold.

# ./sslcertcheck -e admin@me.com -f sslcertcheck.cfg 

Host                                            Status       Expires      Days
----------------------------------------------- ------------ ------------ ----
10.10.8.1:443                                 Valid        Nov 14 2013  972
10.10.8.7:443                                 Valid        Jan 20 2014  1039
172.2.1.1:443                               Valid        Jul 24 2011  128

Thats all folks!

Basic networking TCP test using telnet

When telneting to an IP at a given port, there are various telnet responses. Knowing the difference in telnet responses could easily point you in the right direction when a telnet to a host on a particular port in unsuccessful.

There are a distinct differences in getting ‘refused’ or ‘timeout’ responses.

You will get a connection refused message for one of the following reasons:

• The application you are trying to test hasn’t been started/installed on the remote server.
• There is a firewall rejecting the connection attempt by terminating the connection setup.

Example output from a Linux box:

$ telnet server2 7063
Trying 172.1.1.1...
telnet: connect to address 172.1.1.1: Connection refused
telnet: Unable to connect to remote host: Connection refused

The similar Connection refused message from a Solaris box :

$ telnet server3 7055
Trying 172.2.1.1...
telnet: Unable to connect to remote host: Connection refused

The Connect failed message is the equivalent but from a Windows box :

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\vickwan>telnet 172.2.1.1 7062
Connecting To 172.2.1.1...Could not open connection to the host, on port 7062: Connect failed

The telnet command will abort the attempted connection after waiting a predetermined time for a response. This is called a timeout response.

In some cases, telnet won’t abort, but will just wait indefinitely. This is also known as hanging. These symptoms can be caused by the one of the following reasons:

• The remote server doesn’t exist on the destination network. It could be turned off.
• The could be a routing issue, either the request or the response never gets to the destination.
• A firewall could be blocking the connection attempt, causing it to timeout instead of being quickly refused.

Here is an example of the output:

$ telnet server3 7055
Trying 172.2.1.1...
telnet: connect to address 172.2.1.1: Connection timed out
telnet: Unable to connect to remote host: Connection timed out

The script, command file and input file.

Reference Adapted from : http://blog.ru.co.za/2009/09/29/telnet/

This little script is written to helps cut down time needed to test if ACL allows connection from server A to server B at a given port. It will attempt to suggest the remedy actions.

Script tested on AIX 6.1, AIX 7.1, RHEL 4.6 and Solaris 9.

#!/bin/ksh
# Written By   : Victor Kwan
# Written On   : 25 Oct 2009
# EMAIL   : victorkk [AT] gmail [DOT] com
# Description  : Utility to test TCP ACL via telnet
# Updated On   : 27 Oct 2009 : Attempt to interpret telnet response.
#              : 09 Mar 2011 : Test if telnet command is executable
#              : 07 Apr 2011 : Support AIX, Improve code to be not chatty and terminate telnet session properly.

FILE=${1}
OUTPUTFILE="$0.output"
LOGFILE="$0.log"
TELNETCMD="$0.telnetcmd"
TELNET=`which telnet`
CAT=`which cat`
ECHO=`which echo`
OS="`uname -s`"

#UNIX Normal "Connection to 10.106.50.10 closed."
#UNIX No route "No route to host"
#UNIX Conn refused "telnet: Unable to connect to remote host: Connection refused"
#UNIX timed out "telnet: Unable to connect to remote host: Connection timed out"

RESPONSE_NORMAL="gn host."
RESPONSE_NO_ROUTE="to host"
RESPONSE_CONN_REFUSED="refused"
RESPONSE_TIMED_OUT="med out"

#AIX Normal "Connection closed."
#AIX No route "No route to host"
#AIX Conn refised "telnet: connect: A remote host refused an attempted connect operation."
#AIX timed out "telnet: connect: A remote host did not respond within the timeout period."

AIXRESPONSE_NORMAL="Connection closed."
AIXRESPONSE_NO_ROUTE="No route to host"
AIXRESPONSE_CONN_REFUSED="connect operation."
AIXRESPONSE_TIMED_OUT="he timeout period."

THISRESPONSE_NORMAL="$RESPONSE_NORMAL"
THISRESPONSE_NO_ROUTE="$RESPONSE_NO_ROUTE"
THISRESPONSE_CONN_REFUSED="$RESPONSE_CONN_REFUSED"
THISRESPONSE_TIMED_OUT="$RESPONSE_TIMED_OUT"

COLOR_BLUE="\033[0;34m"
COLOR_GREEN="\033[32m"
COLOR_RED="\033[31m"
COLOR_BRIGHTRED="\033[1;31m"
COLOR_WHITE="\033[0m"
COLOR_BRIGHTWHITE="\033[1;37m"

if [ ! -x $TELNET ]
then
        echo "${COLOR_BRIGHTRED}Telnet command is not executable!!${COLOR_WHITE}"
        echo "${COLOR_WHITE}Script will now terminate.${COLOR_WHITE}"
        exit
fi

echo "Commence telnet test based on [$FILE] file."
echo

cat ${FILE} | grep -v "#" | while read LINE do {
        IP=`echo $LINE | awk -F: '{print $1}'`
        PORT=`echo $LINE | awk -F: '{print $2}'`

        ($CAT $TELNETCMD) | $TELNET $IP $PORT >> $OUTPUTFILE 2>&1

        RESPONSE=`tail -1 $OUTPUTFILE | tr -d "\r" | tr -d "\n"`
	if [ "$OS" = "SunOS" ]
	then
	{
		STR_TO_CMP=`echo "$RESPONSE" | awk '{print substr($0,length-7)}'`
	}
	elif [ "$OS" = "AIX" ]
	then
	{
		STR_TO_CMP=`echo "$RESPONSE" | awk '{print substr($0,length-18)}'`
		THISRESPONSE_NORMAL="$AIXRESPONSE_NORMAL"
		THISRESPONSE_NO_ROUTE="$AIXRESPONSE_NO_ROUTE"
		THISRESPONSE_CONN_REFUSED="$AIXRESPONSE_CONN_REFUSED"
		THISRESPONSE_TIMED_OUT="$AIXRESPONSE_TIMED_OUT"
	}
	fi

        if [ ! "$STR_TO_CMP" = "$THISRESPONSE_NORMAL" ]
        then
        {
                echo "Telnet ${COLOR_BRIGHTRED}FAILED${COLOR_WHITE} for ${COLOR_BRIGHTWHITE}$IP:$PORT${COLOR_WHITE}."
                echo "${COLOR_BRIGHTRED}Error Message${COLOR_WHITE} : [$RESPONSE]!"

                if [ "$STR_TO_CMP" = "$THISRESPONSE_NO_ROUTE" ]
                then
                {
                        echo "${COLOR_GREEN}Suggestion${COLOR_WHITE}: Check routing at both source and destination"
                }
                fi

                if [ "$STR_TO_CMP" = "$THISRESPONSE_CONN_REFUSED" ]
                then
                {
                        echo "${COLOR_GREEN}Suggestion${COLOR_WHITE}: Destination may not be listening, routable or firewall is blocking the connection."
                }
                fi

                if [ "$STR_TO_CMP" = "$THISRESPONSE_TIMED_OUT" ]
                then
                {
                        echo "${COLOR_GREEN}Suggestion${COLOR_WHITE}: Destination may not be listening, routable or firewall is blocking the connection."
                }
                fi
        }
        fi
     	echo "Done for $IP:$PORT."
        echo " "
}
done
echo "Telnet test ends."

For the input file, e.g. IP_PORT It is okay to have commented lines as the script will ignore them.

~$more IP_PORT
#WLS
server2:7003
server2:7004
server2:7022
server2:7023
...
...

For the command file, the 2 telnet control commands must be used.

~$ more testACL.telnetcmd
^]
quit

Final outcome. Output may look similar to the following. No output for telnet success.

> ./testACL_telnet.ksh IP_PORT
Commence telnet test based on [IP_PORT] file.
Telnet FAILED for server4:7053.
Error Message : [telnet: Unable to connect to remote host: Connection refused]!
Suggestion: Destination may not be listening on this IP and Port, routable or firewall is blocking the connection.
...
...
...
telnet test ends.

Swap partition in AIX

By default, swap space is defined as 512 Mb which is a little bit low when oracle, websphere or any other enterprise application would be running. Hence, it will be important to increase the swap space before the installation of these applications.

 lsps -a
Page Space      Physical Volume   Volume Group Size %Used Active  Auto  Type Chksum
hd6             hdisk0            rootvg       512MB     1   yes   yes    lv     0

AIX put swap space on hd6 by default.

To increase the swap space, use 'chps -s xxx hd6' command where xxx mean by how many PP you want to increase.

lsps -a
Page Space      Physical Volume   Volume Group Size %Used Active  Auto  Type Chksum
hd6             hdisk0            rootvg       12480MB     1   yes   yes    lv     0

swap -l
device              maj,min        total       free
/dev/hd6              10,  2     12480MB     12448MB

tmp in AIX

We know that /tmp is using tmpfs which is essentially swap in solaris. However, in AIX, /tmp is using the usual file system 'jfs2'. This means that whatever is placed in /tmp/ would not be wiped out after reboot as the space really have a allocated disk space.

AIX put /tmp on hd3 by default.

df -k | grep tmp
/dev/hd3          6291456   2706240   57%     4993     1% /tmp

See that the files from 2010 are still around.

ls -lt /tmp/ | tail -5
d-w-------    2 root     system          256 Nov 12 12:12 errmbatch
-rw-r--r--    1 root     system          530 Nov 12 11:33 IBM.CSMAgentRM_dr.sh.dbg
-rw-------    1 root     system            0 Nov 12 11:30 .strload.mutex
-rw-r--r--    1 root     system          406 Nov 12 11:26 .sr_migrate.log
drwx------    2 root     system          256 Nov 12 10:41 lost+found

Uptime is only 15 days, which prove that the above files from 2010 are really not wiped out.

uptime
  02:59PM   up 15 days,   3:03,  3 users,  load average: 1.50, 1.77, 1.87

Timestamp for the sake of completeness.

date
Thu Apr  7 14:59:55 GMT+08:00 2011

Good or bad? i don't think this is critical to the performance nor operation of the system hence i recommend to leave it alone.

Why SSHD account cannot be removed

In modern SSHD, the privilege separation security feature is provided to allow SSHD to create unprivileged child process to deal with incoming network traffic. After successful authentication, another process will be created that has the privilege of the authenticated user. Privilege separation is to prevent privilege escalation by containing any corruption within the unprivileged processes.

Default setting in SSHD is 'yes', meaning its enabled. Hence, the account 'sshd' account with 'sshd' group is required by SSHD.

How to Remove Unwanted route to 169.254.0.0 in RHEL Linux

Every time the system boots, You may have seen the following with the route to 169.254.0.0.

# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.10.2.0      *               255.255.255.0   U     0      0        0 bond0
10.10.2.0      *               255.255.255.0   U     0      0        0 eth3
169.254.0.0     *               255.255.0.0     U     0      0        0 eth3
default         10.10.2.254    0.0.0.0         UG    0      0        0 bond0

This is the zeroconf route (169.254.0.0). You manually disable it by turning off the firewall and remove the route with 169.254.0.0 / 255.255.0.0 using the route command.

Permanent Solution:

To disable the zeroconf route during system boot, edit the /etc/sysconfig/network file and add the following NOZEROCONF value to the end of the file:

NETWORKING=YES
HOSTNAME=localhost.localdomain
NOZEROCONF=yes

Layman Explanation:

Zeroconf, or Zero Configuration Networking, is a set of techniques that automatically create a usable IP network without configuration or special servers.

This allows inexpert users to connect computers, networked printers, and other network devices and expect a functioning network to be established automatically. Without Zeroconf, a user must either set up special services, like DHCP and DNS, or set up each computer's network settings manually, which may be challenging for non-technical or novice users.

Additional Information: wiki more about zeroconf

How to transfer MQ objects from one site to another using saveqmgr

Question

You used the SupportPac MS03 to save your queue manager object definitions to a file that you named TEST.TST. You want to use the saveqmgr output file (TEST.TST) to create the same queue manager object definitions on a new server machine.

Whats the process to redefine your objects?
First, copy the saveqmgr output file (TEST.TST) onto the new server machine.

Create the queue manager

crtmqm TEST

Start the queue manager

strmqm TEST

Define your objects

runmqsc TEST < TEST.TST

where TEST is the name of the new QMGR and TEST.TST is the name of the queue manager object definitions file. Reference: MQ Reference

Simple AIX LVM concept and Disk structure

The LVM controls everything here through logical volume device drive. It takes a complex structure of PP, including mirroring, and presents a simple LP to the application.

Physical Volume (hdiskX)

Each physical disk is called a physical volume (PV).

Every PV in use belongs to a VG unless the PV is used as raw storage device or a hot spare.
Each PV contains 1 or more disks (or platters) stacked on top of each other.

PV is also known as direct access storage devices (DASDs). It can be fixed or removable.
A block (sector) is a continuous 512-byte region on PV that corresponds in size to a DASD sector.
A Partition is a set of blocks (with sequential cylinder, head, and sector numbers) contained within a single PV.

Physical Parition

Each PV is divided into PP of a fixed size (e.g. 4Mb partition size).
This is the smallest unit a PV can have.

Volume Group (rootvg, datavg)

Each VG is a collection of 1 or more PV.

A VG can consist of any mixture of physical disk types, though performance is most consistent if devices of the same size are used in a single VG.

PV cannot be shared between VG, therefore entire PV becomes part of the VG.

Each system can have up to 32 VG only.
Information about all LVs and PVs within a VG are stored in VGDA area.
first 64K of a PV is reserved for this area as defined in .
The VGDA consists of bootrecord (first 512 bytes), bad block directory () and LVM record ()

Informations about which PP are stale and which PV are missing within a VG is stored in VGSA area.
The LVM adn SCSI driver reserves somewhere between 7 - 10% of the available disk space for LVM maps, etc.

Lets go deeper.

With each VG, 1 or more LV are defined.

Logical Volume (hd1, hd2var, etc)

Each LV group together 1 or more Logical Partition (LP).

LV is an area of disk used to store data, appearing contiguous to the application but may not be contiguous on the actual PV.
A file system resides on top of an LV, there only 1 LV is mapped to a file system.
LV reside only within VG.
LV can span on multiple PV within the VG.
LV cab also have mirrored copies on different PV within the VG.
Each VG can contain up to 255 LV.

All system LV should reside in the VG named rootvg, under 1 PV if possible. This will allow complete re-installation of the system from backup without affecting the application data that resides on other VG.

There is no difference in response time between a single or multiple VG. When a LV is accessed, the ODM db (/etc/objrepos/*) is searched to determine which VG the LV belongs to, then the VGDA for the VG is read to determine the physical placement of the LV on that PV and ultimately the physical track and sector where the data resides.

Logical Partition

Each LP contains 1 or more physical partition (PP).

If LV is mirrored, then additional PP are allocated.
These mirrors usually reside on different PV (for availability) but may reside on same PV (for performance).

Up to 2 copies of PP can be mirrored, where you have LP count of 3 (2 mirrors plus original)
Space Region in physical Volume

For space allocation purposes, each PV is divided into 5 regions, namely the outer_edge, out_middle, center, inner_middle, and inner_edge. These can be viewed as cylindrical segments cut perpendicularly through the disk platters. The number of physical partitions in each region depends on the total capacity of the disk drive.

AIX Useful commands

For my own references

Commands for memory

shows how much RAM does my machine has (as root)

bootinfo –r 

shows how much RAM does my machine have (as non root)

lsattr –E –l sys0 –a realmem

Simulates a system with various sizes of memory for performance testing of applications.

To set the memory size to 512 MB

rmss -c 512

To reset the memory size to the original one

rmss -r 

Commands for Disk / file system

Shows all disks

lsdev -Ccdisk

Gives info about the disk manufacture type

lscfg -vp -l hdisk0 | grep Machine

Displays disk size (even if no volume group is assigned)

bootinfo –s hdisk0

Displays information about a physical volume within a volume group (Display PP's used, location on disk, mount point)

lspv -p hdisk0

List of Filesystem items of jfs2 file system type.

lsfs -v jfs2  

List the parameter the filesystem of jfs2 file system type

lsfs -q -v jfs2

From here, we can see if e.g. /backup was or is a big_filesystem_enabled one. Important for the 2GB File limit.

Displays all exported volumes

showmount –e    

Show who's got my file systems mounted over IP !

showmount -a

Shows all mounted filesystems (nfs+local)

mount

Synchronize the device configuration database (ODM) with the logical volume manager information for rootvg

synclvodm -vP rootvg

Redfined VG definition in ODM

redefinevg rootvg

Reads logical volumes info from disk

lqueryvg -p hdisk0 –Avt  

Increases /var FS by about 100MB (each unit is 512 bytes)

chfs -a size=+200000 /var

Mounts all FS stated in /etc/filesystems

mount all

Unmounts a FS

umount /mnt/

Identifies processes using a file or file structure

fuser –c /dataVolumes/test

Releases a CD that will not unmount by sending SIGKILL signal to each local process. Must be root.

fuser -k /dev/cd0

Shows when the file was last created/modified/accessed

istat 

Reports input/output statistics for logical partitions, logical volumes and volume groups every 2 seconds

lvmstat –v hd2 2

Enable / Disable LVM statistic collection

lvmstat -e hd2

lvmstat -d hd2

Commands for Network

Displays en0 driver params

lsattr  -El  en0

Displays ent0 Hardware params

lsattr  -El  ent0

Display specific device specified by logical name ent0

lscfg  -v -l ent0

Display firmware device tree for the corresponding node to the ent0 device

lscfg -vp -lent0

Displays en0 driver params

ifconfig en0

Displays network interfaces setting

netstat  -i

Display all interface information - host:addr:mask:_rawname:nameserv:domain:gateway:cost:activedgd:type:start

for i in `lsdev | grep en | grep "Standard Ethernet" | awk '{print $1}'`; do echo Checking $i;mktcpip -S $i; done

Displays all IP oriented processes status

lssrc –g tcpip

Display any communication errors on ent2

entstat -drt ent2 |grep –i error

Resets all the statistics back to their initial values.

entstat -r

Shows a local arp cache

arp -a

Shows IP statistics

netstat  -ptcp

Shows UDP statistics

netstat  -pudp

Show network buffer cache statistic

netstat  -c

Show network statistic for each protocol

netstat  -s

Shows statistics recorded by the memory management routines

netstat -m

Trace en2 every 3 seconds

netstat -I en2 3

Ping with displaying the routing info

ping -R -c 1 myserver

Display routing info

netstat -rn

Shows the state of all configured interfaces

netstat -in

Display routing info with full hostnames

netstat -r

Trace all hobs (interconnections=routers) to the destination IP

traceroute 149.115.39.1

Monitors activity and reports statistics on network I/O and network-related CPU usage.

To start

netpmon -o netpmon.out

To Stop

trcstop

Displays a fully qualified domain name of a host

namerslv -s | grep domain | awk '{ print $2 }'

Shows the status of a remote host on the local network

rup myserver

Commands for Tape Device

Displays tape params

lsattr -El rmt0

all information about a tape drive

lscfg -vp -l rmt0

Shows all tapes lsdev -Cctape

Commands for NFS

Resets NFS stats without reboot

nfsstat  –z

To stop / start NFS services on a client

stopsrc –g NFS

startsrc –g NFS

Mount an NFS filesystem

mount hostname:/filesystem /mount-point

Exports a directory to NFS clients

mknfsexp -d /directory

Mounts a directory from an NFS server

mknfsmnt

Changes the configuration of the system to stop running NFS daemons

rmnfs 

Configures the system to run NFS

mknfs

Un-exports a filesystem

exportfs -u (filesystem)

Lists all exported filesystems

exportfs

Exports all directories listed in /etc/exports file

exportfs -a

Commands for Devices

Displays system type, firmware, etc driver params

lsattr  -El  sys0

Lists all system HW config (NVRAM)

lscfg –v

List all scsi devices

lsdev –Csscsi

List all pci devices

lsdev –Cspci

List all scsi adapters

lsparent –Ck scsi

List fiberchannel devices

lsdevfc

Configures devices

cfgmgr

This is similar to devfsadm in Solaris Specifies verbose output. The cfgmgr command writes information about what it is doing to standard output.

cfgmgr -v -l device –v

Name Specifies the named device to configure along with its children.

cfgmgr -v -l device

If you only turned on a disk tower at e.g. scsi2 cfgmgr -v -l scsi2 will only configure this with detailed output.

Commands for Graphic adapter

To check which graphic adapter is installed.

lsdisp

List all information about a adapter

lscfg -vp -l mga0

Commands for boot

Change the default bootlist

bootlist -m normal cd0  rmt0 hdisk0

Commands for Firmwares

Display the system firmware level and service processor

lsmcode -c

Display the adapter microcode levels for a RAID adapter scraid0

lsmcode -r -d scraid0

Display the microcode level for all supported devices

lsmcode -A

Commands for System Information

Get machine ID

/usr/bin/uname -m

Get platform type

/usr/bin/uname -M

Determines the system serial number

lsattr -El sys0 -a systemid | awk '{print $2}' | awk -F, '{print $2}'

Displays current AIX level

oslevel

Displays current AIX maintenance level

oslevel -r

List filesets at levels later than maintenance level !!!

oslevel -g

List all information about a processors

lscfg -vp -l proc0  (1,2,3)   

List all information about memory modules installed

lscfg -vp -l mem0 |pg

Determines the system Firmware level

lscfg –vp|grep ROM|grep -v CD  

Check last system dump status

sysdumpdev -L

Check system dump device settings

sysdumpdev -l

List contents of the package

lslpp -f Upd_Timna_DTM.obj

Shows create/modify/access file info

istat 

Display system boot log

alog -o -t boot | more

Identifies the users currently logged in

who

Shows recent 5 lines

last –5

Shows username ‘root’ login/logout record

last root

Displays local system statistics in interactive mode and records system statistics in recording mode

nmon

The nmon command provides the following views in interactive mode:

    * System resource view (using the r key)
    * Process view (using the t and u keys)
    * AIO processes view (using the A key)
    * Processor usage small view (using the c key)
    * Processor usage large view (using the C key)
    * Shared-processor logical partition view (using the p key)
    * NFS panel (using the N key)
    * Network interface view (using the n key)
    * WLM view (using the W key)
    * Disk busy map (using the o key)
    * Disk groups (using the g key)
    * ESS vpath statistics view (using the e key)
    * JFS view (using the j key)
    * Kernel statistics (using the k key)
    * Long term processor averages view (using the l key)
    * Large page analysis (using the L key)
    * Paging space (using the P key)
    * Volume group statistics (using the V key)
    * Disk statistics (using the D key)
    * Disk statistics with graph (using the d key)
    * Memory and paging statistics (using the m key)
    * Adapter I/O statistics (using the a key)
    * Shared Ethernet adapter statistics (using the O key)
    * Verbose checks OK/Warn/Danger view (using the v key)
    * Detailed Page Statistics (using the M key)
    * Fibre channel adapter statistics (using the ^ key)

Reports selected local and remote system statistics

topas

Monitors system 10 top processes with 2 seconds

monitor -top 10 -s 2

Displays disks activity every 2 seconds refresh interval

iostat 2

Displays the adapter and disk throughput report every 2 seconds

iostat –a 2

Monitors virtual memory statistics every 2 seconds

vmstat 2

Show all CPU’s activity on an SMP machine 2 times separated by 2 seconds.
Each row is for 1 logical processor.
'U' row is for system-wide Unused capacity.
Last line is for system-wide statistics.

sar –P ALL 2 2

Monitors real and virtual memory

svmon –i 2

Shows top 10 memory usage by process

ps auxw | sort –r +3 |head –10  

Shows top 10 CPU usage by process

ps auxw | sort –r +2 |head –10

Traces FS,LV,disks,files activity of a “find” command into a logfile (filemon.out). Must be preceded by a trcstop command.

filemon –O all –o filemon.out ; find / -name core ; trcstop

Traces CPU activity of a “find” command Several logfile are created. Must be preceded by a trcstop command.

tprof –x find / -name core ; trcstop

Shows the DNS server name and address

nslookup hostname

Trace CPU activity for next 30 seconds. Results in file sleep.tprof

tprof -ske -x "sleep 30"

Commands for paging.

Paging space settings.

lsps -a

Commands for user accounts

Environment setings - show user ulimit

env ulimit

Shows all user parameters (max .file size,etc)

lsuser –f root

Lists login users and their programs.

w

Commands to browse errlog

Generates a report of logged errors. Default command

errpt

All details

errpt -a

Moderate details

errpt -A

Eliminates double entries

errpt –D

Browse the errlog in detail for all errors within a timeframe

errpt -a  -s 0604090601  -e 0605090901

where date format is mmddhhmmyy Generates a report of resource names specified by the ResourceNameList variable (SYSPROC)

errpt -a  -N SYSPROC |more

Browse the errlog by the identifier

errpt -j 5DFED6F1

>h4>Misc Find all files containing "10.10.10.13" IP address

ksh
find / -type f|xargs grep "10.10.10.13" 2> /dev/null

Compresses the files while keeps the original

compress -c file > file.Z

Returns full path of program (Similar to 'which' command in Solaris / Linux. Also available in AIX)

whereis  

Finding out which HMC your server is attached to

Find out which HMC your server is attached to (in AIX 6.1 / 7.1)

bash-3.2# lsrsrc IBM.MCP
Resource Persistent Attributes for IBM.MCP
resource 1:
        MNName           = "10.10.10.15"
        NodeID           = 4935249694248815870
        KeyToken         = "10.10.10.10"
        IPAddresses      = {"10.0.3.20","10.10.10.15","10.10.10.16","fe80::e62f:13ff:f231:6218","fe80::e62f:13ff:fe33:24b0","fe80::e62f:13ff:fe33:24b2"}
        ActivePeerDomain = ""
        NodeNameList     = {"myserver"}

In this example, we see that myserver is connected to HMC via 10.10.10.15 and 10.10.10.16.

How a small difference in NFS can cause sleepless night

Setting up NFS to share to non-AIX hosts

I wanted to setup NFS between AIX and non-AIX systems using NFS. being more familiar with NFS setup in Solaris and Linux, i thought setup would be a breeze.

what went wrong

AIX is very strict on who can access the partition, i.e. root. Otherwise, even though the NFS share is created but you will not be able to mount the partition from the other hosts.

Also ensure that both NFS server and client can use the same security protocol, otherwise the result is same as above. As the NFSv3 is the more commonly used protocol, it is recommended for the current setup.

Lastly, put in explicitly who can read-write or read-only to control access.

How NPIV can save on fibre cabling for SAN

What is NPIV

NPIV, which stands for N-Port ID Virtualisation is a fibre channel facility allowing multiple N-Port IDs to share a singale physic N-Port. Hence this allows multiple fibre channel to occupy a single physical port, easing hardware requirements in the SAN design.

It is noted that NPIV is an extension to a standard already defined in the fibre channel protocols that allow one to get past single initiator/single target design limitations.

In order to take advantage of this, both the HBA card (from the host and SAN array) and the switch must support NPIV to generate and publish an additional WWPN in a virtual fashion.

Why is it good

Traditionally, we provide at least 2 fibre link for each host, 1 link on 1 controller which is connected to 1 SAN switch in the production environment. With 4 LPARs in the p7 server requiring SAN connection, potentially, we need at least 8 fibre links with 8 SAN ports allocation. Additional links and SAN ports are needed to connect to the SAN arrays.

With the NPIV protocol, we can use just 2 fibre links to connect between the SAN switch and the p7 server. This is a savings of 75%!

Why it may be bad

In the event that there are lots of host sharing the same fibre link, the nightmare of link failure will be catastrophic. It may be mitigated by having 2 HBA controllers with 2 links each, and distributed connection to 2 different SAN switches.

How Disk Expansion on AIX can fail

IBM has been boasting about the ease of disk space management on its AIX hosts. Sure, we are able to expand the file system on AIX for many occasions. i.e. /opt, /home, /tmp and etc. However, trying on /opt/oracle/oradata failed. Let see why.

The story.

my oravg was full due to rapid expansion of business, hence needed more space. So no impact is expected as disk space expansion should be on the fly. I have personally tried on /tmp, / and /var, so i am pretty confident that this is OK.

Expansion was completed but from "df" command, we cant see the expected changes. Then the phone started ringing. Oracle DB has crashed.

Binary installation was still intact but the DB is gone.

Troubleshooting and recovery

fsck showed that the "bit allocation map" is corrupted. Tried to creat another VG and try to copy the LV over to the new VG, trying to see if the inodes, files, links and bit allocation map can be recreated. The map couldn't be recreated.

Searching in google did not yield any useful information other than creating snapshot to backup the system or apply snapshot to recover the system. So this is a provision for quick recovery rather than what i need.

It seems that oracle DB or IBM DB2 are "aware" of the disk physical boundaries and when the boundaries are changed, they do not know how to react and hence crashed. In addition, there was no space for the OS to process the file system expansion since it was 100% full. The next recovery step is to wipe the oravg clean and rebuild the DB.

The painful thing was that the building of DB took more than 3 days and this rebuild took another weekend. ouch.

The take away

Before doing any disk expansion for DB or on any other disk where the applications running off the disk are "aware" of the disk setup, please shut it down before doing any disk changes. In addition, should clear up some space so that the OS has some leeway to process the file system expansion on the fly.

I'm lucky that this is a trial setup.

So do not believe blindly that disk space management is on the fly without strings attached. There are bound to be some obscure conditions that will break it.

Murphy law - "Anything that can go wrong, will go wrong"

My thoughts on how can 802.1q can save on network cablings and potential problems.

IEEE 802.1Q or commonly known as VLAN tagging is a networking standard for sharing of physical Ethernet network link by multiple independent logical networks.

The protocol works with the MAC layer and Spanning Tree Protocol (802.1D) to allow nodes / hosts on different VLAN to communicate with each other through network switch or router on Network Layer.

VLAN tagging

If i have 2 different environment where servers are members of different network segments, in order to allow the different hosts which are in the p750 server to share the same physical ethernet link, the switch or router need to understand and route the network traffic for both the 10.10.10.* and 10.10.20.* to the p750 machine.

Within the p750 machine, the NIC is capable of deciphering the VLAN ID and route the traffic to the designated hosts or LPARs for inward traffic. For outward traffic, the NIC would tag the VLAN ID to the traffic and route it out to the gateway.

This is also similar to the VTP or ISL protocol that is proprietary to Cisco.

Why is it good.

With 2 VIO servers, the different LPARs in the same IBM p750 machine would traditionally need more than 20 UTP cables. With 802.1Q, we need only 2 cables for all the LPARs and 2 cables for HMC. This is more than 80% savings!

Why it may be bad

In the event that there are lots of host sharing the same UTP link, the nightmare of link failure will be catastrophic. It may be mitigated by having 2 NIC controllers with 2 links each, and distributed connection to 2 different switches.

i guess you cant have your cake and eat it!

How to automatically redirect HTTP to HTTPS in Apache

Redirecting HTTP to HTTPS is one common and popular way to protect user privacy and sensitive information without making user typing 'https' manually to access your site.

First, we verify that Apache is configured for HTTPS connection and necessary SSL certificates are already in placed.

Then, either we use redirect or mod_rewrite.

• Using mod_rewrite. Add these directives to your configuration file:

          RewriteEngine On
          RewriteCond %{SERVER_PORT} !^443$
          RewriteRule ^/(.*) https://%{SERVER_NAME}/$1 [L,R]

Make sure you have loaded mod_rewrite module into Apache.


• Using redirect. Add these directives to your configuration file:

          SSLRequireSSL
          Redirect permanent /secure https://www.domain.com/secure

The 2nd method which uses redirect uses one less module, so security wise could be better. In addition, you don't need to worry about re-writing on logs and etc. Now we restart Apache and go test it out.

How to check LVM health in AIX

What to check for LVM

We can check the state, the utilisation of the PV and VG, whether is the state open, closed, stale, syncd or whether the utilisation is max out already.

As LVM consist of many components like, PP, PV, LV, VG and so on, the best way is to script the things to check. For details of what is the acronyms, please refer to AIX LVM concept and Disk structure.

My newbie demo script for checking LVM health. Gurus out there can really help me along if you think i can do better with some suggestions. :D

#!/bin/ksh
# FILENAME   : checkLVM.ksh
# AUTHOR     : Victor Kwan
# EMAIL     : victorkk [AT] gmail [DOT] com
# PURPOSE    : To check the health of PV, VG and LV
#            : and alert sys admin if threshold is breached.
# DATE       : Feb 2011
#

#
# Parameter setup
OUTPUTFILE="checkLVM.PV.`hostname`.`date '+%d%b%Y'`.output"
NOTIFICATION_MSG="checkLVM.PV.`hostname`.`date '+%d%b%Y'`.message"
isFOUND=0
isERROR=0

if [ $# -ne 2 ]
then
        printf "Usage: \n\t$0  \n\n"
        exit
fi

PV_THRESHOLD=$1
EMAIL="$2"

#
# Extract PV Information from ODM
lspv | while read PV; do
        printf "\n$PV\n" >> $OUTPUTFILE
        printf "--------------------------\n" >> $OUTPUTFILE
        lspv $PV >> $OUTPUTFILE
done

#
# Check for PV Errors
grep -n "PV STATE" $OUTPUTFILE > $OUTPUTFILE.PV

printf "\n\n\n------------------------------\n" > $NOTIFICATION_MSG
printf " Check for PV errors\n" >> $NOTIFICATION_MSG
printf "------------------------------\n" >> $NOTIFICATION_MSG

cat $OUTPUTFILE.PV | while read PVLINE
do
        isLOGICAL_CHECK=`echo $PVLINE | grep "PV STATE" | awk -F: '{print $3}' | grep -v "active" | wc -l`
        #printf "[DEBUG]isLOGICAL_CHECK is %d.\n" $isLOGICAL_CHECK

        if [ $isLOGICAL_CHECK == 1 ]
        then
                PV_STATUS=`echo $PVLINE | grep "PV STATE" | awk -F: '{print $3}' | grep -v "active"`
                PV_LINE=`echo $PVLINE | grep "PV STATE" | awk -F: '{print $1}'`
                PV=`head -$PV_LINE $OUTPUTFILE | tail -3 | head -1 | awk '{print $3}'`
                VG=`head -$PV_LINE $OUTPUTFILE | tail -3 | head -1 | awk '{print $6}'`
                PV_LINE_TOTALPP=`echo $PV_LINE + 3 | bc`
                PV_SIZE=`head -$PV_LINE_TOTALPP $OUTPUTFILE | tail -1 | awk '{print $4}' | awk -F\( '{print $2}'`
                PV_LINE_USEDPP=`echo $PV_LINE + 5 | bc`
                PV_USED=`head -$PV_LINE_USEDPP $OUTPUTFILE | tail -1 | awk '{print $4}' | awk -F\( '{print $2}'`

                #printf "[DEBUG]The line is $PVLINE\n"
                printf "Volume Group: %s\n" $VG >> $NOTIFICATION_MSG
                printf "Physical Volume: %s\n" $PV >> $NOTIFICATION_MSG
                printf "Status     | Size (Mb) | Used (Mb)\n" >> $NOTIFICATION_MSG
                printf "%-10s | %-9d | %-8d\n\n" $PV_STATUS $PV_SIZE $PV_USED >> $NOTIFICATION_MSG
                isFOUND=1
                isERROR=1
        fi
done

if [ $isERROR == 0 ]
then
        printf "All Physical Volumes are clean.\n" >> $NOTIFICATION_MSG
fi

# Check for PV full
grep -n "PV STATE" $OUTPUTFILE > $OUTPUTFILE.PV
isERROR=0

printf "\n\n\n------------------------------\n" >> $NOTIFICATION_MSG
printf " Check for PV utilisation\n" >> $NOTIFICATION_MSG
printf " PV Threshold: $PV_THRESHOLD \n" >> $NOTIFICATION_MSG
printf "------------------------------\n" >> $NOTIFICATION_MSG

cat $OUTPUTFILE.PV | while read PVLINE
do
        isLOGICAL_CHECK=`echo $PVLINE | grep "PV STATE" | awk -F: '{print $3}' | wc -l`
        #printf "[DEBUG]isLOGICAL_CHECK is %d.\n" $isLOGICAL_CHECK

        if [ $isLOGICAL_CHECK == 1 ]
        then

                PV_STATUS=`echo $PVLINE | grep "PV STATE" | awk -F: '{print $3}'`
                PV_LINE=`echo $PVLINE | grep "PV STATE" | awk -F: '{print $1}'`
                PV=`head -$PV_LINE $OUTPUTFILE | tail -3 | head -1 | awk '{print $3}'`
                VG=`head -$PV_LINE $OUTPUTFILE | tail -3 | head -1 | awk '{print $6}'`
                PV_LINE_TOTALPP=`echo $PV_LINE + 3 | bc`
                PV_SIZE=`head -$PV_LINE_TOTALPP $OUTPUTFILE | tail -1 | awk '{print $4}' | awk -F\( '{print $2}'`
                PV_LINE_USEDPP=`echo $PV_LINE + 5 | bc`
                PV_USED=`head -$PV_LINE_USEDPP $OUTPUTFILE | tail -1 | awk '{print $4}' | awk -F\( '{print $2}'`
                PV_PERCENTAGE=$(echo "scale=8; $PV_USED / $PV_SIZE * 100" | bc)

                if [ $PV_PERCENTAGE -ge $PV_THRESHOLD ]
                then
                        #printf "[DEBUG]The line is $PVLINE\n" >> $NOTIFICATION_MSG
                        printf "Volume Group: %s\n" $VG >> $NOTIFICATION_MSG
                        printf "Physical Volume: %s\n" $PV >> $NOTIFICATION_MSG
                        printf "Status     | Size (Mb) | Used (%%)\n" >> $NOTIFICATION_MSG
                        printf "%-10s | %-9d | %-5.2f\n\n" $PV_STATUS $PV_SIZE $PV_PERCENTAGE >> $NOTIFICATION_MSG
                        isFOUND=1
                        isERROR=1
                fi
        fi
done

if [ $isERROR == 0 ]
then
        printf "All Physical Volume within threshold.\n" >> $NOTIFICATION_MSG
fi

rm $OUTPUTFILE
rm $OUTPUTFILE.PV

# Extract VG Information from ODM
lsvg | while read VG
do
        print "\nListing $VG:\n" >> $OUTPUTFILE
        lsvg $VG >> $OUTPUTFILE
        lsvg -l $VG >> $OUTPUTFILE
        #lsvg -l $VG | egrep -v "^$VG:" | egrep -v "^LV NAME" | while read LV JUNK
        #do
        #       lslv $LV >> $OUTPUTFILE
        #done
done

# Check for VG errors in ODM
grep -n "VG STATE" $OUTPUTFILE > $OUTPUTFILE.VG
isERROR=0

printf "\n\n\n------------------------------\n" >> $NOTIFICATION_MSG
printf " Check for VG errors\n" >> $NOTIFICATION_MSG
printf "------------------------------\n" >> $NOTIFICATION_MSG

cat $OUTPUTFILE.VG | while read VGLINE
do
        isVG_CHECK=`echo $VGLINE | grep "VG STATE" | awk -F: '{print $3}' | wc -l`
        #printf "[DEBUG]isVG_CHECK is %d.\n" $isVG_CHECK

        if [ $isVG_CHECK == 1 ]
        then
                VG_STATUS=`echo $VGLINE | grep "VG STATE" | awk '{print $3}'`
                VG_LINE=`echo $VGLINE | grep "VG STATE" | awk -F: '{print $1}'`
                VG=`head -$VG_LINE $OUTPUTFILE | tail -2 | head -1 | awk '{print $3}'`
                VG_LINE_TOTALPP=`echo $VG_LINE + 1 | bc`
                VG_TOTALPP=`head -$VG_LINE_TOTALPP $OUTPUTFILE | tail -1 | awk '{print $7}' | awk -F\( '{print $2}'`
                VG_LINE_USEDPP=`echo $VG_LINE + 3 | bc`
                VG_USEDPP=`head -$VG_LINE_USEDPP $OUTPUTFILE | tail -1 | awk '{print $6}' | awk -F\( '{print $2}'`
                VG_LINE_TOTALPV=`echo $VG_LINE + 5 | bc`
                VG_TOTALPV=`head -$VG_LINE_TOTALPV $OUTPUTFILE | tail -1 | awk '{print $3}'`
                VG_LINE_STALEPV=`echo $VG_LINE + 6 | bc`
                VG_STALEPV=`head -$VG_LINE_STALEPV $OUTPUTFILE | tail -1 | awk '{print $3}'`
                VG_STALEPP=`head -$VG_LINE_STALEPV $OUTPUTFILE | tail -1 | awk '{print $6}'`
                VG_LINE_ACTIVEPV=`echo $VG_LINE + 7 | bc`
                VG_ACTIVEPV=`head -$VG_LINE_ACTIVEPV $OUTPUTFILE | tail -1 | awk '{print $3}'`

                PV_LINE=`lsvg -p $VG | wc -l`
                PV_NUMOFMEMBERS=`echo $PV_LINE - 2 | bc`
                PV_NAME="`lsvg -p $VG | tail -$PV_NUMOFMEMBERS | awk '{print $1}' | xargs`"
                PV_STALENAME=`lsvg -p $VG | tail -$PV_NUMOFMEMBERS | grep -v active | awk '{print $1}' | xargs`

                if [ -z "$PV_NAME" ]
                then
                        $PV_NAME="NA"
                fi

                LV_LINE=`lsvg -l $VG | wc -l`
                LV_NUMOFMEMBERS=`echo $LV_LINE - 2 | bc`
                LV_NUMOFPROBLEM=`lsvg -l $VG | tail -$LV_NUMOFMEMBERS | grep -v "open/syncd" |  wc -l`
                LV_NUMOFOPEN=`echo $LV_NUMOFMEMBERS - $LV_NUMOFPROBLEM | bc`
                LV_PROBLEMNAME=`lsvg -l $VG | tail -$LV_NUMOFMEMBERS | grep -v "open/syncd" | awk '{print $1}' | xargs`
                LV_NAME="`lsvg -l $VG | tail -$LV_NUMOFMEMBERS | awk '{print $1}' | xargs`"

                if [ -z "$LV_NAME" ]
                then
                        $LV_NAME="NA"
                fi

                #printf "[DEBUG]The line is $VGLINE\n"

                if [ $VG_STALEPP -ge 1 -o $VG_STALEPV -ge 1 -o $LV_NUMOFPROBLEM -ge 1 ]
                then
                        printf "Volume Group: %s\nVolume Group Status: %s\n\n" $VG $VG_STATUS >> $NOTIFICATION_MSG

                        printf "Total PP Size (Mb) | Used PP Size (Mb) | Stale PP\n" >> $NOTIFICATION_MSG
                        printf "%-18d | %-17d | %-5d \n\n" $VG_TOTALPP $VG_USEDPP $VG_STALEPP >> $NOTIFICATION_MSG

                        printf "Total PV | Active PV | All PV members\n" >> $NOTIFICATION_MSG
                        printf "%-8d | %-9d | %-s\n\n" $VG_TOTALPV $VG_ACTIVEPV "$PV_NAME" >> $NOTIFICATION_MSG

                        printf "Total LV | Open LV | All LV members\n" >> $NOTIFICATION_MSG
                        printf "%-8d | %-7d | %-s\n\n" $LV_NUMOFMEMBERS $LV_NUMOFOPEN "$LV_NAME" >> $NOTIFICATION_MSG

                        if [ $VG_STALEPV -ge 1 ]
                        then
                                printf "Status of PV with problems:\n" >> $NOTIFICATION_MSG
                                for i in $PV_STALENAME
                                do
                                        THIS_PV=`lspv $i | grep "PV STATE" | awk '{print $1,$2,$3}'`
                                        printf "$i ($THIS_PV) \n" >> $NOTIFICATION_MSG
                                done
                        fi
                        printf "\n" >> $NOTIFICATION_MSG

                        if [ $LV_NUMOFPROBLEM -ge 1 ]
                        then
                                LV_FLAG=0
                                for i in $LV_PROBLEMNAME
                                do
                                        THIS_LV=`lslv $i | grep "LV STATE" | awk '{print $4,$5,$6}'`
                                        THIS_STATE=`lslv $i | grep "LV STATE" | awk '{print $6}'`
                                        THIS_BOOT=`lslv $i | grep "TYPE" | awk '{print $2}' | grep "boot" | wc -l`
                                        THIS_DUMP=`lslv $i | grep "TYPE" | awk '{print $2}' | grep "sysdump" | wc -l`
                                        if [ "$THIS_STATE" != "closed/syncd" ]
                                        then
                                                if [ $LV_FLAG = 0 ]
                                                then
                                                        printf "Status of LV with problems:\n" >> $NOTIFICATION_MSG
                                                        LV_FLAG=1
                                                fi
                                                printf "$i ($THIS_LV) \n" >> $NOTIFICATION_MSG
                                        elif [ "$THIS_STATE" != "open/syncd" -a $THIS_DUMP = "1" ]
                                        then
                                                if [ $LV_FLAG = 0 ]
                                                then
                                                        printf "Status of LV with problems:\n" >> $NOTIFICATION_MSG
                                                        LV_FLAG=1
                                                fi
                                                printf "$i ($THIS_LV) \n" >> $NOTIFICATION_MSG
                                        elif [ "$THIS_STATE" != "closed/syncd" -a $THIS_BOOT = "1" ]
                                        then
                                                if [ $LV_FLAG = 0 ]
                                                then
                                                        printf "Status of LV with problems:\n" >> $NOTIFICATION_MSG
                                                        LV_FLAG=1
                                                fi
                                                printf "$i ($THIS_LV) \n" >> $NOTIFICATION_MSG
                                        fi
                                done
                        fi
                        printf "\n\n" >> $NOTIFICATION_MSG
                        isFOUND=1
                        isERROR=1
                fi
        fi
done

if [ $isERROR == 0 ]
then
        printf "All Volume Groups are clean.\n" >> $NOTIFICATION_MSG
fi

if [ $isFOUND == 1 ]
then
        cat $NOTIFICATION_MSG | mailx -s "[`hostname`] LVM Errors" $EMAIL
fi

rm $OUTPUTFILE
rm $OUTPUTFILE.VG
rm $NOTIFICATION_MSG

How to check status of subgroup TCPIP in AIX

To get the status of the subsystem group TCPIP

# lssrc -g tcpip
Subsystem         Group            PID          Status
 inetd            tcpip            3997848      active
 hostmibd         tcpip            2752750      active
 aixmibd          tcpip            3539102      active
 xntpd            tcpip            9633868      active
 muxatmd          tcpip                         inoperative
 rwhod            tcpip                         inoperative
 snmpd            tcpip                         inoperative
 snmpmibd         tcpip                         inoperative
 dpid2            tcpip                         inoperative
 dhcpcd           tcpip                         inoperative
 dhcpcd6          tcpip                         inoperative
 ndpd-host        tcpip                         inoperative
 ndpd-router      tcpip                         inoperative
 tftpd            tcpip                         inoperative
 gated            tcpip                         inoperative
 named            tcpip                         inoperative
 routed           tcpip                         inoperative
 iptrace          tcpip                         inoperative
 timed            tcpip                         inoperative
 dhcpsd           tcpip                         inoperative
 dhcpsdv6         tcpip                         inoperative
 dhcprd           tcpip                         inoperative
 mrouted          tcpip                         inoperative
 pxed             tcpip                         inoperative
 binld            tcpip                         inoperative
 dfpd             tcpip                         inoperative

Easy?! :)

How to configure link aggregation in AIX

Link aggregation means you can give one IP address to two network cards and connect to two different switches for redundancy purpose. In this only one network card will be active in one time, and when it got failed the other network card goes active and let us continue our work.

It is better to use through SMIT. (For newbies like me :D )

# smit

then goto Devices > Communication > EtherChannel / IEEE 802.3ad Link Aggregation > Add An EtherChannel / Link Aggregation

Here select the network card that you want to use, ie active.

Eg: select ent0

IMP : then select Mode as 8023ad

then select backup adapter for redundancy.(press F4 to show N/W adapters.)

Eg: ent1

press enter.

now ent0 and ent1 got bonded.

then automatically a virtual adapter will be created named ent2.

then put IP address and all to this virtual adapter.

# smit

Communications Applications and Services > TCP/IP > Minimum Configuration & Startup

here select ent2 ( new bonded virtual adapter )

put IP Address and all,

give start now option.

Now you are successfully completed Link aggregation and check whether it works or not by removing the 2nd cable to the network card and check ping, then put the 2nd cable and remove 1st cable. 2 - 3 drops normally occurs.

How to configure NTP in AIX 7.1

First, verify that you have a server suitable for synchronisation. Do note that the offset must be less than 1000 seconds for xntpd to sync. If the offset is more than 1000 seconds, change the time manually on the client and try again. Ensure that firewall is opened for port 123.

# ntpq -d 10.10.10.254
ntpq> ^C

# ntpdate -d 10.10.10.254
23 Feb 14:41:24 ntpdate[4653300]: 3.4y
transmit(10.10.10.254)
receive(10.10.10.254)
transmit(10.10.10.254)
receive(10.10.10.254)
transmit(10.10.10.254)
receive(10.10.10.254)
transmit(10.10.10.254)
receive(10.10.10.254)
transmit(10.10.10.254)
server 10.10.10.254, port 123
stratum 2, precision -18, leap 00, trust 000
refid [10.10.10.250], delay 0.02632, dispersion 0.00002
transmitted 4, in filter 4
reference time:      d10f27e7.15e9bbb7  Wed, Feb 23 2011 14:29:59.085
originate timestamp: d10f2a94.f62731bd  Wed, Feb 23 2011 14:41:24.961
transmit timestamp:  d10f2a94.f62e7000  Wed, Feb 23 2011 14:41:24.961
filter delay:  0.02638  0.02632  0.02637  0.02637
               0.00000  0.00000  0.00000  0.00000
filter offset: -0.00046 -0.00047 -0.00050 -0.00051
               0.000000 0.000000 0.000000 0.000000
delay 0.02632, dispersion 0.00002
offset -0.000479

23 Feb 14:41:24 ntpdate[4653300]: adjust time server 10.10.10.254 offset -0.000479

Now specify xntp server in /etc/ntp.conf. Add in the line "server ip-address-of-the-server prefer". Leave the driftfile and tracefile at their defaults.

vi /etc/ntp.conf

Now, ensure xntpd is started after every reboot.

vi /etc/rc.tcpip

Uncomment the following line

start /usr/sbin/xntpd "$src_running"

Then, start the client and verify.

# lssrc -s xntpd
Subsystem         Group            PID          Status
 xntpd            tcpip                         inoperative

startsrc -s xntpd
0513-059 The xntpd Subsystem has been started. Subsystem PID is 4391134.

#lssrc -s xntpd
Subsystem         Group            PID          Status
 xntpd            tcpip            4391134      active

# ntpq -p
     remote           refid      st t when poll reach   delay   offset    disp
==============================================================================
*10.10.10.254   10.10.10.250      2 u   25   64  377     0.87    0.266    0.02

For AIX NTP server, Firstly, verify that you have a suitable NTP server. "Sys peer" should show a valid server or 127.127.1.0. Otherwise, if the server is "insane", you will need to ad a server line to /etc/ntp.conf and restart xntpd.

# lssrc -ls xntpd
 Program name:    /usr/sbin/xntpd
 Version:         3
 Leap indicator:  00 (No leap second today.)
 Sys peer:        10.10.10.254
 Sys stratum:     3
 Sys precision:   -18
 Debug/Tracing:   DISABLED
 Root distance:   0.002701
 Root dispersion: 0.001129
 Reference ID:    10.10.10.254
 Reference time:  d10f3206.cca44000  Wed, Feb 23 2011 15:13:10.799
 Broadcast delay: 0.003906 (sec)
 Auth delay:      0.000122 (sec)
 System flags:    bclient pll monitor filegen
 System uptime:   2653 (sec)
 Clock stability: 1.338257 (sec)
 Clock frequency: 0.000000 (sec)
 Peer: 10.10.10.254
      flags: (configured)(sys peer)(preferred)
      stratum:  2, version: 3
      our mode: client, his mode: server
Subsystem         Group            PID          Status
 xntpd            tcpip            4391134      active

To add server line into /etc/ntp.conf

vi /etc/ntp.conf

Add the following ilne and ensure "broadcast client" is commented out.

server 127.127.1.0

Restart xntpd

stopsrc -s xntpd
startsrc -s xntpd

If the server runs databases, use the -x flag to prevent the clock from changing in a negative direction. Add into /etc/rc.tcpip if necessary. Remember to use the double quote as in "-x". The whole process may take up to 12 minutes.