How to list the top 5 largest directories

# du -sk /opt/* | sort +0nr | head -5
25456556        /opt/data
12634192        /opt/read
5483564 /opt/download
196104  /opt/scripts
132964  /opt/freeware

tip.. if you want to include the hidden files and directories, try this.

# du -sk .[a-z]* * | sort +0nr | head -5

My one liner to extracting lines using sed, perl or awk

While working on extracting data from large amount of files, i have compiled some commands over the years to really helps a lot.

Most of the time, we use head, tail, grep. However, these commands are good at wholesale extracting or just by some keywords. For more complex extraction, we may use sed, perl or awk instead.

Using myfile as example,

myserver:/tmp/:>head -10 myfile
# IBM_PROLOG_BEGIN_TAG
# This is an automatically generated prolog.
#
# bos61D src/bos/usr/sbin/netstart/hosts 1.2
#
# Licensed Materials - Property of IBM
#
# COPYRIGHT International Business Machines Corp. 1985,1989
# All Rights Reserved
#

myserver:/tmp:>tail -2 myfile
10.1.1.123     host1
10.2.1.124     host2


myserver:/tmp/:>grep host1 myfile
10.1.1.123     host1

Say, for more complicated stuffs, like extracting 2nd line PLUS 5th to 7th line, i find it tough to code using the above commands.

h2. sed, perl or awk?

Do note that sed will transverse the entire file, hence if you have a very large file, this might take some time.

Say, we want to extract the 2nd line, we can use sed or awk

myserver:/tmp/:>sed 2p myfile
# IBM_PROLOG_BEGIN_TAG
# This is an automatically generated prolog.
# This is an automatically generated prolog.
#
# bos61D src/bos/usr/sbin/netstart/hosts 1.2
...
...

myserver:/tmp/:>awk 'NR==2' myfile
# This is an automatically generated prolog.





If you have try it out, you will see that for sed, the 2nd line is indeed extracted but the rest of the file is also printed out! Use the following to disable printing out the old file.

myserver:/tmp/:>sed -n 2p myfile
# This is an automatically generated prolog.

Alternatively, you might want to 'delete' whatever that you don't want by using the '!d' parameter.

myserver:/tmp/:>sed '2!d' myfile
# This is an automatically generated prolog.

I wouldn't want to use this method as i have difficulty converting the line to use variables. Do give me suggestions or advice if you think otherwise. I don't claim to be expert in writing scripts. :)

IMPORTANT: Note that the single quotes are required. Else '!d' will bring back the last command you have executed with the letter 'd'.



If you only want one and only line from the file, you can get awk to exit after getting that line, otherwise the awk will transverse through the whole file.

myserver:/tmp/:>awk 'NR==6 {print; exit}' myfile
# Licensed Materials - Property of IBM

If we try to extract line 5 to 7 using sed or awk

myserver:/tmp/:>sed -n 5,7p myfile
#
# Licensed Materials - Property of IBM
#

myserver:/tmp/:>awk 'NR==5,NR==7' myfile
#
# Licensed Materials - Property of IBM
#

Here's another trick that i read from Mr Google. If you want to extract every 5th line of a file starting from the top of the file, perl or awk does the job easily.


myserver:/tmp/:>perl -ne 'print unless (0 != $. % 5)' myfile
#
#
# IBM_PROLOG_END_TAG
#
# Licensed Materials - Property of IBM
#  /etc/hosts
#
#
...
...


myserver:/tmp/:>awk '0 == NR % 5'   myfile
#
#
# IBM_PROLOG_END_TAG
#
# Licensed Materials - Property of IBM
#  /etc/hosts
#
#
...
...


Tip: If you don't want to start from the top of the file, you can put (NR + 1), which means to start from line 1.

Thats all folks.

How to list Network statistics

This is actually useful across all platforms like Solaris, AIX, Linux and etc to list out the network statistics of the NICs. We can tell if there are any potential network issues and spawn off necessary actions.

root@myserver:/> netstat -i
Name  Mtu   Network     Address            Ipkts Ierrs    Opkts Oerrs  Coll
en2   1500  link#2      d2.48.a8.b8.c9.2    13862     0    14038     0     0
en2   1500  10.10.10   myserver          13862     0    14038     0     0
lo0   16896 link#1                          10644     0    10644     0     0
lo0   16896 127         loopback            10644     0    10644     0     0
lo0   16896 loopback                        10644     0    10644     0     0

MPT major number error

If you hit the following errors when you reboot the server, just as the server is almost ready. They will come in a large numbers repeatedly.

"WARNING: add_spec: No major number for mpt"

You may want to try the solutions below. I have tried out solution 2 successfully.

Document Audience: SPECTRUM
Document ID: 74181
Title: SolarisTM: "WARNING: add_spec: No major number for mpt"
Update Date: Wed Sep 29 00:00:00 MDT 2004
Products: Solaris 8 Operating System, Solaris 9 Operating System
Technical Areas: Patch

--------------------------------------------------------------------------------

--------------------------------------------------------------------------------

Keyword(s):add_spec, mpt, major number

Problem Statement:

After installing a patch to update /etc/driver_classes with
an "mpt" entry (for example, 108528-xx with xx>21), the system may
generate the following WARNING messages at boot time:

SunOS Release 5.8 Version Generic_108528-29 64-bit
Copyright 1983-2003 Sun Microsystems, Inc. All rights reserved.
WARNING: add_spec: No major number for mpt
WARNING: add_spec: No major number for mpt
WARNING: add_spec: No major number for mpt
WARNING: add_spec: No major number for mpt
[...]
WARNING: add_spec: No major number for mpt
WARNING: add_spec: No major number for mpt
WARNING: add_spec: No major number for mpt
configuring IPv4 interfaces: hme0.
configuring IPv6 interfaces: hme0.

The following message appears in the /var/sadm/patch//log file
when the entry has not been successfully added to the /etc/name_to_major
file:
SUNWcsr: failed to add mpt to /etc/name_to_major:
(mpt) already in use as a driver or alias.

Explanation:
============
The problem is due to a reference in /etc/driver_classes to the
mpt driver. The mpt driver isn't added to the system because the
/etc/driver_aliases file already had an entry with the mpt driver. Because
of this, the add_drv failed when trying to install the driver.

This is an example of the inconsistencies described in Bug ID 4939994,
"Inconsistency between name_to_major and driver_aliases."

Resolution:

To solve this problem, a script has been created to identify the
inconsistencies between /etc/driver_aliases and /etc/name_to_major.
This script can also add the missing entries if they are listed in a
"reference" name_to_major file.

Here is an example of the way to use the attached script:

    cksum InconsitencyFixTool.tar.gz
    1091730651 4410 InconsitencyFixTool.tar.gz

    gzip -dc InconsitencyFixTool.tar.gz | tar -xvf -
    x InconsitencyFixTool, 0 bytes, 0 tape blocks
    x InconsitencyFixTool/README, 1364 bytes, 3 tape blocks
    x InconsitencyFixTool/name_to_major.InconsitencyFix, 4928 bytes, 10 tape blocks
    x InconsitencyFixTool/name_to_major.i386.5.8, 841 bytes, 2 tape blocks
    x InconsitencyFixTool/name_to_major.i386.5.9, 889 bytes, 2 tape blocks
    x InconsitencyFixTool/name_to_major.sparc.5.8, 1831 bytes, 4 tape blocks
    x InconsitencyFixTool/name_to_major.sparc.5.9, 1815 bytes, 4 tape blocks

    cd InconsitencyFixTool
    ./name_to_major.InconsitencyFix
    Saving original files into .
    Inconsistency found between //etc/driver_aliases and //etc/name_to_major on
    the following driver(s):
    mpt

Add mpt to //etc/name_to_major ? y y
Adding the following devices to //etc/name_to_major :
mpt 215

     

Reboot the system.

Temporary Workaround:

If the above script cannot be used, there are two other ways to fix the
problem:

1. Remove the "mpt" lines from /etc/driver_classes and /etc/driver_aliases.

They should look like the following:
driver_aliases:mpt "pci1000,30"
driver_classes:mpt scsi

Next, install (or remove and reinstall) a patch updating all
those files [/etc/driver_aliases /etc/driver_classes
/etc/name_to_major]. For example, install the mpt patch 115275-01 (or
above) or a new kernel Update patch.

OR

2. Add the missing "mpt" entry in the /etc/name_to_major file
to correct the problem with the patch installation.

You must manually append the "mpt" entry at the end
of the /etc/name_to_major file as follows:

mpt XXX

Where XXX is the maximum+1 of the numbers already given to
the other drivers in this file. Separate "mpt" and the given number with
a space character.

For example:

    tail /etc/name_to_major
    fasttrap 223
    dmfe 224
    todds1307 225
    pool 226
    zcons 227
    ipf 228
    pfil 229
    ctsmc 230
    bl 231
    mpt 232

Reboot the system.

Additional Information:

History:
========
Two patches have been identified that can cause the problem:

108974-31(or greater): SunOS 5.8: dada, uata, dad, sd, ssd and scsi
drivers patch

OR

109885-14 SunOS 5.8: glm patch

Only on top of a kernel patch 108528-xx where xx <= 21.

Note: If you have a more recent version than -21 before installing
the above patches, the problem will not occur.

This problem can also exist on Solaris 9 - if S9 was installed with
an upgrade from a S8 system where the inconsistency existed.

Impact:
=======
The mpt driver would not be installed correctly and cannot be used.

SSL Certificate Monitoring

Here's a guide on how to setup monitoring for SSL certificates

It is important to ensure that the SSL certificates used in services that are fronting users or for secure communication are valid otherwise, we risk service outage because of expired certificates.

The SSL monitoring script

Using the monitoring script "SSL Certificate Check" written by Matty, we can use it to monitor the SSL certificates either by verifying the certificate itself or querying the status through the application services.

Link to detailed documentation at SSL Certificate Check

How to set it up.

I am using the script v3.21 dated Oct 2010 in the example.

1) Download at SSL Certificate Check Script

2) Deploy it to a suitable location. Give it execute permission at least 0700.

I have configured the script to the following

# Who to page when an expired certificate is detected (cmdline: -e)
ADMIN="admin@myserver.com"

# Number of days in the warning threshhold  (cmdline: -x)
WARNDAYS=100

# If QUIET is set to TRUE, don't print anything on the console (cmdline: -q)
QUIET="FALSE"

# Don't send E-mail by default (cmdline: -a)
ALARM="TRUE"

# Don't run as a Nagios plugin by default (cmdline: -n)
NAGIOS="FALSE"

where the script will notify via email (default) when the certificate has less than 100 days of validity. It will print out to console. If you don't need it, change QUIET to "TRUE".

If you require to override the default settings in the script, you can use the following switches

#./sslcertcheck
Usage: ./sslcertcheck [ -e email address ] [ -x days ] [-q] [-a] [-b] [-h] [-i] [-n] [-v]
       { [ -s common_name ] && [ -p port] } || { [ -f cert_file ] } || { [ -c certificate file ] }

  -a                : Send a warning message through E-mail
  -b                : Will not print header
  -c cert file      : Print the expiration date for the PEM or PKCS12 formatted certificate in cert file
  -e E-mail address : E-mail address to send expiration notices
  -f cert file      : File with a list of FQDNs and ports
  -h                : Print this screen
  -i                : Print the issuer of the certificate
  -k password       : PKCS12 file password
  -n                : Run as a Nagios plugin
  -p port           : Port to connect to (interactive mode)
  -s commmon name   : Server to connect to (interactive mode)
  -q                : Don't print anything on the console
  -v                : Only print validation data
  -x days           : Certificate expiration interval (eg. if cert_date < days)

Requirements.

mktemp package needs to be available in the server.

Usage

1) Running the script against the certificate file.

$ sslcertcheck -c /etc/httpd/conf/ssl.crt/abc.pem
Host                                            Status       Expires      Days Left
----------------------------------------------- ------------ ------------ ----------
FILE:/etc/httpd/conf/ssl.crt/abc.pem            Valid        Jan 2 2010   807   

sslcertcheck will print the file or hostname in the first column, a value to indicate if the certifciate is valid in the second column, the date the certificate will expire in the third column, and the number of days remaining until the certificate expires in the fourth column.

2) If you do not have local access to the certificate files, you can use sslcertcheck's network connectivity option to extract the certificate expiration date from a live server. To check when the certificate used by the web server will expire, the server name or IP address and a port number can be passed to sslcertcheck's "-s" (server name) and "-p" (tcp port) options:

#./sslcertcheck -s 172.21.41.136 -p 443

Host                                            Status       Expires      Days
----------------------------------------------- ------------ ------------ ----
172.2.1.1:443                               Valid        Jul 24 2011  128

3) You may want to manage dozens of SSL-enabled servers, you can place the server names and port numbers in a file, and run sslcertcheck against that file:

The configuration file.

$ cat sslcertcheck.cfg
10.10.8.1 443
10.10.8.7 443
172.2.1.1 443

The output from the script with setting whom to email when any entry has validity less than threshold.

# ./sslcertcheck -e admin@me.com -f sslcertcheck.cfg 

Host                                            Status       Expires      Days
----------------------------------------------- ------------ ------------ ----
10.10.8.1:443                                 Valid        Nov 14 2013  972
10.10.8.7:443                                 Valid        Jan 20 2014  1039
172.2.1.1:443                               Valid        Jul 24 2011  128

Thats all folks!

Basic networking TCP test using telnet

When telneting to an IP at a given port, there are various telnet responses. Knowing the difference in telnet responses could easily point you in the right direction when a telnet to a host on a particular port in unsuccessful.

There are a distinct differences in getting ‘refused’ or ‘timeout’ responses.

You will get a connection refused message for one of the following reasons:

• The application you are trying to test hasn’t been started/installed on the remote server.
• There is a firewall rejecting the connection attempt by terminating the connection setup.

Example output from a Linux box:

$ telnet server2 7063
Trying 172.1.1.1...
telnet: connect to address 172.1.1.1: Connection refused
telnet: Unable to connect to remote host: Connection refused

The similar Connection refused message from a Solaris box :

$ telnet server3 7055
Trying 172.2.1.1...
telnet: Unable to connect to remote host: Connection refused

The Connect failed message is the equivalent but from a Windows box :

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\vickwan>telnet 172.2.1.1 7062
Connecting To 172.2.1.1...Could not open connection to the host, on port 7062: Connect failed

The telnet command will abort the attempted connection after waiting a predetermined time for a response. This is called a timeout response.

In some cases, telnet won’t abort, but will just wait indefinitely. This is also known as hanging. These symptoms can be caused by the one of the following reasons:

• The remote server doesn’t exist on the destination network. It could be turned off.
• The could be a routing issue, either the request or the response never gets to the destination.
• A firewall could be blocking the connection attempt, causing it to timeout instead of being quickly refused.

Here is an example of the output:

$ telnet server3 7055
Trying 172.2.1.1...
telnet: connect to address 172.2.1.1: Connection timed out
telnet: Unable to connect to remote host: Connection timed out

The script, command file and input file.

Reference Adapted from : http://blog.ru.co.za/2009/09/29/telnet/

This little script is written to helps cut down time needed to test if ACL allows connection from server A to server B at a given port. It will attempt to suggest the remedy actions.

Script tested on AIX 6.1, AIX 7.1, RHEL 4.6 and Solaris 9.

#!/bin/ksh
# Written By   : Victor Kwan
# Written On   : 25 Oct 2009
# EMAIL   : victorkk [AT] gmail [DOT] com
# Description  : Utility to test TCP ACL via telnet
# Updated On   : 27 Oct 2009 : Attempt to interpret telnet response.
#              : 09 Mar 2011 : Test if telnet command is executable
#              : 07 Apr 2011 : Support AIX, Improve code to be not chatty and terminate telnet session properly.

FILE=${1}
OUTPUTFILE="$0.output"
LOGFILE="$0.log"
TELNETCMD="$0.telnetcmd"
TELNET=`which telnet`
CAT=`which cat`
ECHO=`which echo`
OS="`uname -s`"

#UNIX Normal "Connection to 10.106.50.10 closed."
#UNIX No route "No route to host"
#UNIX Conn refused "telnet: Unable to connect to remote host: Connection refused"
#UNIX timed out "telnet: Unable to connect to remote host: Connection timed out"

RESPONSE_NORMAL="gn host."
RESPONSE_NO_ROUTE="to host"
RESPONSE_CONN_REFUSED="refused"
RESPONSE_TIMED_OUT="med out"

#AIX Normal "Connection closed."
#AIX No route "No route to host"
#AIX Conn refised "telnet: connect: A remote host refused an attempted connect operation."
#AIX timed out "telnet: connect: A remote host did not respond within the timeout period."

AIXRESPONSE_NORMAL="Connection closed."
AIXRESPONSE_NO_ROUTE="No route to host"
AIXRESPONSE_CONN_REFUSED="connect operation."
AIXRESPONSE_TIMED_OUT="he timeout period."

THISRESPONSE_NORMAL="$RESPONSE_NORMAL"
THISRESPONSE_NO_ROUTE="$RESPONSE_NO_ROUTE"
THISRESPONSE_CONN_REFUSED="$RESPONSE_CONN_REFUSED"
THISRESPONSE_TIMED_OUT="$RESPONSE_TIMED_OUT"

COLOR_BLUE="\033[0;34m"
COLOR_GREEN="\033[32m"
COLOR_RED="\033[31m"
COLOR_BRIGHTRED="\033[1;31m"
COLOR_WHITE="\033[0m"
COLOR_BRIGHTWHITE="\033[1;37m"

if [ ! -x $TELNET ]
then
        echo "${COLOR_BRIGHTRED}Telnet command is not executable!!${COLOR_WHITE}"
        echo "${COLOR_WHITE}Script will now terminate.${COLOR_WHITE}"
        exit
fi

echo "Commence telnet test based on [$FILE] file."
echo

cat ${FILE} | grep -v "#" | while read LINE do {
        IP=`echo $LINE | awk -F: '{print $1}'`
        PORT=`echo $LINE | awk -F: '{print $2}'`

        ($CAT $TELNETCMD) | $TELNET $IP $PORT >> $OUTPUTFILE 2>&1

        RESPONSE=`tail -1 $OUTPUTFILE | tr -d "\r" | tr -d "\n"`
	if [ "$OS" = "SunOS" ]
	then
	{
		STR_TO_CMP=`echo "$RESPONSE" | awk '{print substr($0,length-7)}'`
	}
	elif [ "$OS" = "AIX" ]
	then
	{
		STR_TO_CMP=`echo "$RESPONSE" | awk '{print substr($0,length-18)}'`
		THISRESPONSE_NORMAL="$AIXRESPONSE_NORMAL"
		THISRESPONSE_NO_ROUTE="$AIXRESPONSE_NO_ROUTE"
		THISRESPONSE_CONN_REFUSED="$AIXRESPONSE_CONN_REFUSED"
		THISRESPONSE_TIMED_OUT="$AIXRESPONSE_TIMED_OUT"
	}
	fi

        if [ ! "$STR_TO_CMP" = "$THISRESPONSE_NORMAL" ]
        then
        {
                echo "Telnet ${COLOR_BRIGHTRED}FAILED${COLOR_WHITE} for ${COLOR_BRIGHTWHITE}$IP:$PORT${COLOR_WHITE}."
                echo "${COLOR_BRIGHTRED}Error Message${COLOR_WHITE} : [$RESPONSE]!"

                if [ "$STR_TO_CMP" = "$THISRESPONSE_NO_ROUTE" ]
                then
                {
                        echo "${COLOR_GREEN}Suggestion${COLOR_WHITE}: Check routing at both source and destination"
                }
                fi

                if [ "$STR_TO_CMP" = "$THISRESPONSE_CONN_REFUSED" ]
                then
                {
                        echo "${COLOR_GREEN}Suggestion${COLOR_WHITE}: Destination may not be listening, routable or firewall is blocking the connection."
                }
                fi

                if [ "$STR_TO_CMP" = "$THISRESPONSE_TIMED_OUT" ]
                then
                {
                        echo "${COLOR_GREEN}Suggestion${COLOR_WHITE}: Destination may not be listening, routable or firewall is blocking the connection."
                }
                fi
        }
        fi
     	echo "Done for $IP:$PORT."
        echo " "
}
done
echo "Telnet test ends."

For the input file, e.g. IP_PORT It is okay to have commented lines as the script will ignore them.

~$more IP_PORT
#WLS
server2:7003
server2:7004
server2:7022
server2:7023
...
...

For the command file, the 2 telnet control commands must be used.

~$ more testACL.telnetcmd
^]
quit

Final outcome. Output may look similar to the following. No output for telnet success.

> ./testACL_telnet.ksh IP_PORT
Commence telnet test based on [IP_PORT] file.
Telnet FAILED for server4:7053.
Error Message : [telnet: Unable to connect to remote host: Connection refused]!
Suggestion: Destination may not be listening on this IP and Port, routable or firewall is blocking the connection.
...
...
...
telnet test ends.

Why SSHD account cannot be removed

In modern SSHD, the privilege separation security feature is provided to allow SSHD to create unprivileged child process to deal with incoming network traffic. After successful authentication, another process will be created that has the privilege of the authenticated user. Privilege separation is to prevent privilege escalation by containing any corruption within the unprivileged processes.

Default setting in SSHD is 'yes', meaning its enabled. Hence, the account 'sshd' account with 'sshd' group is required by SSHD.

How to add a mirror disk back to VxVM

I have 3 disks which are part of the mirror group.

root@myserver # vxdisk list | grep ^c
c1t0d0s2 sliced rootdisk rootdg online
c1t1d0s2 sliced rootmirror1 rootdg online
c2t0d0s2 sliced - - online
c2t1d0s2 sliced - - online

Naming conventions:
- rootdisk
- rootmirror1
- rootmirror2
- rootmirror3

Now i want to add the mirror back. We are going to attach back device c2t0d0 into the rootdg disk group.

Always run the command vxdisk -ht to check what is the actual volume name to attach the disk to.

Setup the disk
root@myserver # vxdisksetup -if c2t0d0 (for VxVM 3.5)
root@myserver # vxdisksetup -if c2t0d0 format=sliced (for VxVM 4.0)
root@myserver # /usr/lib/vxvm/bin/vxdisksetup -if c2t0d0 format=sliced (for VxVM 4.1)

Add the disk into rootdg.
root@myserver # vxdg -g rootdg adddisk rootmirror2=c2t0d0

Mirror each of the partitions. Here i throttle the synchronisation speed to have less performance impact on the server.
root@myserver # vxassist -o slow=1500 -g rootdg mirror rootvol rootmirror2 &
root@myserver # vxassist -o slow=1500 -g rootdg mirror opt rootmirror2 &
root@myserver # vxassist -o slow=1500 -g rootdg mirror swapvol rootmirror2 &
root@myserver # vxassist -o slow=1500 -g rootdg mirror usr rootmirror2 &
root@myserver # vxassist -o slow=1500 -g rootdg mirror var rootmirror2 &

To monitor the progress of synchronisation.
root@myserver # vxtask list
TASKID PTID TYPE/STATE PCT PROGRESS
166 ATCOPY/R 07.88% 0/25166079/1982464 PLXATT opt opt-01
167 ATCOPY/R 25.29% 0/4194828/1060864 PLXATT swapvol swapvol-01
168 ATCOPY/R 15.13% 0/4194828/634880 PLXATT usr usr-01
169 ATCOPY/R 02.82% 0/16779312/473088 PLXATT var var-01

How to ensure VTOC is updated from VxVM in SunOS?

It is important to keep the VTOC in sync with the setup in VxVM else you risk not being able to boot up the disk after un-encapsulating.

SUN also advised that if the VxVM disk mirrors are not identical to the first disk, this would result in unusable mirrors.

Also after creation of VxVM rootdisk and rootmirrors, you may want to create the underlying partition on the rootdisk and rootmirror.

Here, i will show the steps for setting up new disk as RAID1 mirror

root@myserver # vxdisksetup -if c2t0d0
root@myserver # vxdg -g rootdg adddisk rootmirror2=c2t0d0
root@myserver # vxassist -g rootdg mirror rootvol rootmirror2 &
root@myserver # vxassist -g rootdg mirror usr rootmirror2 &
root@myserver # vxassist -g rootdg mirror var rootmirror2 &
root@myserver # vxassist -g rootdg mirror opt rootmirror2 &

Take a look at the VTOC

root@myserver # prtvtoc /dev/dsk/c1t0d0s2
* /dev/dsk/c1t0d0s2 partition map
*
* Dimensions:
* 512 bytes/sector
* 107 sectors/track
* 27 tracks/cylinder
* 2889 sectors/cylinder
* 24622 cylinders
* 24620 accessible cylinders
*
* Flags:
* 1: unmountable
* 10: read-only
*
* Unallocated space:
* First Sector Last
* Sector Count Sector
* 0 2889 2888
* 8395434 4286577640 5777
* 71127180 4232235550 8395433
* 58730481 12396699 71127179
*
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
0 2 00 5778 8389656 8395433
1 3 01 8395434 4194828 12590261
2 5 00 0 71127180 71127179
3 15 01 2889 2889 5777
4 14 01 5778 71121402 71127179
5 0 00 33564402 25166079 58730480
6 4 00 12590262 4194828 16785089
7 7 00 16785090 16779312 33564401

Now, the VTOC is not updated. we need to use VxVM command to update VTOC. not really advisable to update using "format" command manually since its more error prone.

Tag Description (2nd column from right)
0x00 UNASSIGNED
0x01 BOOT
0x02 ROOT
0x03 SWAP
0x04 USR
0x05 BACKUP
0x06 STAND
0x07 VAR
0x08 HOME

Flag Description (right-most column)
0x00 MOUNTABLE
0x01 UNMOUNTABLE
0x10 READ-ONLY

Note that Swap is NOT mountable.


root@myserver # vxprint -htg rootdg
DG NAME NCONFIG NLOG MINORS GROUP-ID
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
V NAME RVG KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg rootdg all all 0 1076053596.1025.myserver

dm rootdisk c1t0d0s2 sliced 2632 71121402 -
dm rootmirror1 c1t1d0s2 sliced 2632 71121402 -
dm rootmirror2 c2t0d0s2 sliced 2632 71121402 -

v opt - ENABLED ACTIVE 25166079 ROUND - fsgen
pl opt-01 opt ENABLED ACTIVE 25166079 CONCAT - RW
sd rootdisk-05 opt-01 rootdisk 33558624 25166079 0 c1t0d0 ENA
pl opt-02 opt ENABLED ACTIVE 25166079 CONCAT - RW
sd rootmirror1-05 opt-02 rootmirror1 12584484 25166079 0 c1t1d0 ENA
pl opt-03 opt ENABLED ACTIVE 25166079 CONCAT - RW
sd rootmirror2-05 opt-03 rootmirror2 33558624 25166079 0 c2t0d0 ENA

v rootvol - ENABLED ACTIVE 8389656 ROUND - root
pl rootvol-01 rootvol ENABLED ACTIVE 8389656 CONCAT - RW
sd rootdisk-01 rootvol-01 rootdisk 0 8389656 0 c1t0d0 ENA
pl rootvol-02 rootvol ENABLED ACTIVE 8389656 CONCAT - RW
sd rootmirror1-01 rootvol-02 rootmirror1 0 8389656 0 c1t1d0 ENA
pl rootvol-03 rootvol ENABLED ACTIVE 8389656 CONCAT - RW
sd rootmirror2-01 rootvol-03 rootmirror2 0 8389656 0 c2t0d0 ENA

v swapvol - ENABLED ACTIVE 4194828 ROUND - swap
pl swapvol-01 swapvol ENABLED ACTIVE 4194828 CONCAT - RW
sd rootdisk-02 swapvol-01 rootdisk 8389656 4194828 0 c1t0d0 ENA
pl swapvol-02 swapvol ENABLED ACTIVE 4194828 CONCAT - RW
sd rootmirror1-02 swapvol-02 rootmirror1 8389656 4194828 0 c1t1d0 ENA
pl swapvol-03 swapvol ENABLED ACTIVE 4194828 CONCAT - RW
sd rootmirror2-02 swapvol-03 rootmirror2 8389656 4194828 0 c2t0d0 ENA

v usr - ENABLED ACTIVE 4194828 ROUND - fsgen
pl usr-01 usr ENABLED ACTIVE 4194828 CONCAT - RW
sd rootdisk-03 usr-01 rootdisk 12584484 4194828 0 c1t0d0 ENA
pl usr-02 usr ENABLED ACTIVE 4194828 CONCAT - RW
sd rootmirror1-03 usr-02 rootmirror1 37750563 4194828 0 c1t1d0 ENA
pl usr-03 usr ENABLED ACTIVE 4194828 CONCAT - RW
sd rootmirror2-03 usr-03 rootmirror2 12584484 4194828 0 c2t0d0 ENA

v var - ENABLED ACTIVE 16779312 ROUND - fsgen
pl var-01 var ENABLED ACTIVE 16779312 CONCAT - RW
sd rootdisk-04 var-01 rootdisk 16779312 16779312 0 c1t0d0 ENA
pl var-02 var ENABLED ACTIVE 16779312 CONCAT - RW
sd rootmirror1-04 var-02 rootmirror1 41945391 16779312 0 c1t1d0 ENA
pl var-03 var ENABLED ACTIVE 16779312 CONCAT - RW
sd rootmirror2-04 var-03 rootmirror2 16779312 16779312 0 c2t0d0 ENA

For automated step, try this
# vxbootsetup

For manual step, try these
Notice that you did NOT create the disk partitions manually when you initialized the mirror disk. This is because VxVM automatically does its partitioning for you. This is also true for the root disk after it has been re-initialized and brought into rootdg. That is, the original rootdisk partition scheme has been replaced by Veritas's partition scheme. This means that the underlying partitions for each volume(swap, /opt, /var, /home) are not physically laid out on the drives corresponding to the veritas volumes.

In some situations though, you might want to be able to access the partitions corresponding to each volume by manually mounting the slice. In a recovery situation for example, you may want to boot from cdrom and access the /opt partition by manually mounting the slice without the veritas drivers loaded. This is not possible unless you create the underlying partition mapping directly to the opt volume using the "vxmksdpart" command.

All underlying partitions except / should be created. The / partition is automatically created and is mapped correctly to the corresponding veritas root volume.

One Caveat, you cannot use partition 3 or 4 because this is typically where Veritas stores is private and public region. The following is snipped from the "format" utility listing the Veritas partitions. This shows that the root partition exists but the swap, /var, /opt, home partitions are not mapped on the drive(Veritas knows where they exist on the drive because of its private and public regions):

Part      Tag    Flag     Cylinders        Size            Blocks

  0       root    wm       1 - 2282        3.91GB    (2282/0/0)  8194662        <=== "/" exists
  1 unassigned    wm       0               0         (0/0/0)           0        <=== Unassigned
  2     backup    wm       0 - 4923        8.43GB    (4924/0/0) 17682084
  3          -    wu       0 -    0        1.75MB    (1/0/0)        3591        <=== Private Region
  4          -    wu       1 - 4923        8.43GB    (4923/0/0) 17678493        <=== Public Region
  5 unassigned    wm       0               0         (0/0/0)           0        <=== Unassigned
  6 unassigned    wm       0               0         (0/0/0)           0        <=== Unassigned
  7 unassigned    wm       0               0         (0/0/0)           0        <=== Unassigned

You may use the unassigned partitions which are 1, 5, 6, and 7 to map to the veritas volumes. Be sure to document which partitions map to the Veritas volumes for future reference. The "vxmksdpart" arguments are the subdisk (which maps directly to the volume), slice, tag and flag. Be sure to use the correct tag and flag for your particular Solaris version. Reference the man page for "fmthard" for correct options. Use the "vxprint" command to get the correct subdisk names: Generic command # /usr/lib/vxvm/bin/vxmksdpart

In my case
Rootdisk:
# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootdisk-02 1 0x03 0x01 <=== swap
# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootdisk-05 5 0x00 0x00 <=== /unassigned
# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootdisk-03 6 0x04 0x00 <=== /usr
# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootdisk-04 7 0x07 0x00 <=== /var

RootMirror:

# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootmirror1-02 1 0x03 0x01 <=== swap
# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootmirror1-05 5 0x00 0x00 <=== /unassigned
# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootmirror1-03 6 0x04 0x00 <=== /usr
# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootmirror1-04 7 0x07 0x00 <=== /var

References:
http://sysunconfig.net/unixtips/jmr_Mirroring_Boot_Disks_Veritas_Volume_Manager.htm
http://unixway.com/vm/veritasvm/rootdg.html

HowTo setup Link-based IPMP with VCS for Solaris 10.

Thanks Bret Sanders for the knowledge. Have used this to setup and like to share.

Original: http://www.runningunix.com/2009/01/link-based-ipmp-setup-with-vcs/

We know that Solaris 10 allows us to setup link-based IPMP. Though a nice feature which the OS determines NIC availability solely on the NIC driver reporting the physical link status - UP or DOWN, setting link-based IPMP in VCS is a little different.

Previous versions used “probe-based” IPMP, where connectivity is tested by pinging something on the network from each interface.

While probe-based is actually a more thorough test (tests network layer 3 as well as 2), it is much more cumbersome to configure, and you need an extra IP address for each interface for “test” addresses. usually Link-based IPMP is sufficient for most applications.

For some reason, configuring link-based IPMP in VCS is somewhat tricky, and the documentation doesn’t seem to help much. It seems all the default values for VCS are for probe-based IPMP only.

To achieve link-based IPMP, There are values you need to change from the defaults in the MultiNICB resource:

Tells VCS to use mpathd for network link status
UseMpathd: 1

The default, /usr/sbin/in.mpathd is just incorrect - it doesn’t live there.
MpathCommand: /usr/lib/inet/in.mpathd -a

If you leave this at 1, it will overwrite your /etc/hostname.xxx files with probe-based IPMP configuration
ConfigCheck: 0

The “interface alias” for each device is not needed, leave them blank.
Device: (your IPMP interfaces here)

You want VCS to NOT ignore link status, since this is how link-based IPMP works.
IgnoreStatus: 0

Do not use your IPMP group name here, it’s not needed. VCS is not monitoring the group, mpathd is.
GroupName:

Here’s how it looks in main.cf:

MultiNICB csgmultinic (
UseMpathd = 1
MpathdCommand = “/usr/lib/inet/in.mpathd -a”
ConfigCheck = 0
Device = { ce0 = “”, ce4 = “” }
IgnoreLinkStatus = 0
)

Check how sendmail is configured

# /usr/lib/sendmail -d0.99 -bt < /dev/null

Using sendmail to check fully qualified hostname and permission

root@myserver:/>/usr/lib/mail/sh/check-hostname
Hostname myserver OK: fully qualified as myserver.

root@myserver:/>/usr/lib/mail/sh/check-permissions
No unsafe directories found.

How to resolve "Unable to qualify my own domain name" error

You may have received this error in /var/adm/messages

Symptom 1:
/var/adm/messages
Mar 13 11:15:16 myserver sendmail[8420]: [ID 702911 mail.crit] My unqualified host name (myserver.) unknown; sleeping for retry
Mar 13 11:15:16 myserver sendmail[8421]: [ID 702911 mail.crit] My unqualified host name (myserver.) unknown; sleeping for retry
Mar 13 11:16:16 myserver sendmail[8421]: [ID 702911 mail.alert] unable to qualify my own domain name (myserver.) -- using short name
Mar 13 11:16:16 myserver sendmail[8420]: [ID 702911 mail.alert] unable to qualify my own domain name (myserver.) -- using short name

Symptom 2:
when you try to manually telnet localhost 25 or mailx, either you can send to a particular domain or the respond is very slow after entering the "mail from" command.

Resolution:
update /etc/hosts ( and /etc/inet/ipnodes if you are on solaris 10) to use the following.

root@myserver:/>more /etc/hosts
# internet host table
#====================
127.0.0.1 localhost
10.106.127.105 myserver. myserver
10.106.63.32 myserver-rsc


#
# Internet host table
#
::1 localhost
127.0.0.1 localhost
10.106.127.105 myserver. myserver loghost

Description:
Usually i would add the "myserver." with the dot behind to /etc/hosts but in Solaris 10, things work a little differently.

It turn out that in solaris 10, the OS will go through /etc/inet/ipnodes for IPv4 address before going to /etc/hosts.

In this case, since ldap does not have the entry, OS will go straight to /etc/inet/ipnodes.

This also mean that if you change the host IP of the solaris 10 server, please change in /etc/inet/ipnodes as well otherwise you have conflict of IP addresses.

Note that this is Solaris specific.

# man ipnodes
...
...
NOTES
IPv4 addresses can be defined in the ipnodes file or in the
hosts file. See hosts(4). The ipnodes file will be searched
for IPv4 addresses when using the getipnodebyname(3SOCKET)
API. If no matching IPv4 addresses are found in the ipnodes
file, then the hosts file will be searched. To prevent
delays in name resolution and to keep /etc/inet/ipnodes and
/etc/inet/hosts synchronized, IPv4 addresses defined in the
hosts file should be copied to the ipnodes file.
...
..

# more /etc/nsswitch.conf
...
...
# consult /etc "files" only if ldap is down.
hosts: ldap [NOTFOUND=continue] files
...
...
# Note that IPv4 addresses are searched for in all of the ipnodes databases
# before searching the hosts databases.
...
...

Moving large directories in Solaris

Learnt a neat way to transfer a large directory of files, esp useful if you are going to move more than 8Gb of data.

1) cd into that directory.
2) use the following command

# tar cpBEf - * | ( cd /; tar xBEf -)

Do note that the "E" flag is very important if you are copying files that are larger than 8Gb.

Persistent route in Solaris 10

Found a neat feature in Solaris 10.
An improvement for static routing.
Simply type 'route -p' for persistent route which Solaris will store the entry in /etc/inet/static_routes.

One more excuse not to use your own routing file and putting into /etc/rc3.d/.

i.e.
# route -p add net 10.10.10.0 10.50.50.254 1
# more /etc/inet/static_routes
# File generated by route(1M) - do not edit.
net 10.10.10.0 10.50.50.254 1

Have fun!

Getting "Host key verification failed" error message when you try to ssh to any server including localhost

In case any of you ever get the "Host key verification failed" error message when you try to ssh to any server including localhost. Thanks kaisen.

Symptoms:
Unable to ssh to any server including localhost when using a non-root account.
Consistently getting the "Host key verification failed" error message even when

~/known_hosts is already empty or doesn't exist
~/known_hosts never gets updated with new host keys

Reason:
Device pointed to by /dev/tty is not of 666 permission.

Resolution:
chmod 666 /dev/tty

Example:
Failed ssh attempts to localhost and other servers:

root@myserver:/etc/ssh>su - account1
myserver% ssh devusr@localhost
Host key verification failed.

myserver% ssh devusr@myserver
Host key verification failed.

root@myserver:/etc/ssh>su - myname
-bash-3.00$ ssh -x myname@myserver
Host key verification failed.

-bash-3.00$ ssh -x myname@myserver
Host key verification failed.

The culprit:
root@myserver:/etc/ssh>ls -l /dev/tty
lrwxrwxrwx 1 root other 26 Sep 11 2006 /dev/tty -> ../devices/pseudo/sy@0:tty

root@myserver:/etc/ssh>ls -l /devices/pseudo/sy@0:tty
crw--w---- 1 root tty 22, 0 Dec 15 20:20 /devices/pseudo/sy@0:tty
chmod 666 /dev/tty:

root@myserver:/etc/ssh>chmod 666 /devices/pseudo/sy@0:tty
root@myserver:/etc/ssh>ls -l /devices/pseudo/sy@0:tty
crw-rw-rw- 1 root tty 22, 0 Dec 15 20:20 /devices/pseudo/sy@0:tty

root@myserver:/etc/ssh>su - account2
-bash-3.00$ ls -l /devices/pseudo/sy@0:tty
crw-rw-rw- 1 root tty 22, 0 Dec 15 20:20 /devices/pseudo/sy@0:tty

ssh should now works and you are able to add new host keys to ~/known_hosts:

-bash-3.00$ ssh -x myname@myserver
The authenticity of host 'myserver (10.100.10.100)' can't be established.
RSA key fingerprint is 43:27:c1:aa:b9:c9:f1:21:33:11:aa:dd:ee:ff:10:9f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'myserver,10.100.10.100' (RSA) to the list of known hosts.

Password:
Last login: Mon Dec 15 19:20:41 2008 from myserver2
Sun Microsystems Inc. SunOS 5.9 Generic May 2002
$ Connection to myserver closed.
-bash-3.00$ logout

Retrieving errors generated in FMA in Solaris 10

Found this script circulating.

NB: Matty, hope you don't mind me listing this useful stuff to share.

#!/bin/bash
#
# Program: E-mail fault manager errors # # Author: Matty < matty91 at gmail dot com > # # Current Version: 1.0 # # Revision History:
#
# Version 1.0
# Initial Release
#
# Last Updated: 08-18-2006
#
# Purpose:
# Fmadm.sh queries the fault manager to see if errors have been
# generated. If an error is detected, the script will email the
# admininstrator defined in the ADMIN vairable with the error
# details.
#
# License:
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
#
# Installation:
# Copy the shell script to a suitable location
#
# Usage:
# To check for events once per hour, add a cron job similar to the following:
#
# $ crontab -l | grep fmadmnotifier.sh
# 0 * * * * /etc/scripts/fmadmnotifier.sh
#
PATH=/usr/bin:/sbin:/usr/sbin:/usr/sfw/bin
# Who to E-mail with new updates
ADMIN="root"
# Location of binaries
AWK=$(which awk)
FMADM=$(which fmadm)
HOSTNAME=$(which hostname)
MAIL=$(which mailx)
MKTEMP=$(which mktemp)
# Check to make sure the mail binary exists if [ ! -f ${MAIL} ] then
echo "Cannot find ${MAIL}"
exit 1
fi
# Check to make sure the fmadm utility exists if [ ! -f ${FMADM} ] then
echo "Cannot find ${FMADM}"
exit 1
fi
# Verify that mktemp exists
if [ ! -f ${MKTEMP} ]
then
echo "Cannot find ${MKTEMP}"
exit 1
fi
# Run fmadm fauly to check for hardware errors FMADMOUTPUT=$(${FMADM} faulty | ${AWK} '$0 !~ /STATE/ && $0 !~ /^----/ { print $0 }')
if [ -n "${FMADMOUTPUT}" ]
then
WORK=`${MKTEMP} /tmp/fmadm.results.XXXXXX`
echo "The fault manager detected a problem with the system hardware." >> ${WORK}
echo "The fmadm and fmdump utilities can be run to retrieve additional" >> ${WORK}
echo "details on the faults and recommended next course of action. " >> ${WORK}
echo "" >> ${WORK}
echo "fmadm faulty output:" >> ${WORK}
echo "" >> ${WORK}
# TODO: Find a way to do all of the above in memory
${FMADM} faulty >> ${WORK}
HOST=$($HOSTNAME)
cat ${WORK} | ${MAIL} -s "Hardware fault on ${HOST}" ${ADMIN}
rm -f ${WORK}
fi

How to fix X forwarding broken by Aug 09 EIS on Solaris

Quite a while back, encountered X forwarding broken for many of the servers.

Basically, this is caused by the SSH patch due to regression in SSHD

Solaris 9 with patch 122300-28 or later
Solaris 10 with patch 126133-03 or later
Solaris 10 with patch 138060-03 and without patch 138060-04

To work around this issue, the following command can be executed as root user:
# ifconfig lo0 inet6 plumb up

More information can be found in this article - sun link

Link based IPMP vs probe based IPMP in solaris 10

From the sunsolve doc http://sunsolve.sun.com/search/document.do?assetkey=1-9-86869-1 , there are 3 modes of IPMP that we can configure,

Probe based IPMP - active standby setup
Probe based IPMP - active active setup
Linke based IPMP - active standby setup

The only difference on Probe based IPMP active-active' setup versus active-standby's setup is the word "deprecated" in the second configuration file. When you add in the "deprecated" tag, the network traffic would actually NOT go through the physical IP. When you snoop on the interface the traffic will go out on your virtual IPs.

Link-based IPMP

For link-based failure detection, only the link between local interface and the link partner is checked on hardware layer. Neither IP layer nor any further network path will be monitored.

No test addresses are required for link-based failure detection. So the pro here is that you save on the number of IP. But then if you are on your own private network, are you sure you have some many IPs that you would ran out of it? Most likely the reason is the ease of IP management.

Probe-based IPMP

Probe-based failure detection is performed on each interface in the IPMP group that has a test address. Using this test address, ICMP probe messages go out over this interface to one or more target systems on the same IP link.

The in.mpathd daemon determines which target systems to probe dynamically. The whole network path up to the gateway (router) is monitored on IP layer. With all interfaces in the IPMP group connected via redundant network paths (switches etc.), you get full redundancy.

On the other hand the default router can be a single point of failure, resulting in 'All Interfaces in group have failed'.

Conclusion

Meaning that probe based IPMP monitors the path up to the gateway while link based IPMP monitors only up to the next physical link. Nothing more nothing less.

Link based IPMP cant 'see' what's after this physical link.

I still prefer probe based IPMP as i have more ease when troubleshooting to determine whether i have connection all the way to the destination. Using link based IPMP means that i would have to get the network guys to check for me if the connection is down.

Note: netstat -k seem to be dropped in solaris 10.

Link based IPMP on Solaris 10

Setting up Link based IPMP in solaris 10 is much more easier than probe based IPMP.

Lets see what NIC i have in my server..


root ~>#dladm show-dev
bge0 link: up speed: 1000 Mbps duplex: full
bge1 link: up speed: 1000 Mbps duplex: full
bge2 link: unknown speed: 0 Mbps duplex: unknown
bge3 link: up speed: 100 Mbps duplex: full


So i have 3 NIC connected, lets use bge0 and bge1 for our link based IPMP. Just use the following configuration.


root ~># more /etc/hostname.bge*
::::::::::::::
/etc/hostname.bge0
::::::::::::::
myserver netmask + broadcast + group production up
::::::::::::::
/etc/hostname.bge1
::::::::::::::
group production up


Remember to put the IPs in the /etc/hosts, netmask in /etc/defaultrouter.

Verify that IPMP daemon is running.


root ~>#pgrep -lf mpathd
165 /usr/lib/inet/in.mpathd -a


Another indication that you are using link based IPMP instead of probe based IPMP is the following message appearing in your console or /var/adm/messages.


Aug 14 10:41:44 in.mpathd[155]: No test address configured on interface bge1; disabling probe-based failure detection on it
Aug 14 10:41:44 in.mpathd[155]: No test address configured on interface bge0; disabling probe-based failure
detection on it

Now, we are ready to do some fail over test.


# if_mpadm -d bge0

root@png2gw2:~>#ifconfig -a
bge0: flags=89000842 mtu 0 index 2
inet 0.0.0.0 netmask 0
groupname production
ether 0:14:4f:91:d:5c
bge1: flags=1000843 mtu 1500 index 3
inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255
groupname production
ether 0:14:4f:91:d:5d
bge1:1: flags=1000843 mtu 1500 index 3
inet 10.55.9.192 netmask ffffff00 broadcast 10.55.9.255

Noticed that the IP in bge0 has been transfered into bge1:1. You may also notice the following will appear in your console or /var/adm/messages.


Aug 14 10:05:10 myserver in.mpathd[165]: [ID 832587 daemon.error] Successfully failed over from NIC bge0 to NIC bge1


So we are quite done. Lets recover and restore the IP


root ~>#if_mpadm -r bge0

In /var/adm/message,

Aug 14 10:05:10 myserver in.mpathd[165]: [ID 832587 daemon.error] Successfully failed over from NIC bge0 to NIC bge1
Aug 14 10:07:26 myserver in.mpathd[165]: [ID 620804 daemon.error] Successfully failed back to NIC bge0

We have restored the NIC.

root ~>#ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
bge0: flags=1000843 mtu 1500 index 2
inet 10.55.9.192 netmask ffffff00 broadcast 10.55.9.255
groupname production
ether 0:14:4f:91:d:5c
bge1: flags=1000843 mtu 1500 index 3
inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255
groupname production
ether 0:14:4f:91:d:5d

While doing this test, i also noticed that when the failover was done, bge1 is still 0.0.0.0 and bge0's IP was plumbed to bge1:1.

From some research and experiment, i found out bge1 can be acutally configured with its own IP, such that it can still provide service, thus allowing 1 more NIC to work.

When the failover happens, both IP will serve business as usual.Here's the test.

bge0 and bge1 has IP plumbed on them. let do a Telnet to 10.55.9.192 (bge0)

myclient -> myserver TCP D=22 S=57645 Syn Seq=2664010663 Len=0 Win=49640 Options=

Also do a Telnet to 10.55.9.198 (bge1)

myclient -> 10.55.9.198 TCP D=22 S=57656 Syn Seq=2669824191 Len=0 Win=49640 Options=
10.55.9.198 -> myclient TCP D=57656 S=22 Syn Ack=2669824192 Seq=478613400 Len=0 Win=49640 Options=

myclient -> 10.55.9.198 TCP D=22 S=57656 Ack=478613401 Seq=2669824192 Len=0 Win=49640
10.55.9.198 -> myclient TCP D=57656 S=22 Push Ack=2669824192 Seq=478613401 Len=20 Win=49640

myclient -> 10.55.9.198 TCP D=22 S=57656 Ack=478613421 Seq=2669824192 Len=0 Win=49640

Because I did not log in, /var/adm/message 'complain'

Aug 14 10:16:27 myserver sshd[24576]: [ID 800047 auth.info] Did not receive identification string from 10.10.140.36
Aug 14 10:16:40 myserver sshd[24579]: [ID 800047 auth.info] Did not receive identification string from 10.10.140.36

Lets fail Bge0 now, monitor the /var/adm/message and snoop output. Acutally all traffic is now going on bge1:1.

Aug 14 10:19:38 myserverin.mpathd[165]: [ID 832587 daemon.error] Successfully failed over from NIC bge0 to NIC bge1

myclient -> myserver TCP D=22 S=57682 Syn Seq=2724219601 Len=0 Win=49640 Options=
myserver -> myclient TCP D=57682 S=22 Syn Ack=2724219602 Seq=2859863037 Len=0 Win=49640 Options=

Traffic on 10.55.9.198 (bge1) is unaffected.

myclient -> 10.55.9.198 TCP D=22 S=57685 Syn Seq=2740048358 Len=0 Win=49640 Options=
10.55.9.198 -> myclient TCP D=57685 S=22 Syn Ack=2740048359 Seq=1234692052 Len=0 Win=49640 Options=

myclient -> 10.55.9.198 TCP D=22 S=57685 Ack=1234692053 Seq=2740048359 Len=0 Win=49640
10.55.9.198 -> myserver TCP D=57685 S=22 Push Ack=2740048359 Seq=1234692053 Len=20 Win=49640
myclient -> 10.55.9.198 TCP D=22 S=57685 Ack=1234692073 Seq=2740048359 Len=0 Win=49640

We restore the NIC.

Aug 14 10:35:08 myserver in.mpathd[165]: [ID 620804 daemon.error] Successfully failed back to NIC bge0

Some of my references are:
http://sunsolve.sun.com/search/document.do?assetkey=1-61-211105-1
http://raulsg.wikispaces.com/ipmp-link-based
http://os.miamano.eu/node/25