How to resolve MQ Channel Sequence Mismatch Error (AMQ9526)

This error would like occur when the local and remote queue managers do not agree on the next message sequence number. For example, when channels goes down or restarted due to network or other issues.

You will see the following error in the Queue manager error log.

AMQ9526 Message sequence number error for channel

So, how to correct this problem??

1) Do a display channel status
DIS CHS(XXXXXXX)
for the sender channel

2. If the channel is still running,
STOP CHANNEL(XXXXXXX) MODE(FORCE).

This will stop transmission of the current batch and will likely result in message being in-doubt.

From my experience so far, i would only issue

stop chl(XXX)

I have not encountered the need to use FORCE yet. Maybe use FORCE when you cant stop the channel the first time round.

3. Lets clear up any in-doubt messages

RESOLVE CHANNEL (XXXXXXX) ACTION(BACKOUT).

This requests channels to commit or back out in-doubt messages. This command can be used only for sender (SDR), server (SVR), and cluster-sender (CLUSSDR) channels (including those that have been defined automatically). The messages are backed out, that is, they are restored to the transmission queue.

So far, i have no need to do step 3.

** IMPORTANT **
From researching the internet, in 99.9% of cases you never use the facility and it's actually dangerous to do so. In-doubt channel problems are usually resolved automatically.

Under certain unusual conditions, the channel can't resolve the situation itself. For this situtation the resolve channel command has been provided so the MQ administrator, using facilities unavailable to the MCAs, can determine what to do to resolve the channel and instruct the system accordingly.

** IMPORTANT **

Do not use RESOLVE CHANNEL (XXXXXXX) ACTION(COMMIT) without considering that the messages are committed, that is, they are deleted from the transmission queue


4. Reset the sequence number

RESET CHANNEL(XXXXXXXX) SEQNUM(1).

This resets the message sequence number for a MQ channel with an optionally specified sequence number to be used the next time the channel is started.

As the sequence number is set by the sender channel so there is no need to reset the receiver.

The sequence number of the last committed message or LUWID is recorded at the receiving end. This is used not only at the sending end but also during re-sequencing, on startup and restarts to ensure that both ends of the link agree on which message have been sent.


5. ALmost there. Start up the channel.

START CHANNEL(XXXXXXX)

6. Done.

Additional information:

Each persistent message that is sent using message sequencing is tagged with an individual sequence number. These are incremented by 1 for each message sent, first on the sending end then on the receiving end. These numbers must match for sequential delivery to work correctly. So, if a message is sent with sequence number of 412 then the receiver side needs to be looking for message number 412.


more references from my favourite places -

Find and Replace Script

#!/bin/ksh
#
# Modify to your needs if different from *.txt
#

for THISFILE in *.txt; do
mv $THISFILE $THISFILE.backup
sed 's/findstr/replacestr/g' $THISFILE.backup > $THISFILE
done

Recover or Reset root password in Sun solaris (SPARC)

How many times have you encountered root password expired and got yourself logged out?

There is every little chance that one loses or forget the root password of his Sun Solaris servers. But then, in the enterprise world, we normally set the password aging so there, the password expired before you know it. My team put up a cron to monitor things like this as Solaris tend to warn you by putting a message in /var/adm/messages.

In the event, root password still expires, too bad. You need to shutdown your applications before performing the below procedures. This is the only way.

1) GO to OBP and Boot the server into Single User mode

OK boot cdrom -s
or
OK boot net -s


2) Mount the root file system (e.g. /dev/dsk/c0t0d0s0) to say, /a

For step 1, please use a Sun Solaris OS CD 1 or a jumpstat server in your environment! No password needed here.

Note that /a is a temporary mount point that is available when you boot from CD or a JumpStart server

edit /a/etc/shadow. If you remember your password, edit the expiry age. Else just blank out the encrypted password portion.

Once done. umount and reboot.

# cd /
# umount /a
# sync
# sync
# init s

If you have blank out the password, press enter at the prompt to enter a password for root else let the server progress to level 3. Reboot again to verify. Once in multi-user mode, log in and test the root password now.

Why we change the root password at level S is only to ensure that the root user at the state without password is not exposed to others. If started in multi-user mode, there is a degree of security risk till the root password is set.

Should say that the servers should be kept out of reach from casual users so that your server will not be so easily 'defeated', worse, lost, like the case in HK recently (May 2008).

----------------

One trick i tried, if somehow, someone still have a root session with the server, use the login command. This should prompt you that password has expired and force you to change. Follow accordingly and you should be able to use the new root password now... :) Good luck!

Change IP Address without rebooting in Solaris 10

Wonderful Solaris 10 huh? normally i would have to reboot the server whenever host IP is changed, now adding or editing the IP address on a Solaris 10 server need not need a reboot.

before Solaris 10, you need to edit the following files

/etc/hosts
/etc/nodenames
/etc/hostname.[device]
/etc/defaultrouter
/etc/defaultdomain
/etc/nodename

and add or modify the entries for the IP address and the hostname.

Be sure to check your /etc/netmasks too if you have network changes.

Example:
192.168.1.1 myserver

In Solaris 10, we need to edit BOTH

/etc/hosts (symlink to /etc/inet/hosts file) AND
/etc/inet/ipnodes

adding an entry for IP address and hostname plue those above mentioned ones.

Once done, do either of the following

1) svcadm restart network/physical <--- restart network service
2) reboot the server

From the man page, ipnodes file is read first before the system check hosts file. I hit this problem before when i changed the hostname/IP of a Solaris 10 server and got the network errors on my ALOM console.

Although, the /etc/inet/ipnodes files is primarily for IPv6 only, without adding an entry to the file, the IP address (IPv4) doesn’t become active in Solaris 10, at least on the release version that i'm using. This seems to be solved in Solaris 10 U4 (08/07 build).

If you need to add addresses, you must add IPv4 addresses to both the hosts and ipnodes files. You add only IPv6 addresses to the ipnodes file.

IPMP Configuration (Probe based)

Lets set up using 2 physical interface with 2 virtual IPs.

Physical interfaces ce0, ce3
Logical interfaces ce0:1, ce0:1

Lets put the IPs into the /etc/hosts.

172.20.1.10 mynic-ce0
172.20.1.11 mynic-ce3

172.20.1.12 application1-service
172.20.1.13 application2-service

Verify local-mac-address? is set to true.
# eeprom local-mac-address?
local-mac-address?=true

Impt! - setting local-mac-address to true will not take effect until next reboot*

If this host will not be forwarding packets set the following.
# touch /etc/notrouter

Manually plumb the interfaces (ignore if you plumb them already)

# ifconfig ce0 plumb mynic-ce0 netmask + broadcast + -failover deprecated up
# ifconfig ce3 plumb mynic-ce3 netmask + broadcast + -failover deprecated up
# ifconfig ce0 addif application1-service + broadcast + failover up
# ifconfig ce3 addif application2-service netmask + broadcast + failover up

Check using /usr/sbin/ifconfig -a.

Lets make the configuration persistant across reboots.

--- /etc/hostname.ce0 ---
mynic-ce0 netmask + broadcast + group production deprecated -failover up \
addif application1-service netmask + broadcast + failover up \
addif application2-service netmask + broadcast + failover up
--- EOF ---

--- /etc/hostname.ce3 ---
mynic-ce3 netmask + broadcast + group production deprecated -failover up
--- EOF ---

8) Now lets test the failover by pulling the ce0 cable. watch /var/adm/messages for the errors.

9) you can now get the applications and users to use the IP application1-service and application2-service

Impt!! network traffic on application1-service can be incoming and outgoing BUT network traffic on application2-service can only be incoming. I have tried to play around with the deprecated tag but only managed to get network traffic going out from application1-service. Well, if you have any advise on this, do let me know!! :)

Enable/Disable IP Forwarding in Solaris 10 without reboot

In Solaris 10, there is this feature of IP forwarding.

This is the process of forward/routing the packets between network interfaces on one system. Meaning to say that the packet for a host on a different network arrive on one of the network interface. This will be forwarded to the appropriate network interface.

We can enable or disable using the following commands:

1) routeadm
2) ifconfig

Read from other websites that in Solaris 9, ndd command is used.

The advantage here in Solaris 10 is that the change is dynamic, real-time and the change is persistant across reboot unlike the ndd command.

Example: Enable/Disable IP Forwarding Globally

# routeadm -e ipv[4|6]-forwarding
# routeadm -d ipv[4|6]-forwarding

Use either 4 or 6 for [4|6] options.
The switches “-e” enables IP Forwarding.
The switches “-d” enables IP Forwarding.

Once done, use either one of the steps below let the new setting take effect.

1) reboot
2) routeadm -u
3) svcadm enable svc:/network/ipv[4|6]-forwarding

The option -u as digged from the man page.
Apply the currently configured options to the running system. These options might include enabling or disabling IP forwarding and launching or killing routing daemons, if any are specified. It does not alter the state of the system for those settings that have been set to default. This option is meant to be used by administrators who do not want to reboot to apply their changes. In addition, this option upgrades on-SMF configurations from the invocations of daemon stop commands, which might include a set of arguments, to a simple enabling of the appropriate service.

To revert? do the following:

# routeadm -r ipv[4|6]-forwarding
# routeadm -u

Example: Enable/Disable IP Forwarding on a particular interface

If we want to work on the ce0 interface using the ifconfig command.

In IPv4
# ifconfig ce0 router <--- enable
# ifconfig ce0 -router <--- disable

In IPv6
# ifconfig ce0 inet6 router <--- enable
# ifconfig ce0 inet6 -router <--- disable

More References:
http://gibbs.acu.edu/2007/02/24/using-solaris-10-as-a-firewallrouter/

Save an expired VxVM system

VxVM is normally installed with 30 days demo licence. After which if you reboot, only the partitions essential to booting up will be loaded. meaning the following partition will be disabled and not mounted. This also mean that without a reboot, your system will work as though VxVM has not expired.

/opt
/var
swapvol

My Experience with this was that i cant even use 'vi' as swapvol is not available. So Basically, there's nothing much you can do.

So, go to single user more. Check your Volume Manager License. Should see that your temp / demo license has expired.

# /usr/sbin/vxlicrep | more

So now we try to install the permanent license. Key in your license key when prompted.

# /usr/sbin/vxlicinst

VERITAS License Manager vxlicinst utility version 3.02.005
Copyright (C) 1996-2004 VERITAS Software Corp. All Rights reserved.
 
Enter your license key : XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XX
 
License key successfully installed for VERITAS Volume Manager

Note: The license keys can coexist on the same machine. ie. if you had installed a previous license, you can continue adding in new ones.
Note: Licences are stored in /etc/vx/licenses/lic

Tip:
If your system do not have the vxlicinst command, you do not have the VRTSvlic package. Try mounting a remote /opt/VRTS/ from a server in the same network segment.


Now you can check the status, you will see (shown only portion of the listing)

# vxprint -ht
 
v  opt          -            DISABLED ACTIVE   94372224 ROUND     -        fsgen
pl opt-01       opt          DISABLED ACTIVE   94372224 CONCAT    -        RW
sd rootdisk-05  opt-01       rootdisk 146798976 94372224 0        c1t0d0   ENA
pl opt-02       opt          DISABLED ACTIVE   94372224 CONCAT    -        RW
sd rootmirror-05 opt-02      rootmirror 146798976 94372224 0      c1t1d0   ENA
 
v  swapvol      -            DISABLED ACTIVE   31423488 ROUND     -        swap
pl swapvol-01   swapvol      DISABLED ACTIVE   31423488 CONCAT    -        RW
sd rootdisk-02  swapvol-01   rootdisk 41945472 31423488 0         c1t0d0   ENA
pl swapvol-02   swapvol      DISABLED ACTIVE   31423488 CONCAT    -        RW
sd rootmirror-02 swapvol-02  rootmirror 41945472 31423488 0       c1t1d0   ENA
 
v  var          -            DISABLED  ACTIVE   52447104 ROUND     -        fsgen
pl var-01       var          DISABLED  ACTIVE   52447104 CONCAT    -        RW
sd rootdisk-04  var-01       rootdisk 94351872 52447104 0         c1t0d0   ENA
pl var-02       var          DISABLED  ACTIVE   52447104 CONCAT    -        RW
sd rootmirror-04 var-02      rootmirror 94351872 52447104 0       c1t1d0   ENA

To save yourself (haha...) and the server, try the following on all the partition that is disabled,

# vxvol -g  start 
i.e. vxvol -g rootdg start opt
i.e. vxvol -g rootdg start var
i.e. vxvol -g rootdg start swapvol

The volumes should now be enabled.

vxprint -ht

Thats it. Reboot and check your system...

To clean up defunct process

Use this command preap .

Find out the environment the process is running in.

For the process 27480 for example, you can try the below to check out the environment the process is running in.

pargs -e 27480 
envp[0]: PWD=/opt/oracle/product/9.2.0/dbs
envp[1]: TZ=Singapore 
...
...

This is useful when used together with the following commands as well. Here you can find the full path of the process. Also the connections and sub processes that it has made.

/usr/ucb/ps -auxxw | grep 27480
lsof | grep 27480

"unable to qualify my own domain name" error

It is not so long ago that i saw this errors on my console..

Symptom 1:

/var/adm/messages
Mar 13 11:15:16 pnsgsit1gw1 sendmail[8420]: [ID 702911 mail.crit] My unqualified host name (pnsgsit1gw1.) unknown; sleeping for retry
Mar 13 11:15:16 pnsgsit1gw1 sendmail[8421]: [ID 702911 mail.crit] My unqualified host name (pnsgsit1gw1.) unknown; sleeping for retry
Mar 13 11:16:16 pnsgsit1gw1 sendmail[8421]: [ID 702911 mail.alert] unable to qualify my own domain name (pnsgsit1gw1.) -- using short name
Mar 13 11:16:16 pnsgsit1gw1 sendmail[8420]: [ID 702911 mail.alert] unable to qualify my own domain name (pnsgsit1gw1.) -- using short name

Symptom 2:
when you try to manually telnet localhost 25 or mailx, either you can send to a particular domain or the respond is very slow after entering the "mail from" command.

Resolution:
update /etc/hosts ( and /etc/inet/ipnodes if you are on solaris 10) to use the following.

root@abc:/>more /etc/hosts
# internet host table
#====================
127.0.0.1       localhost
10.100.127.105  abc. abc
10.100.63.32            abc-rsc

 
#
# Internet host table
#
::1     localhost
127.0.0.1       localhost
10.100.127.105  abc.    abc     loghost 

Description:
it turn out that in solaris 10, the OS will go through /etc/inet/ipnodes for IPv4 address before going to /etc/hosts. in this case, if you have ldap configure and ldap does not have the entry, OS will go straight to /etc/inet/ipnodes.

This also mean that if you change the host IP of the solaris 10 server, please change in /etc/inet/ipnodes as well otherwise you have conflict of IP addresses.

note that this is Solaris specific.

# man ipnodes
...
...
NOTES
IPv4 addresses can be defined in the ipnodes file or in the
hosts file. See hosts(4). The ipnodes file will be searched
for IPv4 addresses when using the getipnodebyname(3SOCKET)
API. If no matching IPv4 addresses are found in the ipnodes
file, then the hosts file will be searched. To prevent
delays in name resolution and to keep /etc/inet/ipnodes and
/etc/inet/hosts synchronized, IPv4 addresses defined in the
hosts file should be copied to the ipnodes file.
...
..

# more /etc/nsswitch.conf
...
...
# consult /etc "files" only if ldap is down.
hosts: ldap [NOTFOUND=continue] files
...
...
# Note that IPv4 addresses are searched for in all of the ipnodes databases
# before searching the hosts databases.
...
...


---------------------------
Addition.

Here are 2 commands that you can hitch on to check if the host has fully qualified name.

root:/>/usr/lib/mail/sh/check-hostname
Hostname abc OK: fully qualified as abc.

root:/>/usr/lib/mail/sh/check-permissions
No unsafe directories found.

"WARNING: add_spec: No major number for mpt" error

If you hit the following errors when you reboot the server, just as the server is almost ready. They will come in a large numbers repeatedly. The errors are usually on your console and /var/adm/messages.

"WARNING: add_spec: No major number for mpt"

You may want to try the solutions below. This is from sunsolve.. I have tried out solution 2 successfully.

Document Audience: SPECTRUM 
Document ID: 74181 
Title: Solaris[TM]: "WARNING: add_spec: No major number for mpt" 
Update Date: Wed Sep 29 00:00:00 MDT 2004 
Products:  Solaris 8 Operating System,  Solaris 9 Operating System 
Technical Areas:  Patch 

--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

Keyword(s):add_spec, mpt, major number

Problem Statement:  

After installing a patch to update /etc/driver_classes with
an "mpt" entry (for example, 108528-xx with xx>21), the system may
generate the following WARNING messages at boot time:

 SunOS Release 5.8 Version Generic_108528-29 64-bit
 Copyright 1983-2003 Sun Microsystems, Inc.  All rights reserved.
 WARNING: add_spec: No major number for mpt
 WARNING: add_spec: No major number for mpt
 WARNING: add_spec: No major number for mpt
 WARNING: add_spec: No major number for mpt
 [...]
 WARNING: add_spec: No major number for mpt
 WARNING: add_spec: No major number for mpt
 WARNING: add_spec: No major number for mpt
 configuring IPv4 interfaces: hme0.
 configuring IPv6 interfaces: hme0.

The following message appears in the /var/sadm/patch//log file
when the entry has not been successfully added to the /etc/name_to_major
file:
          SUNWcsr: failed to add mpt to  /etc/name_to_major:
          (mpt) already in use as a driver or alias.

Explanation:
============
The problem is due to a reference in /etc/driver_classes to the
mpt driver. The mpt driver isn't added to the system because the
/etc/driver_aliases file already had an entry with the mpt driver. Because
of this, the add_drv failed when trying to install the driver.

This is an example of the inconsistencies described in Bug ID 4939994,
"Inconsistency between name_to_major and driver_aliases."


Resolution:  

To solve this problem, a script has been created to identify the
inconsistencies between /etc/driver_aliases and /etc/name_to_major.
This script can also add the missing entries if they are listed in a
"reference" name_to_major file.

Here is an example of the way to use the attached script:

# cksum InconsitencyFixTool.tar.gz
1091730651      4410    InconsitencyFixTool.tar.gz

# gzip -dc InconsitencyFixTool.tar.gz | tar -xvf -
x InconsitencyFixTool, 0 bytes, 0 tape blocks
x InconsitencyFixTool/README, 1364 bytes, 3 tape blocks
x InconsitencyFixTool/name_to_major.InconsitencyFix, 4928 bytes, 10 tape blocks
x InconsitencyFixTool/name_to_major.i386.5.8, 841 bytes, 2 tape blocks
x InconsitencyFixTool/name_to_major.i386.5.9, 889 bytes, 2 tape blocks
x InconsitencyFixTool/name_to_major.sparc.5.8, 1831 bytes, 4 tape blocks
x InconsitencyFixTool/name_to_major.sparc.5.9, 1815 bytes, 4 tape blocks

# cd InconsitencyFixTool
# ./name_to_major.InconsitencyFix
Saving original files into .
Inconsistency found between //etc/driver_aliases and //etc/name_to_major on
the following driver(s):
mpt

Add  mpt  to //etc/name_to_major ? [y] y
Adding the following devices to //etc/name_to_major :
mpt 215
# 

Reboot the system.


Temporary Workaround:  

If the above script cannot be used, there are two other ways to fix the
problem:

1. Remove the "mpt" lines from /etc/driver_classes and /etc/driver_aliases.

   They should look like the following:
     driver_aliases:mpt "pci1000,30"
     driver_classes:mpt      scsi

   Next, install (or remove and reinstall) a patch updating all
   those files [/etc/driver_aliases /etc/driver_classes  
   /etc/name_to_major]. For example, install the mpt patch 115275-01 (or
   above) or a new kernel Update patch.

   OR

2. Add the missing "mpt" entry in the /etc/name_to_major file
   to correct the problem with the patch installation.

   You must manually append the "mpt" entry at the end
   of the /etc/name_to_major file as follows:

          mpt XXX

   Where XXX is the maximum+1 of the numbers already given to
   the other drivers in this file. Separate "mpt" and the given number with
   a space character.

   For example:

   # tail /etc/name_to_major
   fasttrap 223
   dmfe 224
   todds1307 225
   pool 226
   zcons 227
   ipf 228
   pfil 229
   ctsmc 230
   bl 231
   mpt 232

   Reboot the system.



Additional Information:  

History:
========
Two patches have been identified that can cause the problem:

    108974-31(or greater): SunOS 5.8: dada, uata, dad, sd, ssd and scsi
    drivers patch
    
    OR

    109885-14 SunOS 5.8: glm patch

    Only on top of a kernel patch 108528-xx where xx <= 21.

Note: If you have a more recent version than -21 before installing
the above patches, the problem will not occur.

This problem can also exist on Solaris 9 - if S9 was installed with
an upgrade from a S8 system where the inconsistency existed.

Impact:
=======
The mpt driver would not be installed correctly and cannot be used.


Attachments:  

76676 - InconsistencyFixTool.tar.gz 

Creation of metadevice

Here are the steps i used for creation of Metadevices for n-way mirror.

Assuming c0t0d0 is already set up, and c0t1d0 is not. You might want to save a copy of the vtoc if you want to.

prtvtoc /dev/rdsk/c0t0d0s2 | fmthard –s - /dev/rdsk/c0t1d0s2

we need (0.5n+1) replicas for the recovery to work so we need minimum 3 state database replicas per disk. creating in the last slice.

metadb -f -a -c 3 c0t0d0s7
metadb -a -c 3 c0t1d0s7

Creating the mirror for the root disk

metainit -f d10 1 1 c0t0d0s0
metainit d20 1 1 c0t1d0s0
metainit d0 -m d10

Configure the system to boot the root filesystem from the metadevice, using the "metaroot" command. This will make the necessary changes to /etc/vfstab and /etc/system:

metaroot d0

You may want to get the name of what is now the raw root disk, in case we need it later:

ls -l /dev/rdsk/c0t1d0s0

creating a 1-way mirror for the other slices. use the "-f" option to force the creation for the first mirror. You can create subsequent mirror slices using this method..

metainit -f d13 1 1 c0t0d0s3
metainit d23 1 1 c0t1d0s3
metainit d3 -m d13

Please modify /etc/vfstab so that the system knows how to load the disks..

#device  device  mount  FS fsck mount  mount
#to mount to fsck  point  type pass at boot  options
#
/dev/md/dsk/d3 /dev/md/rdsk/d3 /opt/ ufs 2 yes logging
/dev/md/dsk/d4 /dev/md/rdsk/d4 /var ufs 2 yes logging
/dev/md/dsk/d5 /dev/md/rdsk/d5 /usr/ ufs 2 yes logging

To be on the safe side, get the system to unlock file system and flush any UFS logging of the master file system before you reboot.

lockfs -fa
reboot

Now attach the secondary mirror to create 2-way mirror for s3, s4, s5

metattach d3 d23
metattach d4 d24
metattach d5 d25

After completion of the syncing process (you can monitor using metastat), you may want to reboot the server to check that all is well.

More reference:
http://bradthemad.org/tech/notes/disksuite_mirroring.php
http://www.sun.com/bigadmin/content/submitted/metadevices.html


---------------------------------------
Addition.

From Solaris 10 6/06 (Update 2) onwards, if you have a production server that root drive has not been mirrored and the root and swap take up the entire disk...

You may try to steal a few cylinders from swap to create the metadb but be in for a big surprise.. :) Solaris 10 does not allow you to do that... see below..

# swap -d /dev/dsk/c0t0d0s1
# format
Searching for disks…done
 
AVAILABLE DISK SELECTIONS:
0. c0t0d0 
1. c0t1d0 
Specify disk (enter its number): 0
selecting c0t0d0
[disk formatted]
Warning: Current Disk has mounted partitions.
/dev/dsk/c0t0d0s0 is currently mounted on /. Please see umount(1M).
/dev/dsk/c0t0d0s1 is currently used by swap. Please see swap(1M).
 
format> par
partition> 1
Part Tag Flag Cylinders Size Blocks
1 swap wu 0 - 429 989.34MB (430/0/0) 2026160
 
Enter partition id tag[swap]:
Enter partition permission flags[wu]:
Enter new starting cyl[0]:
Enter partition size[2026160b, 430c, 429e, 989.34mb, 0.97gb]: 414c
partition> 7
Part Tag Flag Cylinders Size Blocks
7 unassigned wm 0 0 (0/0/0) 0
 
Enter partition id tag[unassigned]:
Enter partition permission flags[wm]:
Enter new starting cyl[0]: 414
Enter partition size[141360b, 30c, 443e, 69.02mb, 0.07gb]: 16c
 
partition> l
Cannot label disk when partitions are in use as described.

haha.. here's the workaround i found on the web...

1) Start format
2) select your new mirror disk, in this case c0t1d0
3) Partition it like your current root disk, except you really want to set it up with a smaller swap and a new slice 7 for metadb. Leave all other slices alone.

Here’s the interesting bit…. you write the VTOC from our mirror disk to the root disk, usually you’d do it the other way….

# prtvtoc -h /dev/rdsk/c0t1d0s2 | fmthard -s - /dev/rdsk/c0t0d0s2

That’s it, you now have two identical VTOCs and a slice 7 for metadb’s... I think SUN is trying to reduce the number of incident where we mess up our systems??

Taking IPMP offline for maintenance

Found this gem for use in Solaris 10 from some sun blog site..

Solaris IPMP (IP multipathing) allows the servers keep operating in the event that a network interface or switch were to fail. Periodically you may need to take IPMP managed interfaces offline but still need to keep the IP addresses attached to those interface up and operational.

Solaris now come with the if_mpadm utility, which provides a simple and straight forward way to take IPMP managed interfaces online and offline.

Prior to using the if_mpadm utility, it is useful to check the status of the interface you want to take online or offline. This can be done by running the ifconfig utility, and checking the status of the interface you are interested in taking online or offline (in this case ni0):

$ ifconfig -a

lo0: flags=2001000849 mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
ni0: flags=201000843 mtu 1500 index 2
        inet 192.168.1.5 netmask ffffff00 broadcast 192.168.1.255
        groupname ipmp0
        ether 0:45:e8:33:3c:97
ni1: flags=201000843 mtu 1500 index 3
        inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255
        groupname ipmp0
        ether 0:72:b6:d3:ee:35
lo0: flags=2002000849 mtu 8252 index 1
        inet6 ::1/128

To take the interface ni0 offline for maintenance, the if_mpadm utility can be run with the “-d” option (take interface offline), and the name of the interface to take offline:

$ if_mpadm -d ni0

Once if_mpadm does it’s job, the interface will be in the OFFLINE state, and the IP addresses attached to that interface will have migrated to another device in the IPMP group:

$ ifconfig -a

lo0: flags=2001000849 mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
ni0: flags=289000842 mtu 0 index 2
        inet 0.0.0.0 netmask 0
        groupname ipmp0
        ether 0:45:e8:33:3c:97
ni1: flags=201000843 mtu 1500 index 3
        inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255
        groupname ipmp0
        ether 0:72:b6:d3:ee:35
ni1:1: flags=201000843 mtu 1500 index 3
        inet 192.168.1.5 netmask ffffff00 broadcast 192.168.1.255
lo0: flags=2002000849 mtu 8252 index 1
        inet6 ::1/128

After you finish your maintenance, you can use the if_mpadm “-r” option (bring interface online) to bring the interface online:

$ if_mpadm -r ni0

Once if_mpadm completes, you can use the ifconfig utility to verify the interface is back up, and the IP addresses have migrated back to the original adaptor (you can disable automatic failback by setting FAILBACK to no in /etc/default/mpathd):

$ ifconfig -a

lo0: flags=2001000849 mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
ni0: flags=201000843 mtu 1500 index 2
        inet 192.168.1.5 netmask ffffff00 broadcast 192.168.1.255
        groupname ipmp0
        ether 0:45:e8:33:3c:97
ni1: flags=201000843 mtu 1500 index 3
        inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255
        groupname ipmp0
        ether 0:72:b6:d3:ee:35
lo0: flags=2002000849 mtu 8252 index 1
        inet6 ::1/128

Check out http://blogs.sun.com/meem/date/20070425

Forward/Reverse Proxying

This post is actually a brain dump of what i know so far from reading the numerous 'guides' out there..

Forward Proxy (e.g. Webmail)

The term Forward meant in the direction from "outside world" to "inside world".
Term to use in /usr/local/apache2/conf/httpd.conf : --> ProxyPass

Scenario

1) User --> webmail.com

2) therefore --> actually the framework is as follows

Webmail.com/

3) This is used to control incoming access (with respect to the web server)


Reverse Proxy (e.g. Internal programs accessing resources outside the network)

The term used is --> ProxyPassReverse

1) This is to control internal Application to access Internet / Extranet.

2) Can use this to control the type of access and the type of traffic allowed.

Moving Large directories in Solaris

When moving or copying really really large directories on Solaris, you can sometimes run into trouble, especially when some of the files in those directories are larger than 8 Gigabytes.

One solution is to use a “ufsdump pipe to ufsrestore” command, but this might even had problems from time to time.

experimented and checked out from some forums, think the safer method is to use a “tar pipe to tar” command. Be aware, however, that on Solaris you will need to include the “E” flag if the directory you are copying contains files larger than 8 Gigabytes.

Here is an example. Simply “CD” into the directory you want to copy and execute the following command, replacing “/new/directory” with path to the new destination directory.

tar cpBEf - * | (cd /new/directory; tar xBEf -)

Solaris 10 Fibre Channel Management

On Solaris 10 Fibre Channel Management is easy, cos the storage foundation kit is now integrated into the base OS, name is ”fcinfo“.

”fcinfo” utility is available to view fibre channel connectivity information.
”fcinfo” is especially useful, since it provides a tool with the base Operating System to view HBA and connectivity information, include HBAs from Emulex, JNI and Qlogic.

Warning!! You should do so only when the server fibre links are online or offline, and never do it continually while you are disrupting the fibre link. else you may never be able to bring up the fibre link till you reboot the server..

# uname -a
SunOS 5.10 

# fcinfo -V
fcinfo: Version 1.0
For more information, please see fcinfo(1M) 

# fcinfo hba-port 

HBA Port WWN: 210000e08b8f29bf
OS Device Name: /devices/pci@84,4000/fibre-channel@3:devctl
Manufacturer: QLogic Corporation
Model: QLA2340
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb
Current Speed: 2Gb
Node WWN: 200000e08b8f29bf 

HBA Port WWN: 10000000c9581765
OS Device Name: /dev/cfg/c3
Manufacturer: Emulex
Model: LP9802
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb
Current Speed: 2Gb
Node WWN: 20000000c9581765 

...
...

Option “-l” is Lists the link error statistics information for the port

# fcinfo hba-port -l 

HBA Port WWN: 210000e08b8f29bf
OS Device Name: /devices/pci@84,4000/fibre-channel@3:devctl
Manufacturer: QLogic Corporation
Model: QLA2340
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb
Current Speed: 2Gb
Node WWN: 200000e08b8f29bf
Error: SendRLS failed for 210000e08b8f29bf 

HBA Port WWN: 10000000c9581765
OS Device Name: /dev/cfg/c3
Manufacturer: Emulex
Model: LP9802
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb
Current Speed: 2Gb
Node WWN: 20000000c9581765
Link Error Statistics:
Link Failure Count: 1
Loss of Sync Count: 6
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 120
Invalid CRC Count: 0 

HBA Port WWN: 10000000c9582596
OS Device Name: /dev/cfg/c4
Manufacturer: Emulex
Model: LP9802
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb
Current Speed: 2Gb
Node WWN: 20000000c9582596
Link Error Statistics:
Link Failure Count: 1
Loss of Sync Count: 6
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 8
Invalid CRC Count: 0 

To check the remote port

# fcinfo remote-port -slp 2100001b320616d2 
Remote Port WWN: 50060e8005626100 
        Active FC4 Types: SCSI 
        SCSI Target: yes 
        Node WWN: 50060e8005626100 
        Link Error Statistics: 
                Link Failure Count: 0 
                Loss of Sync Count: 0 
                Loss of Signal Count: 0 
                Primitive Seq Protocol Error Count: 0 
                Invalid Tx Word Count: 0 
                Invalid CRC Count: 0 
        LUN: 0 
          Vendor: HITACHI 
          Product: OPEN-V      -SUN 
          OS Device Name: /dev/rdsk/c2t50060E8005626100d0s2 
        LUN: 1 
          Vendor: HITACHI 
          Product: OPEN-V      -SUN 
          OS Device Name: /dev/rdsk/c2t50060E8005626100d1s2 
        LUN: 2 
          Vendor: HITACHI 
          Product: OPEN-V      -SUN 
          OS Device Name: /dev/rdsk/c2t50060E8005626100d2s2 
        LUN: 3 
          Vendor: HITACHI 
          Product: OPEN-V      -SUN 
          OS Device Name: /dev/rdsk/c2t50060E8005626100d3s2 
        LUN: 4 
          Vendor: HITACHI 
          Product: OPEN-V      -SUN 
          OS Device Name: /dev/rdsk/c2t50060E8005626100d4s2 
        LUN: 5 
          Vendor: HITACHI 
          Product: OPEN-V      -SUN 
          OS Device Name: /dev/rdsk/c2t50060E8005626100d5s2 
        LUN: 6 
          Vendor: HITACHI 
          Product: OPEN-V      -SUN 
          OS Device Name: /dev/rdsk/c2t50060E8005626100d6s2 
        LUN: 7 
          Vendor: HITACHI 
          Product: OPEN-V      -SUN 
          OS Device Name: /dev/rdsk/c2t50060E8005626100d7s2 

To check the details of a particular fibre especially during setup, use luxadm to match the fibre and its WWN.

root # luxadm -e dump_map /devices/pci@7,700000/SUNW,qlc@0/fp@0,0:devctl 
Pos  Port_ID Hard_Addr Port WWN         Node WWN         Type 
0    80f00   0         50060e8005626110 50060e8005626110 0x0  (Disk device) 
1    82400   0         2100001b320610cd 2000001b320610cd 0x1f (Unknown Type,Host Bus Adapter) 

root # fcinfo remote-port -slp 2100001b320610cd 
Remote Port WWN: 50060e8005626110 
        Active FC4 Types: SCSI 
        SCSI Target: yes 
        Node WWN: 50060e8005626110 
        Link Error Statistics: 
                Link Failure Count: 0 
                Loss of Sync Count: 0 
                Loss of Signal Count: 0 
                Primitive Seq Protocol Error Count: 0 
                Invalid Tx Word Count: 0 
                Invalid CRC Count: 0 
        LUN: 0 
          Vendor: HITACHI 
          Product: OPEN-V      -SUN 
          OS Device Name: /dev/rdsk/c3t50060E8005626110d0s2 
        LUN: 1 
          Vendor: HITACHI 
          Product: OPEN-V      -SUN 
          OS Device Name: /dev/rdsk/c3t50060E8005626110d1s2 
        LUN: 2 
          Vendor: HITACHI 
          Product: OPEN-V      -SUN 
          OS Device Name: /dev/rdsk/c3t50060E8005626110d2s2 
        LUN: 3 
          Vendor: HITACHI 
          Product: OPEN-V      -SUN 
          OS Device Name: /dev/rdsk/c3t50060E8005626110d3s2 
        LUN: 4 
          Vendor: HITACHI 
          Product: OPEN-V      -SUN 
          OS Device Name: /dev/rdsk/c3t50060E8005626110d4s2 
        LUN: 5 
          Vendor: HITACHI 
          Product: OPEN-V      -SUN 
          OS Device Name: /dev/rdsk/c3t50060E8005626110d5s2 
        LUN: 6 
          Vendor: HITACHI 
          Product: OPEN-V      -SUN 
          OS Device Name: /dev/rdsk/c3t50060E8005626110d6s2 
        LUN: 7 
          Vendor: HITACHI 
          Product: OPEN-V      -SUN 
          OS Device Name: /dev/rdsk/c3t50060E8005626110d7s2 

Check where a user is logged in and its MAC address

This windows command can find out who logs into a particular machine and it's MAC address.

nbstat.exe -a username

Using Telnet to test SMTP

We normally use telnet to open up a connection to the smtp server to send/retrieve the mails. Below is an example of the commands used..

telnet mail.domain 25
Trying ???.???.???.???...
Connected to mail.domain.
Escape character is '^]'.
220 mail.domain ESMTP Sendmail ?version-number?; ?date+time+gmtoffset?

be polite and give the smtp server a 'hello' although the mail server would take your word for it as of RFC822-RFC1123

HELO local.domain.name
250 mail.domain Hello local.domain.name [loc.al.i.p], pleased to meet you

lets send an email..

MAIL FROM: mail@domain.com
250 2.1.0 mail@domain.com... Sender ok
RCPT TO: mail@otherdomain.com
250 2.1.0 mail@otherdomain.com... Recipient ok

If it doesn't please see possible problems.

To start composing the message issue the command DATA

If you want a subject for your email type Subject:-type subject here- then press enter twice (these are needed to conform to RFC 882)

You may now proceed to type the body of your message

To tell the mail server that you have completed the message enter a single "." on a line on it's own.

.
250 2.0.0 ???????? Message accepted for delivery

You can close the connection by issuing the QUIT command.

quit
221 2.0.0 mail.domain.ext closing connection
Connection closed by foreign host.

Here are a list of problems that i have encountered before....

The domain that you are sending from must exist

501 nouser@nosuchplace.here... Sender domain must exist

A recipient has been specified before a sender.

503 Need MAIL before RCPT

The mail server has refused to relay mail for you, this may be for any number of reasons but typical resons include:
Not using this provider for an internet connection and/or
Not using an email address provided by the owner of the server.
ACL in the mail configuration.

550 mail@domain.ext... Relaying Denied

Solaris NIC speed and duplex settings

In solaris 10, we have this very useful command to check the state of the NIC..

# dladm   show-dev 
bge0            link: up        speed: 1000  Mbps       duplex: full 
bge1            link: unknown   speed: 0     Mbps       duplex: unknown 
e1000g0         link: up        speed: 1000  Mbps       duplex: full 
e1000g1         link: unknown   speed: 0     Mbps       duplex: half 
e1000g2         link: up        speed: 1000  Mbps       duplex: full 
e1000g3         link: up        speed: 100   Mbps       duplex: full 
bge2            link: up        speed: 1000  Mbps       duplex: full 
bge3            link: up        speed: 1000  Mbps       duplex: full 
e1000g4         link: up        speed: 1000  Mbps       duplex: full 
e1000g5         link: unknown   speed: 0     Mbps       duplex: half 

# dladm show-link 
bge0            type: non-vlan  mtu: 1500       device: bge0 
bge1            type: non-vlan  mtu: 1500       device: bge1 
e1000g0         type: non-vlan  mtu: 1500       device: e1000g0 
e1000g1         type: non-vlan  mtu: 1500       device: e1000g1 
e1000g2         type: non-vlan  mtu: 1500       device: e1000g2 
e1000g3         type: non-vlan  mtu: 1500       device: e1000g3 
bge2            type: non-vlan  mtu: 1500       device: bge2 
bge3            type: non-vlan  mtu: 1500       device: bge3 
e1000g4         type: non-vlan  mtu: 1500       device: e1000g4 
e1000g5         type: non-vlan  mtu: 1500       device: e1000g5 

Here's a few good places to check out!!

http://www.brandonhutchinson.com/Solaris_NIC_speed_and_duplex_settings.html
http://forum.java.sun.com/thread.jspa?threadID=5084843

Match User Identity

Here's a little snippet for check for the correct user id before your script goes on with its task.. I have used this so far in ksh scripts..

# Only the root user can run the ndd commands
if [ "`/usr/bin/id | /usr/bin/cut -c1-5`" != "uid=0" ] ; then
   echo "You must be the root user to run `basename $0`."
   exit 1
fi

Sometime, i would use this also..

# Only the root user can run the ndd commands
if [ "`/usr/ucb/whoami != "root" ] ; then
   echo "You must be the root user to run `basename $0`."
   exit 1
fi