CmdLog write failed in VxVM

Ever encountered CmdLog write failure warning from VRTSvcs or VRTSvxvm?

For example,

root@nodeA # grep CmdLog /var/VRTSvcs/log/engine_A.log  | tail -5
VxVM vxdisk WARNING V-5-1-9668 CmdLog: write failed - No space left on device
VxVM vxdctl WARNING V-5-1-9668 CmdLog: write failed - No space left on device
VxVM vxdg WARNING V-5-1-9668 CmdLog: write failed - No space left on device
VxVM vxdisk WARNING V-5-1-9668 CmdLog: write failed - No space left on device
VxVM vxdisk WARNING V-5-1-9668 CmdLog: write failed - No space left on device

The cause of this is due to disk space in the partition where cmdlog resides is used up. In my case, the warning message came from node A even though /var partition on node B was full. Since logs for VCS should be identical for all nodes in a cluster, log could not be written to /var in node B but was able to in node A, hence the notification from node A.

The resolution is to clear up some space in that partition and you are good to go. Do remember to find out why this partition was filled up.

A little more about CmdLog.

 

Typically, cmdlog is located at /var/adm/vx

root@nodeA # ls -l /var/adm/vx/*log
-rw-------   1 root     root      336683 Nov 18 10:19 /var/adm/vx/cmdlog
-rw-r--r--   1 root     root       62423 Nov 17 17:25 /var/adm/vx/dmpevents.log
-rw-------   1 root     root      428837 Nov 18 10:19 /var/adm/vx/translog
-rw-r--r--   1 root     other          0 Aug  5  2009 /var/adm/vx/veacmdlog

And it is ASCII that records the commands that you fire to VCS / VxVM.

root@nodeA # file /var/adm/vx/cmdlog
/var/adm/vx/cmdlog:     ascii text

root@nodeA # tail -3 /var/adm/vx/cmdlog
 /usr/sbin/vxdisk -qag dg_myapp list
# 21304, 8603, Fri Nov 18 10:27:19 2011
 /usr/sbin/vxdisk list c4t50060E1006A11B41d0s2

Hope this helps.

How to add a mirror disk back to VxVM

I have 3 disks which are part of the mirror group.

root@myserver # vxdisk list | grep ^c
c1t0d0s2 sliced rootdisk rootdg online
c1t1d0s2 sliced rootmirror1 rootdg online
c2t0d0s2 sliced - - online
c2t1d0s2 sliced - - online

Naming conventions:
- rootdisk
- rootmirror1
- rootmirror2
- rootmirror3

Now i want to add the mirror back. We are going to attach back device c2t0d0 into the rootdg disk group.

Always run the command vxdisk -ht to check what is the actual volume name to attach the disk to.

Setup the disk
root@myserver # vxdisksetup -if c2t0d0 (for VxVM 3.5)
root@myserver # vxdisksetup -if c2t0d0 format=sliced (for VxVM 4.0)
root@myserver # /usr/lib/vxvm/bin/vxdisksetup -if c2t0d0 format=sliced (for VxVM 4.1)

Add the disk into rootdg.
root@myserver # vxdg -g rootdg adddisk rootmirror2=c2t0d0

Mirror each of the partitions. Here i throttle the synchronisation speed to have less performance impact on the server.
root@myserver # vxassist -o slow=1500 -g rootdg mirror rootvol rootmirror2 &
root@myserver # vxassist -o slow=1500 -g rootdg mirror opt rootmirror2 &
root@myserver # vxassist -o slow=1500 -g rootdg mirror swapvol rootmirror2 &
root@myserver # vxassist -o slow=1500 -g rootdg mirror usr rootmirror2 &
root@myserver # vxassist -o slow=1500 -g rootdg mirror var rootmirror2 &

To monitor the progress of synchronisation.
root@myserver # vxtask list
TASKID PTID TYPE/STATE PCT PROGRESS
166 ATCOPY/R 07.88% 0/25166079/1982464 PLXATT opt opt-01
167 ATCOPY/R 25.29% 0/4194828/1060864 PLXATT swapvol swapvol-01
168 ATCOPY/R 15.13% 0/4194828/634880 PLXATT usr usr-01
169 ATCOPY/R 02.82% 0/16779312/473088 PLXATT var var-01

How to ensure VTOC is updated from VxVM in SunOS?

It is important to keep the VTOC in sync with the setup in VxVM else you risk not being able to boot up the disk after un-encapsulating.

SUN also advised that if the VxVM disk mirrors are not identical to the first disk, this would result in unusable mirrors.

Also after creation of VxVM rootdisk and rootmirrors, you may want to create the underlying partition on the rootdisk and rootmirror.

Here, i will show the steps for setting up new disk as RAID1 mirror

root@myserver # vxdisksetup -if c2t0d0
root@myserver # vxdg -g rootdg adddisk rootmirror2=c2t0d0
root@myserver # vxassist -g rootdg mirror rootvol rootmirror2 &
root@myserver # vxassist -g rootdg mirror usr rootmirror2 &
root@myserver # vxassist -g rootdg mirror var rootmirror2 &
root@myserver # vxassist -g rootdg mirror opt rootmirror2 &

Take a look at the VTOC

root@myserver # prtvtoc /dev/dsk/c1t0d0s2
* /dev/dsk/c1t0d0s2 partition map
*
* Dimensions:
* 512 bytes/sector
* 107 sectors/track
* 27 tracks/cylinder
* 2889 sectors/cylinder
* 24622 cylinders
* 24620 accessible cylinders
*
* Flags:
* 1: unmountable
* 10: read-only
*
* Unallocated space:
* First Sector Last
* Sector Count Sector
* 0 2889 2888
* 8395434 4286577640 5777
* 71127180 4232235550 8395433
* 58730481 12396699 71127179
*
* First Sector Last
* Partition Tag Flags Sector Count Sector Mount Directory
0 2 00 5778 8389656 8395433
1 3 01 8395434 4194828 12590261
2 5 00 0 71127180 71127179
3 15 01 2889 2889 5777
4 14 01 5778 71121402 71127179
5 0 00 33564402 25166079 58730480
6 4 00 12590262 4194828 16785089
7 7 00 16785090 16779312 33564401

Now, the VTOC is not updated. we need to use VxVM command to update VTOC. not really advisable to update using "format" command manually since its more error prone.

Tag Description (2nd column from right)
0x00 UNASSIGNED
0x01 BOOT
0x02 ROOT
0x03 SWAP
0x04 USR
0x05 BACKUP
0x06 STAND
0x07 VAR
0x08 HOME

Flag Description (right-most column)
0x00 MOUNTABLE
0x01 UNMOUNTABLE
0x10 READ-ONLY

Note that Swap is NOT mountable.


root@myserver # vxprint -htg rootdg
DG NAME NCONFIG NLOG MINORS GROUP-ID
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
V NAME RVG KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO

dg rootdg all all 0 1076053596.1025.myserver

dm rootdisk c1t0d0s2 sliced 2632 71121402 -
dm rootmirror1 c1t1d0s2 sliced 2632 71121402 -
dm rootmirror2 c2t0d0s2 sliced 2632 71121402 -

v opt - ENABLED ACTIVE 25166079 ROUND - fsgen
pl opt-01 opt ENABLED ACTIVE 25166079 CONCAT - RW
sd rootdisk-05 opt-01 rootdisk 33558624 25166079 0 c1t0d0 ENA
pl opt-02 opt ENABLED ACTIVE 25166079 CONCAT - RW
sd rootmirror1-05 opt-02 rootmirror1 12584484 25166079 0 c1t1d0 ENA
pl opt-03 opt ENABLED ACTIVE 25166079 CONCAT - RW
sd rootmirror2-05 opt-03 rootmirror2 33558624 25166079 0 c2t0d0 ENA

v rootvol - ENABLED ACTIVE 8389656 ROUND - root
pl rootvol-01 rootvol ENABLED ACTIVE 8389656 CONCAT - RW
sd rootdisk-01 rootvol-01 rootdisk 0 8389656 0 c1t0d0 ENA
pl rootvol-02 rootvol ENABLED ACTIVE 8389656 CONCAT - RW
sd rootmirror1-01 rootvol-02 rootmirror1 0 8389656 0 c1t1d0 ENA
pl rootvol-03 rootvol ENABLED ACTIVE 8389656 CONCAT - RW
sd rootmirror2-01 rootvol-03 rootmirror2 0 8389656 0 c2t0d0 ENA

v swapvol - ENABLED ACTIVE 4194828 ROUND - swap
pl swapvol-01 swapvol ENABLED ACTIVE 4194828 CONCAT - RW
sd rootdisk-02 swapvol-01 rootdisk 8389656 4194828 0 c1t0d0 ENA
pl swapvol-02 swapvol ENABLED ACTIVE 4194828 CONCAT - RW
sd rootmirror1-02 swapvol-02 rootmirror1 8389656 4194828 0 c1t1d0 ENA
pl swapvol-03 swapvol ENABLED ACTIVE 4194828 CONCAT - RW
sd rootmirror2-02 swapvol-03 rootmirror2 8389656 4194828 0 c2t0d0 ENA

v usr - ENABLED ACTIVE 4194828 ROUND - fsgen
pl usr-01 usr ENABLED ACTIVE 4194828 CONCAT - RW
sd rootdisk-03 usr-01 rootdisk 12584484 4194828 0 c1t0d0 ENA
pl usr-02 usr ENABLED ACTIVE 4194828 CONCAT - RW
sd rootmirror1-03 usr-02 rootmirror1 37750563 4194828 0 c1t1d0 ENA
pl usr-03 usr ENABLED ACTIVE 4194828 CONCAT - RW
sd rootmirror2-03 usr-03 rootmirror2 12584484 4194828 0 c2t0d0 ENA

v var - ENABLED ACTIVE 16779312 ROUND - fsgen
pl var-01 var ENABLED ACTIVE 16779312 CONCAT - RW
sd rootdisk-04 var-01 rootdisk 16779312 16779312 0 c1t0d0 ENA
pl var-02 var ENABLED ACTIVE 16779312 CONCAT - RW
sd rootmirror1-04 var-02 rootmirror1 41945391 16779312 0 c1t1d0 ENA
pl var-03 var ENABLED ACTIVE 16779312 CONCAT - RW
sd rootmirror2-04 var-03 rootmirror2 16779312 16779312 0 c2t0d0 ENA

For automated step, try this
# vxbootsetup

For manual step, try these
Notice that you did NOT create the disk partitions manually when you initialized the mirror disk. This is because VxVM automatically does its partitioning for you. This is also true for the root disk after it has been re-initialized and brought into rootdg. That is, the original rootdisk partition scheme has been replaced by Veritas's partition scheme. This means that the underlying partitions for each volume(swap, /opt, /var, /home) are not physically laid out on the drives corresponding to the veritas volumes.

In some situations though, you might want to be able to access the partitions corresponding to each volume by manually mounting the slice. In a recovery situation for example, you may want to boot from cdrom and access the /opt partition by manually mounting the slice without the veritas drivers loaded. This is not possible unless you create the underlying partition mapping directly to the opt volume using the "vxmksdpart" command.

All underlying partitions except / should be created. The / partition is automatically created and is mapped correctly to the corresponding veritas root volume.

One Caveat, you cannot use partition 3 or 4 because this is typically where Veritas stores is private and public region. The following is snipped from the "format" utility listing the Veritas partitions. This shows that the root partition exists but the swap, /var, /opt, home partitions are not mapped on the drive(Veritas knows where they exist on the drive because of its private and public regions):

Part      Tag    Flag     Cylinders        Size            Blocks

  0       root    wm       1 - 2282        3.91GB    (2282/0/0)  8194662        <=== "/" exists
  1 unassigned    wm       0               0         (0/0/0)           0        <=== Unassigned
  2     backup    wm       0 - 4923        8.43GB    (4924/0/0) 17682084
  3          -    wu       0 -    0        1.75MB    (1/0/0)        3591        <=== Private Region
  4          -    wu       1 - 4923        8.43GB    (4923/0/0) 17678493        <=== Public Region
  5 unassigned    wm       0               0         (0/0/0)           0        <=== Unassigned
  6 unassigned    wm       0               0         (0/0/0)           0        <=== Unassigned
  7 unassigned    wm       0               0         (0/0/0)           0        <=== Unassigned

You may use the unassigned partitions which are 1, 5, 6, and 7 to map to the veritas volumes. Be sure to document which partitions map to the Veritas volumes for future reference. The "vxmksdpart" arguments are the subdisk (which maps directly to the volume), slice, tag and flag. Be sure to use the correct tag and flag for your particular Solaris version. Reference the man page for "fmthard" for correct options. Use the "vxprint" command to get the correct subdisk names: Generic command # /usr/lib/vxvm/bin/vxmksdpart

In my case
Rootdisk:
# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootdisk-02 1 0x03 0x01 <=== swap
# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootdisk-05 5 0x00 0x00 <=== /unassigned
# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootdisk-03 6 0x04 0x00 <=== /usr
# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootdisk-04 7 0x07 0x00 <=== /var

RootMirror:

# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootmirror1-02 1 0x03 0x01 <=== swap
# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootmirror1-05 5 0x00 0x00 <=== /unassigned
# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootmirror1-03 6 0x04 0x00 <=== /usr
# /usr/lib/vxvm/bin/vxmksdpart -g rootdg rootmirror1-04 7 0x07 0x00 <=== /var

References:
http://sysunconfig.net/unixtips/jmr_Mirroring_Boot_Disks_Veritas_Volume_Manager.htm
http://unixway.com/vm/veritasvm/rootdg.html

How to unmount a online / mounted VXFS on 5.0 MP3

You may have encountered this

# umount -f /opt/app/oracle
UX:vxfs umount: ERROR: V-3-26360: file system /opt/app/oracle has been mount locked

Taken from the symantec support site - http://seer.entsupport.symantec.com/docs/313851.htm

Details:
Storage Foundation 5.0MP3 introduced a new feature called VxFS filesystem lock which disallows accidental unmounts when the file system resource is online. New umount option mntunlock is used to clear the lock and then unmount the filesystem. The offline script for the Mount resource will use this new option.

How to check if the filesystem is locked by VCS:
# mount -v | grep mntlock

Sometimes, it may be necessary to unmount a mount locked filesystem. This is for cases where VCS service groups having DiskGroup resources configured with UnMountVolumes attribute set and the volumes are mounted outside of VCS control (this is not very common). Use the VxFS umount command for Solaris and vxumount command for Linux to manually unmount the file system

Solaris:
# /opt/VRTS/bin/umount -o mntunlock=VCS

Linux:
# /usr/lib/fs/vxfs/umount -o mntunlock=VCS

Example:
# mount -v | grep mntlock
/dev/vx/dsk/oracledg/oracle on /opt/app/oracle type vxfs read/write/setuid/devices/mincache=direct/convosync=direct/delaylog/largefiles/qio/ioerror=mwdisable/mntlock=VCS/dev=46c2ee0 on Mon Sep 15 18:28:18 2008
# /opt/VRTS/bin/umount -o mntunlock=VCS /opt/app/oracle

Check Disk Space in VxVM (Solaris)

Was trying to find out how much disk space i have left on a system as the partition was nearing max capacity..

root # df -h /opt/myapp
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/mydg/myapp
961M 760M 201M 79% /opt/myapp

Tried this to get the free space in VxVM.. it looks pretty intimidating right? not good for the eyes. Here the "LENGTH" attribute shows the balance free space in 512Kb blocks.

root # vxdg -g mydg free
DISK DEVICE TAG OFFSET LENGTH FLAGS
disk1_1 HDS_6 HDS_6 4195200 12574080 -
disk1_2 HDS1_4 HDS_4 2098560 14670720 -
disk2_1 HDS_6 HDS_6 4195200 12574080 -
disk2_2 HDS0_4 HDS_4 2098560 14670720 -

so i have roughly 12574080 * 512 / 1024 = 6Gb free and 2Gb used in disk1_1.

For each individual disk, the balance free disk space is

# vxassist maxsize disk1_1
Maximum volume size: 12572672 (6139Mb)
etc...

so this pretty match up with the one reported by vxdg.

Therefore i only have 13Gb left in total, taking into consideration that i have a RAID 1.

This one only show the total space for the whole disk group. Again, here the figure matches.

root # vxassist -g myapp maxsize
Maximum volume size: 54489088 (26606Mb)

Something interesting. found from http://www.hyborian.demon.co.uk/notes/vx_cli.html#disksp that vxprint can also display the disk space.

root # vxprint -g myapp -dF "%publen" awk 'BEGIN {s = 0} {s += $1} END {print s/2097152, "GB"}'
31.9849 GB

And the total amount of diskspace in this system, i have...

root # vxprint -AQqdF "%publen" awk 'BEGIN {s = 0} {s += $1} END {print s/2097152, "GB"}'
99.8143 GB


Wow! great. the output from vxprint and vxassist disagree with each other. i think i would prefer to trust vxdg or vxassist since this is really the true size of SAN disk allocated.

Now to see when i can 'grow' the volume... :)

Save an expired VxVM system

VxVM is normally installed with 30 days demo licence. After which if you reboot, only the partitions essential to booting up will be loaded. meaning the following partition will be disabled and not mounted. This also mean that without a reboot, your system will work as though VxVM has not expired.

/opt
/var
swapvol

My Experience with this was that i cant even use 'vi' as swapvol is not available. So Basically, there's nothing much you can do.

So, go to single user more. Check your Volume Manager License. Should see that your temp / demo license has expired.

# /usr/sbin/vxlicrep | more

So now we try to install the permanent license. Key in your license key when prompted.

# /usr/sbin/vxlicinst

VERITAS License Manager vxlicinst utility version 3.02.005
Copyright (C) 1996-2004 VERITAS Software Corp. All Rights reserved.
 
Enter your license key : XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XXXX-XX
 
License key successfully installed for VERITAS Volume Manager

Note: The license keys can coexist on the same machine. ie. if you had installed a previous license, you can continue adding in new ones.
Note: Licences are stored in /etc/vx/licenses/lic

Tip:
If your system do not have the vxlicinst command, you do not have the VRTSvlic package. Try mounting a remote /opt/VRTS/ from a server in the same network segment.


Now you can check the status, you will see (shown only portion of the listing)

# vxprint -ht
 
v  opt          -            DISABLED ACTIVE   94372224 ROUND     -        fsgen
pl opt-01       opt          DISABLED ACTIVE   94372224 CONCAT    -        RW
sd rootdisk-05  opt-01       rootdisk 146798976 94372224 0        c1t0d0   ENA
pl opt-02       opt          DISABLED ACTIVE   94372224 CONCAT    -        RW
sd rootmirror-05 opt-02      rootmirror 146798976 94372224 0      c1t1d0   ENA
 
v  swapvol      -            DISABLED ACTIVE   31423488 ROUND     -        swap
pl swapvol-01   swapvol      DISABLED ACTIVE   31423488 CONCAT    -        RW
sd rootdisk-02  swapvol-01   rootdisk 41945472 31423488 0         c1t0d0   ENA
pl swapvol-02   swapvol      DISABLED ACTIVE   31423488 CONCAT    -        RW
sd rootmirror-02 swapvol-02  rootmirror 41945472 31423488 0       c1t1d0   ENA
 
v  var          -            DISABLED  ACTIVE   52447104 ROUND     -        fsgen
pl var-01       var          DISABLED  ACTIVE   52447104 CONCAT    -        RW
sd rootdisk-04  var-01       rootdisk 94351872 52447104 0         c1t0d0   ENA
pl var-02       var          DISABLED  ACTIVE   52447104 CONCAT    -        RW
sd rootmirror-04 var-02      rootmirror 94351872 52447104 0       c1t1d0   ENA

To save yourself (haha...) and the server, try the following on all the partition that is disabled,

# vxvol -g  start 
i.e. vxvol -g rootdg start opt
i.e. vxvol -g rootdg start var
i.e. vxvol -g rootdg start swapvol

The volumes should now be enabled.

vxprint -ht

Thats it. Reboot and check your system...