CmdLog write failed in VxVM

Ever encountered CmdLog write failure warning from VRTSvcs or VRTSvxvm?

For example,

root@nodeA # grep CmdLog /var/VRTSvcs/log/engine_A.log  | tail -5
VxVM vxdisk WARNING V-5-1-9668 CmdLog: write failed - No space left on device
VxVM vxdctl WARNING V-5-1-9668 CmdLog: write failed - No space left on device
VxVM vxdg WARNING V-5-1-9668 CmdLog: write failed - No space left on device
VxVM vxdisk WARNING V-5-1-9668 CmdLog: write failed - No space left on device
VxVM vxdisk WARNING V-5-1-9668 CmdLog: write failed - No space left on device

The cause of this is due to disk space in the partition where cmdlog resides is used up. In my case, the warning message came from node A even though /var partition on node B was full. Since logs for VCS should be identical for all nodes in a cluster, log could not be written to /var in node B but was able to in node A, hence the notification from node A.

The resolution is to clear up some space in that partition and you are good to go. Do remember to find out why this partition was filled up.

A little more about CmdLog.

 

Typically, cmdlog is located at /var/adm/vx

root@nodeA # ls -l /var/adm/vx/*log
-rw-------   1 root     root      336683 Nov 18 10:19 /var/adm/vx/cmdlog
-rw-r--r--   1 root     root       62423 Nov 17 17:25 /var/adm/vx/dmpevents.log
-rw-------   1 root     root      428837 Nov 18 10:19 /var/adm/vx/translog
-rw-r--r--   1 root     other          0 Aug  5  2009 /var/adm/vx/veacmdlog

And it is ASCII that records the commands that you fire to VCS / VxVM.

root@nodeA # file /var/adm/vx/cmdlog
/var/adm/vx/cmdlog:     ascii text

root@nodeA # tail -3 /var/adm/vx/cmdlog
 /usr/sbin/vxdisk -qag dg_myapp list
# 21304, 8603, Fri Nov 18 10:27:19 2011
 /usr/sbin/vxdisk list c4t50060E1006A11B41d0s2

Hope this helps.