Saturday, June 30, 2012

Installing IBM Websphere Applicationi Server (WAS) 6 Plugin on Apache 2.2.22

Learnt that plugin setup for WAS is different from the setting up in BEA weblogic environment.

WAS plugin require an installation while we just need to put in the plugin for BEA weblogic. Here's the steps to get WAS running.

Installing the Plugin in the web server.


Download the installation file from IBM. i.e. "Tiv_Middl_Inst_750_1of3_Linux_x86-64.tar" since i'm using linux for my web server.

Transfer the tar file into the web server and unpack.

Was only able to install using the supplied GUI so be prepared to export display.

# cd linux64/WS-WAS_ND_7.0_Supplemental
# gunzip C1G36ML.tar.gz
# tar xfp C1G36ML.tar
# cd plugin
# export BROWSER=/usr/bin/mozilla
# export DISPLAY=10.10.10.16:0.0
# ./launchpad.sh
need to have your xwin or xmanager ready.

My template cfg.xml file is now at "/opt/IBM/WebSphere/Plugins/config/myapp/plugin-cfg.xml"

Copy the configuring script to the WAS server.

There is a customised configuration script that you need to run at the WAS server to generate the real cfg.xml file. It is usually in the plugin bin, i.e. "/opt/IBM/WebSphere/Plugins/bin/configuremyappserver.sh"
copied this to the root directory of WAS, i.e. /opt/was/IBM/WebSphere/AppServer/bin and run it.

Next, generating the plugin xml file in WAS server.

root@myappserver:/opt/was/IBM/WebSphere/AppServer/bin> ./configuremyappserver.sh
Realm/Cell Name: <default>
Username: wasuser
Password:                                                                                                                                                    
WASX7209I: Connected to process "dmgr" on node myappserverCellManager01 using SOAP connector;  The type of process is: DeploymentManager
WASX7303I: The following options are passed to the scripting environment and are available as arguments that are stored in the argv variable: "[myapp, APACHE, /usr/local/apache2222, /usr/local/apache2222/conf/httpd.conf, 80, MAP_ALL, /opt/IBM/WebSphere/Plugins, unmanaged, mywebserver-node, mywebserver, linux]"

Input parameters:

   Web server name             - myappserver
   Web server type             - APACHE
   Web server install location - /usr/local/apache2222
   Web server config location  - /usr/local/apache2222/conf/httpd.conf
   Web server port             - 80
   Map Applications            - MAP_ALL
   Plugin install location     - /opt/IBM/WebSphere/Plugins
   Web server node type        - unmanaged
   Web server node name        - mywebserver-node
   Web server host name        - mywebserver
   Web server operating system - linux

Creating the unmanaged node mywebserver-node .
Unmanged node mywebserver-node is created.

Creating the web server definition for myapp.
Web server definition for myapp is created.

Start computing the plugin properties ID.
Plugin properties ID is computed.

Start updating the plugin install location.
Plugin install location is updated.

Start updating the plugin log file location.
Plugin log file location is updated.

Start updating the RemoteConfigFilename location.
Plugin remote config file location is updated.

Start updating the RemoteKeyRingFileName location.
Plugin remote keyring file location is updated.

Start saving the configuration.

Configuration save is complete.

Computed the list of installed applications.

Processing the application myapp.
Get the current target mapping for the application myapp.
Computed the current target mapping for the application myapp.
Start updating the target mappings for the application myapp.
Target mapping is updated for the application myapp.

Start saving the configuration.

Configuration save is complete.

Transfer the plugin-cfg.xml file to the web server.

 scp /opt/was/IBM/WebSphere/AppServer/profiles/Dmgr01/config/cells/myappserverCell01/nodes/mywebserver-node/servers/myapp/plugin-cfg.xml user@mywebserver:/tmp/
The generated file is generally in "profiles_install_root/config/cells/cell_name/nodes/node_name/servers/web_server_name" directory
The place to put the plugin-cfg.xml file is generally in "plugins_install_root/config/web_server_name" directory

Start up Apache and test.

you should be good to go.

Friday, June 29, 2012

AIX powerHA auto-verification

powerHA 7.1 will automatically runs a verification every night, usually around mid-night. With a very simple command you can check the status of this verification run:

# tail -10 /var/hacmp/log/clutils.log 2>/dev/null|grep detected|tail -1

If this shows a returncode of 0, the cluster verification ran without any errors. Anything else, you'll have to investigate. You can use this command on all your HACMP clusters, allowing you to verify your HACMP cluster status every day.

With the following smitty menu you can change the time when the auto-verification runs and if it should produce debug output or not:

    # smitty clautover.dialog
                                                        [Entry Fields]
* Automatic cluster configuration verification        Enabled                                                                                             +
* Node name                                           Default                                                                                             +
* HOUR (00 - 23)                                     [00]                                                                                                 +#
  Debug                                               yes        

You can check with:

    # odmget HACMPcluster
    # odmget HACMPtimersvc

Be aware that if you change the runtime of the auto-verification that you have to synchronize the cluster afterwards to update the other nodes in the cluster.

Source Reference (If the original author do not agree that i post on my blog, please let me know. :) )

http://www.aixhealthcheck.com/blog.php?id=116

Thursday, June 28, 2012

How to list Network statistics

This is actually useful across all platforms like Solaris, AIX, Linux and etc to list out the network statistics of the NICs. We can tell if there are any potential network issues and spawn off necessary actions.

 
root@myserver:/> netstat -i
Name  Mtu   Network     Address            Ipkts Ierrs    Opkts Oerrs  Coll
en2   1500  link#2      d2.48.a8.b8.c9.2    13862     0    14038     0     0
en2   1500  10.10.10   myserver          13862     0    14038     0     0
lo0   16896 link#1                          10644     0    10644     0     0
lo0   16896 127         loopback            10644     0    10644     0     0
lo0   16896 loopback                        10644     0    10644     0     0

How to check powerHA settings and events from ODM

In ODM, the object class definition that will be displayed when you query a particular object class. For example, here i queried the HACMPevent object.

root@myserver:/> odmshow HACMPevent
class HACMPevent {
        char name[256];                              /* offset: 0xc ( 12) */
        char desc[256];                              /* offset: 0x10c ( 268) */
        short setno;                                 /* offset: 0x20c ( 524) */
        short msgno;                                 /* offset: 0x20e ( 526) */
        char catalog[256];                           /* offset: 0x210 ( 528) */
        char cmd[1024];                              /* offset: 0x310 ( 784) */
        char notify[1024];                           /* offset: 0x710 ( 1808) */
        char pre[1024];                              /* offset: 0xb10 ( 2832) */
        char post[1024];                             /* offset: 0xf10 ( 3856) */
        char recv[1024];                             /* offset: 0x1310 ( 4880) */
        short count;                                 /* offset: 0x1710 ( 5904) */
        long event_duration;                         /* offset: 0x1714 ( 5908) */
        };
/*
        descriptors:    12
        structure size: 0x1718 (5912) bytes
        data offset:    0x380
        population:     89 objects (89 active, 0 deleted)
*/

i.e. what script to run when a node is attempting to join a cluster.

root@myserver:/> odmget -q name=node_up HACMPevent

HACMPevent:
        name = "node_up"
        desc = "Script run when a node is attempting to join the cluster."
        setno = 101
        msgno = 7
        catalog = "events.cat"
        cmd = "/usr/es/sbin/cluster/events/node_up"
        notify = ""
        pre = ""
        post = ""
        recv = ""
        count = 0
        event_duration = 0
 
 
and the powerHA 7.1 events from ODM database,

root@myserver:/> odmget HACMPevent | awk '/name/ {print $3}' | sed 's/"//g'
swap_adapter
swap_adapter_complete
network_up
network_down
network_up_complete
network_down_complete
node_up
node_down
node_up_complete
node_down_complete
join_standby
fail_standby
acquire_service_addr
acquire_takeover_addr
get_disk_vg_fs
node_down_local
node_down_local_complete
node_down_remote
node_down_remote_complete
node_up_local
node_up_local_complete
node_up_remote
node_up_remote_complete
release_service_addr
release_takeover_addr
release_vg_fs
start_server
stop_server
config_too_long
event_error
reconfig_topology_start
reconfig_topology_complete
reconfig_resource_release
reconfig_resource_release_primary
reconfig_resource_release_secondary
reconfig_resource_acquire_secondary
reconfig_resource_complete_secondary
reconfig_resource_release_fence
reconfig_resource_acquire_fence
reconfig_resource_acquire
reconfig_resource_complete
migrate
migrate_complete
acquire_aconn_service
swap_aconn_protocols
get_aconn_rs
release_aconn_rs
server_restart
server_restart_complete
server_down
server_down_complete
rg_move
rg_move_release
rg_move_acquire
rg_move_fence
rg_move_complete
site_down
site_down_complete
site_down_local
site_down_local_complete
site_down_remote
site_down_remote_complete
site_up
site_up_complete
site_up_local
site_up_local_complete
site_up_remote
site_up_remote_complete
site_merge
site_merge_complete
site_isolation
site_isolation_complete
fail_interface
join_interface
cluster_notify
resource_add
resource_modify
resource_delete
resource_online
resource_offline
resource_state_change
resource_state_change_complete
external_resource_state_change
external_resource_state_change_complete
intersite_fallover_prevented
reconfig_configuration_complete
forced_down_too_long
start_udresource
stop_udresource
 

Wednesday, June 27, 2012

Access Controls and Virtual Hosts for WebSphere Application Server

WebSphere Virtual Host


WebSphere applications are assigned to virtual hosts during installation and configuration process.

The virtual hosts are bound to one or more aliases (host names and ports), allowing a single application server to respond to multiple inbound request formats. 

The virtual host aliases that are configured within WebSphere define the pattern match possibilities for which WebSphere application server will respond with data. 

For example, if we have 7 aliases defined for a given virtual host 'foo-app' on application server, ‘fooserver′.

Virtual Host:  'foo-app'
Hostname         Port
thisport         8080
thatport         8081
secureport       8443
secureport2      9443
otherport        10001
otherport2       10002
otherport3       10003


Hence, for any inbound request for 'foo-app', if the URL does not match against the list above, the request is denied, else the client gets the access. The Application Server is the gatekeeper here.


Usually all default application server ports are removed from the ‘default_host’ virtual host once you do some 'hardening', forcing all inbound requests to be channeled through a proxy mechanism.

Hence, you will not be able to access the application 'foo-app' directly, if the default listening port is not in the list above. e.g. 'foo-app' listens on 1234. To get to 'foo-app', you need to get through the aliases above to get to 'foo-app'.

I use Apache in the web layer above Websphere Application Server. Need to install the Websphere Proxy-Plugin and generate the xml file.

When client access the URL, the GET request reaches Apache, which then checks the request against the Websphere Proxy Plugin for the host or virtual host corresponding to the requested URL. 

The Websphere Proxy Plugin has a list of valid WebSphere virtual hosts and virtual-host associated resources. Here, if matched, those requests are forwarded to the appropriate WebSphere application server. If not matched, 404 is generated.

How to use iptrace

The iptrace command, like tcpdump or snoop can be very useful to find out what network traffic flows to and from an AIX system.

You can use any combination of these options, but you do not need to use them all:
  • -a Do NOT print out ARP packets.
  • -s source IP Limit trace to source/client IP address, if known.
  • -d destination IP Limit trace to destination IP, if known.
  • -b Capture bidirectional network traffic (send and receive packets).
  • -p port Specify the port to be traced.
  • -i interface Only trace for network traffic on a specific interface.
Example:
Run iptrace on AIX interface en1 to capture port 80 traffic to file trace.out from a single client IP to a server IP:

root@myserver:/> iptrace -a -i en0 -s 10.10.10.19 -b -d 10.10.10.11 -p 80 /tmp/trace.out
[17957068]

This trace will capture both directions of the port 80 traffic on interface en1 between the client IP and server IP and sends this to the raw file of trace.out.

To stop the trace:
root@myserver:/> ps -aef | grep iptra
    root 17957068        1   0 11:09:09      -  0:00 iptrace -a -i en0 -s 10.10.10.19 -b -d 10.10.10.11 -p 80 /tmp/trace.out 
 
root@myserver:/> kill -15 17957068
 
root@myserver:/> iptrace: unload success!
 
Leaving it running too long would require a large amount of disk space!

The ipreport command can be used to transform the trace file generated by iptrace to human readable format:
 
root@myserver:/> ipreport /tmp/trace.out /tmp/trace.report
IPTRACE version: 2.0


++++++ END OF REPORT ++++++

processed 0 packets

Tuesday, June 26, 2012

How to resolve gethostbyaddr IPv6 error

What to do when sendmail log "gethostbyaddr(IPv6:::1) failed: 1" warning messages to syslog?

In AIX 5.3 TL11 and AIX 6.1 TL4 and later, sendmail is IPv6 enabled. When sendmail attempts to resolve local interfaces, it will encounter the IPv6 loopback interface (::1) and perform an IPv6 lookup, which fails and thus the gethostbyaddr warning is logged to syslog.

To resolve this matter, add this entry into the /etc/hosts file
 
::1 loopback localhost

Future releases of AIX will automatically include this entry in the /etc/hosts file.

Also, add the following entry to /etc/netsvc.conf :
hosts=local

How to determine File system creation time

To determine the time and date a file system was created, try this.

Find the LV for that file system.

Lets try /opt.

root@myserver:/> lsfs /opt
Name            Nodename   Mount Pt               VFS   Size    Options    Auto Accounting
/dev/hd10opt    --         /opt                   jfs2  10485760 --         yes  no

Since /opt is located on LV hd10opt. we then try next
 
root@myserver:/> getlvcb -AT hd10opt
         AIX LVCB
         intrapolicy = c
         copies = 1
         interpolicy = m
         lvid = 00f603d800002c000000012f34187103.9
         lvname = hd10opt
         label = /opt
         machine id = 603C84A00
         number lps = 160
         relocatable = y
         strict = y
         stripe width = 0
         stripe size in exponent = 0
         type = jfs2
         upperbound = 32
         fs =
         time created  = Thu Aug 25 04:48:35 2011
         time modified = Fri Sep 23 10:16:13 2011

Now we can tell that creation time aka "time created" for /opt is in Aug 2011.

How to recreate BOOT LOGICAL VOLUME (BLV) in AIX

If a Boot Logical volume (BLV) is corrupted, the machine will not boot.(Eg:bad block in a disk might cause a corrupted BLV)
Therefore, to fix this situation, You must boot your machine in maintenance mode, from a CD or Tape. If a NIM has been setup for a machine, you can also boot the machine from a NIM master in maintenance mode.

The bootlists are set using the bootlist command or through the System Management Services Progam (SMS). pressing F1 will go to SMS Mode.

then change the bootlist for service(maintenance) mode as 1st device to CD ROM.

# bootlist -m service cd0 hdisk0 hdisk1

then start maintenance mode for system recovery,

Access rootvg,
access this volum group to start a shell, then recreate BLV using bosboot command.

# bosboot -ad /dev/hdisk0

it's important that you do a proper shutdown, All changes need to be written from memory to disk.
 
# shutdown -Fr

Important!! bosboot command requires that boot logical volume hd5 exists. If you want create a BLV ( may be it had been deleted by mistake ), do the following,

1. boot your machine in maintenance mode,
2. Create a new hd5 logical volume, one PP size, must be in rootvg,specify boot as logical volume type,
# mklv -y hd5 -t boot rootvg 1
3. Then run bosboot command as described.

If you have an HMC, then at the time of booting select boot as SMS in the properties of that partition.

Monday, June 25, 2012

How to update AIX-rpm database

AIX-rpm is actually a "virtual" package which reflects what has been installed on the system by installp. It is created by the /usr/sbin/updtvpkg script when the rpm.rte is installed.

Since AIX-rpm has to have some sort of version number, it simply reflects the level of bos.rte on the system where /usr/sbin/updtvpkg is being run. It's just informational.

However, if you are on some slower systems with lots of software installed, /usr/sbin/updtvpkg may take a long time so wait a while. :)

If you want to run the command manually:

# /usr/sbin/updtvpkg

If you get an error similar to "cannot read header at 20760 for lookup" when running updtvpkg, run a rpm rebuilddb:

    # rpm --rebuilddb

Once you run updtvpkg, you can run a rpm -qa to see your new AIX-rpm package.

Thats all folks!

What is ODM?

This is a database that house many information in AIX. The information is categorised in object classes.

ODM Device Configuration Object Classes


A sample list of the ODM Device Configuration Object Classes follows:

PdDv
Predefined Devices

PdCn
Predefined Connection

PdAt
Predefined Attribute

Config_Rules
Configuration Rules

CuDv
Customized Devices

CuDep
Customized Dependency

CuAt
Customized Attribute

CuDvDr
Customized Device Driver

CuVPD
Customized Vital Product Data


A more complete list of the objects are at /etc/objrepos

root@myserver:/> ls /etc/objrepos/
ATM_PVC                 FRUB                    HACMPlogs               HACMPrules              HACMPudresource         SRCnotify
CDiagAtt                FRUB_SRC                HACMPmonitor            HACMPsa                 HACMPx25                SRCodmlock
CDiagAtt.vc             FRUs                    HACMPnetwork            HACMPsa_metadata        HACMPxd_mirror_group    SRCsubsvr
CDiagDev                FRUs_src                HACMPnim                HACMPsdisksubsys        MenuGoal                SRCsubsys
Config_Rules            HACMPadapter            HACMPnode               HACMPserver             PDiagAtt                SWservAt
CuAt                    HACMPcluster            HACMPnpp                HACMPsircol             PDiagAtt.vc             SWservAt.vc
CuAt.vc                 HACMPcommadapter        HACMPoemfilesystem      HACMPsite               PDiagDev                TMInput
CuData                  HACMPcommlink           HACMPoemfsmethods       HACMPsiteinfo           PDiagDev.vc             config_lock
CuData.vc               HACMPcsserver           HACMPoemvgmethods       HACMPsna                PDiagRes                crypto_module
CuDep                   HACMPcustom             HACMPoemvolumegroup     HACMPsp2                PDiagRes.vc             crypto_module.vc
CuDv                    HACMPdaemons            HACMPpager              HACMPspprc              PDiagTask               diag_lock
CuDvDr                  HACMPdisksubsys         HACMPpairtasks          HACMPsr                 PDiagTask.vc            errnotify
CuPath                  HACMPdisktype           HACMPpathtasks          HACMPsvc                PdAt                    history
CuPath.vc               HACMPercmf              HACMPport               HACMPsvcpprc            PdAt.vc                 history.vc
CuPathAt                HACMPercmfglobals       HACMPpprc               HACMPsvcrelationship    PdAtXtd                 inventory
CuPathAt.vc             HACMPevent              HACMPpprcconsistgrp     HACMPtape               PdAtXtd.vc              inventory.vc
CuVPD                   HACMPeventmgr           HACMPras                HACMPtc                 PdCn                    lpp
CuWxt                   HACMPfcfile             HACMPresource           HACMPtimer              PdDv                    lpp.vc
DAVars                  HACMPfcmodtime          HACMPresourcetype       HACMPtimersvc           PdDv.vc                 product
DSMOptions              HACMPfilecollection     HACMPrg_loc_dependency  HACMPtopsvcs            PdPathAt                product.vc
DSMOptions.vc           HACMPgpfs               HACMPrgdependency       HACMPude                PdPathAt.vc
DSMenu                  HACMPgroup              HACMPrresmethods        HACMPudres_def          SRCextmeth


Usage example (Gateway)


Checking number of gateway setup.


If you have more than one gateway listed, may need to remove the extra ones. Here myserver uses only the 10.10.10.254 gateway which is the correct one.

root@myserver:/> odmget -q "attribute=route" CuAt

CuAt:
        name = "inet0"
        attribute = "route"
        value = "net,-hopcount,0,,0,10.10.10.254"
        type = "R"
        generic = "DU"
        rep = "s"
        nls_index = 0


Here's a tip, in case you have more than one gateway, here's how to remove

# chdev -l inet0 -a delroute="net,-hopcount,0,,0,"
Method error (/usr/lib/methods/chginet):
        0514-068 Cause not known.
0821-279 writing to routing socket: The process does not exist.
route: not in table or multiple matches
0821-207 chginet: Cannot add route record to CuAt.


Automatically e-mail error report entries using ODM

We can automatically forward all error report entries to email.

Create a file like this:


    # cat /tmp/you
    errnotify:
      en_name="you"
      en_persistenceflg=1
      en_method="errpt -a -l $1|mail -s \"errpt: $9\" you@my.email.com"


Add this to the ODM:

# odmadd /tmp/you

Now log an entry in the error report:

# errlogger "Where are you?"

You will see in the error report:

# errpt -a
    ----------------------------------------------------
    LABEL:          OPMSG
    IDENTIFIER:     AA8AB241

    Date/Time:       Tue Oct  6 15:57:58 CDT 2011
    Sequence Number: 585
    Machine Id:      0004D6EC4C00
    Node Id:         hostname
    Class:           O
    Type:            TEMP
    Resource Name:   OPERATOR

    Description
    OPERATOR NOTIFICATION

    User Causes
    ERRLOGGER COMMAND

            Recommended Actions
            REVIEW DETAILED DATA

    Detail Data
    MESSAGE FROM ERRLOGGER COMMAND
    Where are you?


Clear the error log again (because we logged a fake test-entry in the error report):

# errclear 0

Watch for the email. You should receive the same error report entry in your email.

By the way, you can delete this from the ODM like this:

# odmdelete -q 'en_name=you' -o errnotify

Restoring individual file or directory from mksysb image


TO retrieve just one single file from a mksysb image backup.

Go to the directory that contains the mksysb image file

# cd /

Try locating that file in the image first

i.e. the hosts file

# restore -Tqlf myserver.mksysb | grep hosts
    New volume on myserver.mksysb:
    Cluster size is 51200 bytes (100 blocks).
    The volume number is 1.
    The backup date is: Thu Oct  9 23:00:28 MST 2011
    Files are backed up by name.
    The user is padmin.
    -rw-r--r--- 10   root  Sep 23  08:37  2903 ./etc/hosts


Here we see the original file was located in /etc/hosts.

Now recover this file:


# restore -xqf myserver.mksysb ./etc/hosts
    x ./etc/hosts


Please note that it is important to add the dot "." before the filename that needs to be recovered. Otherwise it won't work.

Your file is now restore to ./etc/hosts, which is a relative folder from the current directory you're in right now.

How to restore individual directory from a file name archive


restore -xdvqf /dev/rmt0 /home/mike/tools

A tip here, if you are restoring from file-system archive, remove the 'd' option.

All the files and subdirectories will be extracted to /home/mike/tools. Directories that don't exist will be created.


How to rectify stale VG

Encountered a VG going stale recently. Take me a while to figure how to resolve it with some google since there are very few people in the team that knows about AIX and stuffs..

Here, ff rootvg becomes stale as a result of a temporary loss of one of the disk (i.e. mirror), the following command can be used to get the ODM to refresh itself and synchronise the disks.

# varyonvg -bu >vg_name>

You can use the following command to monitor the progress of the refresh. Once all the LVs are refreshed, there should be no more opened/stale LVs.

lsvg -l 

AIX NFS setup

How to setup NFS server in AIX

This guide is recorded to help myself remember how to set up NFS service in AIX 7.1.

Turn on NFS server service

The easier way is to use smitty.

root@myserver:/> smitty nfs

Select "Network File system NFS"

Select "Configure NFS on This System"

Go to the "Start NFS" menu

Decide if you want NFS to start up automatically after each reboot. I selected "both" as i want it online all the time.


Done. You may see some output, unless there are any errors, you may want to check the NFS configurations otherwise, just see the "OK" status on the top left hand corner.


You should now have some services listening for NFS, like nfs and mountd processes
 
root@myserver:/> rpcinfo -p
   program vers proto   port  service
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    200006    1   udp   2049
    200006    4   udp   2049
    200006    1   tcp   2049
    200006    4   tcp   2049
    100005    1   tcp  32768  mountd
    100005    2   tcp  32768  mountd
    100005    3   tcp  32768  mountd
    100005    1   udp  32772  mountd
    100005    2   udp  32772  mountd
    100005    3   udp  32772  mountd
    400005    1   udp  32773
    100024    1   tcp  32769  status
    100024    1   udp  32779  status
    100133    1   tcp  32769
    100133    1   udp  32786
    200001    1   tcp  32769
    200001    1   udp  32793
    200001    2   tcp  32769
    200001    2   udp  32800
    100021    1   udp  32803  nlockmgr
    100021    2   udp  32803  nlockmgr
    100021    3   udp  32803  nlockmgr
    100021    4   udp  32803  nlockmgr
    100021    1   tcp  32770  nlockmgr
    100021    2   tcp  32770  nlockmgr
    100021    3   tcp  32770  nlockmgr
    100021    4   tcp  32770  nlockmgr
 
The file "/etc/rc.nfs" is used to control what NFS / NIS services can be started up.

Adding a directory into NFS service.

Method 1: use smitty

Start up smitty like above then go to the "Add a Directory to Exports List" menu.

Fill up the Pathname of directory to export, leave UID as "-2", set the NFS version (i used 3 as this is the best version i think all my servers can talk), then set which accounts from which servers can access this directory.



Method 2: use the CLI way.

Edit the exports file
 
root@myserver:/> more /etc/exports
/opt/myfolder/data -sec=sys:krb5p:krb5i:krb5:dh,rw,root=myserver
/opt/myfolder/binaries -sec=sys:krb5p:krb5i:krb5:dh,ro=myserver,root=myserver
 
AIX is very strict on who can access the partition, so make sure you add in who can mount the partition, i.e. root otherwise, you will not be able to mount the partition.
Also ensure that both NFS server and client can use the same security protocol, otherwise the result is same as above.
Lastly, put in explicitly who can read-write or read-only to control access.

Make sure that both the server and client end has each other's hostname and IP in the hosts files otherwise the next step will fail.

then restart the exports service
 
exportfs -a

Now we check and confirm NFS is shared as we want it.

root@myserver:/> exportfs
/opt/myfolder/data -sec=sys:krb5p:krb5i:krb5:dh,rw,root=myserver
/opt/myfolder/binaries     -sec=sys:krb5p:krb5i:krb5:dh,ro=myserver,root=myserver
 
If you are sharing between AIX and Solaris or Linux, use NFS version 3 for now to have full compatibility. For AIX NFS server, set in "Allow Access by NFS versions" entry. For Solaris client, mount with "-o vers=3" option.

How to mount logical volume from another node in the cluster

Stop using the logical volume in the first node.

# umount 
# varyoffvg


Import the disk at node 2


# importvg -L
# varyonvg

Pls Note

The 'L' option takes a volume group and learns about possible changes performed to that volume group. Any new logical volumes created as a result of this command emulate the ownership, group identification, and permissions of the /dev special file for the volume group listed in the -y flag. The -L flag performs the functional equivalent of the -F and -n flags during execution.

Restrictions:

    * The volume group must not be in an active state on the system executing the -L flag.
    * The volume group's disks must be unlocked on all systems that have the volume group varied on and operational. Volume groups and their disks may be unlocked, remain active and used via the varyonvg -b -u command.
    * The physical volume name provided must be of a good and known state, the disk named may not be in the missing or removed state.
    * If a logical volume name clash is detected, the command will fail. Unlike the basic importvg actions, clashing logical volume names will not be renamed.





 

Extra Info:

The steps also assume that the Major number on both nodes are the same. Otherwise, please watch potential issue when importing the VG into the node 2. You can check for next available major number using 'lvlstmajor' command.
h4. Now the LVs can be mounted and use.

# mount

If you have configured powerHA, this should be taken care of and transparent when doing a switch over. The steps above is the crude and manual way of doing what powerHA can do.

How to Collect system information

This is a tool taken from IBM performance manager suite to extract all configured system information. The script takes no parameters and output is saved in config.html. Good to have a snapshot of what is in the host.

# ls -ltr
total 40
-rwxr--r--    1 root     system        19719 Jan 27 2009  config.sh

Output of the script execution.
 
# ./config.sh
21:37:28-09/01/11 :     config.sh begins
21:37:29-09/01/11 :     BASIC_CONFIG
21:37:32-09/01/11 :     UNAME_PROCTYPE
./config.sh[14]: ./lsc:  not found
21:37:32-09/01/11 :     WHO
21:37:32-09/01/11 :     BOOTLIST
21:37:32-09/01/11 :     LSDEV
21:37:32-09/01/11 :     LSPV_DETAIL
21:37:32-09/01/11 :     LSVG
21:37:33-09/01/11 :     LSLV
21:37:36-09/01/11 :     SSA
21:37:36-09/01/11 :     LSATTR
21:37:37-09/01/11 :     FILESYSTEMS
21:37:39-09/01/11 :     LSSRC
21:37:39-09/01/11 :     PAGING_SPACE
21:37:39-09/01/11 :     NETSTAT
21:37:39-09/01/11 :     RPC
rpcinfo: 1831-256 unknownhost is unknown host
21:37:39-09/01/11 :     TUNABLES
21:37:40-09/01/11 :     SHOWMOUNT
21:37:40-09/01/11 :     RAS
21:37:40-09/01/11 :     WLM
21:37:40-09/01/11 :     AUDIT
21:37:40-09/01/11 :     SHELL_ENV
21:37:40-09/01/11 :     ERRPT
21:37:40-09/01/11 :     JAVA
java full version "J2RE 1.5.0 IBM AIX build pap32devifx-20110211b (SR12 FP3 +IZ94331)"
21:37:40-09/01/11 :     LSSLOT
21:37:40-09/01/11 :     LSCFG_VP
21:37:41-09/01/11 :     MICROCODE
21:37:42-09/01/11 :     PROCESSOR
./config.sh[14]: ./lsc:  not found
21:37:42-09/01/11 :     DUMP
21:37:42-09/01/11 :     BOSDEBUG
21:37:42-09/01/11 :     PROCESSES
21:37:42-09/01/11 :     MEMORY
./config.sh[14]: ./memdetails.sh:  not found
./config.sh[14]: ./getmempool.sh:  not found
21:37:42-09/01/11 :     FIBRE_CHANNEL
21:37:42-09/01/11 :     LOCKTRACE
21:37:42-09/01/11 :     UNIX_KERNEL
21:37:42-09/01/11 :     PCMCTRL
21:37:42-09/01/11 :     RSET
21:37:42-09/01/11 :     USERS
21:37:43-09/01/11 :     CRONTAB
21:37:43-09/01/11 :     FILES
21:37:43-09/01/11 :     LICENSES
21:37:43-09/01/11 :     VIO
21:37:43-09/01/11 :     TCBCK
21:37:43-09/01/11 :     MPIO
21:37:43-09/01/11 :     LPP_OSLEVEL
21:37:45-09/01/11 :     INSTFIX_TL_SP
21:37:45-09/01/11 :     DEVICES
21:37:45-09/01/11 :     MONITOR
21:37:45-09/01/11 :     HACMP
21:37:45-09/01/11 :     config.sh completed

     CONFIG.SH: Report is in file >>> config.html <<<

How to Add SAN or local volume in VIO Server and LPAR

The fact is, i'm still learning how to administer a IBM box.

The following procedure i am trying to record is very similar be it adding a partition that is SAN LUN or local disk. If in doubt, check using smitty.

The requirement for this is that all the disk or LUNs are presented to the VIO server before using VIO server to assign them as volumes to the LPARs.

If you have 2 VIO servers, which is usually the case for redundancy purpose, do this on both VIO server A good practice would be to mirror them at the LPAR level.

1) Log in to the VIO server. Verify if the disk or LUNs is visible to the VIO server

'Refresh' the device tree.
$ cfgdev

List out the PV. Those that are labeled as "None" are ones not assigned to any use yet.
$ lspv 

You can also double check using the "-free" option to list those that are not in use yet. This is only available in VIO in the restricted shell only.
$ lspv -free

2) Create VG in VIO server

Once the disk or LUNs is visible, go on to create VG. Notice that the PVID is randomly created for the disk or LUNs to uniquely identify them in the ODM.
$ mkvg -vg dbvg_clients hdisk8
dbvg_clients
0516-1254 mkvg: Changing the PVID in the ODM.

3) Next, Create LV in VIO server

Here we are creating the LV with "db_vg" LV name to the "dbvg_clients" VG name with 20Gb space from hdisk8.
$ mklv -lv db_vg dbvg_clients 20000M hdisk8
db_vg

4) Now, Assign the VG to the LPAR.

Create a virtual adapter in HMC to map to the LPAR, then reload in VIOS and assign the VG to the LPAR.
$ mkvdev -vdev db_vg -vadapter vhost5
vtscsi0 Available
 



For easy management you may want to separate the different VG that is assigned to the LPAR via the vhostX. i.e. rootvg assigned to vhost4 (myserver) and datavg assigned to vhost5 (myserver too). Also if there are multiple SAN devices you want to connect to, recommendation to use a different vhost so that in the event that you need to remove access to that SAN device, the virtual resources can be removed easily without downtime.
vhost is the "Virtual SCSI Server Adapter" that maps to the LPAR. To know which LPAR is mapped to which vhost, check the HMC.
In HMC, Select the LPAR > Hardware Information > Virtual I/O adapters > SCSI
Look for your host(s) in "remote partition" and note their 'vhost' number another way is to check from the VIO server.
$ lsmap -all | grep vhost | grep 0005
vhost4          U8233.E8B.1003D8P-V1-C40                     0x00000005
vhost5          U8233.E8B.1003D8P-V1-C41                     0x00000005
vhost12         U8233.E8B.1003D8P-V1-C42                     0x00000005

5) Lets login to the LPAR "vhost5" aka myserver.

Refresh the device tree.
# cfgmgr

The disk should now appear. Here, we see hdisk6
root@myserver:> lspv
hdisk0          00f603d83df2e2f6                    rootvg          active
hdisk1          00f603d843c55fa7                    rootvg          active
hdisk2          00f603d852925bfd                    oravg           active
hdisk3          00f603d891cf7b8a                    oravg2          active
hdisk4          00f603d852925c71                    oravg           active
hdisk5          00f603d891cf7c09                    oravg2          active
hdisk6          none                                None

6) Create VG in LPAR

We will create a VG using hdisk6. Noticed that the PVID is generated for the PV.
root@myserver:/> mkvg -y oravg3 hdisk6
0516-1254 mkvg: Changing the PVID in the ODM.
oravg3

7) Next, create LV in LPAR

The command below creates the LV named "oratmplv" belonging to oravg3 VG with 750 LP from hdisk6. As each LP is 256Mb, total LV size is 192Gb.
# /usr/sbin/mklv -y oratmplv -t jfs2 oravg3 750 hdisk6

8) Create journal log in LPAR if required.

Journal log partition is the log device used to provide the partition protection. It is used by OS to recover the partition when required.
This LV oratemplv has 3 LP from hdisk6.
# /usr/sbin/mklv -y oradjlogv3  -t jfs2log oravg3 3 hdisk6
General guide is 2Mb for every 1Gb or 1 partition per 512 partition as a baseline.
In some cases where the filesystem activities are too heavy or too frequent for the log device, you might wrap errors like the below in errpt
LABEL: J2_LOG_WAIT IDENTIFIER: CF71B5B3
LABEL: J2_LOG_WRAP IDENTIFIER: 061675CF

Increase the size of the journal log as follows
# extendlv  

Check and unmount the log device
# mount
# unmount /dev/

Format and mount back the log device
# logform /dev/
#mount 

My recommendation is to quieten the LV data partition before working on the LV journal log device.

9) Format the log device

We will use the log device as it is and not use it as "inline" for data LV. (Not required if you don't need to specify a separate log device)
root@myserver:/> logform  /dev/oradjloglv3
logform: destroy /dev/roradjloglv3 (y)?y

10) Create file-system with mount-point.

create mount-point.
# mkdir /opt/oracle11

We create the file system to mount to /opt/oracle11.

If you are using "smitty storage" to create file system, do the following
  • put in the log device in "Logical Volume for Log"
  • leave the "Inline Log size" blank.
Alternatively, use crfs with "-a logname='oradjloglv3'" or mkfs.
# mkfs -o log=/dev/oradjloglv3 -V jfs2 /dev/oratmplv

11) Mount the file system and ready for use.

Confirm file-system integrity
# fsck -p /dev/oratemplv

Then finally create mountpoint and mount the filesystem
# chown -R oracle:dba /opt/oracle11
# mount -o log=/dev/oradjloglv3 /dev/oratmplv /opt/oracle11
 


Remember to add into /etc/filesystem if you want to mount it after every restart.
If you used smitty to mount the filesystem, choose "Mount AUTOMATICALLY at system restart?" so that you don't need to meddle with the /etc/filesystem file.

Extracting public SSH key from its private key

There are numerous situations where we set up controlled password-less SSH access between 2 different accounts.

For example, we might use the private key 'Identity' to access the relevant servers to implement updates to applications.

ssh –i identity … … …

And that the identity is actually a DSA encrypted private key.

[root@PROD ~]# more id_dsa
-----BEGIN DSA PRIVATE KEY-----
MIIBugIBAAKBgQCHSKGUulKjivv+k6V6v3AosDG2fR/tpVQMQy5zgCk+6qu7cnzr
COJsJ7Kgwo5hGnh84CMoJrBvTt+EgYm+j+M42hJlBw3T3Y2mLZPm41cctLwmkV81
3WLaLLrwgCVlFGojPqVZp+rTX7X2+7/Y7UwyjjyS82FeYMuJY0+JqxkJEwIVALJO
t1gjM9qhAffvHLuQCv1a+jVpAoGAU/luROlbuzsPsXmxOCFTZUQCKSKB7gCHMqYS
3+S38omZiKyit4iYrtpbVKosH9IhPX8YJvYJya/UKCVllkuMHR0qybEizDbzRmWQ
4onBim3GyFq9C71i72XWHcVrndrwuHp+7rUcA2J4REkft8kodKs2SOZCKlnUduOz
K3ZwRRMCgYByS35wDwH6b7dym1eGj/3YCAKg8iGrfbBgZRMhvg3Z1+nxr9Uk5Vbl
1aOuLZ9/cmF5aEgBbSvW1+04DQh0cS+qCnUaIBGhMDzWFYIrGF7pfoJTfhS58SfT
YBcFo14hpU+SYCYaxjJI2KIMh85y+5pdJBGMvfsLBDo1k7YSxxvVRwIUMIBGrgmK
EgtczjE0j6N9zpozxCv=
-----END DSA PRIVATE KEY-----

If you do not have or misplaced the public key (rare case i think), we can actually extract the public key from the private key by the following command.

[root@PROD ~]# ssh-keygen -y -f id_dsa
ssh-dss AAAAB3NzaC1kc3MAAACBAIcvBnM6UqOK+/6TpXq/cCiwMbZ9H+2lVAxDLnOAKT7qq7tyfOsI4mwnsqDCjmEaeHzgIygmsG9O34SBib6P4zjaEmUHDdPdjaYtk+bjVxy0vCaRXzXdYtosuvCAJWUUaiM+pVmn6tNftfb7v9jtTDKOPJLzYV5gy4ljT4mrGQkTAAAAFQCyTrdYIzPaoQH37xy7kAr9Wvo1aQAAAIBT+W5E6Vu7Ow+xebE4IVNlRAIpIoHuAIcyphLf5LfygQbU3KK3iJiu2ltUqiwf0iE9fxgm9gnJr9QoJWWWS4wdHSrJsSLMNvNGZZDiicGKbcbIWr0LvWLvZdYdxWud2vC4en7utRwDYnhESR+3ySh0qzZI5kIqWdR247MrdnBFEwAAAIByS35wDwH6b7dym1eGj/3YCAKg8iGrfbBgZRMhvg3Z1+nxr9Uk5Vbl1aOuLZ9/cmF5bCjzbSvW1+04DQh0cS+qCnUaIBGhMDzWFYIrGF7pfoJTfhS58SfTYBcFo14hpU+SYCYaxjJI2KIMh85y+5pdJBGMvfsLBDowErtYxxvVRw==

Then you can use the output in the authorized_keys for setting up the password-less access.

~end

How to rename disk device

For ease of managing the disks, especially in a cluster is very important.

Advantage of this will be allow admins to regconise the disk fast and less errors for 'varyon' or 'varyoff' when importing or exporting the disks.

Disadvantage is that you need to plan ahead the "numbering" of the disks so that the number actually mean something.


> lspv
hdisk0          00f603d2341862f6                    rootvg          active
hdisk1          00f603d243581d29                    rootvg          active
 
> rendev -l hdisk1 -n hdisk99
 
> lspv
hdisk0          00f603d2341862f6                    rootvg          active
hdisk99          00f603d243581d29                    rootvg          active