Apache on Windows cant start because of SSLSessionCache error

Received a call that a new setup of Apache could not start up no matter how she tweak the configurations.

She is on MS Windows Server 64bit, Apache 2.2.24 64 bit, etc, etc..

Event log:















Another event log that points the exact problem.


















As i administer a large set of servers that are of different flavours, first i did a comparison between the UNIX httpd.conf and this Windows one. Noticed that there is this chuck "(x86)" is in the windows configuration file.

Microsoft used this (x86) string to differentiate between 64bit and 32bit programs and Apache is unable to interpret it.

In the UNIX world, we can usually use escaping to workaround but i'm not sure if this can work in windows world as setting to 8.3 format using \~1 or \~2 does not work 100% of the time.

In the end, just advise her to point the path to another path that does not have any parenthesis. The Apache work like a charm.

Job done!

How to list the top 5 largest directories

# du -sk /opt/* | sort +0nr | head -5
25456556        /opt/data
12634192        /opt/read
5483564 /opt/download
196104  /opt/scripts
132964  /opt/freeware

tip.. if you want to include the hidden files and directories, try this.

# du -sk .[a-z]* * | sort +0nr | head -5

How to un-mount a volume forcefully

If you ever want to unmount a volume forcefully when the system does not allow, you need to find out who is holding on to the resource, terminate it and then unmount. This is very important to prevent data loss.

Who is holding on to the volume

# lsof | grep "/opt/download"


# fuser -cu /opt/data/System.log

Now terminate them

# kill -9

The PID will be determined from the fuser or lsof command above.

Now unmount the volume peacefully

# umount /opt/download

If you disregard who's active and want to unmount right away,

  # fuser -km /opt/download

In case, you want to unmount a NFS volume that is unreachable, try this.

# umount -f /opt/download

Done...

How to retrieve the list of failed logins in AIX

short and sharp.

# /usr/sbin/acct/fwtmp < /etc/security/failedlogin  | more

How to modify permission for queue on MS MQ

Recently, a request was made to modify the permission on a queue on MS MQ. Has been administering IBM Websphere MQ so this one is new to me.

Failed attempt

Open up Computer Management console, navigated to the queues under services and application, right click to the security properties and started tweaking.


Ops. System says "Access to Message Queuing System is denied" that essential mean i do not have the rights. Probably administrator account was not used to create the queues? So how?

Found the solution!

Turn out that i just need to click "Advanced" from the security tab, then go to the owner tab, take over control using administrator account, apply and ok. Going back to assign permission to the queue now is OK.

Done. :D

Default password in Websphere Application Server plug-in for Apache expiry

If you are using the key store installed by default on your web server for the Websphere Application Server plug-in and that you have never  changed the key store's password, you will hit the following problem.

[root@server ]# gsk7capicmd_64 -keydb -expiry -db ./plugin-key.kdb -pw WebAS
Validity:  Thursday, 26 April 2012 23:20:31 PM SGT

Do note that default password for the key DB is "WebAS".

To resolve this issue, you can just change the password without setting the expiry date.

Here's the steps that i used. (only CLI since i don't usually access UI.)

Get your LD library path

[root@server ]# export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/IBM/WebSphere/Plugins/gsk7/gsk7_64/lib64/

Check that the Key DB really expired

 [root@server ]# gsk7capicmd_64 -keydb -expiry -db ./plugin-key.kdb -pw WebAS
Validity:  Thursday, 26 April 2012 23:20:31 PM SGT

Let's change the password. Take note that you must stash the key for the password to be saved.

[root@server ]# gsk7capicmd_64 -keydb -changepw -pw WebAS -new_pw WebAS2 -stash -db plugin-key.kdb

If you really fancy, change the password back. (But i don't recommend this as this pose a big security risk!)

[root@server ]# gsk7capicmd_64 -keydb -changepw -pw WebAS2 -new_pw WebAS -stash -db plugin-key.kdb

Let's verify.

[root@server ]# gsk7capicmd_64 -keydb -expiry -db ./plugin-key.kdb -pw WebAS
Validity:  0


We are now done with the password change.

IBM Reference: http://www-01.ibm.com/support/docview.wss?uid=swg21579757

AIX su restriction using sugroup

In AIX, we can restrict who can access a particular user account using the SUGROUP parameter.

Background

Using the below example to explain.

We have normal user account ‘user1’.

We are going to create the admin account for ‘user1’, this account is ‘admin1’.

We do not want to allow any NON-admin to access ‘admin1’, hence we use SUGROUP to restrict. Here, I used ‘admingrp’ group since all admin are in this group.

Not to worry if another DBA access ‘admin1’ since /var/log/authlog would show who used that account. Below example log showed someone using root accessing ‘user2’ account before using ‘admin1’.

devserver:/:>tail -2 /var/log/authlog

Feb 14 10:12:11 devserver auth|security:notice su: from root to user2 at /dev/pts/0

Feb 14 10:12:15 devserver auth|security:notice su: from user2 to admin1 at /dev/pts/0

Parameters and Procedure

The account creation via smitty in its entirety. Further notes

-          Requirement is ‘admin1’ should be member of the group ‘appgrp’ is in so that he can access files in appgrp.

-          Also ‘admin1’ needs to be member of staff group to access ‘user1’ files.

-          Since this is a privilege account, we set the following

o   No remote login (SSH, telnet, etc)

o   No local login (physical, console)

o   Shorter account expiry

-          SU GROUP set to ‘admingrp’

 If you want to user command line, which I don’t really recommend unless you script.

devserver:/:>mkuser "id=11705" "pgrp=appgrp" "groups=appgrp,staff" "home=/home/admin1" "shell=/usr/bin/bash" "gecos=name name name" "login=false" "su=true" "rlogin=false" "admin=false" "sugroups=admingrp" "maxage=5" admin1

devserver:/:>passwd admin1

devserver:/:>pwdadm -c admin1

Result

Remote login will fail.

me@server [~]

~$ssh admin1@devserver

admin1@devserver's password:

Received disconnect from 10.10.50.10: 2: Remote login for account admin1 is not allowed.

Access to root will fail.

devserver:/:>su -user1

$ su - admin1

admin1's Password:

-bash-3.2$ su -

root's Password:

You are not allowed to su to this account.

Cannot su to "root" : Account is not accessible.

Non admin will not be able to access this account.

me@devserver [~]

~$su - admin1

admin1's Password:

You are not allowed to su to this account.

Cannot su to "admin1" : Account is not accessible.

Another admin can access this account.

devserver:/:>su - user2

$ su - admin1

admin1's Password:

-bash-3.2$ id

uid=11705(admin1) gid=101(appgrp) groups=1(staff)

  

Switching LDAP repository in Websphere Application Server 7

This is to record the steps i used to switch LDAP repository in Websphere Application Server 7 and enabled LDAP over SSL.

Lets Start

Point your browser to the WAS console and login using admin account.

Add in the new LDAP server configurations

As i use a few repositories in my environment, i would be updating the repositories in the "Federated repositories" section.

Click on left column's link: Security -> Global Security -> Configure (the drop down box is pointed to "Federated repositories"

The page refreshed, look for "Manage repositories" -> Add

Fill the following.
- Repository identifier
- Directory type (here, i used IBM TDS)
- Primary hostname (put in IP, if you have the entry in /etc/hosts, can use hostname)
- Port (389 for a start. Later will be updated to 636)
- Bind DN (The account to connect to LDAP server)

Click Apply when you are done. Then Click Save. Wait for sync to finish and click OK.

If there is something wrong, WAS will complain.

i.e.
cannot reach the LDAP server. (ACL/firewall??)
wrong port
Bind DN is wrong
etc

Import LDAP server SSL

 Store the SSL certificate as a flat file in the WAS server. You may need to convert the SSL certificate to "der" format and deposit the certificate to the deployment manager directory.

Create WAS Truststore

We will create the key store in WAS to store keys and certificate for LDAP. Idea is to separate key store for different functions.

By java definition, keystore is an object that holds personal certificate. truststore is a Java object that holds signer certificates. I gather we will only create the truststore since WAS security guide listed this step.

Click Security -> SSL Certificate and key management -> keystores and certificates

Then click New

Fill up the following
- Name (i used LDAPTruststore)
- Management scope (IBM security guide recommended cell level)
- Path (Where you want to store this key store)
- Password

Import the LDAP SSL certiticate into LDAPTruststore

From the breadcrumb of the previous step, Click on "LDAPTruststore" and Signer certificates.


Click Add


Fill up the following
- Alias (i used ldapcert)
- File Name (the path to the LDAP ssl certificate you put in previous step.)


Click Apply when you are done. Then Click Save. Wait for sync to finish and click OK

Creating SSL alias link to the Trust store

We will create Click Security -> SSL certificate and key management -> SSL configuration.

Click New

Fill up the following
- Name (I used LDAPSSLSettings)
- Trust store name (its LDAPTruststore)
- Key store name (its LDAPTruststore)
- Management Scope (Its Cell Level)

Click OK then Save then OK.

We are now ready to enable LDAP over SSL communication to LDAP server

Go back to the repository.

Click Security -> Global Security -> Configure (drop down bar should point to "Federated repositories")

When the page refreshed, Click "Manage repositories"

Fill up the following
- Port (change to 636)

Check the "Require SSL communication"

Choose the radio button "use specific SSL alias" and select LDAPSSLSettings from the drop down menu.

Click Apply when you are done. Then Click Save. Wait for sync to finish and click OK

Add the Base DN

Here, we need to configure from where in LDAP server we should make the queries.

Click Security -> Global Security -> Configure (drop down bar should point to "Federated repositories")

Click the "Add base entry to realm" button

Fill up the following
- Repository (Put in the name you used for "Repository identifier")
- OU (Put in the Base DN here)

Click Apply when you are done. Then Click Save. Wait for sync to finish and click OK 

A little housekeeping

Removethe base DN for the old LDAP server.

Click on the base DN -> remove
Then Click Save. Wait for sync to finish and click OK  

Click on Manage repositories
Check the old repository -> delete
Then Click Save. Wait for sync to finish and click OK  

Restart AppSvr, NodeMgr and Dmgr

To be safe, i would prefer to restart everything and make sure i still can log in as administrator and the application would have no problem working with the new LDAP server.

Check the Dmgr logs, AppSvr logs for signs of errors.

A small test

Click on "users and groups" -> manage users
Search for some valid users and verify that they come from the new LDAP server.
Get the software team to verify too in case the problem is subtle enough not to be caught in the application logs.

Force refresh of remote device's ARP cache

Background

ARP.
This is short for Address Resolution Protocol.
This is a method for finding the hardware address (MAC) when only its IP is known.

On OSI model, ARP is a Data Link Layer Protocol. (Layer 2). IP is on Network Layer (layer 3).

There are times when you need to force the remote devices, such as the routers, switches or gateways to refresh their ARP cache when you perform some network changes on servers. Some of these changes includes

• Physical Box relocation (requiring a new network IP)
• Logical IP relocation (especially in cluster or DR)

ARP cache will be updated usually when either of the following happens

• Expiry of the entry in the cache
• Host updates the router about the change in IP
• ARP cache is manually cleared.

Without forcing through an immediate ARP cache refresh, sometimes the network will not know the changes until very much later and end result is service unaccessible, causing losses to busineseses and much headaches for the IT team.

Procedures (RHEL)

On RHEL, usually arping is available in the default installation. You need to execute the command in root.

# /sbin/arping -s -b

where

-b : keep broadcasting, don't go unicast
-s source : source ip address

Procedures (Windows XP, Vista, 7)

netsh interface IP delete arpcache

What is linux-gate.so.1?

Recently, was tracing some faults on a RHEL system and noticed that there is this "linux-gate.so.1" library that is pointed by many of the binaries.

# ldd /sbin/vxconfigd

linux-gate.so.1 => (0xffffe000)
libcrypt.so.1 => /lib/libcrypt.so.1 (0x0054e000)
libm.so.6 => /lib/tls/libm.so.6 (0x004de000)
libc.so.6 => /lib/tls/libc.so.6 (0x003aa000)
libdl.so.2 => /lib/libdl.so.2 (0x004d8000)
libvxscsi.so => /etc/vx/slib/libvxscsi.so (0xf7fcd000)
libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00515000)
/lib/ld-linux.so.2 (0x00390000)

Seems that this is similar even on RHEL 4.2 and 4.6.
Asked Mr. Google and this URL turned up. [http://www.trilithium.com/johan/2005/08/linux-gate/]

In summary, this file is a virtual DSO, dynamically shared objected exposed by the kernel at a fixed address in every process memory.
If you want more details, check out the above link. :)

What does clfileprop process do

May have seen the following process

myserver:/:>ps -aef | grep clfile
root 4063286 5570752 0 00:59:56 - 0:00 /usr/es/sbin/cluster/utilities/clfileprop -a

This process, in essense,
- belongs to powerHA.
- runs every 10 minutes on my servers. Think this is default.
- propagate changes of configuration files to all other nodes.
- caveat is, if you run manually on node A, this node A will propagate files from node A to other nodes regardless of last modified date. e.g. if you run verification, etc.

My understanding of RBAC in AIX

What is RBAC?

It stand for Role Based Access Control.
There is major differences between RBAC in AIX 5.3 and older AND RBAC in AIX 6.1/7.1. No value in discussing older RBAC. Will explain for "enhanced RBAC" instead.
Three primary rules are defined for RBAC:

• Role assignment: A subject can exercise a permission only if the subject has selected or been assigned a role.
• Role authorization: A subject's active role must be authorized for the subject.
  With rule 1 above, this rule ensures that users can take on only roles for which they are authorized.
• Permission authorization: A subject can exercise a permission only if the permission is authorized for the subject's active role.
  With rules 1 and 2, this rule ensures that users can exercise only permissions for which they are authorized.

The traditional DAC

Traditional access control, as we call it DAC (discretional access control) has been used for ages and taken for granted. The familiar string r-x------ is fundamental for all sys admin. DAC provides SUID, GUID, etc but the control scope deals with All, GROUP or OWNER access.

AIX RBAC

RBAC provides precise access control such that the target role can only be assumed by a particular user. The range of commands the role can access could be a subset of all the commands that a root or any other account actually have. 

Difference from SUDO

SUDO is another means to control the access to privilege commands. However, it can be tedious to configure each and every commands that you want to allow an account to access.

Difference from Solaris RBAC

In essence, both Solaris RBAC and AIX RBAC are similar. The main difference is the way to implement it.
In Solaris, we use mainly the following files to setup RBAC.

root:/ #ls -l /etc/user_attr /etc/security/exec_attr /etc/security/prof_attr /etc/security/auth_attr
-rw-r--r--   1 root     sys        11855 Mar 28  2012 /etc/security/auth_attr
-rw-r--r--   1 root     sys        20934 Aug 15 11:40 /etc/security/exec_attr
-rw-r--r--   1 root     sys         8433 Aug 15 11:42 /etc/security/prof_attr
-rw-r--r--   1 root     sys         1292 Aug 30 12:01 /etc/user_attr

Authorisation file for Solaris.

root:/ #tail -3 /etc/security/auth_attr
solaris.system.:::Machine Administration::help=SysHeader.html
solaris.system.date:::Set Date & Time::help=SysDate.html
solaris.system.shutdown:::Shutdown the System::help=SysShutdown.html

In AIX, this authorisation list is kept in a DB. You can create custom ones, especially for those not already in the DB. AIX don't provide the help HTML file. In reality, do we use them?

server:/: lsauth ALL | tail -3
wpar.mobility.appli id=10014
wpar.mobility.appli.other id=10016
wpar.mobility.appli.owner id=10015

The Solaris file that manage the effective privilege level to execute the command

root:/ #tail -3 /etc/security/exec_attr
Zone Management:solaris:cmd:::/usr/sbin/zoneadm:uid=0
Zone Management:solaris:cmd:::/usr/sbin/zonecfg:uid=0
DisasterRecovery Admin:suser:cmd:::/opt/sysadmin/Portnet_DR_Scripts/*:uid=root

Next, not much meaning in this profile file but only to maintain the profile and description of the role

root:/: #tail -5 /etc/security/prof_attr
ZFS Storage Management:::Create and Manage ZFS Storage Pools:help=RtZFSStorageMngmnt.html
Zone Management:::Zones Virtual Application Environment Administration:help=RtZoneMngmnt.html
dtwm:::Do not assign to users. Actions and commands required for the window manager (dtwm).:help=Rtdtwm.html
shutdown:::Do not assign to users. Contains actions requiring shutdown authorization.:auths=solaris.system.shutdown;help=Rtshutdown.html
DisasterRecovery Admin:::For running DisasterRecovery scripts:

In AIX, here it is though we can set much more information, like password control, access to smitty and all that.

server:/: lsrole -f appadmin
appadmin:
        authorizations=aix.system.cluster
        rolelist=
        groups=admingrp
        visibility=1
        screens=*
        dfltmsg=role to manage Application resources
        msgcat=
        auth_mode=NONE
        id=11

In Solaris, the file that assign who can assume the role.

root:/ #tail -3 /etc/user_attr
me::::type=normal;profiles=DNS Admin
you::::type=normal;profiles=DNS Admin
her::::type=normal;profiles=DNS Admin

AIX keep this information in the ODM too.

server:/: lsuser -f meuser | grep role
        default_roles=
        roles=appadmin

How to setup

Say for instance, powerHA can only be accessed by root. but to allow menu control of cluster resources, we need to have a means to start/stop/restart/suspend/resume/failover the resources without using root. It is a bad security idea to allow menu to manage the cluster resources via root account.

Hence, we authorise, say meuser to access powerHA administrative commands by giving it ibm.hacmp.admin authorisation. How do we do that?

Check that Enhanced RBAC is enabled.

# lsattr -El sys0 -a enhanced_RBAC

enhanced_RBAC true Enhanced RBAC Mode True 

Let's create the authorisations.

/:> mkauth dfltmsg='IBM custom' ibm
/:> mkauth dfltmsg='IBM custom hacmp' ibm.hacmp
/:> mkauth dfltmsg='IBM custom hacmp admin' ibm.hacmp.admin 

Then check out what privileges that the commands that you are using requires.

# tracepriv -ef /usr/es/sbin/cluster/utilities/clRGinfo

-----------------------------------------------------------------------------
Group Name     State                        Node
-----------------------------------------------------------------------------
apps_rg     ONLINE                       servera
               OFFLINE                      serverb

9568366: Used privileges for /usr/es/sbin/cluster/utilities/clRGinfo:
  PV_AU_ADMIN                        PV_NET_CNTL
  PV_NET_PORT
 

# tracepriv -ef /usr/es/sbin/cluster/events/utils/cl_RMupdate 

...

...

...

if you need to use your own shell script, you may need to add it into the privileged command database. Allow EUID to be equal to the owner of that script.

Now we add the commands into the privileged command database.

/:> setsecattr -c innateprivs=PV_AU_ADMIN,PV_NET_PORT,PV_NET_CNTL accessauths=ibm.hacmp.admin /usr/es/sbin/cluster/utilities/clRGinfo

/:> setsecattr -c innateprivs=PV_AU_ADMIN,PV_KER_ACCT,PV_PROC_PRIV accessauths=ibm.hacmp.admin euid=0 /usr/es/sbin/cluster/events/utils/cl_RMupdate 

/:> setsecattr -c innateprivs=PV_AU_ADMIN,PV_KER_ACCT,PV_PROC_PRIV accessauths=ibm.hacmp.admin euid=0 /admin.sh
/:> setsecattr -c innateprivs=PV_AU_ADMIN accessauths=ibm.hacmp.admin euid=0 /dlpar.sh

You can verify by using lssecattr.

/:> lssecattr -F -c /dlpar.sh
/dlpar.sh:
        euid=0
        accessauths=ibm.hacmp.admin
        innateprivs=PV_AU_ADMIN

Now, we create a role with the above authorisations.

# mkrole authorizations=ibm.hacmp.admin dfltmsg="Custom role to do admin with hacmp" appadmin

if its for automation, you may want to remove password access to the role by the following command. chrole auth_mode=NONE appadmin By default, auth_mode is INVOKER which means that is password control.

Next, allow meuser to be able to assume the role

 chuser roles=appadmin meuser

Before you try it out, you need to update the kernel for all these to take effect. As AIX kernel is RBAC aware for all the IBM system commands, without updating the kernel, any changes will not take effect.

setket

Try it out

swrole 

If you are not allow to assume the role you will receive the following error. In this example, thatuser should not assume appadmin role.

server:/HAapps: su - thatuser
-bash-3.2$ swrole appadmin
swrole: 1420-052 appadmin is not a valid role for thatuser.

It is authorised to assume meuser role instead.

server:/HAapps: su - meuser
-bash-3.2$ swrole appadmin
bash-3.2$ /usr/es/sbin/cluster/events/utils/cl_RMupdate suspend_appmon apps apps_rg
Suspend HA Monitoring for apps.
2012-10-22T15:58:03.289727
2012-10-22T15:58:03.309369
Oct 22 2012 15:58:03 cl_RMupdate: Completed request to suspend monitor(s) for application apps.
Oct 22 2012 15:58:03 cl_RMupdate: The following monitor(s) are in use for application apps:
apps_svr
apps_dm

Reference: http://aixhelp.blogspot.sg/2010/12/aix6-rbac.html

Good reference link of powerHA

This is a bookmark of powerHA links i find it useful. Keep them here just in case. :)

http://aix4admins.blogspot.sg/2011/10/commands.html

Allowing longer web session going through Apache to Websphere Application Server

Had a tough one last month when migrating the system to WAS. I'm still new to WAS, hit a few problems and take this chance to document down so that this form my reference and hopefully it help you too.

Users has been complaining that the web service keep getting time out, returning a 500 error.

What i found out was that i can actually, tune the "ServerIOTimeout" parameter in the WAS plugin for Apache beyond the default. i used 900, which is 15min in seconds.

In addition,a little performance fine tuning was done using "LoadBalanceWeight" to keep the application servers from being 'hit' random when they are just started up, especially when i have a cluster of them. As recommended by IBM specialist, i used some numbers, with one of the application server assigned to an odd number, different from the rest.

The idea is to make one of the application server the first one to serve, instead of randomising it.

      Server CloneID="179d3la" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="20" MaxConnections="-1" Name="Node1" ServerIOTimeout="900" WaitForContinue="false"

...
...
     Server CloneID="179d5sb" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="20" MaxConnections="-1" Name="Node2" ServerIOTimeout="900" WaitForContinue="false"

...
...
      Server CloneID="179d8gc" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="21" MaxConnections="-1" Name="Node3" ServerIOTimeout="900" WaitForContinue="false"

...
...



Do let me know if you have better ideas of solving it.

Recovering Websphere Application Server (WAS) after hitting JMS error

Recently, the following errors were logged in the SystemOut.log of WAS version 7. The whole cluster of WAS came to a halt with a flurry of angry calls asking why the web service is down.

- BMXAA1580E - A Java Message System (JMS) error occurred
- CWSIT0088E: There are currently no messaging engines in bus intjmsbus running.

Apparently, the messaging engine went down. All the application servers can communicate via JMS and sat there idling.

Used the following way to recover the system. even though i still don't the reason even after logging a PMR yet, here's how.

1) bring down the Application Server
2) bring down the node agent if you have.
3) to be safe, i bring down the Deployment manager too.
4) move the messagestore Log file so that WAS can recreate it upon start up.
5) remove the transaction/tranlog/log1 and log2 file so that WAS can recreate it upon start up.
6) start up WAS and node agent.