Default password in Websphere Application Server plug-in for Apache expiry

If you are using the key store installed by default on your web server for the Websphere Application Server plug-in and that you have never  changed the key store's password, you will hit the following problem.

[root@server ]# gsk7capicmd_64 -keydb -expiry -db ./plugin-key.kdb -pw WebAS
Validity:  Thursday, 26 April 2012 23:20:31 PM SGT

Do note that default password for the key DB is "WebAS".

To resolve this issue, you can just change the password without setting the expiry date.

Here's the steps that i used. (only CLI since i don't usually access UI.)

Get your LD library path

[root@server ]# export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/IBM/WebSphere/Plugins/gsk7/gsk7_64/lib64/

Check that the Key DB really expired

 [root@server ]# gsk7capicmd_64 -keydb -expiry -db ./plugin-key.kdb -pw WebAS
Validity:  Thursday, 26 April 2012 23:20:31 PM SGT

Let's change the password. Take note that you must stash the key for the password to be saved.

[root@server ]# gsk7capicmd_64 -keydb -changepw -pw WebAS -new_pw WebAS2 -stash -db plugin-key.kdb

If you really fancy, change the password back. (But i don't recommend this as this pose a big security risk!)

[root@server ]# gsk7capicmd_64 -keydb -changepw -pw WebAS2 -new_pw WebAS -stash -db plugin-key.kdb

Let's verify.

[root@server ]# gsk7capicmd_64 -keydb -expiry -db ./plugin-key.kdb -pw WebAS
Validity:  0


We are now done with the password change.

IBM Reference: http://www-01.ibm.com/support/docview.wss?uid=swg21579757

Switching LDAP repository in Websphere Application Server 7

This is to record the steps i used to switch LDAP repository in Websphere Application Server 7 and enabled LDAP over SSL.

Lets Start

Point your browser to the WAS console and login using admin account.

Add in the new LDAP server configurations

As i use a few repositories in my environment, i would be updating the repositories in the "Federated repositories" section.

Click on left column's link: Security -> Global Security -> Configure (the drop down box is pointed to "Federated repositories"

The page refreshed, look for "Manage repositories" -> Add

Fill the following.
- Repository identifier
- Directory type (here, i used IBM TDS)
- Primary hostname (put in IP, if you have the entry in /etc/hosts, can use hostname)
- Port (389 for a start. Later will be updated to 636)
- Bind DN (The account to connect to LDAP server)

Click Apply when you are done. Then Click Save. Wait for sync to finish and click OK.

If there is something wrong, WAS will complain.

i.e.
cannot reach the LDAP server. (ACL/firewall??)
wrong port
Bind DN is wrong
etc

Import LDAP server SSL

 Store the SSL certificate as a flat file in the WAS server. You may need to convert the SSL certificate to "der" format and deposit the certificate to the deployment manager directory.

Create WAS Truststore

We will create the key store in WAS to store keys and certificate for LDAP. Idea is to separate key store for different functions.

By java definition, keystore is an object that holds personal certificate. truststore is a Java object that holds signer certificates. I gather we will only create the truststore since WAS security guide listed this step.

Click Security -> SSL Certificate and key management -> keystores and certificates

Then click New

Fill up the following
- Name (i used LDAPTruststore)
- Management scope (IBM security guide recommended cell level)
- Path (Where you want to store this key store)
- Password

Import the LDAP SSL certiticate into LDAPTruststore

From the breadcrumb of the previous step, Click on "LDAPTruststore" and Signer certificates.


Click Add


Fill up the following
- Alias (i used ldapcert)
- File Name (the path to the LDAP ssl certificate you put in previous step.)


Click Apply when you are done. Then Click Save. Wait for sync to finish and click OK

Creating SSL alias link to the Trust store

We will create Click Security -> SSL certificate and key management -> SSL configuration.

Click New

Fill up the following
- Name (I used LDAPSSLSettings)
- Trust store name (its LDAPTruststore)
- Key store name (its LDAPTruststore)
- Management Scope (Its Cell Level)

Click OK then Save then OK.

We are now ready to enable LDAP over SSL communication to LDAP server

Go back to the repository.

Click Security -> Global Security -> Configure (drop down bar should point to "Federated repositories")

When the page refreshed, Click "Manage repositories"

Fill up the following
- Port (change to 636)

Check the "Require SSL communication"

Choose the radio button "use specific SSL alias" and select LDAPSSLSettings from the drop down menu.

Click Apply when you are done. Then Click Save. Wait for sync to finish and click OK

Add the Base DN

Here, we need to configure from where in LDAP server we should make the queries.

Click Security -> Global Security -> Configure (drop down bar should point to "Federated repositories")

Click the "Add base entry to realm" button

Fill up the following
- Repository (Put in the name you used for "Repository identifier")
- OU (Put in the Base DN here)

Click Apply when you are done. Then Click Save. Wait for sync to finish and click OK 

A little housekeeping

Removethe base DN for the old LDAP server.

Click on the base DN -> remove
Then Click Save. Wait for sync to finish and click OK  

Click on Manage repositories
Check the old repository -> delete
Then Click Save. Wait for sync to finish and click OK  

Restart AppSvr, NodeMgr and Dmgr

To be safe, i would prefer to restart everything and make sure i still can log in as administrator and the application would have no problem working with the new LDAP server.

Check the Dmgr logs, AppSvr logs for signs of errors.

A small test

Click on "users and groups" -> manage users
Search for some valid users and verify that they come from the new LDAP server.
Get the software team to verify too in case the problem is subtle enough not to be caught in the application logs.

Allowing longer web session going through Apache to Websphere Application Server

Had a tough one last month when migrating the system to WAS. I'm still new to WAS, hit a few problems and take this chance to document down so that this form my reference and hopefully it help you too.

Users has been complaining that the web service keep getting time out, returning a 500 error.

What i found out was that i can actually, tune the "ServerIOTimeout" parameter in the WAS plugin for Apache beyond the default. i used 900, which is 15min in seconds.

In addition,a little performance fine tuning was done using "LoadBalanceWeight" to keep the application servers from being 'hit' random when they are just started up, especially when i have a cluster of them. As recommended by IBM specialist, i used some numbers, with one of the application server assigned to an odd number, different from the rest.

The idea is to make one of the application server the first one to serve, instead of randomising it.

      Server CloneID="179d3la" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="20" MaxConnections="-1" Name="Node1" ServerIOTimeout="900" WaitForContinue="false"

...
...
     Server CloneID="179d5sb" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="20" MaxConnections="-1" Name="Node2" ServerIOTimeout="900" WaitForContinue="false"

...
...
      Server CloneID="179d8gc" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="21" MaxConnections="-1" Name="Node3" ServerIOTimeout="900" WaitForContinue="false"

...
...



Do let me know if you have better ideas of solving it.

Recovering Websphere Application Server (WAS) after hitting JMS error

Recently, the following errors were logged in the SystemOut.log of WAS version 7. The whole cluster of WAS came to a halt with a flurry of angry calls asking why the web service is down.

- BMXAA1580E - A Java Message System (JMS) error occurred
- CWSIT0088E: There are currently no messaging engines in bus intjmsbus running.

Apparently, the messaging engine went down. All the application servers can communicate via JMS and sat there idling.

Used the following way to recover the system. even though i still don't the reason even after logging a PMR yet, here's how.

1) bring down the Application Server
2) bring down the node agent if you have.
3) to be safe, i bring down the Deployment manager too.
4) move the messagestore Log file so that WAS can recreate it upon start up.
5) remove the transaction/tranlog/log1 and log2 file so that WAS can recreate it upon start up.
6) start up WAS and node agent.

How to Configure WebSphere Application Server hung thread detector to automatically produce javacores or thread dump

When ever Websphere Application Server has a thread that runs for a long time (600s), by default it will report with a WSVR0605W message. This is a similar behvaiour we see in weblogic. WSVR0605W: Thread has been active for and may be hung. There are in total in the server that may be hung. Reading from the following source, it is possible to get Websphere Application Server to generate a javacore when a potentially hung thread is reported. On Solaris, we call this thread dump. Source : [http://www-01.ibm.com/support/docview.wss?uid=swg21448581] This core file can be helpful in troubleshooting server hangs and performance issues. If the jobs running in your system usually take a long time, you may want to tune the monitoring time to more than 600s else you might get false reports. The website described that property "com.ibm.websphere.threadmonitor.dump.java" should be enabled. h4. Steps to enable auto thread dump. Log in to administrative console, click Servers > Application Servers > server_name. Under Server Infrastructure, click Administration > Custom Properties. Click New. Add the following property: Name: com.ibm.websphere.threadmonitor.dump.java Value: true Click Apply. Click OK and save the configuration changes. Restart the Application Server for the changes to take effect. done. In case you want to manually trigger a thread dump, try kill -3 .

Restarting Application Server by Node Agent

Learnt that in websphere application server 7, by default, the node agent will not take any action when an application server fails. 

In order to get the node agent to monitor and automatically restart a failed application server instance, we must setup the monitoring policy for that application server.

Go to the deployment manager console, and do the following:

1 . –> Java and Process Management –> Monitoring Policy
2.  Check the “Automatic Restart” box
3.  In the “Node Restart State“, set the state to “STOPPED”

Whenever you have a failed or killed application servers, node agent will now auto-restart the application server. 

If the state is set to "RUNNING", not only will the node agent restart a failed or killed application server, it WILL ALSO auto start the application server upon a node agent restart. 

Resolving ADMR0104E for Application Server

This write up serve to record the resolution for the ADMR0104E error encountered by Websphere Application server during start up. The Application Server eventually is unable to start up.

From "SystemOut.log", we see that the system is unable to read some properties file.

[6/27/12 12:08:35:103 SGT] 00000000 FileDocument  E   ADMR0104E: The system is unable to read document cells/Cell01/nodes/Node01/node-metadata.properties: java.io.IOException: Permission denied
        at java.io.File.checkAndCreate(File.java:1715)
        at java.io.File.createTempFile(File.java:1803)
        at com.ibm.ws.management.repository.FileDocument.createTempFile(FileDocument.java:564)
        at com.ibm.ws.management.repository.FileDocument.read(FileDocument.java:500)
        at com.ibm.ws.management.repository.FileRepository.extractInternal(FileRepository.java:1134)

Some research and checks revealed that the permissions on the temp directory under the application server profile had been changed. The application server would then be no longer able to write to the temp directory for the node in the below directory.

# ls -ltr
total 0
drwxr-xr-x    3 root     system          256 Jun 27 11:54 download

The cause of this is the start up of the application server using root. That's the reason why the above temp directory is owned by root.

Potentially, you should check the ffdc directory as well.

# ls -l //AppSrv01/logs/ | grep ffdc
drwxr-xr-x    2 appusr   appgrp        49152 Jun 27 13:50 ffdc

Research from the internet, the directory owner and the process execution user should be in the same group and be at least of permission 774. TO be fail safe, change the ownership/group as required under //profiles/ and //profiles/.

Once the ownership is reverted back to "appusr", we should see the result as below.

# chown -R appusr:appgrp  download

# ls -ltr
total 0
drwxr-xr-x    3 appusr   appgrp          256 Jun 27 11:54 download

The Application server is able to start up now.

[6/27/12 12:21:56:692 SGT] 00000000 AdminTool     A   ADMU3000I: Server appsrv open for e-business; process id is 4128910{code}

We can also check the process execution of the application server in order to compare to the file system permissions, one can do the following:

1. Open the admin console
2. Open Servers –> Application Servers –>
3. Open Java Process Management –> Server Execution
4. Look for username and group of executing user

Thats all.

Recover websphere password

Google online and found this interesting step to recover websphere 7.1 password.

For encrypting the password we have,

//java/bin/java -Djava.ext.dirs=//deploytool/itp/plugins/com.ibm.websphere.v7_7.0.1.v20100710_0411/wasJars/ -cp securityimpl.jar:iwsorb.jar  com.ibm.ws.security.util.PasswordEncoder secret


The output is

decoded password == "secret", encoded password == "{xor}LDo8LTor"

Hence, you can use the same method to decrypt the encrypted password.

//java/bin/java -Djava.ext.dirs=//deploytool/itp/plugins/com.ibm.websphere.v7_7.0.1.v20100710_0411/wasJars/ -cp securityimpl.jar:iwsorb.jar  com.ibm.ws.security.util.PasswordDecoder {xor}LDo8LTor






The output is

encoded password == "{xor}LDo8LTor", decoded password == "secret"

If you want to know, you can update the password for the deployment manager and nodes without knowing the password. Check out /.../config/cells//security.xml. :)

Encrypting the ID and Password for Websphere Application Server

By default, you need to supply the ID and password when starting up/shutting down the deployment manager, node or application server. Example of the command as below

Deployment Manager
//bin/startManager.sh -username XXX -password XXX

Node
//bin/startNode.sh -username XXX -password XXX

Application Server
//bin/startServer.sh -username XXX -password XXX

The steps to encrypt the password and ID is as follows.

Insert the ID and password in clear text into the SOAP properties file at //properties/soap.client.props.

# grep SOAP.login soap.client.props | grep -v "#"
com.ibm.SOAP.loginUserid=wasadm
com.ibm.SOAP.loginPassword=wasadm
com.ibm.SOAP.loginSource=prompt

We use the IBM provided script to encode the password.

//bin/PropFilePasswordEncoder.sh //profiles/default/properties/soap.client.props com.ibm.SOAP.loginPassword -Backup

Taking a look at the same property file again, the password is now encrypted.

# grep SOAP.login soap.client.props | grep -v "#"
com.ibm.SOAP.loginUserid=wasadm
com.ibm.SOAP.loginPassword={xor}Es4zPjwS
com.ibm.SOAP.loginSource=prompt

Now, we can start up websphere and shut down without using the password.

su wasadm -c "//bin/startManager.sh"
su wasadm -c "//bin/startNode.sh"
su wasadm -c "//bin/startServer.sh "


su wasadm -c "//bin/stopManager.sh"
su wasadm -c "//bin/stopNode.sh"
su wasadm -c "//bin/stopServer.sh "
end.

======================

Some trival.
How come IBM prefers to use XOR instead of some stronger algorithm like how weblogic uses 3DES? XOR is good enough only to prevent casual snooping. 

Someone demonstrated that with a online decoder

http://www.poweredbywebsphere.com/decoder.html


Change WebSphere Ports without Reinstalling

Scenario: you have WebSphere Application Server 7.1 installed as ND. If the cells are using default ports on the same host and you want to access the different cells concurrently, you may want to change the ports on one of the cell.

1.  Go to the master config repository for the server ports (Dmgr profiles directory)

2.  Backup the current serverindex.xml

3.  Edit each of the ports in this file. (Dmgr will use the new ports)

4.  Repeat this process for all nodes in the master repository (Node profiles directories)

5.  For all cells,

6.  Backup virtualhosts.xml

7.  edit all the ports.  (nodes will use this ports to connect with Dmgr)

7.  Start the dmgr (startServer.sh)

8.  For each node,  executue a syncNode so that nodes get their new port assignments from the master repository

//bin/syncNode.sh

Use the new SOAP ports used in step #3.

9.  Start up each node

10.  Start up each application server.

Confirm which new ports you want to use before you start.
To make it easier to remember, maybe instead of the usual 80, can try prepending like 9080, 19080, etc.

Done.

Installing IBM Websphere Applicationi Server (WAS) 6 Plugin on Apache 2.2.22

Learnt that plugin setup for WAS is different from the setting up in BEA weblogic environment. WAS plugin require an installation while we just need to put in the plugin for BEA weblogic. Here's the steps to get WAS running. Installing the Plugin in the web server. Download the installation file from IBM. i.e. "Tiv_Middl_Inst_750_1of3_Linux_x86-64.tar" since i'm using linux for my web server. Transfer the tar file into the web server and unpack. Was only able to install using the supplied GUI so be prepared to export display. # cd linux64/WS-WAS_ND_7.0_Supplemental # gunzip C1G36ML.tar.gz # tar xfp C1G36ML.tar # cd plugin # export BROWSER=/usr/bin/mozilla # export DISPLAY=10.10.10.16:0.0 # ./launchpad.sh need to have your xwin or xmanager ready. My template cfg.xml file is now at "/opt/IBM/WebSphere/Plugins/config/myapp/plugin-cfg.xml" Copy the configuring script to the WAS server. There is a customised configuration script that you need to run at the WAS server to generate the real cfg.xml file. It is usually in the plugin bin, i.e. "/opt/IBM/WebSphere/Plugins/bin/configuremyappserver.sh" copied this to the root directory of WAS, i.e. /opt/was/IBM/WebSphere/AppServer/bin and run it. Next, generating the plugin xml file in WAS server. root@myappserver:/opt/was/IBM/WebSphere/AppServer/bin> ./configuremyappserver.sh Realm/Cell Name: <default> Username: wasuser Password: WASX7209I: Connected to process "dmgr" on node myappserverCellManager01 using SOAP connector; The type of process is: DeploymentManager WASX7303I: The following options are passed to the scripting environment and are available as arguments that are stored in the argv variable: "[myapp, APACHE, /usr/local/apache2222, /usr/local/apache2222/conf/httpd.conf, 80, MAP_ALL, /opt/IBM/WebSphere/Plugins, unmanaged, mywebserver-node, mywebserver, linux]" Input parameters: Web server name - myappserver Web server type - APACHE Web server install location - /usr/local/apache2222 Web server config location - /usr/local/apache2222/conf/httpd.conf Web server port - 80 Map Applications - MAP_ALL Plugin install location - /opt/IBM/WebSphere/Plugins Web server node type - unmanaged Web server node name - mywebserver-node Web server host name - mywebserver Web server operating system - linux Creating the unmanaged node mywebserver-node . Unmanged node mywebserver-node is created. Creating the web server definition for myapp. Web server definition for myapp is created. Start computing the plugin properties ID. Plugin properties ID is computed. Start updating the plugin install location. Plugin install location is updated. Start updating the plugin log file location. Plugin log file location is updated. Start updating the RemoteConfigFilename location. Plugin remote config file location is updated. Start updating the RemoteKeyRingFileName location. Plugin remote keyring file location is updated. Start saving the configuration. Configuration save is complete. Computed the list of installed applications. Processing the application myapp. Get the current target mapping for the application myapp. Computed the current target mapping for the application myapp. Start updating the target mappings for the application myapp. Target mapping is updated for the application myapp. Start saving the configuration. Configuration save is complete. Transfer the plugin-cfg.xml file to the web server. scp /opt/was/IBM/WebSphere/AppServer/profiles/Dmgr01/config/cells/myappserverCell01/nodes/mywebserver-node/servers/myapp/plugin-cfg.xml user@mywebserver:/tmp/ The generated file is generally in "profiles_install_root/config/cells/cell_name/nodes/node_name/servers/web_server_name" directory The place to put the plugin-cfg.xml file is generally in "plugins_install_root/config/web_server_name" directory Start up Apache and test. you should be good to go.

Access Controls and Virtual Hosts for WebSphere Application Server

WebSphere Virtual Host

WebSphere applications are assigned to virtual hosts during installation and configuration process.

The virtual hosts are bound to one or more aliases (host names and ports), allowing a single application server to respond to multiple inbound request formats. 

The virtual host aliases that are configured within WebSphere define the pattern match possibilities for which WebSphere application server will respond with data. 

For example, if we have 7 aliases defined for a given virtual host 'foo-app' on application server, ‘fooserver′.

Virtual Host:  'foo-app'
Hostname         Port
thisport         8080
thatport         8081
secureport       8443
secureport2      9443
otherport        10001
otherport2       10002
otherport3       10003

Hence, for any inbound request for 'foo-app', if the URL does not match against the list above, the request is denied, else the client gets the access. The Application Server is the gatekeeper here.


Usually all default application server ports are removed from the ‘default_host’ virtual host once you do some 'hardening', forcing all inbound requests to be channeled through a proxy mechanism.

Hence, you will not be able to access the application 'foo-app' directly, if the default listening port is not in the list above. e.g. 'foo-app' listens on 1234. To get to 'foo-app', you need to get through the aliases above to get to 'foo-app'.

I use Apache in the web layer above Websphere Application Server. Need to install the Websphere Proxy-Plugin and generate the xml file.

When client access the URL, the GET request reaches Apache, which then checks the request against the Websphere Proxy Plugin for the host or virtual host corresponding to the requested URL. 

The Websphere Proxy Plugin has a list of valid WebSphere virtual hosts and virtual-host associated resources. Here, if matched, those requests are forwarded to the appropriate WebSphere application server. If not matched, 404 is generated.