Recently, a request was made to modify the permission on a queue on MS MQ. Has been administering IBM Websphere MQ so this one is new to me.
Failed attempt
Open up Computer Management console, navigated to the queues under services and application, right click to the security properties and started tweaking.
Ops. System says "Access to Message Queuing System is denied" that essential mean i do not have the rights. Probably administrator account was not used to create the queues? So how?
Found the solution!
Turn out that i just need to click "Advanced" from the security tab, then go to the owner tab, take over control using administrator account, apply and ok. Going back to assign permission to the queue now is OK.
Here's a guide on how to setup monitoring for SSL certificates
It is important to ensure that the SSL certificates used in services that are fronting users or for secure communication are valid otherwise, we risk service outage because of expired certificates.
The SSL monitoring script
Using the monitoring script "SSL Certificate Check" written by Matty, we can use it to monitor the SSL certificates either by verifying the certificate itself or querying the status through the application services.
I am using the script v3.21 dated Oct 2010 in the example.
1) Download at SSL Certificate Check Script
2) Deploy it to a suitable location. Give it execute permission at least 0700.
I have configured the script to the following
# Who to page when an expired certificate is detected (cmdline: -e)
ADMIN="admin@myserver.com"
# Number of days in the warning threshhold (cmdline: -x)
WARNDAYS=100
# If QUIET is set to TRUE, don't print anything on the console (cmdline: -q)
QUIET="FALSE"
# Don't send E-mail by default (cmdline: -a)
ALARM="TRUE"
# Don't run as a Nagios plugin by default (cmdline: -n)
NAGIOS="FALSE"
where the script will notify via email (default) when the certificate has less than 100 days of validity. It will print out to console. If you don't need it, change QUIET to "TRUE".
If you require to override the default settings in the script, you can use the following switches
#./sslcertcheck
Usage: ./sslcertcheck [ -e email address ] [ -x days ] [-q] [-a] [-b] [-h] [-i] [-n] [-v]
{ [ -s common_name ] && [ -p port] } || { [ -f cert_file ] } || { [ -c certificate file ] }
-a : Send a warning message through E-mail
-b : Will not print header
-c cert file : Print the expiration date for the PEM or PKCS12 formatted certificate in cert file
-e E-mail address : E-mail address to send expiration notices
-f cert file : File with a list of FQDNs and ports
-h : Print this screen
-i : Print the issuer of the certificate
-k password : PKCS12 file password
-n : Run as a Nagios plugin
-p port : Port to connect to (interactive mode)
-s commmon name : Server to connect to (interactive mode)
-q : Don't print anything on the console
-v : Only print validation data
-x days : Certificate expiration interval (eg. if cert_date < days)
Requirements.
mktemp package needs to be available in the server.
Usage
1) Running the script against the certificate file.
$ sslcertcheck -c /etc/httpd/conf/ssl.crt/abc.pem
Host Status Expires Days Left
----------------------------------------------- ------------ ------------ ----------
FILE:/etc/httpd/conf/ssl.crt/abc.pem Valid Jan 2 2010 807
sslcertcheck will print the file or hostname in the first column, a value to indicate if the certifciate is valid in the second column, the date the certificate will expire in the third column, and the number of days remaining until the certificate expires in the fourth column.
2) If you do not have local access to the certificate files, you can use sslcertcheck's network connectivity option to extract the certificate expiration date from a live server. To check when the certificate used by the web server will expire, the server name or IP address and a port number can be passed to sslcertcheck's "-s" (server name) and "-p" (tcp port) options:
#./sslcertcheck -s 172.21.41.136 -p 443
Host Status Expires Days
----------------------------------------------- ------------ ------------ ----
172.2.1.1:443 Valid Jul 24 2011 128
3) You may want to manage dozens of SSL-enabled servers, you can place the server names and port numbers in a file, and run sslcertcheck against that file:
You used the SupportPac MS03 to save your queue manager object definitions to a file that you named TEST.TST. You want to use the saveqmgr output file (TEST.TST) to create the same queue manager object definitions on a new server machine.
Whats the process to redefine your objects?
First, copy the saveqmgr output file (TEST.TST) onto the new server machine.
Create the queue manager
crtmqm TEST
Start the queue manager
strmqm TEST
Define your objects
runmqsc TEST < TEST.TST
where TEST is the name of the new QMGR and TEST.TST is the name of the queue manager object definitions file.
Reference: MQ Reference
On MQ v6, MCAUSER in SYSTEM.DEF.SVRCONN is blank by default, hence this would not allow anyone to connect to it.
This is an example of the error from the application when it couldn't connect even though LDAP setup is correct.
Basic network connectivity test by telnet is OK.
2008-04-06 11:11:10,452 [main] ERROR - Hit exception at init JMSSubscriber, System reconnecting after [10] seconds ....
com.psa.infra.messaging.PMException
at com.psa.infra.messaging.PMSessionFactory.createConnection(PMSessionFactory.java:260)
at com.psa.infra.messaging.PMSessionFactory.getConnection(PMSessionFactory.java:168)
at com.psa.infra.messaging.PMSession.getPMJMSSession(PMSession.java:216)
at com.psa.infra.messaging.PMSession.createHandler(PMSession.java:101)
at com.psa.infra.messaging.PMSession.createHandler(PMSession.java:58)
at com.psa.common.utility.jms.subscriber.JMSSubscriber.init(JMSSubscriber.java:95)
at com.psa.common.utility.core.mesg.MessageTransfer.process(MessageTransfer.java:94)
at com.psa.common.utility.core.mesg.MessageTransfer.main(MessageTransfer.java:160)
Caused by: javax.jms.JMSException: MQJMS2005: failed to create MQQueueManager for 'png1comm2_ica:PSA.QUEUE.MANAGER'
at com.ibm.mq.jms.services.ConfigEnvironment.newException(ConfigEnvironment.java:434)
at com.ibm.mq.jms.MQQueueConnection.createQM(MQQueueConnection.java:478)
at com.ibm.mq.jms.MQQueueConnection.(MQQueueConnection.java:182)
at com.ibm.mq.jms.MQQueueConnectionFactory.createQueueConnection(MQQueueConnectionFactory.java:166)
at com.psa.infra.messaging.PMSessionFactory.createConnection(PMSessionFactory.java:234)
... 7 more
2008-04-06 11:11:10,453 [main] INFO - -- Close JMS subscriber handler
To allow applications to connect, enter the following code after getting in 'runmqsc'.
> alter CHANNEL ('SYSTEM.DEF.SVRCONN') CHLTYPE(SVRCONN) MCAUSER('mqm')
^^ dn --> distinguished name. X.500 distinguished name enclosed in double quotes.
Note that only the CN attribute is required.
You can supply multiple OU attributes.
*** VERY IMPT
For MQ server, please use ibmwebspheremq'queue manager name' without the quotes. Do not try to "learn" from the hard way. All small letters and do follow exactly.
For MQ clients, please use ibmwebspheremq'userid' without the quotes.
1. RUNMQLSR - MQ TCP listener (multi-threaded) 2. AMQCLMAA - MQ TCP listener (single-threaded) 3. AMQRMPPA - Channel process pooling job 4. RUNMQCHI - MQ channel initiator 5. AMQCRSTA - MQ receiving MCA jobs 6. RUNMQCHL - MQ sending MCA jobs 7. AMQCRS6B - LU62 receiver channel 8. AMQPCSEA - MQ command server 9. RUNMQTRM - Application trigger monitor 10. RUNMQDLQ - Dead letter queue handler 11. AMQFCXBA - MQ Broker Worker Job 12. RUNMQBRK - MQ Broker Control Job 13. AMQZMUC0 ('0' is a zero) - MQ Utility Manager 14. AMQZMUR0 ('0' is a zero) - MQ Utility Manager 15. AMQZMGR0 ('0' is a zero) - MQ Process Controller 16. AMQRRMFA - MQ cluster repository manager 17. AMQZDMAA - MQ deferred message manager 18. AMQALMPX - MQ Log Manager 19. AMQZFUMA - MQ Object Authority Manager 20. AMQZLAS0 ('0' is a zero) - MQ LQM agents 21. AMQZLAA0 ('0' is a zero) - MQ LQM agents 22. AMQZXMA0 ('0' is a zero) - MQ Execution Controller
Problem
Q1. What is process amqzlaa0?
Q2. Why does it take so much cpu time? Solution A1. The 'amqzlaa0' process is the WebSphere® MQ agent.
A2. Agent processes act as buffers between the QMGR and applications. This is where most of the WebSphere® MQ work gets done.
Additional Information
The 'amqzlaa0' process is the agent process spawned by the execution controller ('amqzxma0') on behalf of the application. These agent processes are the buffer between the QMGR and applications, and it is where most of the WebSphere® MQ work gets done. Expected to see heavy CPU and memory utilization for amqzlaa0 on a busy box.
Each agent thread handles a request for one connection and these threads can be used to service multiple applications. A new agent is spawned when no threads are available for a new connection request.
There are times where we need to quickly display all the queue depth, trying to look for queues that have more than X messages. Note that this is applicable for MQ 6 and newer.
We used this command:
dis q(*) curdepth
Result is a very long list of output with much of it unwanted.
In MQ6, the 'WHERE' parameter in introduced. The same example, say X is 1.
dis q(*) where (curdepth gt 1)
1 : dis q(*) where (curdepth gt 1)
AMQ8409: Display Queue details.
QUEUE(SYSTEM.AUTH.DATA.QUEUE) TYPE(QLOCAL)
CURDEPTH(70)
AMQ8409: Display Queue details.
QUEUE(SYSTEM.CHANNEL.SYNCQ) TYPE(QLOCAL)
CURDEPTH(4)
This time, the result returns only those of more than 1 message in the queue.
If you wish to find out which channels are having problems, can try something like this
dis chs(*) where (status ne running)
Say if we want to trace messages mysteriously disappearing from a particular queue but we don't know which application is reading from it, we can try this..
Here, we listed all connections by the current Queue Manager where application type is USER. This excludes all system processes and objects. Also only display the PID, MQ connection option, name of binary, user ID and perhaps the channel that the application is connecting over.
Operators allowed for boolean comparison is as follows. Note that only simple boolean allowed. No compound usage yet.
operator := [ LT | GT | EQ | NE | LE | GE | CT | EX | LK | NL | CTG | EXG ]
Found this quite a while back. Try to share this during my free time.
This is a good tool that will help to tune the system for MQ. Has tried it for MQ 5.3, 6 and even 7 plus Solaris 8, 9 and 10. There is a attachment which you need to download and try.
This error would like occur when the local and remote queue managers do not agree on the next message sequence number. For example, when channels goes down or restarted due to network or other issues.
You will see the following error in the Queue manager error log.
AMQ9526 Message sequence number error for channel
So, how to correct this problem??
1) Do a display channel status DIS CHS(XXXXXXX) for the sender channel
2. If the channel is still running, STOP CHANNEL(XXXXXXX) MODE(FORCE).
This will stop transmission of the current batch and will likely result in message being in-doubt.
From my experience so far, i would only issue
stop chl(XXX)
I have not encountered the need to use FORCE yet. Maybe use FORCE when you cant stop the channel the first time round.
3. Lets clear up any in-doubt messages
RESOLVE CHANNEL (XXXXXXX) ACTION(BACKOUT).
This requests channels to commit or back out in-doubt messages. This command can be used only for sender (SDR), server (SVR), and cluster-sender (CLUSSDR) channels (including those that have been defined automatically). The messages are backed out, that is, they are restored to the transmission queue.
So far, i have no need to do step 3.
** IMPORTANT ** From researching the internet, in 99.9% of cases you never use the facility and it's actually dangerous to do so. In-doubt channel problems are usually resolved automatically.
Under certain unusual conditions, the channel can't resolve the situation itself. For this situtation the resolve channel command has been provided so the MQ administrator, using facilities unavailable to the MCAs, can determine what to do to resolve the channel and instruct the system accordingly.
** IMPORTANT **
Do not use RESOLVE CHANNEL (XXXXXXX) ACTION(COMMIT) without considering that the messages are committed, that is, they are deleted from the transmission queue
4. Reset the sequence number
RESET CHANNEL(XXXXXXXX) SEQNUM(1).
This resets the message sequence number for a MQ channel with an optionally specified sequence number to be used the next time the channel is started.
As the sequence number is set by the sender channel so there is no need to reset the receiver.
The sequence number of the last committed message or LUWID is recorded at the receiving end. This is used not only at the sending end but also during re-sequencing, on startup and restarts to ensure that both ends of the link agree on which message have been sent.
5. ALmost there. Start up the channel.
START CHANNEL(XXXXXXX)
6. Done.
Additional information:
Each persistent message that is sent using message sequencing is tagged with an individual sequence number. These are incremented by 1 for each message sent, first on the sending end then on the receiving end. These numbers must match for sequential delivery to work correctly. So, if a message is sent with sequence number of 412 then the receiver side needs to be looking for message number 412.
^^ dn --> distinguished name. X.500 distinguished name enclosed in double quotes. Note that only the CN attribute is required. You can supply multiple OU attributes.
*** For MQ server, please use ibmwebspheremq'queue manager name' without the quotes. Do not try to "learn" from the hard way. All small letters and follow exactly. For MQ clients, please use ibmwebspheremq'userid' without the quotes.
3) Sent to CA to sign the certificate you have generated.
AMQ5882: WebSphere MQ Publish/Subscribe broker has written a message to the dead-letter queue.
EXPLANATION: The broker has written a message to the dead-letter queue (SYSTEM.DEAD.LETTER.QUEUE ) for reason '2053:MQRC_Q_FULL'. Note. To save log space, after the first occurrence of this message for stream SYSTEM.BROKER.DEFAULT.STREAM ), it will only be written periodically. ACTION: If the message was not deliberately written to the dead-letter queue, for example by a message broker exit, determine why the message was written to the dead-letter queue, and resolve the problem that is preventing the message from being sent to its destination.
Explanation - The broker might be putting the subscriber's publications to the dead-letter queue. (Proven because the DLQ is increasing in number)
There might be a problem with the subscriber's queue. For example, it might be put-inhibited or the publications might be too large for the queue. In this case the broker, by default, puts these messages to the dead-letter queue (DLQ). Check the DLQ at the subscriber's broker. The broker also issues message AMQ5882 if it has to put a message to the DLQ.
Hey, Notice that CURDEPTH = MAXDEPTH!! we have queue that is full!! From here, we should check if the receiving MQ is probably down or network issue.. it may be because the backend processes died or something.
Be sure to check the impact before proceeding to restart the receiving MQ channel or the backend processes otherwise you may cause more damages to the whole business flow especially if the messages are critical.
09/09/07 04:07:08 - Process(18642.58) User(mqm) Program(amqrmppa) AMQ9526: Message sequence number error for channel 'xxxxx'.
EXPLANATION: The local and remote queue managers do not agree on the next message sequence number. A message with sequence number 1484994 has been sent when sequence number 1484986 was expected. ACTION: Determine the cause of the inconsistency. It could be that the synchronization information has become damaged, or has been backed out to a previous version. If the situation cannot be resolved, the sequence number can be manually reset at the sending end of the channel using the RESET CHANNEL command. ----- amqrmtra.c : 3812 ------------------------------------------------------- 09/09/07 04:07:08 - Process(18642.58) User(mqm) Program(amqrmppa) AMQ9999: Channel program ended abnormally.
EXPLANATION: Channel program 'xxxxx' ended abnormally. ACTION: Look at previous error messages for channel program 'xxxxxx' in the error files to determine the cause of the failure.
How do we resolve this? Here's how.
bash-2.05$ runmqsc QMGR 5724-H72 (C) Copyright IBM Corp. 1994, 2005. ALL RIGHTS RESERVED. Starting MQSC for queue manager QMGR.
