Troubleshooting AMQ5882 error

This error points towards a overflow of messages from the queues into the system dead letter queue.. Here's the symptom.

> tail -50 AMQERR01.LOG 
----- amqccita.c : 3263 --------------------------------------------
08/02/07 21:38:48 - Process(20733.10) User(mqm) Program(amqfcxba)

AMQ5882: WebSphere MQ Publish/Subscribe broker has written a message to the 
dead-letter queue. 

EXPLANATION: 
The broker has written a message to the dead-letter queue 
(SYSTEM.DEAD.LETTER.QUEUE                        ) for reason 
'2053:MQRC_Q_FULL'. Note. To save log space, after the first occurrence of this 
message for stream SYSTEM.BROKER.DEFAULT.STREAM                    ), it will only be written periodically. 
ACTION: 
If the message was not deliberately written to the dead-letter queue, 
for example by a message broker exit, determine why the message was written to the dead-letter queue, 
and resolve the problem that is preventing the message from being sent to its destination.



Explanation - 
The broker might be putting the subscriber's publications to the dead-letter queue. 
(Proven because the DLQ is increasing in number)

There might be a problem with the subscriber's queue. 
For example, it might be put-inhibited or the publications might be too large for the queue. 
In this case the broker, by default, puts these messages to the dead-letter queue (DLQ). 
Check the DLQ at the subscriber's broker. The broker also issues message AMQ5882 if 
it has to put a message to the DLQ. 

We check further..

     1 : dis ql(*) curdepth 
...
...
AMQ8409: Display Queue details. 
   QUEUE(yyy) 
   TYPE(QLOCAL)                            CURDEPTH(0) 
AMQ8409: Display Queue details. 
   QUEUE(xxx) 
   TYPE(QLOCAL)                            CURDEPTH(5000) 
AMQ8409: Display Queue details. 
   QUEUE(zzz) 
   TYPE(QLOCAL)                            CURDEPTH(0) 
… … … … 
… … … … 

Notice that one of the queues has a very high CURDEPTH. So we investigate more.

dis ql('xxx') 
     1 : dis ql('SYSTEM.xxxx') 
AMQ8409: Display Queue details. 
   QUEUE(xxx) 
   TYPE(QLOCAL)                            ACCTQ(QMGR) 
   ALTDATE(2007-04-09)                     ALTTIME(09.44.28) 
   BOQNAME( )                              BOTHRESH(0) 
   CLUSNL( )                               CLUSTER( ) 
   CLWLPRTY(0)                             CLWLRANK(0) 
   CLWLUSEQ(LOCAL)                         CRDATE(2007-04-09) 
   CRTIME(09.44.28)                        CURDEPTH(5000) 
   DEFBIND(OPEN)                           DEFPRTY(0) 
   DEFPSIST(NO)                            DEFSOPT(SHARED) 
   DEFTYPE(PERMDYN) 
   DESCR(Websphere MQ - JMS Classes - Model queue) 
   DISTL(NO)                               GET(ENABLED) 
   HARDENBO                                INITQ( ) 
   IPPROCS(0)                              MAXDEPTH(5000) 
   MAXMSGL(15728640)                       MONQ(QMGR) 
   MSGDLVSQ(PRIORITY)                      NOTRIGGER 
   NPMCLASS(NORMAL)                        OPPROCS(1) 
   PROCESS( )                              PUT(ENABLED) 
   QDEPTHHI(80)                            QDEPTHLO(20) 
   QDPHIEV(DISABLED)                       QDPLOEV(DISABLED) 
   QDPMAXEV(ENABLED)                       QSVCIEV(NONE) 
   QSVCINT(999999999)                      RETINTVL(999999999) 
   SCOPE(QMGR)                             SHARE 
   STATQ(QMGR)                             TRIGDATA( ) 
   TRIGDPTH(1)                             TRIGMPRI(0) 
   TRIGTYPE(FIRST)                         USAGE(NORMAL) 

Hey, Notice that CURDEPTH = MAXDEPTH!! we have queue that is full!! From here, we should check if the receiving MQ is probably down or network issue.. it may be because the backend processes died or something.

Be sure to check the impact before proceeding to restart the receiving MQ channel or the backend processes otherwise you may cause more damages to the whole business flow especially if the messages are critical.