PK49155: LACK OF RECOVERABILITY IN CICS TS 3.2 SOAP / WEBSERVICES REQUEST

Obtain the fix for this APAR.

APAR status

• Closed as new function.

Error description

• The current support for WMQ transport in CICS Web Services does
  not provide provision for the recovery of the request message
  data in certain failure circumstances. Once the message is
  removed from the request queue it is not possible to recover
  this data should any form of failure occur, either program or
  system.
  
  ADDTIONAL SYMPTOMS:
  You are using SOAP to run programs in CICS.  After coming in
  through DFHIPIPM and going through INVOKE_STUB processing, you
  enter the INVOKE_PROGRAM function.  Your application program
  gets invoked, that in turn LINKs to a second program.  This
  program makes an DSNSQL (SQL) call to DB2 and performs an update
  or add of a record.  On return from DB2 your application program
  abends with a program check.  This is percolated up to the
  calling program and to the INVOKE_PROGRAM function where a
  non-exception trace entry is made indicating that an ASRA
  occurred.  If PI tracing is turned on you can see a
  FUNCTION(SOAPFAULT_CREATE) trace entry, but when the INVOKE_STUB
  EXIT is traced, an OK response is given.  This causes the work
  on DB2 to be committed FORWARD, instead of backed out.
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED: All CICS users                               *
  ****************************************************************
  * PROBLEM DESCRIPTION: In the current design for Web Services  *
  *                      a system outage or failure, when using  *
  *                      the MQ transport, may lead to the loss  *
  *                      of Soap requests.                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  The MQ transport for Web Services utilizes a request queue onto
  which outbound Soap messages are placed. Transaction CPIL
  removes these messages and places the request data into a series
  of standard channel containers. The Soap pipeline is then
  started and a SYNCPOINT performed. At this point the MQGET is
  committed and the assumption is that any MQ persistent messages
  will be delivered. However in the current implementation, should
  a systems failure occur at this point it is possible to lose a
  SOAP request as the standard channels and their containers are
  not recoverable.
  
  This APAR also enhances task recovery in a Context-switching
  environment. A context switch occurs within Web Services when an
  alteration to the transaction ID and or the Userid has been
  effected by a pipeline handler.
  Additional Keywords: CPIQ
  

Problem conclusion

Temporary fix

• FIX AVAILABLE BY PTF ONLY
  

Comments

•  In the "CICS Web Services Guide" manual, SC34-6838-00, the
   section in Chapter 6, "Configuring your CICS system for
   Web Services." is amended as follows;
  
   MQ Persistent Message support
  
   CICS Web Services provides support for MQ persistent messages.
  
   CICS Business Transaction Services (BTS) is used to ensure that
   MQ persistent messages sent to an MQ Transport Web Service on
   CICS TS 3.2 are recovered in the event of a CICS system
   failure. For one-way messages, if the Web Service abends or
   backs out, sufficient information is retained to allow the
   installation to retry the failing request, or to report the
   failure appropriately.
  
   For this to work correctly, BTS must be configured as described
   in "Configuring BTS". If an error occurs when creating the
   required BTS process, message DFHPI0118 is issued, and CICS
   continues without BTS, using the existing channel-based
   container mechanism.
  
   The retention of the MQ persistent message information within
   the BTS process is affected by the use of the SYNCPOINT or
   SYNCPOINT ROLLBACK commands within application code. The
   encompassing BTS process is classed as having completed
   successfully as soon as either of these commands has been issue
   successfully. All the retained data, as extracted from the
   incoming MQ persistent message, is then discarded. This
   completion of the BTS process does not affect the running
   application in any way.
  
  
   Configuring BTS
  
   You must configure BTS as follows:
  
     1. Use IDCAMS to define the Local Request Queue (LRQ) and
        Repository File to MVS.
  
     2. Use RDO to define the LRQ and Repository File to CICS. A
        suitable value for STRINGS must be specified for the file
        definition. The default value of 1 is unlikely to be
        sufficient, and you are recommended to use 10.
  
     3. Use RDO to define a PROCESSTYPE with name DFHMQSOA to
        CICS. Specify the Repository File name as the FILE
        parameter on this definition.
  
   See the "CICS TS 3.2 Business Transaction Services" manual for
   full details of BTS configuration.
  
  
   Processing Overview
  
   When a WebService request is received in an MQ persistent
   message, a unique BTS process is created with PROCESSTYPE
   DFHMQSOA, and data relating to the inbound request is captured
   in BTS containers associated with the process. The process is
   then scheduled to run asynchronously. If the Web Service
   completes normally and commits, the BTS process is deleted.
    **Note that this includes the case when a SOAP fault is
      generated without abending.
  
   In the event of a CICS failure before the Web Service starts or
   before the Web Service completes processing, BTS recovery
   ensures that the process is rescheduled when CICS is restarted.
  
   If the Web Service abends and backs out, the BTS process is
   marked complete with status ABENDED. For two way messages, a
   SOAP fault is returned to the requester, which can then take
   appropriate action. The BTS process is cancelled, and CICS
   retains no information about the failed request. CICS issues
   message DFHBA0104 on transient data queue CSBA, and message
   DFHPI0117 on transient data queue CPIO.
  
   For one way messages, because there is no way to return
   information about the failure to the requester, the BTS process
   is retained in COMPLETE ABENDED state. CICS issues message
   DFHBA0104 on transient data queue CSBA, and DFHPI0116 on
   transient data queue CPIO.
  
  
  Recovery Techniques.
  
  
  You can use the CBAM transaction to display any COMPLETE ABENDED
  processes, or you can supply a recovery transaction to check for
  COMPLETE ABENDED processes of the DFHMQSOA processtype and take
  appropriate action. For example, it could:
  
     - RESET the process and issue RUN ASYNC to retry the failing
       Web Service. If so, it could keep a retry count in another
       container on the process, to avoid repeated failure.
  
     - use information in the process's containers to report on
       the problem:
  
       -The DFHMQORIGINALMSG container contains the message
         received from MQ (which may contain RFH2 headers).
  
       - The DFHMQMSG container contains the MQ message with any
         RFH2 headers removed.
  
       - The DFHMQDLQ container contains the name of the MQ Dead
         Letter Queue associated with the original message.
  
       - DFHMQCONT container contains the MQ MQMD control block
         relating to the MQ GET for the original message.
  
  
  
   The following messages are added to the CICS Messages and Codes
   manual, GC34-6827-00.
  
   DFHPIxxxx messages
  
   DFHPI0116 date time applid A one-way request has been received
     as an MQ persistent message, but the provider pipeline has
     abended or backed out changes to recoverable resources. The
     BTS process "processname" of processtype "processtype" has
     completed with status ABENDED and this process can be retried
     or used to provide information for reporting the "failure"
  
   Explanation: A one way SOAP message has been received as a
     persistent MQ message, but the provider pipeline has abended
     or forced a backout of changes to recoverable resources. As
     there is no reply to queue, there is no way of informing the
     requester of the failure. A BTS process processname in
     completion state ABENDED contains information about the
     original MQ message, and this process and its containers can
     be used to retry the failing provider pipeline, or to report
     the failure appropriately.
  
   System action: CICS has kept the information necessary to retry
     the failing provider pipeline in the named process. CICS
     takes no further action for the process.
  
   User response: A user-written transaction can ACQUIRE the
     PROCESS named in the message and take appropriate
     installation defined action. This could include one or more
     of the following:
  
       - retry the failing process by issuing RESET ACQPROCESS
         followed by RUN ACPROCESS ASYNC. Of course, the process
         could fail again in exactly the same way, and so an
         installation might wish to limit the number of retries.
         This can be done by using a container to contain a retry
         count, which would be incremented on each attempt, and
         taking a different action when the count exceeds an
         installation defined threshold.
  
       - recover information about the original MQ message
         received and record this on an installation defined log
         file. There are a number of containers associated with
         the PROCESS, DFHMQORIGINALMSG contains the message as
         received from MQ, DFHMQMSG contains the inbound message
         with RFH2 header removed, that is, the SOAP message.
         DFHMQCONT contains the MQMD control block with data
         relating to the MQ GET that was issued by CICS pipeline
         processing MQ transport. After recovering any necessary
         information for reporting or logging the failure, the
         PROCESS can be cancelled (CANCEL ACQPROCESS).
  
   Destination: CPIO
   Module: DFHPIDSQ
   XMEOUT Parameters: date,  time, applid
  
  
   DFHPI0117 date time applid BTS process "processname" of
     processtype "processtype", which has completed with status
     ABENDED, has been cancelled. A provider pipeline started with
     a persistent MQ message has abended or backed out, but a
     response has been sent to the requester.
  
   Explanation: A provider pipeline was started by a persistent MQ
     message, and a BTS process was created to ensure that the
     information was not lost on a system failure. The provider
     pipeline abended or backed out changes to recoverable
     resources, and the BTS process completed with status ABENDED.
     As a response message has been returned to the requester -in
     the form of an MQ message on the reply-to queue-, the
     requester is aware of the failure and can take appropriate
     action. The BTS process in the provider system has been
     cancelled, as there is no longer any need to retain it. And
     earlier message DFHBA0104 reported the ABENDED state of the
     process.
  
   System action: CICS cancels the process that completed with
     status ABENDED. All the data associated with the process is
     removed.
  
   User response: None. This is an informational message,
     indicating that the process that complete with status ABENDED
     has been cancelled. CICS has sent a response to the requester
     indicating the failure.
  
   Destination: CPIO
   Module: DFHPIDSQ
   XMEOUT Parameters: date,  time, applid
  
   DFHPI0118 applid CICS has attempted to use BTS
     processes to support pipelines started with WebSphere MQ
     persistent messages. This attempt failed. CICS will continue,
     using channel-based containers for the pipeline, but there is
     a risk of data loss in the even of a system failure. Ensure
     that BTS PROCESSTYPE, repository, and local request queue are
     correctly defined and installed.
  
   Explanation: A persistent MQ message has been received to start
     a CICS provider pipeline, and CICS has attempted to create a
     BTS process to reliably hold the message data until the
     provider pipeline completes. However, an error occurred when
     attempting to create the BTS process. CICS will continue
     processing persistent MQ messages to start provider
     pipelines, using channel based containers, but there is a
     risk that data from the persistent message may be lost in the
     event of a system failure.
  
   System action: CICS continues to process inbound MQ persistent
     messages to drive provider pipelines, but in the event of a
     system failure, data originating in the persistent messages
     may be lost. This message is issued once on the first
     occurrence of a failure to create an BTS process. It is not
     issued for every occurrence of such a failure.
  
   User response: Check that the steps stated in the section
     "Configuring your CICS system for Web Services" in the
     "CICS Web Services Guide" manual have been taken.
  
   Destination: CONSOLE
   Module: DFHPILSQ
   XMEOUT Parameters: applid
  
  ž**** AE07/12/03 FIX IN ERROR. SEE APAR PK57426  FOR DESCRIPTION
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

  PK37266

• APAR is sysrouted TO one or more of the following:

  UK29110

Modules/Macros

•    DESBAACT DESBASP  DESBAXM  DESEISP  DESPIDM
  DESPIDSQ DESPIDUF DESPIIM  DESPIIS  DESPILSQ DESPIPM  DESPITL
  DFHAPCOM DFHAPTRY DFHBAACC DFHBASP  DFHBAUWC DFHBAXM  DFHCSAD
  DFHCSAPS DFHEISP  DFHMEPIC DFHMEPIE DFHMEPIK DFHPIAP  DFHPIDM
  DFHPIDSQ DFHPIDUF DFHPIIMC DFHPIIMI DFHPIISC DFHPIISI DFHPILSQ
  DFHPIPM  DFHPIPMA DFHPIPMM DFHPIPMT DFHPITL  DFHPIUCC DFHPIUCD
  DFH49155
  

Fix information

• Fixed component name

  CICSTS V3 Z/OS

• Fixed component ID

  5655M1500

Applicable component levels

• R500 PSY UK29110

     UP07/09/21 P F709

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.