PI83680: CPSM WLM: RECYCLING TARGET AORS ONE BY ONE MAY LEAD TO MAXTASK DURING HIGH TRANSACTIONS PER SECOND (TPS) RATE

Obtain the fix for this APAR.

APAR status

• Closed as program error.

Error description

• You have a CICSplex environment consisting of multiple lpars,
  with multiple router regions and target regions running on
  each.  When you recycle the target regions one by one, when the
  new recycled AOR is restarted, majority of the workload is
  directed to this AOR.  If this occurs during a time when the
  transactions rate is high, the AOR may reach MAXTASK.  As each
  AOR is recycled, multiple target regions may reach maxtask
  causing the work to back up and the CPSM routing to be come
  chaotic.
  .
  .
  .
  Additional Symptom(s) Search Keyword(s): KIXREVGJT
  

Local fix

• n/a
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All CICSPlex SM V5R2M0, V5R3M0 and V5R4M0    *
  *                 Users                                        *
  ****************************************************************
  * PROBLEM DESCRIPTION: When the transaction rate for a         *
  *                      workload managed by CPSM WLM is high,   *
  *                      and optimized workload processing is    *
  *                      being used, one or more target regions  *
  *                      in the AORSCOPE may invalidly receive   *
  *                      more work than other target regions in  *
  *                      the AORSCOPE for a period of time.      *
  *                      This can result in maxtask (MXT)        *
  *                      occurrences in the targeted regions,    *
  *                      which can lead to decreased throughput  *
  *                      for the transactions that are part of   *
  *                      the workload.                           *
  ****************************************************************
  * RECOMMENDATION: After applying the PTF that resolves this    *
  *                 APAR, all CMASes and MASes must be           *
  *                 restarted.  Note that the restarts do not    *
  *                 need to occur at the same time, but until    *
  *                 all regions are restarted, the problem may   *
  *                 occur.                                       *
  ****************************************************************
  When using CPSM optimized workload processing, the routing
  regions depend upon data provided by the target regions in the
  region status (RS) block, to determine where to route.  The main
  factor in the RS block is the current task load in the target
  region.
  
  When the workload transaction rate is high, it is possible that
  multiple tasks in multiple routing regions will make routing
  decisions that do not take into account previous routes, because
  the decisions are being made before the target regions have the
  opportunity to update the RS block to take those previous routes
  into account.  This can result in target regions being given
  more work than other regions, possibly resulting in MXT.
  

Problem conclusion

• To address this, work sent to a target region between updates of
  its RS block will be kept track of by routing regions, and will
  be used during the route determination process.
  
  To affect this change, the following updates have been made:
  
  -  Method EYU0WDTM (WDTM), running in a routing region during
     route processing, will increment a counter in the WLM data
     space for the target region the route is being directed to.
  
     As documented below, this new counter will reside in the WLM
     data space copy of the RS block, and its field name is
     TOR_RTE_CNT.
  
     The counter is never decremented by the routing regions, but
     will be reset to zero (0) depending upon whether the target
     region is connected to the same CMAS as the routing region:
  
     -  if not connected to the same CMAS, the count will be reset
        when the target region's READRS value expires.  The READRS
        default value is 200 milliseconds.
  
     -  if connected to the same CMAS, the count will be reset
        when the target region next updates its RS block, or when
        the target region's READRS value expires, whichever comes
        first.
  
  -  Method EYU0WNAL (WNAL), running in a routing region during
     route determination processing, will add the counter being
     set by WDTM for a target region to the task load in the RS
     block for the target region, to set the total task load for
     the target region.
  
  -  Methods EYU0WABA (WABA) and EYU0WABB (WABB), which run in a
     CMAS when the WLMAWAOR (WABA) or WLMATARG (WABB) resource
     tables are requested through the WUI, Explorer or a CPSM API
     program, have been updated to not cause the counter to be
     reset prematurely.  The Routing Load attributes of the
     tables, WLMAWAOR_ROUTINGLOAD and WLMATARG_ROUTINGLOAD, do not
     include the new counter, due to its transient nature.
  
  -  DFHRSBC, which defines the RS block, has been updated to add
     field TOR_RTE_CNT.  Additionally, since this fix is being
     delivered by PTF, and there is no guarantee that a specific
     target region has been restarted with the PTF code, the RS
     block version has been updated form 1 to 2, so that routing
     regions know whether a target has the fix applied.
  
     The RS block for a target region is located in three
     locations:
  
     -  in the RS domain anchor block in the target region.
  
     -  in the CPSM optimized workload coupling facility data
        table (CFDT) for the CICSplex the target region is
        associated with.  This is set by the target region
        from the copy in the anchor block, when it is updated.
  
     -  in the WLM data space for each CMAS that manages the
        workload(s) that the target region is associated with.
        This is set based upon whether the target region is
        connected to the CMAS whose WLM data space it is:
  
        -  if the target region is connected to the CMAS, then
           after updating the CFDT, it will also update the
           WLM data space from the copy in the anchor block.
  
        -  if the target region is not connected to the CMAS, then
           during the route determination process for a route
           request for which the target region is in the AORSCOPE,
           routing regions connected to the CMAS will read the RS
           block into the WLM data space based upon the target
           region's READRS value.
  
     TOR_RTE_CNT will only be set by WDTM in the WLM data space
     copy of the RS block.
  
  -  DFHRSCM contains common subroutines used by other CICS RS
     domain modules.  One of those routines, update_CF_record, is
     called to update the CFDT copy of the RS block.  This has
     been changed to also update the local WLM data space copy of
     the RS block after updating the CFDT.
  
     Other RS domain modules that call update_CF_record, which
     previously had been updating the local WLM data space copy
     outside of the update_CF_record routine, have been updated to
     remove those now extraneous updates.  Modules updated for
     this are noted below.
  
  -  DFHRSDC, which defines the RS domain anchor block and RS
     domain trace point IDs, has been updated to allow for new
     exception traces that will be issued by the RS domain when it
     is notified for MXT start or end in a target region.
  
     The new exception trace point IDs are x'0406' for MXT start
     and x'0407' for MXT end.
  
  -  DFHRSDM, which is the RS domain initialization program, has
     been updated to set the new TOR_RTE_CNT field in the anchor
     block copy of the RS block to zero (0).  As this field is
     never updated by the target region, this will ensure that
     when a target region updates the CFDT and local WLM data
     space copies of the RS block, that TOR_RTE_CNT will be reset.
  
  -  DFHRSDU, which is called when dumps start or end in a target
     region, and which calls subroutine update_CF_record if
     needed, has been updated to remove now extraneous updates of
     the local WLM data space copy of the RS block.
  
  -  DFHRSSR, which is called at various times to update the RS
     block in a target region, and which calls subroutine
     update_CF_record to do this, has been updated to remove now
     extraneous updates of the local WLM data space copy of the RS
     block.
  
  -  DFHRSTP, which is called under the RS domain long running
     task at various times to update the RS block in a target
     region, and which calls subroutine update_CF_record to do
     this, has been updated to remove now extraneous updates of
     the local WLM data space copy of the RS block.
  
  -  DFHRSXM, which is called when a task starts and ends and when
     MXT starts or ends in a target region, and which calls
     subroutine update_CF_record if needed, has been updated to
     remove now extraneous updates of the local WLM data space
     copy of the RS block.
  
     Additionally:
  
     -  when called for MXT start, it will now issue the new MXT
        start exception trace, and will also update the RS block
        task count to match the MXT count before updating the RS
        block.
  
     -  when called for MXT end, it will now issue the new MXT end
        exception trace.
  
  -  DFHRSXRI, which is the RS domain trace interpretation module,
     has been updated to allow for formatting of the new RS domain
     MXT start and end exception traces.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

  UI52175 UI52176 UI52177 UI52178 UI52179 UI52180

Modules/Macros

• CJA0WNAL CJB0WNAL CJC0WNAL CJD0WNAL CJE0WNAL CJF0WNAL CJG0WNAL
  DFHRSDM  DFHRSDU  DFHRSDUT DFHRSSR  DFHRSSRT DFHRSTP  DFHRSTPT
  DFHRSXM  DFHRSXMT DFHRSXRI EYU0WABA EYU0WABB EYU0WDTM EYU0WNAL
  

Fix information

• Fixed component name

  CICS TS Z/OS V5

• Fixed component ID

  5655Y0400

Applicable component levels

• R000 PSY UI52179

     UP17/11/30 P F711

• R00M PSY UI52180

     UP17/11/30 P F711

• R100 PSY UI52177

     UP17/12/01 P F711

• R10M PSY UI52178

     UP17/12/01 P F711

• R900 PSY UI52175

     UP17/11/30 P F711

• R90M PSY UI52176

     UP17/11/30 P F711

Fix is available

• Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.