A fix is available
APAR status
Closed as program error.
Error description
Our 3.1 CMAS reached a short on storage condition DFHSM0133 and did not recover, so was cancelled. There was a batch job (EFGJOBB) running at the time of the issue (to refresh our production proving environment in CPSM). This job was cancelled as well but on CMAS restart we ran into multiple errors: 09.38.57 STC23720 +EYUXD0806E CMASABDC API ESSS Connection has failed. ESSS Response (6) 09.38.57 STC23720 +EYUXD0804E CMASABCD Connection task unable to recover from failure(s). Task being terminated EYUXL0905E CMASQABCD ASRA IN XCLD, OFFSET 000005E4 09.38.57 STC23720 +EYUXL0112E CMASABCD CMAS initialization failed 09.38.57 STC23720 +DFHDU0201 CMASABCD ABOUT TO TAKE SDUMP. DUMPCODE: EYUXL001, DUMPID: 27/0003 . The subsequent CMAS restart failed with : EYUXC0006S CMASABCD Unable to create Data Cache Management Dataspace, DSPSERV RC=00000008, REAS=6B000911 and many dumps issued by ESSS : Dump Title: CICSPlex SM 310 Lock Management Abend,SSRCVLOCK ,CMASABCD,LPAR,10/23/12,09:40:26 . DSPSERV RC=00000008, REAS=6B000911 codes mean that the DMDS dataspace create attempt has failed since the name is not unique, already exists. Running IPCS RSMDATA DPSACE command on the dump gives a list of the dataspaces owned by the ESSS and shows they they all exist but are orphaned and the CMAS restart attempts fail.
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: All CICSPlex SM V5R1M0 Users * **************************************************************** * PROBLEM DESCRIPTION: Unpredicatble results may occur if a * * CMAS that was active is restarted at * * the same time that the last MAS or * * batch API program that was connected to * * the CMAS when it was originally active * * is being terminated. * * * * Results that may occur include but are * * not limited to the following: * * * * - CMAS initialization may fail after * * receiving messages EYUXD0806E and * * EYUXD0804E: * * * * EYUXD0806E API ESSS Connection has * * failed. ESSS Response * * (6) * * * * EYUXD0804E Connection task unable * * to recover from * * failure(s). Task being * * terminated * * * * - CMAS initialization may fail after * * receiving messages EYUXC0006S and * * EYUXC0002E: * * * * EYUXC0006S Unable to create Data * * Cache Management * * Dataspace, DSPSERV * * RC=00000008, * * REAS=6B000911 * * * * EYUXC0002E Data Cache * * initialization failed * * * * - System dumps may be taken on behalf * * of the CPSM Lock Management PC * * routines for various CPSM regions on * * the LPAR. The dump title of these * * abends will be similar to the * * following: * * * * CICSPlex SM <cpsmversion> Lock * * Management Abend,<routine>, * * <jobname>,<lparid>,<date>,<time> * * * * Note that the failure to restart the * * CMAS on the LPAR and the abends will * * occur until the LPAR is IPLed or the * * CPSM ESSS is terminated through the * * EYU9XEUT utility. Note also that if * * the EYU9XEUT utility is used to * * terminate the ESSS, most likely the * * FORCE option will be required. * **************************************************************** * RECOMMENDATION: After applying the PTF that resolves this * * APAR, the CPSM ESSS (Environment Service * * System Services) subsystem address space * * (EYUX510) must be terminated on every MVS * * image where a CMAS has executed, since the * * last IPL. * * * * This can be accomplished in either of two * * ways: * * * * 1 - IPL the MVS Image. * * * * Or: * * * * 2 - Stop all CMASes and MASes on an MVS * * image. * * * * - Use EYU9XENF to check that no address * * spaces are connected to the ESSS * * subsystem. * * * * - Use the EYU9XEUT TERMINATE function to * * stop the ESSS. * * * * - Refresh LLA to ensure the updated * * version of EYU9X510 is picked up. * * * * - Restart any CMASes and MASes which * * execute on the MVS image, insuring * * that updated libraries are being * * picked up. When the first CMAS is * * started the EYUX510 address space will * * be started automatically. * * * * For details on EYU9XENF and the EYU9XEUT * * TERMINATE function refer to "CICS * * Transaction Server for z/OS CICSPlex SM * * Problem Determination Version 5 Release 1" * * (GC34-2890-xx). * * * * Note that each MVS image can be updated * * separately, and systems on an MVS image that * * are using the new code can communicate with * * systems on other MVS images that are not yet * * using the new code. * **************************************************************** When a CMAS first comes up on an LPAR, a connected address space element (CASTE) is allocated for the CMAS, activated and placed in the active CMAS chain. Additionally, private (COMx, MONx, RTAx, BASx) and shared (DMDS, DATx, QUEx, TOPx, WLMx, MASx) dataspaces are allocated for the CMAS. When the CMAS terminates while no MASes or batch API programs are connected to it, its CASTE is deactivated, removed from the active CMAS chain and deallocated, and the private and shared dataspaces are deleted. When the CMAS terminates while at least one MAS or batch API program is connected to it, its CASTE is deactivated but remains allocated and on the active CMAS chain, the private dataspaces are deleted, and the shared dataspaces remain allocated. If the CMAS restarts before all the connected MASes and batch API programs terminate, then the CMAS CASTE is reactivated, the shared dataspaces remain allocated, and the private dataspaces are allocated. If all connected MASes and batch API programs terminate before the CMAS is restarted, then when the last of these terminate the CMAS CASTE is removed from the active CMAS chain and deallocated and the shared dataspaces are deleted. If the CMAS restarts after all the connected MASes and batch API programs terminate, then a new CASTE is allocated for the CMAS, activated and placed in the active CMAS chain, and the private and shared dataspaces are allocated for the CMAS. A problem can occur if the last MAS or batch API program terminates at the same time the CMAS is restarted. In this situation, if the termination request is processed first, a serialization error can allow the termination of the MAS or batch API program to trigger the remove of the CMAS's CASTE from the active CMAS chain and deallocate it, and delete the shared dataspaces, at the same time as the start up of the CMAS triggers the reuse and reactivation of the CASTE and the reuse of the shared dataspaces. If this occurs, the symptoms noted above will occur.
Problem conclusion
A number of updates have been made to ensure that the serialization error is addressed: - Copybook EYURXECA, which defines the CASTE, has been updated to add a new status flag, CAST_CSTA_DEAD, which is set when a CMAS's CASTE is being deallocated. - Copybook EYURXEES, which defines the parameter list for the ESSS connect process, has been updated to return to new status of XEES_CMASDEAD when a CMAS attempts to connect to the ESSS at the same time that a termination of a MAS or batch API program has resulted in the CASTE being deallocated for the CMAS. - Method EYU0XLBV (XLBV), which issues the ESSS connect request during CMAS initialization, has been updated to recognize if a connect request returns the new XEES_CMASDEAD status. If so, XLBV will delay for one second and retry the connect request. XLBV will do this for up to 10 retries. - Copybook EYU2XEEC, which is included in the PC routines program EYUTXEPC and processes the ESSS connect request, has been updated to check for a CASTE status of CAST_CSTA_DEAD when processing a CMAS connect request when the CASTE already exists. If this status is found, the connect is failed with the new XEES_CMASDEAD status. - Copybook EYU2XERT, which is included in the PC routines program EYUTXEPC and processes the deallocation of a CMAS's CASTE and the deleting of the shared dataspaces, has been updated to set the CMAS's CASTE status to CAST_CSTA_DEAD before starting the deallocation and deletion.
Temporary fix
********* * HIPER * ********* FIX AVAILABLE BY PTF ONLY
Comments
APAR Information
APAR number
PM78547
Reported component name
CICS TS Z/OS V5
Reported component ID
5655Y0400
Reported release
80M
Status
CLOSED PER
PE
NoPE
HIPER
YesHIPER
Special Attention
NoSpecatt
Submitted date
2012-12-05
Closed date
2013-01-18
Last modified date
2015-03-04
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
UK91025
Modules/Macros
EYU0XLBV EYU9X510 EYUTXEPC
Fix information
Fixed component name
CICS TS Z/OS V5
Fixed component ID
5655Y0400
Applicable component levels
R80M PSY UK91025
UP13/01/22 P F301
Fix is available
Select the PTF appropriate for your component level. You will be required to sign in. Distribution on physical media is not available in all countries.
[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSGMGV","label":"CICS Transaction Server"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"5.1","Edition":"","Line of Business":{"code":"LOB35","label":"Mainframe SW"}},{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"SG19M","label":"APARs - z\/OS environment"},"Component":"","ARM Category":[],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"5.1","Edition":"","Line of Business":{"code":"","label":""}}]
Document Information
Modified date:
04 March 2015