Fixes are available
16.0.0.2: WebSphere Application Server Liberty 16.0.0.2
16.0.0.3: WebSphere Application Server Liberty 16.0.0.3
16.0.0.4: WebSphere Application Server Liberty 16.0.0.4
17.0.0.1: WebSphere Application Server Liberty 17.0.0.1
17.0.0.2: WebSphere Application Server Liberty 17.0.0.2
17.0.0.3: WebSphere Application Server Liberty 17.0.0.3
17.0.0.4: WebSphere Application Server Liberty 17.0.0.4
18.0.0.1: WebSphere Application Server Liberty 18.0.0.1
18.0.0.2: WebSphere Application Server Liberty 18.0.0.2
18.0.0.3: WebSphere Application Server Liberty 18.0.0.3
18.0.0.4: WebSphere Application Server Liberty 18.0.0.4
19.0.0.1: WebSphere Application Server Liberty 19.0.0.1
19.0.0.2: WebSphere Application Server Liberty 19.0.0.2
19.0.0.3: WebSphere Application Server Liberty 19.0.0.3
19.0.0.4: WebSphere Application Server Liberty 19.0.0.4
19.0.0.5: WebSphere Application Server Liberty 19.0.0.5
19.0.0.6: WebSphere Application Server Liberty 19.0.0.6
19.0.0.7: WebSphere Application Server Liberty 19.0.0.7
19.0.0.8: WebSphere Application Server Liberty 19.0.0.8
19.0.0.9: WebSphere Application Server Liberty 19.0.0.9
19.0.0.10: WebSphere Application Server Liberty 19.0.0.10
19.0.0.11: WebSphere Application Server Liberty 19.0.0.11
19.0.0.12: WebSphere Application Server Liberty 19.0.0.12
20.0.0.1: WebSphere Application Server Liberty 20.0.0.1
20.0.0.2: WebSphere Application Server Liberty 20.0.0.2
20.0.0.3: WebSphere Application Server Liberty 20.0.0.3
20.0.0.4: WebSphere Application Server Liberty 20.0.0.4
20.0.0.5: WebSphere Application Server Liberty 20.0.0.5
20.0.0.6: WebSphere Application Server Liberty 20.0.0.6
20.0.0.7: WebSphere Application Server Liberty 20.0.0.7
20.0.0.8: WebSphere Application Server Liberty 20.0.0.8
20.0.0.9: WebSphere Application Server Liberty 20.0.0.9
20.0.0.10: WebSphere Application Server Liberty 20.0.0.10
20.0.0.11: WebSphere Application Server Liberty 20.0.0.11
20.0.0.12: WebSphere Application Server Liberty 20.0.0.12
21.0.0.3: WebSphere Application Server Liberty 21.0.0.3
21.0.0.4: WebSphere Application Server Liberty 21.0.0.4
21.0.0.5: WebSphere Application Server Liberty 21.0.0.5
21.0.0.6: WebSphere Application Server Liberty 21.0.0.6
21.0.0.7: WebSphere Application Server Liberty 21.0.0.7
21.0.0.8: WebSphere Application Server Liberty 21.0.0.8
21.0.0.9: WebSphere Application Server Liberty 21.0.0.9
21.0.0.1: WebSphere Application Server Liberty 21.0.0.1
21.0.0.2: WebSphere Application Server Liberty 21.0.0.2
21.0.0.10: WebSphere Application Server Liberty 21.0.0.10
21.0.0.11: WebSphere Application Server Liberty 21.0.0.11
21.0.0.12: WebSphere Application Server Liberty 21.0.0.12
22.0.0.1: WebSphere Application Server Liberty 22.0.0.1
22.0.0.2: WebSphere Application Server Liberty 22.0.0.2
22.0.0.3: WebSphere Application Server Liberty 22.0.0.3
22.0.0.4: WebSphere Application Server Liberty 22.0.0.4
APAR status
Closed as program error.
Error description
Customer is running liberty server embedded in a CICS environment on z/OS and reported some CICS tasks that did not complete. £ADESTADS.TASK157.ADHT] State:(BLOCKED) LockName: £java.util. concurrent.atomic.AtomicReference@3b588b9d] LockOwner: £Default Executor-thread-24] Analysis of a console dump showed a deadlock: com/ibm/ws/classloading/internal/util/CanonicalStore@0x 48ee8c9a80 owner thread id: 0x240ee000 name: Default Executor-thread-1109 waiting thread id: 0x24043800 name: Default Executor-thread-1079 .. java/util/concurrent/atomic/AtomicReference@0x4a6e548aa0 owner thread id: 0x24043800 name: Default Executor-thread-1079 waiting thread id: 0x240ee000 name: Default Executor-thread-1109 -- Review of the logic on the threads showed that the out of order lock obtains occur when WAB1 is being added to the webcontainer on one thread and WAB2 is being started on the other thread. The Deadlock can occur if WAB2 needs to add the bundle for WAB1 during OSGi Bundle Resolution.
Local fix
1) Start WAB1 and WAB2 seperately 2) Isolate the function that is common to WAB1 and WAB2 and place it in a bundle repository and make both WAB1 and WAB2 depend on the bundle repository rather than on each other. <bundleRepository> <fileset id="repo1" dir="£{shared.resource.dir}/myRepository" includes="*.jar"/> </bundleRepository> also see "Administering bundle repositories" here https://www-01.ibm.com/support/knowledgecenter/SSAW57_8 .5.5/com.ibm.websphere.osgi.nd.doc/ae/ta_admin_obr.html ?lang=en
Problem summary
**************************************************************** * USERS AFFECTED: All users of IBM WebSphere Application * * Server Liberty * **************************************************************** * PROBLEM DESCRIPTION: Deadlock adding WABs to the * * webcontainer * **************************************************************** * RECOMMENDATION: * **************************************************************** A deadlock can occur when starting a WebSphere Liberty application server with multiple WABs. The deadlock can be reported in a thread dump as such: com/ibm/ws/classloading/internal/util/CanonicalStore@0x48ee8c7a8 0 owner thread id: 0x240ee000 name: Default Executor-thread-1139 waiting thread id: 0x24043800 name: Default Executor-thread- 1379 .. java/util/concurrent/atomic/AtomicReference@0x4a6e545aa0 owner thread id: 0x24043800 name: Default Executor-thread-1379 waiting thread id: 0x240ee000 name: Default Executor-thread- 1139 One callstack will look like this: at com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:46 4(Compiled Code)) at com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:57 (Compiled Code)) at org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun dleTracker.java:469(Compiled Code)) at org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun dleTracker.java:1(Compiled Code)) at org/osgi/util/tracker/AbstractTracked.trackAdding(AbstractTracke d.java:256(Compiled Code)) at org/osgi/util/tracker/AbstractTracked.track(AbstractTracked.java :229(Compiled Code)) at org/osgi/util/tracker/BundleTracker$Tracked.bundleChanged(Bundle Tracker.java:444(Compiled Code)) at org/eclipse/osgi/internal/framework/BundleContextImpl.dispatchEv ent(BundleContextImpl.java:902(Compiled Code)) at org/eclipse/osgi/framework/eventmgr/EventManager.dispatchEvent(E ventManager.java:230(Compiled Code)) at org/eclipse/osgi/framework/eventmgr/ListenerQueue.dispatchEventS ynchronous(ListenerQueue.java:148(Compiled Code)) at org/eclipse/osgi/internal/framework/EquinoxEventPublisher.publis hBundleEventPrivileged(EquinoxEventPublisher.java:165(Compiled Code)) at org/eclipse/osgi/internal/framework/EquinoxEventPublisher.publis hBundleEvent(EquinoxEventPublisher.java:75(Compiled Code)) at org/eclipse/osgi/internal/framework/EquinoxEventPublisher.publis hBundleEvent(EquinoxEventPublisher.java:67(Compiled Code)) at org/eclipse/osgi/internal/framework/EquinoxContainerAdaptor.publ ishModuleEvent(EquinoxContainerAdaptor.java:102(Compiled Code)) at org/eclipse/osgi/container/ModuleContainer.applyDelta(ModuleCont ainer.java:691(Compiled Code)) at org/eclipse/osgi/container/ModuleContainer.resolveAndApply(Modul eContainer.java:491(Compiled Code)) at org/eclipse/osgi/container/ModuleContainer.resolve(ModuleContain er.java:437(Compiled Code)) at org/eclipse/osgi/container/ModuleContainer.resolve(ModuleContain er.java:427(Compiled Code)) at org/eclipse/osgi/container/Module.start(Module.java:416(Compiled Code)) at org/eclipse/osgi/internal/framework/EquinoxBundle.start(EquinoxB undle.java:393(Compiled Code)) at org/eclipse/osgi/internal/framework/EquinoxBundle.start(EquinoxB undle.java:412(Compiled Code)) at com/ibm/ws/classloading/internal/GatewayBundleFactory.start(Gate wayBundleFactory.java:135(Compiled Code)) at com/ibm/ws/classloading/internal/GatewayBundleFactory.createGate wayBundleClassLoader(GatewayBundleFactory.java:92(Compiled Code)) at com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT CCL(ClassLoadingServiceImpl.java:464(Compiled Code)) at com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.access$ 200(ClassLoadingServiceImpl.java:82(Compiled Code)) at com/ibm/ws/classloading/internal/ClassLoadingServiceImpl$4.creat eInstance(ClassLoadingServiceImpl.java:436(Compiled Code)) at com/ibm/ws/classloading/internal/ClassLoadingServiceImpl$4.creat eInstance(ClassLoadingServiceImpl.java:433(Compiled Code)) at com/ibm/ws/classloading/internal/util/CanonicalStore.retrieveOrC reate(CanonicalStore.java:85(Compiled Code)) at com/ibm/ws/classloading/internal/util/CanonicalStore.retrieveOrC reate(CanonicalStore.java:75(Compiled Code)) at com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT hreadContextClassLoader(ClassLoadingServiceImpl.java:433(Compile d Code)) at com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT hreadContextClassLoader(ClassLoadingServiceImpl.java:78(Compiled Code)) at com/ibm/ws/webcontainer/osgi/WebContainer.startModule(WebContain er.java:798(Compiled Code)) at com/ibm/ws/app/manager/web/internal/WebModuleHandlerImpl.deployM odule(WebModuleHandlerImpl.java:165(Compiled Code)) at com/ibm/ws/app/manager/wab/internal/WABInstaller.installIntoWebC ontainer(WABInstaller.java:377(Compiled Code)) at com/ibm/ws/app/manager/wab/internal/WAB.addToWebContainer(WAB.ja va:373(Compiled Code)) at com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:48 0(Compiled Code)) at com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:57 (Compiled Code)) at org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun dleTracker.java:469(Compiled Code)) at org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun dleTracker.java:1(Compiled Code)) at org/osgi/util/tracker/AbstractTracked.trackAdding(AbstractTracke d.java:256(Compiled Code)) at org/osgi/util/tracker/AbstractTracked.trackInitial(AbstractTrack ed.java:183(Compiled Code)) at org/osgi/util/tracker/BundleTracker.open(BundleTracker.java:156( Compiled Code)) at com/ibm/ws/app/manager/wab/internal/WAB$1.run(WAB.java:132(Compi led Code)) at java/util/concurrent/Executors$RunnableAdapter.call(Executors.ja va:483(Compiled Code)) at java/util/concurrent/FutureTask.run(FutureTask.java:274(Compiled Code)) at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExec utor.java:1157) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExe cutor.java:627) at java/lang/Thread.run(Thread.java:798) while the other (and possibly many others) will look like this: at com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:46 4(Compiled Code)) at com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:57 (Compiled Code)) at org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun dleTracker.java:469(Compiled Code)) at org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun dleTracker.java:1(Compiled Code)) at org/osgi/util/tracker/AbstractTracked.trackAdding(AbstractTracke d.java:256(Compiled Code)) at org/osgi/util/tracker/AbstractTracked.track(AbstractTracked.java :229(Compiled Code)) at org/osgi/util/tracker/BundleTracker$Tracked.bundleChanged(Bundle Tracker.java:444(Compiled Code)) at org/eclipse/osgi/internal/framework/BundleContextImpl.dispatchEv ent(BundleContextImpl.java:902(Compiled Code)) at org/eclipse/osgi/framework/eventmgr/EventManager.dispatchEvent(E ventManager.java:230(Compiled Code)) at org/eclipse/osgi/framework/eventmgr/ListenerQueue.dispatchEventS ynchronous(ListenerQueue.java:148(Compiled Code)) at org/eclipse/osgi/internal/framework/EquinoxEventPublisher.publis hBundleEventPrivileged(EquinoxEventPublisher.java:165(Compiled Code)) at org/eclipse/osgi/internal/framework/EquinoxEventPublisher.publis hBundleEvent(EquinoxEventPublisher.java:75(Compiled Code)) at org/eclipse/osgi/internal/framework/EquinoxEventPublisher.publis hBundleEvent(EquinoxEventPublisher.java:67(Compiled Code)) at org/eclipse/osgi/internal/framework/EquinoxContainerAdaptor.publ ishModuleEvent(EquinoxContainerAdaptor.java:102(Compiled Code)) at org/eclipse/osgi/container/ModuleContainer.applyDelta(ModuleCont ainer.java:691(Compiled Code)) at org/eclipse/osgi/container/ModuleContainer.resolveAndApply(Modul eContainer.java:491(Compiled Code)) at org/eclipse/osgi/container/ModuleContainer.resolve(ModuleContain er.java:437(Compiled Code)) at org/eclipse/osgi/container/ModuleContainer.resolve(ModuleContain er.java:427(Compiled Code)) at org/eclipse/osgi/container/Module.start(Module.java:416(Compiled Code)) at org/eclipse/osgi/internal/framework/EquinoxBundle.start(EquinoxB undle.java:393(Compiled Code)) at org/eclipse/osgi/internal/framework/EquinoxBundle.start(EquinoxB undle.java:412(Compiled Code)) at com/ibm/ws/classloading/internal/GatewayBundleFactory.start(Gate wayBundleFactory.java:135(Compiled Code)) at com/ibm/ws/classloading/internal/GatewayBundleFactory.createGate wayBundleClassLoader(GatewayBundleFactory.java:92(Compiled Code)) at com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT CCL(ClassLoadingServiceImpl.java:464(Compiled Code)) at com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.access$ 200(ClassLoadingServiceImpl.java:82(Compiled Code)) at com/ibm/ws/classloading/internal/ClassLoadingServiceImpl$4.creat eInstance(ClassLoadingServiceImpl.java:436(Compiled Code)) at com/ibm/ws/classloading/internal/ClassLoadingServiceImpl$4.creat eInstance(ClassLoadingServiceImpl.java:433(Compiled Code)) at com/ibm/ws/classloading/internal/util/CanonicalStore.retrieveOrC reate(CanonicalStore.java:85(Compiled Code)) at com/ibm/ws/classloading/internal/util/CanonicalStore.retrieveOrC reate(CanonicalStore.java:75(Compiled Code)) at com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT hreadContextClassLoader(ClassLoadingServiceImpl.java:433(Compile d Code)) at com/ibm/ws/classloading/internal/ClassLoadingServiceImpl.createT hreadContextClassLoader(ClassLoadingServiceImpl.java:78(Compiled Code)) at com/ibm/ws/webcontainer/osgi/WebContainer.startModule(WebContain er.java:798(Compiled Code)) at com/ibm/ws/app/manager/web/internal/WebModuleHandlerImpl.deployM odule(WebModuleHandlerImpl.java:165(Compiled Code)) at com/ibm/ws/app/manager/wab/internal/WABInstaller.installIntoWebC ontainer(WABInstaller.java:377(Compiled Code)) at com/ibm/ws/app/manager/wab/internal/WAB.addToWebContainer(WAB.ja va:373(Compiled Code)) at com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:48 0(Compiled Code)) at com/ibm/ws/app/manager/wab/internal/WAB.addingBundle(WAB.java:57 (Compiled Code)) at org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun dleTracker.java:469(Compiled Code)) at org/osgi/util/tracker/BundleTracker$Tracked.customizerAdding(Bun dleTracker.java:1(Compiled Code)) at org/osgi/util/tracker/AbstractTracked.trackAdding(AbstractTracke d.java:256(Compiled Code)) at org/osgi/util/tracker/AbstractTracked.trackInitial(AbstractTrack ed.java:183(Compiled Code)) at org/osgi/util/tracker/BundleTracker.open(BundleTracker.java:156( Compiled Code)) at com/ibm/ws/app/manager/wab/internal/WAB$1.run(WAB.java:132(Compi led Code)) at java/util/concurrent/Executors$RunnableAdapter.call(Executors.ja va:483(Compiled Code)) at java/util/concurrent/FutureTask.run(FutureTask.java:274(Compiled Code)) at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExec utor.java:1157) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExe cutor.java:627) at java/lang/Thread.run(Thread.java:798)
Problem conclusion
The fix for this APAR is to improve the locking mechanism used in the WAB code to avoid this deadlock. The fix for this APAR is currently targeted for inclusion in fix pack 16.0.0.2. Please refer to the Recommended Updates page for delivery information: http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
Temporary fix
Comments
APAR Information
APAR number
PI53419
Reported component name
LIBERTY PROF -
Reported component ID
5655W6514
Reported release
850
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2015-12-02
Closed date
2016-05-02
Last modified date
2016-06-13
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
LIBERTY PROF -
Fixed component ID
5655W6514
Applicable component levels
R850 PSY
UP
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"850","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
04 May 2022