IBM Support

PI33067: Liberty profile server may hang when using the scalingController feature

Fixes are available

8.5.5.5: WebSphere Application Server V8.5.5 Fix Pack 5
8.5.5.6: WebSphere Application Server V8.5.5 Fix Pack 6
8.5.5.7: WebSphere Application Server V8.5.5 Fix Pack 7
8.5.5.8: WebSphere Application Server V8.5.5 Fix Pack 8
8.5.5.9: WebSphere Application Server V8.5.5 Fix Pack 9
8.5.5.10: WebSphere Application Server V8.5.5 Fix Pack 10
8.5.5.11: WebSphere Application Server V8.5.5 Fix Pack 11
8.5.5.12: WebSphere Application Server V8.5.5 Fix Pack 12
8.5.5.13: WebSphere Application Server V8.5.5 Fix Pack 13
8.5.5.14: WebSphere Application Server V8.5.5 Fix Pack 14
8.5.5.15: WebSphere Application Server V8.5.5 Fix Pack 15
8.5.5.14: WebSphere Application Server V8.5.5 Fix Pack 14
8.5.5.17: WebSphere Application Server V8.5.5 Fix Pack 17
8.5.5.20: WebSphere Application Server V8.5.5.20
8.5.5.18: WebSphere Application Server V8.5.5 Fix Pack 18
8.5.5.19: WebSphere Application Server V8.5.5 Fix Pack 19
8.5.5.16: WebSphere Application Server V8.5.5 Fix Pack 16
8.5.5.21: WebSphere Application Server V8.5.5.21

Subscribe

You can track all active APARs for this component.

 

APAR status

  • Closed as program error.

Error description

  • A Liberty profile server may hang when using the
scalingController feature.  A javacore dump of the server
shows a deadlock.

1LKDEADLOCK    Deadlock detected !!!
NULL           ---------------------
NULL
2LKDEADLOCKTHR  Thread "Default Executor-thread-1718"
(0x0000000002DA8200)
3LKDEADLOCKWTR    is waiting for:
4LKDEADLOCKMON      sys_mon_t:0x00007F889435EB88 infl_mon_t:
0x00007F889435EC08:
4LKDEADLOCKOBJ
com/ibm/ws/imf/apc/IMControllerImpl$GroupChangeManager@0x000
00000E2B598F8
3LKDEADLOCKOWN    which is owned by:
2LKDEADLOCKTHR  Thread "Scaling Controller topology update
handler" (0x0000000002302500)
3LKDEADLOCKWTR    which is waiting for:
4LKDEADLOCKMON      sys_mon_t:0x00007F8844C36F88 infl_mon_t:
0x00007F8844C37008:
4LKDEADLOCKOBJ
com/ibm/ws/imf/apc/IMControllerImpl@0x00000000E2B57870
3LKDEADLOCKOWN    which is owned by:
2LKDEADLOCKTHR  Thread "Default Executor-thread-1718"
(0x0000000002DA8200)

The thread names may vary.  The key information is that one
of the threads holds a lock on
com/ibm/ws/imf/apc/IMControllerImpl$GroupChangeManager.

Local fix

  • 



Problem summary


    

  • ****************************************************************
* USERS AFFECTED:  All users of the auto scaling function in   *
*                  IBM WebSphere Application Server Liberty    *
*                  Profile                                     *
****************************************************************
* PROBLEM DESCRIPTION: Liberty profile server may hang when    *
*                      using the scalingController feature     *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
A Liberty profile server may hang when using the
scalingController feature.  A javacore dump of the server shows
a deadlock.

1LKDEADLOCK    Deadlock detected !!!
NULL           ---------------------
NULL
2LKDEADLOCKTHR  Thread "Default Executor-thread-1718"
(0x0000000002DA8200)
3LKDEADLOCKWTR    is waiting for:
4LKDEADLOCKMON      sys_mon_t:0x00007F889435EB88 infl_mon_t:
0x00007F889435EC08:
4LKDEADLOCKOBJ
com/ibm/ws/imf/apc/IMControllerImpl$GroupChangeManager@0x0000000
0E2B598F8
3LKDEADLOCKOWN    which is owned by:
2LKDEADLOCKTHR  Thread "Scaling Controller topology update
handler" (0x0000000002302500)
3LKDEADLOCKWTR    which is waiting for:
4LKDEADLOCKMON      sys_mon_t:0x00007F8844C36F88 infl_mon_t:
0x00007F8844C37008:
4LKDEADLOCKOBJ
com/ibm/ws/imf/apc/IMControllerImpl@0x00000000E2B57870
3LKDEADLOCKOWN    which is owned by:
2LKDEADLOCKTHR  Thread "Default Executor-thread-1718"
(0x0000000002DA8200)

The thread names may vary.  The key information is that one of
the threads holds a lock on
com/ibm/ws/imf/apc/IMControllerImpl$GroupChangeManager.

    



Problem conclusion


    

  • The synchronization within the IMControllerImpl class has been
changed to avoid the deadlock.
The inner class IMControllerImpl$GroupChangeManager has been
deleted as part of this fix.

This fix also resolves the following
ConcurrentModificationException which can occur in the
IMControllerImpl class when iterating through a HashSet.

 java.util.ConcurrentModificationException
    at
java.util.HashMap$HashIterator.nextEntry(HashMap.java:948)
    at java.util.HashMap$KeyIterator.next(HashMap.java:982)
    at
com.ibm.ws.imf.apc.IMControllerImpl.updateContainerTree(IMContro
llerImpl.java:416)
    at
com.ibm.ws.imf.apc.IMControllerImpl.updateContainerTree(IMContro
llerImpl.java:378)
    at
com.ibm.ws.scaling.controller.messaging.ControllerHaService.rece
iveMessage(ControllerHaService.java:218)
    at
com.ibm.ws.collective.singleton.internal.SingletonServiceMesseng
erMBeanImpl$DeliveryTask.run(SingletonServiceMessengerMBeanImpl.
java:287)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec
utor.java:1177)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe
cutor.java:642)
    at java.lang.Thread.run(Thread.java:857)

This fix also reduces the likelihood of the CWWKV0117E message
with text indicating that a server
is already running.

[16/01/15 18:18:30:873 GMT] 00000113
com.ibm.ws.scaling.controller.internal.ScalingExecutorImpl   I
CWWKV0117E: The scaling controller was unsuccessful in an
attempt to start server member1 on host localhost.
 Return code: 1
 Standard output: Server member1 is already running.




The fix for this APAR is currently targeted for inclusion in fix
pack 8.5.5.5.  Please refer to the Recommended Updates page for
delivery information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

    



Temporary fix


    

  • 



Comments


    

  • 



APAR Information




    

  • APAR number
    PI33067
    • 

  • Reported component name
    LIBERTY PROFILE
    • 

  • Reported component ID
    5724J0814
    • 

  • Reported release
    850
    • 

  • Status
    CLOSED  PER
    • 

  • PE
    NoPE
    • 

  • HIPER
    NoHIPER
    • 

  • Special Attention
    NoSpecatt
    • 

  • Submitted date
    2015-01-20
    • 

  • Closed date
    2015-01-29
    • 

  • Last modified date
    2015-01-29
    • 





    

  • APAR is sysrouted FROM one or more of the following:
    

    • 

  • APAR is sysrouted TO one or more of the following:
    

    • 



Fix information


    

  • Fixed component name
    LIBERTY PROFILE
    • 

  • Fixed component ID
    5724J0814
    • 



Applicable component levels


    

  • R850  PSY
       UP
    • 








      
              
                            

              

              [{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"850","Line of Business":{"code":"LOB45","label":"Automation"}}]
              

                          

          
 
        

      

    

      

        

          

            
Document Information


            

            


                        

            Modified date:
            

            28 April 2022
            

            
                      

        


        

        


      

    

  



    

  

  
    
            

          
Page Feedback