IBM Support

Handling certain Out of Memory conditions when migrating an earlier version of WebSphere Application Server

Troubleshooting


Problem

When running WASPreUpgrade or WASPostUpgrade, during the migration process from an earlier release of IBM® WebSphere® Application Server to V6.1, up through V8.5.5, the heap might run out of memory for a complex environment. Either the execution of wsadmin to install the applications, or the migration of the configuration itself, throws java.lang.OutOfMemory errors.

Symptom

  • Out of Memory condition occurs during application deployment
    The WASPostUpgrade migration process uses the operating system default value for the heap. However, in some cases this might not be enough space to install large applications. When the migration trace is set, it will write the following to the specified trace log file:
    MIGR0339I: Using wsadmin to deploy the application application_name.

  • exec <install_root>/bin/wsadmin.sh -conntype NONE -f <backup_directory>/install_application_name.jacl
                                     wsadmin
    MIGR0241I: Output of wsadmin.
    WASX7357I: By request, this scripting client is not connected to any server process. Certain configuration and application operations will be available in local mode.
    JVMDG217: Dump Handler is Processing a Signal - Please Wait.
    JVMDG315: JVM Requesting Heap dump file
    JVMDG318: Heap dump file written to install_root/bin/heapdump467178.1108586637.txt
    JVMDG303: JVM Requesting Java™ core file
    JVMDG304: Java core file written to install_root/bin/javacore467178.1108586640.txt
    JVMDG274: Dump Handler has Processed OutOfMemory.
    WASX7017E: Exception received while running file "backup_directory/install_<app_name>.jacl"; exception information: com.ibm.bsf.BSFException: error while eval'ing Jacl expression: java.lang.OutOfMemoryError: JVMXE006:OutOfMemoryError, stAllocArray for executeJava failed
    MIGR0340E: Unable to deploy the application application_name.


    The log excerpt above shows that the WASPostUpgrade tool invokes the wsadmin utility to handle the deployment of the applications to the new node. However, the task fails, indicating an OutOfMemoryError. Note the highlighted text in the error excerpt -- it is important to observe that the OutOfMemoryError occurs within a few messages of the call to wsadmin.sh.

    Solution: Check the solution 1, 2, or 3 for steps to resolve an Out of Memory condition during application deployment. If the Out of Memory condition occurs during configuration migration instead, then proceed to the following explanation, then check solution 4.
  • Out of Memory condition occurs during configuration migration
    If the application server or cell environment is especially complex, the WASPreUpgrade utility may run out of memory while gathering information about the application server or cell. Also, the WASPostUpgrade utility can run out of memory while migrating the configuration data from the backup directory into the new application server or profile. This is different than running out of memory while deploying applications, because the WASPreUpgrade or WASPostUpgrade process is running out of memory, as opposed to the wsadmin process responsible for deploying applications.

    An Out of Memory condition can manifest itself in a variety of ways when it occurs in WASPreUpgrade or WASPostUpgrade. When the tool runs out of memory, it might not necessarily abort right away. Therefore, any subsequent actions might fail in unexpected ways. So, when examining a WASPreUpgrade or WASPostUpgrade trace, be sure to find the first occurrence of the OutOfMemoryError listed in the trace. Any errors listed after that first OutOfMemoryError might not be relevant.

    Solution: Check solution 4 for steps to resolve an Out of Memory condition during configuration migration.

Resolving The Problem

There are three possible solutions to this problem:

  • Solution 1: Edit the wsadmin.sh script

    Important:
    This solution is recommended when migrating to version V6.0, or V6.0.1, for situations where there is time available to run WASPostMigrate again. See solution 2 if there is not enough time available to run WASPostMigrate again. See solution 3 for suggestions when migrating to version V6.0.2 and V6.1 or later. Solution 3 is easier to implement, but is only available for those versions.
    1. Change directories to the new_install_root/bin directory, where new_install_root represents the new product version being migrated to.

    2. Edit the wsadmin.sh/bat file and navigate to the bottom of the file. Look for the call to "java" near the bottom of the file.

    3. You should find the following command, depending on the platform:
      • On a Microsoft® Windows® system:
        "%JAVA_HOME%\bin\java" %javaoption% %CONSOLE_ENCODING% %DEBUG% "%CLIENTSOAP%" "%CLIENTSAS%" "-Dcom.ibm.ws.scripting.wsadminprops=%WSADMIN_PROPERTIES%" -Dcom.ibm.ws.management.standalone=true
        ...

      • On a UNIX or Linux-based system:
        "$JAVA_HOME/bin/java" \
        -Xbootclasspath/p:"$WAS_BOOTCLASSPATH" \
        $EXTRA_X_ARGS \
        $CONSOLE_ENCODING \
        $javaOption \
        $WAS_DEBUG \
        ...

    4. After the javaoption reference, add the parameter -Xmx512m. The 512 can be increased depending on the amount of memory on your system.
      Note: This solution is not meant for V6.0.2 or later. These edits will have the proper effect only on releases V6.0, and V6.0.1.
      • For example, on a Windows system:
        %JAVA_HOME%\bin\java" %javaoption% -Xmx512m %CONSOLE_ENCODING% %DEBUG%....</code>]
      • On a UNIX® or Linux® based system, the -Xmx512m should be placed directly after the $javaOption, but before the \ character:
        "$JAVA_HOME/bin/java" \
        -Xbootclasspath/p:"$WAS_BOOTCLASSPATH" \
        $EXTRA_X_ARGS \
        $CONSOLE_ENCODING \
        $javaOption
        -Xmx512m \
        $WAS_DEBUG \
        ...


        Note: It is highly recommended that the system have at least 512 megabytes more physical memory than whatever value is specified. If -Xmx512m is specified, than the system should have at least 1024 megabytes of physical memory. The upper limit value for 32-bit editions of WebSphere Application Server is 2048m. The upper limit value for 64-bit editions is orders of magnitude higher, though it is probably not necessary to use a value higher than 4096m.

    5. Save the wsadmin.sh/bat file.

    6. Return your targeted application server (or application server profile) back to the its state before running the WASPostUpgrade command. For example, restore from a configuration backup using the restoreConfig command.

    7. Run the WASPostUpgrade.sh/bat command again.

    8. Verify that the application was installed properly. Then remove the -Xmx512m parameter from the wsadmin.sh/bat file.



  • Solution 2: Run the deployment scripts manually

    Important: This solution is recommended for cases where it is not possible to run WASPostMigrate again, such as in cases where WASPostMigrate takes a long time to complete. This is an alternative to solution 1.
    1. Let the migration process complete. Then, review the migration to make sure that all was successful except for the deployment of the client's applications.

    2. After confirming a successful migration of the configuration (but failure to deploy one or more applications), open the trace file. Find the commands that the WASPostUpgrade program failed on. For example, from the error excerpted in the Cause section of this technote, the following shows up in the trace file:
      exec install_root/bin/wsadmin.sh -conntype NONE -f
      backup_dir/install_application_name.jacl

      Note: The keyword exec shortly before wsadmin in the trace file. This shows an example of the command which the migration tool invoked, but failed to complete its task.

    3. Manually run this command from the command line, adding the -Xmx512m parameter after wsadmin.sh:
      install_root/bin/wsadmin.sh -Xmx512m -conntype NONE -f backup_dir/install_application_name.jacl

      Notes:
      • For notes concerning the use of -Xmx512m, see the discussion of this parameter in solution 1.

      • In this example, install_application_name should be replaced with the appropriate script name for installing an application. List the files in the migration backup directory to determine which scripts are available for migration deployment.
      • Examples of this can be found in the Information Center, section "Java heap size for migrating EAR files":
    4. This will install the client's application. Verify that the application was installed properly and that it functions correctly.

  • Solution 3: Use the -appInstallMaxHeapSize parameter, only valid for v5.x to 6.0.2.x

    Important:
    This is an alternative to Solution 1. WASPostUpgrade script which gives the administrator the opportunity to increase the heap size available to the wsadmin command when it is called from within the migration process.
    1. Discard the migrated profile (target profile) and recreate a default target profile. Since failure to install the applications does not cause the migration to rollback changes, it is unsupported to effectively migrate twice into the same profile..

    2. Prepare to run the WASPostUpgrade.sh/bat command again. This time, add the -appInstallMaxHeapSize parameter when invoking the script, with an integer value representing the amount of memory in megabytes to allocate to the application deployment process.

      For example:
      install_root/bin/WASPostUpgrade.sh backup_dir -appInstallMaxHeapSize 512

    3. Examine the logs and the new application server, or profile, to verify that the applications were deployed without error.

  • Solution 4: Edit the WASPreUpgrade and/or WASPostUpgrade script

    Important: If the WASPreUpgrade script indicates an OutOfMemoryError anywhere, or if the WASPostUpgrade script indicates an OutOfMemoryError which is not related to calling wsadmin for application deployment, then this is the recommended solution. This solution might be necessary in addition to solution 1, 2, or 3.
    1. Change directories to the new_install_root/bin directory, where new_install_root represents the new product version being migrated to.

    2. Open both the WASPreUpgrade.sh/bat and WASPreUpgrade.sh/bat files in a text editor. The following steps apply to editing both files.

    3. If you are editing the V6.1 and later WASPostUpgrade, then follow this step. Otherwise, if you are editing V6.1 WASPreUpgrade or V6.0 scripts, skip this step and proceed to the next step.

      The WASPostUpgrade script for V6.1 and later already contains parameters which increase the maximum heap size available to the WASPostUpgrade tool. To increase the maximum heap size, search the script for every occurrence of -Xmx and replace the number after that phrase with a larger number. For example, replace every occurrence of -Xmx256m with -Xmx1024m. Note that it is not necessary to change the -Xms parameters.

      For Solaris and HP OS use:
      Migration tools might hang and generate out-of-memory exceptions on Solaris or HP-UX operating systems
      For all other OS platforms you change the -Xmx value for the parameter PREF_JVM_OPTIONS (WASP<>Upgrade.sh/unix) or the PERFJAVAOPTION (in the WASP<>Upgrade.bat/windows).
    4. If you are editing V6.0 or V6.1 WASPreUpgrade, or editing V6.0 WASPostUpgrade, then follow this step. At the bottom of the file, you should find a command structured similar to the following examples. The exact command depends on the platform and product version. In all cases, the keywords highlighted in these examples for each platform will be present in the WASPreUpgrade and WASPostUpgrade scripts.

      On a Microsoft Windows system:
      • "%JAVA_HOME%\bin\java" -Dcmd.properties.file=%TMPJAVAPROPFILE% %WAS_LOGGING% %CONSOLE_ENCODING% ...
      • On a UNIX or Linux-based system:
        "$JAVA_HOME/bin/java" \
        ...
        -Dcom.ibm.websphere.migration.serverRoot="$WAS_HOME" \
        -Dws.ext.dirs="$WAS_EXT_DIRS" \
        -classpath "$WAS_CLASSPATH" com.ibm.ws.bootstrap.WSLauncher \

    5. Preceding the reference highlighted above for each platform, add the parameter -Xmx512m. The 512 can be increased depending on the amount of memory available on the system.
      • On a Microsoft Windows system:
        "%JAVA_HOME%\bin\java" -Xmx512m -Dcmd.properties.file=%TMPJAVAPROPFILE% %WAS_LOGGING% %CONSOLE_ENCODING% ...

      • On a UNIX or Linux-based system:
        "$JAVA_HOME/bin/java" \
        ...
        -Xmx512m -Dcom.ibm.websphere.migration.serverRoot="$WAS_HOME" \-Dws.ext.dirs="$WAS_EXT_DIRS" \
        -classpath "$WAS_CLASSPATH" com.ibm.ws.bootstrap.WSLauncher \
        ...


        Note: It is highly recommended that the system have at least 512 megabytes more physical memory whatever value is specified. If -Xmx512m is specified, than the system should have at least 1024 megabytes of physical memory. The upper limit value for 32-bit editions of WebSphere Application Server is 2048m. The upper limit value for 64-bit editions is orders of magnitude higher, though it is probably not necessary to use a value higher than 4096m.

    6. Save the WASPreUpgrade.sh/bat and WASPostUpgrade.sh/bat files.

    7. If a previous WASPostUpgrade attempt failed, then return your targeted application server (or application server profile) back to the its state before running the WASPostUpgrade command (for example, use the restoreConfig command to restore the node from backup).

    8. If a previous WASPreUpgrade attempt failed, then wipe out the previous migration backup and run WASPreUpgrade again.

    9. Complete the migration process by running WASPostUpgrade again.

      Note: If the migration fails again due to an Out of Memory condition, then it may be necessary to increase the memory usage further. If the migration tool continues to encounter the Out of Memory condition at the same stage of migration despite the increase in heap size, then the migration tool may have a different problem than described in this technote. In that case, more assistance may be necessary from IBM Support.

    10. (Optional step) Remove the changes made to WASPreUpgrade.sh/bat and WASPostUpgrade.sh/bat

[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Migration","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"9.0;8.5.5;8.5","Edition":"Edition Independent","Line of Business":{"code":"LOB45","label":"Automation"}},{"Product":{"code":"SSNVBF","label":"Runtimes for Java Technology"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Java SDK","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]

Document Information

Modified date:
15 June 2018

UID

swg21199311