APPL_TOP is set to /u03/oracle/DEVECS/fs1/EBSapps/appl
[STATEMENT] [START 2016/09/27 09:36:50] Determining admin node
[STATEMENT] [END 2016/09/27 09:36:53] Determining admin node
[STATEMENT] [START 2016/09/27 09:36:55] Acquiring lock on sessions table
[STATEMENT] [END 2016/09/27 09:36:57] Acquiring lock on sessions table
[STATEMENT] [START 2016/09/27 09:36:57] Checking for any pending sessions
[STATEMENT] Cleanup is not done in earlier session
[STATEMENT] [START 2016/09/27 09:37:05] Unlocking sessions table
[STATEMENT] [END 2016/09/27 09:37:06] Unlocking sessions table
[STATEMENT] [START 2016/09/27 09:37:06] adzdoptl.pl run
[STATEMENT] Session ID: 8
[STATEMENT] Phase: cleanup
[STATEMENT] Log file: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/adop_20160927_093619.log
[ERROR] Error: Pending session of prepare/apply/cutover exists
[ERROR] Hint: Before trying cleanup complete the pending session
[STATEMENT] [START 2016/09/27 09:37:17] Unlocking sessions table
[STATEMENT] [END 2016/09/27 09:37:19] Unlocking sessions table
[STATEMENT] Log file: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/adop_20160927_093619.log
adop exiting with status = 2 (Fail)
Prepare status is still showing as R (Running) hence the cleanup phase is not completing successfully,
Solution:
Update the value of prepare_status with Y in table ad_adop_sessions, and retry the adop cleanup phase
update table ad_adop_sessions set prepare_status='Y' where prepare_status='R';
COMMIT;
Retry the adop_cleanup phase it will complete.
[applmgr@xxxxxxxx ~]$ adop phase=cleanup cleanup_mode=full
Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:
Please wait. Validating credentials...
ERROR: Invalid apps user credentials.
Warning: Your system may have a limit on the number of failed login attempts before the account is locked out.
Enter the APPS password:
Enter the APPS password:
Please wait. Validating credentials...
Enter the RUN file system context file name [/u03/oracle/DEVECS/fs1/inst/apps/DEVECS_xxxxxxxx/appl/admin/DEVECS_xxxxxxxx.xml]:
Enter the PATCH file system context file name [/u03/oracle/DEVECS/fs2/inst/apps/DEVECS_xxxxxxxx/appl/admin/DEVECS_xxxxxxxx.xml]:
[STATEMENT] [END 2016/09/27 09:46:03] Performing verification of parameters
[STATEMENT] [START 2016/09/27 09:46:04] Checking for the required ENV setup
[STATEMENT] [END 2016/09/27 09:46:04] Checking for the required ENV setup
************* Start of session *************
version: 12.2.0
started at: Tue Sep 27 2016 09:46:04
APPL_TOP is set to /u03/oracle/DEVECS/fs1/EBSapps/appl
[STATEMENT] [START 2016/09/27 09:46:04] Determining admin node
[STATEMENT] [END 2016/09/27 09:46:06] Determining admin node
[STATEMENT] [START 2016/09/27 09:46:09] Acquiring lock on sessions table
[STATEMENT] [END 2016/09/27 09:46:11] Acquiring lock on sessions table
[STATEMENT] [START 2016/09/27 09:46:11] Checking for any pending sessions
[STATEMENT] Cleanup is not done in earlier session
[STATEMENT] [START 2016/09/27 09:46:19] Unlocking sessions table
[STATEMENT] [END 2016/09/27 09:46:20] Unlocking sessions table
[STATEMENT] [START 2016/09/27 09:46:20] adzdoptl.pl run
[STATEMENT] Session ID: 8
[STATEMENT] Phase: cleanup
[STATEMENT] Log file: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/adop_20160927_094523.log
[STATEMENT] [START 2016/09/27 09:46:34] cleanup phase
[STATEMENT] CLEANUP Phase START TIME: 27-09-2016 09:46:36
[STATEMENT] [START 2016/09/27 09:46:39] Performing Cleanup steps
[STATEMENT] [START 2016/09/27 09:46:41] Running CLEANUP ddls in ddl handler table
[STATEMENT] [Start 2016/09/27 09:46:43] ADOP::DefaultsFileUtil::substituteDefaultsFileTokensForPatchApplTop
[STATEMENT] Reading defaults file /u03/oracle/DEVECS/fs1/EBSapps/appl/admin/DEVECS/adalldefaults.txt
[STATEMENT] Substituting tokens ...
[STATEMENT] Regenerating defaults file ...
[STATEMENT] [End 2016/09/27 09:46:43] ADOP::DefaultsFileUtil::substituteDefaultsFileTokensForPatchApplTop
[STATEMENT] Calling: adpatch options=hotpatch,nocompiledb interactive=no console=no workers=2 restart=no abandon=yes defaultsfile=/u03/oracle/DEVECS/fs1/EBSapps/appl/admin/DEVECS/adalldefaults.txt patchtop=/u03/oracle/DEVECS/fs1/EBSapps/appl/ad/12.0.0/patch/115/driver logfile=cleanup.log driver=ucleanup.drv
[STATEMENT] ADPATCH Log directory: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/cleanup_20160927_094523/DEVECS_xxxxxxxx/log
[STATEMENT] Running: adpatch options=hotpatch,nocompiledb interactive=no console=no workers=2 restart=no abandon=yes defaultsfile=/u03/oracle/DEVECS/fs1/EBSapps/appl/admin/DEVECS/adalldefaults.txt patchtop=/u03/oracle/DEVECS/fs1/EBSapps/appl/ad/12.0.0/patch/115/driver logfile=cleanup.log driver=ucleanup.drv stdin=yes
Successfully created out directory
[STATEMENT] [END 2016/09/27 09:47:44] Running CLEANUP ddls in ddl handler table
[STATEMENT] [WARN] Phase cleanup_full is currently not supported from adop.
[STATEMENT] For more information please refer to the adop documentation.
[STATEMENT] Proceeding with normal cleanup mode
[STATEMENT] [START 2016/09/27 09:47:50] Generating Report to display all the DDLs from ad_zd_ddl_handler
[STATEMENT] Report: /u03/oracle/DEVECS/fs1/EBSapps/appl/ad/12.0.0/sql/ADZDALLDDLS.sql
[STATEMENT] Output: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/cleanup_20160927_094523/DEVECS_xxxxxxxx/adzdallddls.out
[STATEMENT] [END 2016/09/27 09:47:54] Generating Report to display all the DDLs from ad_zd_ddl_handler
[STATEMENT] Calling cleanup in STANDARD mode
[STATEMENT] [WARN] Cleanup may take a while. Please wait.
[STATEMENT] [END 2016/09/27 09:48:02] Performing Cleanup steps
[STATEMENT] [START 2016/09/27 09:48:04] Generating Report to display the cleanup status
[STATEMENT] Report: /u03/oracle/DEVECS/fs1/EBSapps/appl/ad/12.0.0/sql/ADZDCLEANUPRP.sql
[STATEMENT] Output: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/cleanup_20160927_094523/DEVECS_xxxxxxxx/adzdcleanuprp.out
[STATEMENT] [END 2016/09/27 09:50:05] Generating Report to display the cleanup status
[STATEMENT] CLEANUP Phase END TIME: 27-09-2016 09:50:11
[STATEMENT] [END 2016/09/27 09:50:13] cleanup phase
[STATEMENT] [START 2016/09/27 09:50:18] Generating Report to spool all logs from ad_zd_logs
[STATEMENT] Report: /u03/oracle/DEVECS/fs1/EBSapps/appl/ad/12.0.0/sql/ADZDSHOWLOG.sql
[STATEMENT] Output: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/cleanup_20160927_094523/DEVECS_xxxxxxxx/adzdshowlog.out
[STATEMENT] [END 2016/09/27 09:50:23] Generating Report to spool all logs from ad_zd_logs
[STATEMENT] [END 2016/09/27 09:50:24] adzdoptl.pl run
[STATEMENT] adop phase=cleanup - Completed Successfully
[STATEMENT] Log file: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/adop_20160927_094523.log
adop exiting with status = 0 (Success)
[STATEMENT] [START 2016/09/27 09:36:50] Determining admin node
[STATEMENT] [END 2016/09/27 09:36:53] Determining admin node
[STATEMENT] [START 2016/09/27 09:36:55] Acquiring lock on sessions table
[STATEMENT] [END 2016/09/27 09:36:57] Acquiring lock on sessions table
[STATEMENT] [START 2016/09/27 09:36:57] Checking for any pending sessions
[STATEMENT] Cleanup is not done in earlier session
[STATEMENT] [START 2016/09/27 09:37:05] Unlocking sessions table
[STATEMENT] [END 2016/09/27 09:37:06] Unlocking sessions table
[STATEMENT] [START 2016/09/27 09:37:06] adzdoptl.pl run
[STATEMENT] Session ID: 8
[STATEMENT] Phase: cleanup
[STATEMENT] Log file: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/adop_20160927_093619.log
[ERROR] Error: Pending session of prepare/apply/cutover exists
[ERROR] Hint: Before trying cleanup complete the pending session
[STATEMENT] [START 2016/09/27 09:37:17] Unlocking sessions table
[STATEMENT] [END 2016/09/27 09:37:19] Unlocking sessions table
[STATEMENT] Log file: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/adop_20160927_093619.log
adop exiting with status = 2 (Fail)
Issue:
1. The adop_phase prepare was run into errors, after the there was a patch applied to the RUN environment in the hotpatch mode, post applying the patch whenever running the prepare phase it is running into errors, hence we aborted the adop phase, after completing the abort we had to run the cleanup phase which was running into errors
1. The adop_phase prepare was run into errors, after the there was a patch applied to the RUN environment in the hotpatch mode, post applying the patch whenever running the prepare phase it is running into errors, hence we aborted the adop phase, after completing the abort we had to run the cleanup phase which was running into errors
| ADOP_SESSION_ID | PREPARE_STATUS | APPLY_STATUS | FINALIZE_STATUS | CUTOVER_STATUS | CLEANUP_STATUS | ABORT_STATUS | STATUS |
| 2 | R | N | N | N | Y | Y | C |
| 4 | R | N | N | N | Y | Y | C |
| 8 | R | N | N | N | N | Y | C |
| 6 | R | N | N | N | Y | Y | C |
| 7 | X | Y | N | X | Y | X | C |
| 5 | R | N | N | N | Y | Y | C |
Prepare status is still showing as R (Running) hence the cleanup phase is not completing successfully,
Solution:
Update the value of prepare_status with Y in table ad_adop_sessions, and retry the adop cleanup phase
update table ad_adop_sessions set prepare_status='Y' where prepare_status='R';
COMMIT;
Retry the adop_cleanup phase it will complete.
[applmgr@xxxxxxxx ~]$ adop phase=cleanup cleanup_mode=full
Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:
Please wait. Validating credentials...
ERROR: Invalid apps user credentials.
Warning: Your system may have a limit on the number of failed login attempts before the account is locked out.
Enter the APPS password:
Enter the APPS password:
Please wait. Validating credentials...
Enter the RUN file system context file name [/u03/oracle/DEVECS/fs1/inst/apps/DEVECS_xxxxxxxx/appl/admin/DEVECS_xxxxxxxx.xml]:
Enter the PATCH file system context file name [/u03/oracle/DEVECS/fs2/inst/apps/DEVECS_xxxxxxxx/appl/admin/DEVECS_xxxxxxxx.xml]:
[STATEMENT] [END 2016/09/27 09:46:03] Performing verification of parameters
[STATEMENT] [START 2016/09/27 09:46:04] Checking for the required ENV setup
[STATEMENT] [END 2016/09/27 09:46:04] Checking for the required ENV setup
************* Start of session *************
version: 12.2.0
started at: Tue Sep 27 2016 09:46:04
APPL_TOP is set to /u03/oracle/DEVECS/fs1/EBSapps/appl
[STATEMENT] [START 2016/09/27 09:46:04] Determining admin node
[STATEMENT] [END 2016/09/27 09:46:06] Determining admin node
[STATEMENT] [START 2016/09/27 09:46:09] Acquiring lock on sessions table
[STATEMENT] [END 2016/09/27 09:46:11] Acquiring lock on sessions table
[STATEMENT] [START 2016/09/27 09:46:11] Checking for any pending sessions
[STATEMENT] Cleanup is not done in earlier session
[STATEMENT] [START 2016/09/27 09:46:19] Unlocking sessions table
[STATEMENT] [END 2016/09/27 09:46:20] Unlocking sessions table
[STATEMENT] [START 2016/09/27 09:46:20] adzdoptl.pl run
[STATEMENT] Session ID: 8
[STATEMENT] Phase: cleanup
[STATEMENT] Log file: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/adop_20160927_094523.log
[STATEMENT] [START 2016/09/27 09:46:34] cleanup phase
[STATEMENT] CLEANUP Phase START TIME: 27-09-2016 09:46:36
[STATEMENT] [START 2016/09/27 09:46:39] Performing Cleanup steps
[STATEMENT] [START 2016/09/27 09:46:41] Running CLEANUP ddls in ddl handler table
[STATEMENT] [Start 2016/09/27 09:46:43] ADOP::DefaultsFileUtil::substituteDefaultsFileTokensForPatchApplTop
[STATEMENT] Reading defaults file /u03/oracle/DEVECS/fs1/EBSapps/appl/admin/DEVECS/adalldefaults.txt
[STATEMENT] Substituting tokens ...
[STATEMENT] Regenerating defaults file ...
[STATEMENT] [End 2016/09/27 09:46:43] ADOP::DefaultsFileUtil::substituteDefaultsFileTokensForPatchApplTop
[STATEMENT] Calling: adpatch options=hotpatch,nocompiledb interactive=no console=no workers=2 restart=no abandon=yes defaultsfile=/u03/oracle/DEVECS/fs1/EBSapps/appl/admin/DEVECS/adalldefaults.txt patchtop=/u03/oracle/DEVECS/fs1/EBSapps/appl/ad/12.0.0/patch/115/driver logfile=cleanup.log driver=ucleanup.drv
[STATEMENT] ADPATCH Log directory: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/cleanup_20160927_094523/DEVECS_xxxxxxxx/log
[STATEMENT] Running: adpatch options=hotpatch,nocompiledb interactive=no console=no workers=2 restart=no abandon=yes defaultsfile=/u03/oracle/DEVECS/fs1/EBSapps/appl/admin/DEVECS/adalldefaults.txt patchtop=/u03/oracle/DEVECS/fs1/EBSapps/appl/ad/12.0.0/patch/115/driver logfile=cleanup.log driver=ucleanup.drv stdin=yes
Successfully created out directory
[STATEMENT] [END 2016/09/27 09:47:44] Running CLEANUP ddls in ddl handler table
[STATEMENT] [WARN] Phase cleanup_full is currently not supported from adop.
[STATEMENT] For more information please refer to the adop documentation.
[STATEMENT] Proceeding with normal cleanup mode
[STATEMENT] [START 2016/09/27 09:47:50] Generating Report to display all the DDLs from ad_zd_ddl_handler
[STATEMENT] Report: /u03/oracle/DEVECS/fs1/EBSapps/appl/ad/12.0.0/sql/ADZDALLDDLS.sql
[STATEMENT] Output: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/cleanup_20160927_094523/DEVECS_xxxxxxxx/adzdallddls.out
[STATEMENT] [END 2016/09/27 09:47:54] Generating Report to display all the DDLs from ad_zd_ddl_handler
[STATEMENT] Calling cleanup in STANDARD mode
[STATEMENT] [WARN] Cleanup may take a while. Please wait.
[STATEMENT] [END 2016/09/27 09:48:02] Performing Cleanup steps
[STATEMENT] [START 2016/09/27 09:48:04] Generating Report to display the cleanup status
[STATEMENT] Report: /u03/oracle/DEVECS/fs1/EBSapps/appl/ad/12.0.0/sql/ADZDCLEANUPRP.sql
[STATEMENT] Output: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/cleanup_20160927_094523/DEVECS_xxxxxxxx/adzdcleanuprp.out
[STATEMENT] [END 2016/09/27 09:50:05] Generating Report to display the cleanup status
[STATEMENT] CLEANUP Phase END TIME: 27-09-2016 09:50:11
[STATEMENT] [END 2016/09/27 09:50:13] cleanup phase
[STATEMENT] [START 2016/09/27 09:50:18] Generating Report to spool all logs from ad_zd_logs
[STATEMENT] Report: /u03/oracle/DEVECS/fs1/EBSapps/appl/ad/12.0.0/sql/ADZDSHOWLOG.sql
[STATEMENT] Output: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/cleanup_20160927_094523/DEVECS_xxxxxxxx/adzdshowlog.out
[STATEMENT] [END 2016/09/27 09:50:23] Generating Report to spool all logs from ad_zd_logs
[STATEMENT] [END 2016/09/27 09:50:24] adzdoptl.pl run
[STATEMENT] adop phase=cleanup - Completed Successfully
[STATEMENT] Log file: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/adop_20160927_094523.log
adop exiting with status = 0 (Success)
