Tuesday, November 22, 2016

R12.2 ONLINE PATCHING (ADOP) PARAMETERS/COMMANDS

ADOP PATCHING PARAMETERS DETAILS:
PARAMETERDEFAULT VALUEPOSSIBLE VALUESDESCRIPTION
 phase N/APHASE=PREPARE
PHASE=APPLY
PHASE=CUTOVER
PHASE=CLEANUP
PHASE=FINALIZE
PHASE=ACTUALIZE_ALL
PHASE=FS_CLONE
PHASE=ABORT
These are the eight phases in which adop can run. It is most important and mandatory parameter that is used with adop.
You can also club multiple phases in single command like ‘PHASE=PREPARE,APPLY’ although abort and fs_clone need to be run alone and can’t be clubbed.Standard phases:
prepare – Prepare the instance for patch application.
apply – Apply patches (to the patch edition).
finalize – Ready the instance for cutover. It is run automatically.
cutover – Make the patch edition the new run edition.
cleanup – Drop obsolete objects and data from old editions. It is run automatically.

There are also three special phases, used as per requirement.
Special phases:
abort – Abort the current patching cycle. The abort phase can be run after either the prepare or apply phases have been run, but not after the cutover phase.
actualize_all – Create new copies of all code objects in the patch
edition.
fs_clone – Copy the run file system to the patch file system.
 loglevelLOGLEVEL=EVENTLOGLEVEL=STATEMENT
LOGLEVEL=PROCEDURE
LOGLEVEL=EVENT
LOGLEVEL=WARNING
LOGLEVEL=ERROR
LOGLEVEL=UNEXPECTED
 STATEMENT > for debugging.
PROCEDURE > for debugging high level procedures.
EVENT > to capture informational messages in normal processing. (default)
WARNING > to capture any internal error that is handled by the system and does not affect processing.
ERROR > indicates action failed, need to be reviewed, but the system continue processing.
UNEXPECTED > indicates an unrecoverable error, requires user intervention before processing can continue.
 cleanup_modeCLEANUP_MODE=STANDARDCLEANUP_MODE=FULL
CLEANUP_MODE=STANDARD
CLEANUP_MODE=QUICK
Cleanup processing needs to happen after adop finishes the patching work.Quick mode > shortest execution time, skips non-essential actions
Standard mode > All quick mode action + drops covered objects
Full mode > All quick mode action +  remove all unused code, data, and old editions and takes much longer
 finalize_modeFINALIZE_MODE=QUICKFINALIZE_MODE=QUICK
FINALIZE_MODE=FULL
Quick mode > shortest execution, skips non-essential actions, no gather statistics.
Full mode > Gather statistics, may improve performance after cutover, can take an hour extra to complete.
 input_file N/AINPUT_FILE=<Absolute input_file path> To specify the name of the input_file supplied to adop. (see details on input_file later in this post)
 workers N/A (depends on number of available CPU cores)WORKERS=<User-specified-value>Number of parallel workers used to execute tasks.In earlier released  adpatch used to prompt for number of workers. With adop in R12.2, if you want  to override the default formula that oracle uses now to calculation number of workers, use the WORKERS parameter. Take care that you don’t specify very high number of workers or else adop will fail.
 maxworkers N/A (depends on number of available CPU cores) MAXWORKERS=<User-specified-value>Maximum parallel workers that can be engaged. maxworkers should always be set to greater than the desired number of workers.
 runcontextfile $CONTEXT_FILE RUNCONTEXTFILE=<Absolute context_file path> To specify the non-default context file patch in RUN filesystem
 patchcontextfile Standard context file path in patch FS PATCHCONTEXTFILE=<Absolute context_file path>  To specify the non-default context file patch in PATCH filesystem
 patches N/A PATCHES=<User-specified-value>For Standard Patch:when>patch directory is  a 6- to 8-digit numberPATCHES=<patch_number>

 
For Non-Standard Patch
when
> patch directory is not a 6- to 8-digit number example NLS patches <patch_number>_<language_code>.
> patch driver files are not named *<patchnum>.drv example merged patchesPATCHES=<patch_number>:<driver_file>.drv
This parameter specifies the patches adop needs to apply.Remember the numbered-only patches (standard) and containing-a-colon categories of patch (non-standard) can be mixed.Like:PATCHES= <patch_number1>,  <patch_number2>:<driver_file2>.drv
 defaultsfile $APPL_TOP/admin/<SID>_patch/adalldefaults.txt DEFAULTSFILE=<Absolute defaults_file path>Default file locations on both the run APPL_TOP and patch APPL_TOP is: $APPL_TOP/admin/<SID>_patch/adalldefaults.txtIn case you have created your own defaults file and want to use that instead, then use this parameter.
 patchtop $NE_BASE/EBSapps/patchPATCHTOP=<Absolute patch_location_file path> Default patch_top location is below.$NE_BASE/EBSapps/patchIf you want to keep your patches in some other lcoation, then you need to use this patrameter to let ADOP know where to search for patches pointed by ‘patches’ parameter.If you have a multi-node environment, you must download and unzip the patches (under $APPL_TOP_NE/EBSapps/patch) on the respective nodes.
 merge MERGE=NOMERGE=NO
MERGE=YES
In R12.2, oracle has integrated patch merging action in the patching command itself. In earlier releases we used to first merge patches using admrgpch command.By using MERGE=YES option ADOP will merge all the unified driver files into a single driver file.
 abandon ABANDON=NOABANDON=YES
ABANDON=NO
If the patch you are applying went into error, you have two option when you start the adop utility again.1) you corrected error and want to continue with previous adop session:ABANDON=NO2) you decided that you don’t want to correct issue for now and want to abandon the previous adop session:ABANDON=YES
 restartRESTART=YESRESTART=NO
RESTART=YES
If the patch you are applying went into error and you corrected the issue and want to restart the previous patching session.It is just the reverse of ABANDON parameter.Remember ABANDON and RESTART will always have opposite value.
 flagsN/AFLAGS=AUTOSKIPUse “flags=autoskip” in conjunction with the “abandon=no” parameter at the command-line to skip a failing patching step to “Continue as if a patch were successful”. You need to review the “autoskip” log that gets generated during the patching cycle in order to make sure that their were no errors and to take required actions in case of any errors
 allnodesALLNODES=NOALLNODES=NO
ALLNODES=YES
This parameter comes into picture when you have multi node setup. If you want to run adop on all nodes then use ALLNODES=YES.
 actionACTION=DBACTION=DB
ACTION=NODB
Use this parameter to specify whether to perform database actions or skip. For example if you are in a multi-node environment and adop has already updated the database so when running on other node just use ACTION=NODB to save time.Remember when you are using ‘allnodes=yes’ in a multi-node ‘action=db’ must be specified.
 apply APPLY=YESAPPLY=YES
APPLY=NO
To run adop in test mode (without applying any patches),specify apply=no
 autoskipAUTOSKIP=YESAUTOSKIP=YES
AUTOSKIP=NO
This parameter control whether the user is prompted about skipping actions in non-interactive patching. This is specifically useful when you are applying patches in multi node setup.
 mtrestartMTRESTART=YESMTRESTART=YES
MTRESTART=NO
This parameter specify whether to restart application tier services after cutover phase or not.
 cm_wait CM_WAIT=INFINITE (will wait forever) CM_WAIT=<user_specified_numberSpecifies the number of minutes to wait until the ICM will be forced down.
 allowcoredump ALLOWCOREDUMP=NOALLOWCOREDUMP=NO
ALLOWCOREDUMP=YES
To specify that a core dump will be generated if adop crashes.
 analyticsANALYTICS=NOANALYTICS=NO
ANALYTICS=YES
To specify that a report will be generated that can help debug certian adop issue.
 preinstallN/APREINSTALL=YThis mode is used only if the patch readme instructs. Generally this mode is used during the upgrade process to update AD utilities, apply pre-upgrade patches, or work around other patching issues.It will Compares version numbers, Copies files, Relinks FND and AD executables, Saves patch information. It also runs autoconfig if required.The dual file system in Release 12.2 means that there is no need to shut down application tier services before running AutoConfig.
 -helpN/AN/AShows the help screen.
 -status-status -detailLatest Session-STATUS (for latest session)
-STATUS <SESSION_ID> (for specific session)
Display status of the latest adop session.Use ‘adop -status -detail’ for detailed info
 -examplesN/AN/ADisplays some commonly used adop ample commnads

adop cutover phase status showing failed in the cloned instance.


Run the following steps:


Node Name       Node Type       Phase           Status          Started                        Finished                       Elapsed
--------------- --------------- --------------- --------------- ------------------------------ ------------------------------ ------------
xxxx12          master          PREPARE         COMPLETED       22-NOV-16 04:00:33 +08:00      22-NOV-16 04:06:20 +08:00      0:05:47
                                APPLY           COMPLETED       22-NOV-16 04:44:02 +08:00      22-NOV-16 05:01:25 +08:00      0:17:23
                                FINALIZE        COMPLETED       22-NOV-16 01:11:06 +08:00      22-NOV-16 01:11:17 +08:00      0:00:11
                                CUTOVER         FAILED          22-NOV-16 01:26:09 +08:00      22-NOV-16 03:19:33 +08:00      1:53:24
                                CLEANUP         NOT STARTED

File System Synchronization Used in this Patching Cycle: Full

File System Information:
========================

File System: SINGLE NODE INSTANCE

Node Name   Node Type   Current Base                   Other Base                     Non Editioned Base             Inst Top
----------- ----------- ------------------------------ ------------------------------ ------------------------------ ------------------------------
xxxx12      master      /u01/appluat/appluat/fs2            /u01/appluat/appluat/fs1            /u01/appluat/appluat/fs_ne          /u01/appluat/appluat/fs2/inst/apps/
                                                                                                                     appluat_xxxx
Custom File System Synchronization script: /u01/appluat/appluat/fs_ne/EBSapps/appl/ad/custom/adop_sync.drv


Solution:
[appluat@XXXXXX PATCH]$ adop phase=cutover mtrestart=no

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

Please wait. Validating credentials...
RUN file system context file: /u01/appluat/UAT/fs1/inst/apps/UAT_XXXXXX/appl/admin/UAT_XXXXXX.xml

PATCH file system context file: /u01/appluat/UAT/fs2/inst/apps/UAT_XXXXXX/appl/admin/UAT_XXXXXX.xml
Execute SYSTEM command : df /u01/appluat/UAT/fs2

Worker count determination...

Validation successful. All expected nodes are listed in ADOP_VALID_NODES table.
[EVENT]     [START 2016/02/23 22:58:56] Performing database sanity checks
[EVENT]     [END   2016/02/23 22:58:57] Finished performing database sanity checks
  There is already a session which is incomplete. Details are:
        Session Id            :   12
        Prepare phase status  :   COMPLETED
        Apply phase status    :   COMPLETED
        Cutover  phase status :   RUNNING
        Abort phase status    :   NOT APPLICABLE
        Session status        :   FAILED
  Will continue with previous session
  [START 2016/02/23 22:58:58] adzdoptl.pl run
    ADOP Session ID: 12
    Phase: cutover
    Log file: /u01/appluat/UAT/fs_ne/EBSapps/log/adop/12/adop_20160223_225831.log
    [EVENT]     [START 2016/02/23 22:58:58] Calling task: "Executing txkADOPValidation script on XXXXXX"; script: "/u01/appluat/UAT/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPValidations.pl"; args: " -contextfile=/u01/appluat/UAT/fs1/inst/apps/UAT_XXXXXX/appl/admin/UAT_XXXXXX.xml -patchctxfile=/u01/appluat/UAT/fs2/inst/apps/UAT_XXXXXX/appl/admin/UAT_XXXXXX.xml -phase=cutover -logloc=/u01/appluat/UAT/fs_ne/EBSapps/log/adop/12/cutover_20160223_225831/UAT_XXXXXX -promptmsg=hide"
      [EVENT]     [START 2016/02/23 22:58:58] Executing txkADOPValidation script on XXXXXX
        [EVENT]     Calling: /u01/appluat/UAT/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPValidations.pl
Oracle E-Business Suite Online Patching
RUN edition environment variables have been set.


calling: /u01/appluat/UAT/fs1/FMW_Home/webtier/perl/bin/perl /u01/appluat/UAT/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkValidateSetup.pl -contextfile=/u01/appluat/UAT/fs1/inst/apps/UAT_XXXXXX/appl/admin/UAT_XXXXXX.xml -patchctxfile=/u01/appluat/UAT/fs2/inst/apps/UAT_XXXXXX/appl/admin/UAT_XXXXXX.xml -logloc=/u01/appluat/UAT/fs_ne/EBSapps/log/adop/12/cutover_20160223_225831/UAT_XXXXXX -phase=cutover -promptmsg=hide

Program : txkValidateSetup.pl started @ Tue Feb 23 22:59:01 2016

*** Log File = /u01/appluat/UAT/fs_ne/EBSapps/log/adop/12/cutover_20160223_225831/UAT_XXXXXX/ADOPValidations_Tue_Feb_23_22_59_01_2016.log

SETUP VALIDATION is in progress. This may take few minutes to complete.

Completed execution of all tests.

=========================== START OF VALIDATION REPORT ====================


Validation Results for Node: XXXXXX
---------------------------------------
All validations were successful, please check /u01/appluat/UAT/fs_ne/EBSapps/log/adop/12/cutover_20160223_225831/UAT_XXXXXX/ADOPValidations_Tue_Feb_23_22_59_01_2016.log

=========================== END OF VALIDATION REPORT ======================
      [EVENT]     [END   2016/02/23 22:59:05] Executing txkADOPValidation script on XXXXXX
    [EVENT]     [END   2016/02/23 22:59:05] Executing txkADOPValidation script on XXXXXX
    [EVENT]     [START 2016/02/23 22:59:05] Checking if all nodes were synchronized in the previous apply phase
    [EVENT]     [END   2016/02/23 22:59:05] Checking if all nodes were synchronized in the previous apply phase
    [START 2016/02/23 22:59:05] cutover phase
      [EVENT]     [START 2016/02/23 22:59:07] Performing Cutover Phase steps
        User specified mtrestart=no, hence middle tier service startup will be skipped.
        User would need to bring up the services manually.
      [EVENT]     [END   2016/02/23 22:59:08] Performing Cutover Phase steps
    [END   2016/02/23 22:59:08] cutover phase
    [START 2016/02/23 22:59:08] Generating Post Cutover Reports
    [END   2016/02/23 22:59:08] Generating Post Cutover Reports
    [EVENT]     [START 2016/02/23 22:59:09] Generating AD_ZD_LOGS Report
      [EVENT]     Report: /u01/appluat/UAT/fs1/EBSapps/appl/ad/12.0.0/sql/ADZDSHOWLOG.sql


      [EVENT]     Output: /u01/appluat/UAT/fs_ne/EBSapps/log/adop/12/cutover_20160223_225831/UAT_XXXXXX/adzdshowlog.out
    [EVENT]     [END   2016/02/23 22:59:09] Generating AD_ZD_LOGS Report

Broadcast message from appluat@XXXXXX.myoracle.com (Tue Feb 23 22:59:09 2016):

UAT environment has changed.
All users must re-source the environment using below command:
source /u01/appluat/UAT/EBSapps.env run|patch
  [END   2016/02/23 22:59:09] adzdoptl.pl run
  adop phase=cutover - Completed Successfully

  Log file: /u01/appluat/UAT/fs_ne/EBSapps/log/adop/12/adop_20160223_225831.log

adop exiting with status = 0 (Success)
==============================


Case2:

ADOP Cutover Failed 

Problem Description 
===========
We were trying to apply hrms patches on our test environment, where cutover phase status showing failed. 
However we noticed fs1 & fs2 got swapped but stauts showing FAILED. Please suggest. 

Solution:
=====

1. Ensure that you have taken a backup of your system before applying the recommended steps. 

2. Please take backup of context files from both run fs and patch fs from application tier. 

3. Edit both context files and for line: 

oacore_jvm_start_options oa_var="s_oacore_jvm_start_options">-d64 -Xms512m -Xmx512m -XX:CompileThreshold=8000 -XX:PermSize=128m 

replace it with: 

oacore_jvm_start_options oa_var="s_oacore_jvm_start_options">-d64 -Xms512m -Xmx2048m -XX:CompileThreshold=8000 -XX:PermSize=128m 

save. 

4. Reboot the server. When the server is rebooted, the changes will be activated at system level and the oacore server will take the new Xmx settings. 
 Obs: there is no need to run autoconfig -- as these are OS system settings for OACORE startup. 

5. Restart cutover phase with option: 

adop phase=cutover mtrestart=no 

Obs: this will continue the cutover phase that previously failed, until completion 

6. After the cutover completes, you can restart application tier services manually on the new run fs. 

cd $ADMIN_SCRIPTS_HOME 

adstrtal.sh 

Monday, October 17, 2016

Time is showing incorrect in oracle EBS , reports however SERVER and db time is correct

Issue: Time is showing incorrect in oracle EBS , reports however SERVER and db time is correct

Cause: Walked through steps to check GUI timezone preferences, existing selection was GMT 00 
This indicates there is no discrepancy, GUI time is showing 4 hours behind OS time but timezone GMT is 4 hours behind GST 

Solution: Change timezone to GMT +04 and test

Login With sysadmin user.
Click on preferences.



Verify if the timezone is GMT +00:00, if it is then modify it to GMT +04:00
Confirm the values are saved.

Retest the issue.

Tuesday, September 27, 2016

Error: Pending session of prepare/apply/cutover exists

APPL_TOP is set to /u03/oracle/DEVECS/fs1/EBSapps/appl
[STATEMENT] [START 2016/09/27 09:36:50] Determining admin node
[STATEMENT] [END   2016/09/27 09:36:53] Determining admin node
[STATEMENT] [START 2016/09/27 09:36:55] Acquiring lock on sessions table
[STATEMENT] [END   2016/09/27 09:36:57] Acquiring lock on sessions table
[STATEMENT] [START 2016/09/27 09:36:57] Checking for any pending sessions
[STATEMENT] Cleanup is not done in earlier session
[STATEMENT] [START 2016/09/27 09:37:05] Unlocking sessions table
[STATEMENT] [END   2016/09/27 09:37:06] Unlocking sessions table
[STATEMENT] [START 2016/09/27 09:37:06] adzdoptl.pl run
  [STATEMENT] Session ID: 8
  [STATEMENT] Phase: cleanup
  [STATEMENT] Log file: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/adop_20160927_093619.log
  [ERROR]     Error: Pending session of prepare/apply/cutover exists
  [ERROR]     Hint: Before trying cleanup complete the pending session
  [STATEMENT] [START 2016/09/27 09:37:17] Unlocking sessions table
  [STATEMENT] [END   2016/09/27 09:37:19] Unlocking sessions table
  [STATEMENT] Log file: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/adop_20160927_093619.log

adop exiting with status = 2 (Fail)

Issue:

1. The adop_phase prepare was run into errors, after the there was a patch applied to the RUN environment in the hotpatch mode, post applying the patch whenever running the prepare phase it is running into errors, hence we aborted the adop phase, after completing the abort we had to run the cleanup phase which was running into errors 
ADOP_SESSION_ID PREPARE_STATUS APPLY_STATUS FINALIZE_STATUS CUTOVER_STATUS CLEANUP_STATUS ABORT_STATUS STATUS
2 R N N N Y Y C
4 R N N N Y Y C
8 R N N N N Y C
6 R N N N Y Y C
7 X Y N X Y X C
5 R N N N Y Y C

Prepare status is still showing as R (Running) hence the cleanup phase is not completing successfully,

Solution:

Update the value of prepare_status with Y in table ad_adop_sessions, and retry the adop cleanup phase

update table ad_adop_sessions set prepare_status='Y' where prepare_status='R';

COMMIT;

Retry the adop_cleanup phase it will complete.

[applmgr@xxxxxxxx ~]$ adop phase=cleanup cleanup_mode=full

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

 Please wait. Validating credentials...


ERROR: Invalid apps user credentials.
Warning: Your system may have a limit on the number of failed login attempts before the account is locked out.



Enter the APPS password:
Enter the APPS password:

 Please wait. Validating credentials...


Enter the RUN file system context file name [/u03/oracle/DEVECS/fs1/inst/apps/DEVECS_xxxxxxxx/appl/admin/DEVECS_xxxxxxxx.xml]:

Enter the PATCH file system context file name [/u03/oracle/DEVECS/fs2/inst/apps/DEVECS_xxxxxxxx/appl/admin/DEVECS_xxxxxxxx.xml]:

[STATEMENT] [END   2016/09/27 09:46:03] Performing verification of parameters
[STATEMENT] [START 2016/09/27 09:46:04] Checking for the required ENV setup
[STATEMENT] [END   2016/09/27 09:46:04] Checking for the required ENV setup

************* Start of  session *************
 version: 12.2.0
 started at: Tue Sep 27 2016 09:46:04

APPL_TOP is set to /u03/oracle/DEVECS/fs1/EBSapps/appl
[STATEMENT] [START 2016/09/27 09:46:04] Determining admin node
[STATEMENT] [END   2016/09/27 09:46:06] Determining admin node
[STATEMENT] [START 2016/09/27 09:46:09] Acquiring lock on sessions table
[STATEMENT] [END   2016/09/27 09:46:11] Acquiring lock on sessions table
[STATEMENT] [START 2016/09/27 09:46:11] Checking for any pending sessions
[STATEMENT] Cleanup is not done in earlier session
[STATEMENT] [START 2016/09/27 09:46:19] Unlocking sessions table
[STATEMENT] [END   2016/09/27 09:46:20] Unlocking sessions table
[STATEMENT] [START 2016/09/27 09:46:20] adzdoptl.pl run
  [STATEMENT] Session ID: 8
  [STATEMENT] Phase: cleanup
  [STATEMENT] Log file: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/adop_20160927_094523.log
  [STATEMENT] [START 2016/09/27 09:46:34] cleanup phase
    [STATEMENT] CLEANUP Phase START TIME: 27-09-2016 09:46:36
    [STATEMENT] [START 2016/09/27 09:46:39] Performing Cleanup steps
      [STATEMENT] [START 2016/09/27 09:46:41] Running CLEANUP ddls in ddl handler table
        [STATEMENT] [Start 2016/09/27 09:46:43] ADOP::DefaultsFileUtil::substituteDefaultsFileTokensForPatchApplTop
          [STATEMENT] Reading defaults file /u03/oracle/DEVECS/fs1/EBSapps/appl/admin/DEVECS/adalldefaults.txt
          [STATEMENT] Substituting tokens ...
          [STATEMENT] Regenerating defaults file ...
        [STATEMENT] [End   2016/09/27 09:46:43] ADOP::DefaultsFileUtil::substituteDefaultsFileTokensForPatchApplTop
        [STATEMENT] Calling: adpatch options=hotpatch,nocompiledb interactive=no console=no workers=2 restart=no abandon=yes defaultsfile=/u03/oracle/DEVECS/fs1/EBSapps/appl/admin/DEVECS/adalldefaults.txt patchtop=/u03/oracle/DEVECS/fs1/EBSapps/appl/ad/12.0.0/patch/115/driver logfile=cleanup.log driver=ucleanup.drv
        [STATEMENT] ADPATCH Log directory: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/cleanup_20160927_094523/DEVECS_xxxxxxxx/log
        [STATEMENT] Running: adpatch options=hotpatch,nocompiledb interactive=no console=no workers=2 restart=no abandon=yes defaultsfile=/u03/oracle/DEVECS/fs1/EBSapps/appl/admin/DEVECS/adalldefaults.txt patchtop=/u03/oracle/DEVECS/fs1/EBSapps/appl/ad/12.0.0/patch/115/driver logfile=cleanup.log driver=ucleanup.drv stdin=yes

Successfully created out directory
      [STATEMENT] [END   2016/09/27 09:47:44] Running CLEANUP ddls in ddl handler table
      [STATEMENT] [WARN] Phase cleanup_full is currently not supported from adop.
      [STATEMENT]        For more information please refer to the adop documentation.
      [STATEMENT]        Proceeding with normal cleanup mode
        [STATEMENT] [START 2016/09/27 09:47:50] Generating Report to display all the DDLs from ad_zd_ddl_handler
          [STATEMENT] Report: /u03/oracle/DEVECS/fs1/EBSapps/appl/ad/12.0.0/sql/ADZDALLDDLS.sql

          [STATEMENT] Output: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/cleanup_20160927_094523/DEVECS_xxxxxxxx/adzdallddls.out

        [STATEMENT] [END   2016/09/27 09:47:54] Generating Report to display all the DDLs from ad_zd_ddl_handler
      [STATEMENT] Calling cleanup in STANDARD mode
      [STATEMENT] [WARN] Cleanup may take a while. Please wait.
    [STATEMENT] [END   2016/09/27 09:48:02] Performing Cleanup steps
      [STATEMENT] [START 2016/09/27 09:48:04] Generating Report to display the cleanup status
        [STATEMENT] Report: /u03/oracle/DEVECS/fs1/EBSapps/appl/ad/12.0.0/sql/ADZDCLEANUPRP.sql

        [STATEMENT] Output: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/cleanup_20160927_094523/DEVECS_xxxxxxxx/adzdcleanuprp.out

      [STATEMENT] [END   2016/09/27 09:50:05] Generating Report to display the cleanup status
    [STATEMENT] CLEANUP Phase END TIME: 27-09-2016 09:50:11
  [STATEMENT] [END   2016/09/27 09:50:13] cleanup phase
    [STATEMENT] [START 2016/09/27 09:50:18] Generating Report to spool all logs from ad_zd_logs
      [STATEMENT] Report: /u03/oracle/DEVECS/fs1/EBSapps/appl/ad/12.0.0/sql/ADZDSHOWLOG.sql

      [STATEMENT] Output: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/cleanup_20160927_094523/DEVECS_xxxxxxxx/adzdshowlog.out

    [STATEMENT] [END   2016/09/27 09:50:23] Generating Report to spool all logs from ad_zd_logs
[STATEMENT] [END   2016/09/27 09:50:24] adzdoptl.pl run
[STATEMENT] adop phase=cleanup - Completed Successfully

[STATEMENT] Log file: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/adop_20160927_094523.log

adop exiting with status = 0 (Success)