Error: Pending session of prepare/apply/cutover exists

APPL_TOP is set to /u03/oracle/DEVECS/fs1/EBSapps/appl
[STATEMENT] [START 2016/09/27 09:36:50] Determining admin node
[STATEMENT] [END   2016/09/27 09:36:53] Determining admin node
[STATEMENT] [START 2016/09/27 09:36:55] Acquiring lock on sessions table
[STATEMENT] [END   2016/09/27 09:36:57] Acquiring lock on sessions table
[STATEMENT] [START 2016/09/27 09:36:57] Checking for any pending sessions
[STATEMENT] Cleanup is not done in earlier session
[STATEMENT] [START 2016/09/27 09:37:05] Unlocking sessions table
[STATEMENT] [END   2016/09/27 09:37:06] Unlocking sessions table
[STATEMENT] [START 2016/09/27 09:37:06] adzdoptl.pl run
  [STATEMENT] Session ID: 8
  [STATEMENT] Phase: cleanup
  [STATEMENT] Log file: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/adop_20160927_093619.log
  [ERROR]     Error: Pending session of prepare/apply/cutover exists
  [ERROR]     Hint: Before trying cleanup complete the pending session
  [STATEMENT] [START 2016/09/27 09:37:17] Unlocking sessions table
  [STATEMENT] [END   2016/09/27 09:37:19] Unlocking sessions table
  [STATEMENT] Log file: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/adop_20160927_093619.log

adop exiting with status = 2 (Fail)

Issue:

1. The adop_phase prepare was run into errors, after the there was a patch applied to the RUN environment in the hotpatch mode, post applying the patch whenever running the prepare phase it is running into errors, hence we aborted the adop phase, after completing the abort we had to run the cleanup phase which was running into errors 

ADOP_SESSION_ID	PREPARE_STATUS	APPLY_STATUS	FINALIZE_STATUS	CUTOVER_STATUS	CLEANUP_STATUS	ABORT_STATUS	STATUS
2	R	N	N	N	Y	Y	C
4	R	N	N	N	Y	Y	C
8	R	N	N	N	N	Y	C
6	R	N	N	N	Y	Y	C
7	X	Y	N	X	Y	X	C
5	R	N	N	N	Y	Y	C

Prepare status is still showing as R (Running) hence the cleanup phase is not completing successfully,

Solution:

Update the value of prepare_status with Y in table ad_adop_sessions, and retry the adop cleanup phase

update table ad_adop_sessions set prepare_status='Y' where prepare_status='R';

COMMIT;

Retry the adop_cleanup phase it will complete.

[applmgr@xxxxxxxx ~]$ adop phase=cleanup cleanup_mode=full

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

 Please wait. Validating credentials...


ERROR: Invalid apps user credentials.
Warning: Your system may have a limit on the number of failed login attempts before the account is locked out.



Enter the APPS password:
Enter the APPS password:

 Please wait. Validating credentials...


Enter the RUN file system context file name [/u03/oracle/DEVECS/fs1/inst/apps/DEVECS_xxxxxxxx/appl/admin/DEVECS_xxxxxxxx.xml]:

Enter the PATCH file system context file name [/u03/oracle/DEVECS/fs2/inst/apps/DEVECS_xxxxxxxx/appl/admin/DEVECS_xxxxxxxx.xml]:

[STATEMENT] [END   2016/09/27 09:46:03] Performing verification of parameters
[STATEMENT] [START 2016/09/27 09:46:04] Checking for the required ENV setup
[STATEMENT] [END   2016/09/27 09:46:04] Checking for the required ENV setup

************* Start of  session *************
 version: 12.2.0
 started at: Tue Sep 27 2016 09:46:04

APPL_TOP is set to /u03/oracle/DEVECS/fs1/EBSapps/appl
[STATEMENT] [START 2016/09/27 09:46:04] Determining admin node
[STATEMENT] [END   2016/09/27 09:46:06] Determining admin node
[STATEMENT] [START 2016/09/27 09:46:09] Acquiring lock on sessions table
[STATEMENT] [END   2016/09/27 09:46:11] Acquiring lock on sessions table
[STATEMENT] [START 2016/09/27 09:46:11] Checking for any pending sessions
[STATEMENT] Cleanup is not done in earlier session
[STATEMENT] [START 2016/09/27 09:46:19] Unlocking sessions table
[STATEMENT] [END   2016/09/27 09:46:20] Unlocking sessions table
[STATEMENT] [START 2016/09/27 09:46:20] adzdoptl.pl run
  [STATEMENT] Session ID: 8
  [STATEMENT] Phase: cleanup
  [STATEMENT] Log file: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/adop_20160927_094523.log
  [STATEMENT] [START 2016/09/27 09:46:34] cleanup phase
    [STATEMENT] CLEANUP Phase START TIME: 27-09-2016 09:46:36
    [STATEMENT] [START 2016/09/27 09:46:39] Performing Cleanup steps
      [STATEMENT] [START 2016/09/27 09:46:41] Running CLEANUP ddls in ddl handler table
        [STATEMENT] [Start 2016/09/27 09:46:43] ADOP::DefaultsFileUtil::substituteDefaultsFileTokensForPatchApplTop
          [STATEMENT] Reading defaults file /u03/oracle/DEVECS/fs1/EBSapps/appl/admin/DEVECS/adalldefaults.txt
          [STATEMENT] Substituting tokens ...
          [STATEMENT] Regenerating defaults file ...
        [STATEMENT] [End   2016/09/27 09:46:43] ADOP::DefaultsFileUtil::substituteDefaultsFileTokensForPatchApplTop
        [STATEMENT] Calling: adpatch options=hotpatch,nocompiledb interactive=no console=no workers=2 restart=no abandon=yes defaultsfile=/u03/oracle/DEVECS/fs1/EBSapps/appl/admin/DEVECS/adalldefaults.txt patchtop=/u03/oracle/DEVECS/fs1/EBSapps/appl/ad/12.0.0/patch/115/driver logfile=cleanup.log driver=ucleanup.drv
        [STATEMENT] ADPATCH Log directory: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/cleanup_20160927_094523/DEVECS_xxxxxxxx/log
        [STATEMENT] Running: adpatch options=hotpatch,nocompiledb interactive=no console=no workers=2 restart=no abandon=yes defaultsfile=/u03/oracle/DEVECS/fs1/EBSapps/appl/admin/DEVECS/adalldefaults.txt patchtop=/u03/oracle/DEVECS/fs1/EBSapps/appl/ad/12.0.0/patch/115/driver logfile=cleanup.log driver=ucleanup.drv stdin=yes

Successfully created out directory
      [STATEMENT] [END   2016/09/27 09:47:44] Running CLEANUP ddls in ddl handler table
      [STATEMENT] [WARN] Phase cleanup_full is currently not supported from adop.
      [STATEMENT]        For more information please refer to the adop documentation.
      [STATEMENT]        Proceeding with normal cleanup mode
        [STATEMENT] [START 2016/09/27 09:47:50] Generating Report to display all the DDLs from ad_zd_ddl_handler
          [STATEMENT] Report: /u03/oracle/DEVECS/fs1/EBSapps/appl/ad/12.0.0/sql/ADZDALLDDLS.sql

          [STATEMENT] Output: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/cleanup_20160927_094523/DEVECS_xxxxxxxx/adzdallddls.out

        [STATEMENT] [END   2016/09/27 09:47:54] Generating Report to display all the DDLs from ad_zd_ddl_handler
      [STATEMENT] Calling cleanup in STANDARD mode
      [STATEMENT] [WARN] Cleanup may take a while. Please wait.
    [STATEMENT] [END   2016/09/27 09:48:02] Performing Cleanup steps
      [STATEMENT] [START 2016/09/27 09:48:04] Generating Report to display the cleanup status
        [STATEMENT] Report: /u03/oracle/DEVECS/fs1/EBSapps/appl/ad/12.0.0/sql/ADZDCLEANUPRP.sql

        [STATEMENT] Output: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/cleanup_20160927_094523/DEVECS_xxxxxxxx/adzdcleanuprp.out

      [STATEMENT] [END   2016/09/27 09:50:05] Generating Report to display the cleanup status
    [STATEMENT] CLEANUP Phase END TIME: 27-09-2016 09:50:11
  [STATEMENT] [END   2016/09/27 09:50:13] cleanup phase
    [STATEMENT] [START 2016/09/27 09:50:18] Generating Report to spool all logs from ad_zd_logs
      [STATEMENT] Report: /u03/oracle/DEVECS/fs1/EBSapps/appl/ad/12.0.0/sql/ADZDSHOWLOG.sql

      [STATEMENT] Output: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/cleanup_20160927_094523/DEVECS_xxxxxxxx/adzdshowlog.out

    [STATEMENT] [END   2016/09/27 09:50:23] Generating Report to spool all logs from ad_zd_logs
[STATEMENT] [END   2016/09/27 09:50:24] adzdoptl.pl run
[STATEMENT] adop phase=cleanup - Completed Successfully

[STATEMENT] Log file: /u03/oracle/DEVECS/fs_ne/EBSapps/log/adop/8/adop_20160927_094523.log

adop exiting with status = 0 (Success)

Metrics Global Cache Blocks lost is at 11

Undesirable Global Cache Statistics
The following are undesirable statistics, or statistics for which the values should always be zero or near-zero.

global cache blocks lost--

This statistic reveals any block losses during transfers and high values may indicate network problems. When using an unreliable IPC protocol such as UDP, the value for 'global cache blocks lost' may be non-zero. If this is the case, then the ratio of 'global cache blocks lost' divided by 'global cache current blocks served' plus 'global cache cr blocks served' should be as small as possible. A non-zero value for 'global cache blocks lost' does not necessarily indicate a problem, because Oracle retries the block transfer operation until it is successful. Unlike TCP/IP, UDP/IP is considered unreliable because UDP/IP provides very few error recovery services.

global cache blocks corrupt--

This statistic indicates whether any blocks were corrupted during transfers. High values for this statistic indicate an IPC, network, or hardware problem. "

Getting "Reached maximum capacity of pool" messages in logs

Issue: Unable to login to Oracle VM Manager. Getting the following error:

Error in the Admin Server Log:

####<2015-02-28T08:52:55.833+0530> <Info> <Common> <servername> <AdminServer> <Odof Tcp Client Thread: /127.x.x.x:54321/2795858> <<anonymous>> <> <1b95d8d1-24dd-4acc-a6f3-4c025158e6a7-000000
03> <1425093775833> <BEA-000627> <Reached maximum capacity of pool "poolname", making "0" new resource instances instead of "1".>
####<2015-02-28T08:52:55.896+0530> <Error> <com.oracle.odof.core.storage.RelationalStore> <servername> <AdminServer> <Odof Tcp Client Thread: /127.x.x.x:54321/2795865> <<anonymous>> <> <1b95
d8d1-24dd-4acc-a6f3-4c025158e6a7-00000003> <1425093775896> <BEA-000000> <Waiting for connection (7 of 720 tries)>
####<2015-02-28T08:52:58.446+0530> <Info> <Common> <servername> <AdminServer> <Odof Tcp Client Thread: /127.x.x.x:54321/2795898> <<anonymous>> <> <1b95d8d1-24dd-4acc-a6f3-4c025158e6a7-000000
03> <1425093778446> <BEA-000627> <Reached maximum capacity of pool "poolname", making "0" new resource instances instead of "1".>
####<2015-02-28T08:52:58.469+0530> <Error> <com.oracle.ovm.mgr.api.exception.FormatException> <servername> <AdminServer> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (sel
f-tuning)'> <<anonymous>> <> <1b95d8d1-24dd-4acc-a6f3-4c025158e6a7-00293343> <1425093778469> <BEA-000000> <com.oracle.odof.exception.TimeoutException: Receive timed out - timeout value(30000)

Solution:

1. Log into the Weblogic console: https://:7002/console. The username should be 'weblogic' and the password should be the same one used to log into the manager.

2. Go to Services --> Data Sources --> ovm-odof-ds --> Monitoring. Click on "Customize this table." Add the following columns to the table: 'Leaked Connection Count', 'Active Connections Current Count', 'Active Connections High Count'.
3. Go to Services --> Data Sources --> ovm-odof-ds --> Connection Pool --> Advanced. Set the "Inactive Connection Timeout" field to 30, and save.

4. Go to Services --> Data Sources --> ovm-odof-ds --> Diagnostics. Check "Profile Connection Leak" and save.

5. Click on "Activate Changes" on the left so that all the config changes take effect.

6. Continue using the manager. Now, weblogic should close any inactive connections that are still open.

7. Go to Diagnostics --> Log Files. View the DataSource log. Information here could help us figure out if the manager is leaving connections open.

Also, as a precaution and to avoid this issue in future, you can increase the maximum capacity of connections for the pool. To do that, follow steps below:

a. Login to Weblogic console
b. Navigate to Domain Structure -> Services -> Data Sources -> -> Connection Pool >> Maximum Capacity
c. Increase the parameter "Maximum Capacity" of the connection pool (If it is 1, increase to 100)

Exception :: ORA-00904: "NEED_BAL_FLAG": invalid identifier

Issue: Issue occurred while running the 9651687 fix script.

SQL> @xla_mulent_fix.sql;
Exception occured :: ORA-20001:
DECLARE
*
ERROR at line 1:
ORA-20001:
ORA-06512: at "APPS.APP_EXCEPTION", line 72
ORA-06512: at line 468

Log Details

Exception :: ORA-00904: "NEED_BAL_FLAG": invalid identifier
in side Print1
Exception occured When Creating xla_ae_headers Backup Table :: ORA-20001:
Exception occured :: ORA-20001: 

Cause:

The backup table is not having the column NEED_BAL_FLAG

Solution:

ALTER TABLE XAH_9651687_BKP ADD(NEED_BAL_FLAG VARCHAR2(1));


SQL> select column_name from ALL_TAB_COLUMNS where table_name='XAH_9651687_BKP';

COLUMN_NAME
------------------------------
NEED_BAL_FLAG
CLOSE_ACCT_SEQ_VERSION_ID
COMPLETION_ACCT_SEQ_VALUE


.....

If the column appears in the output then run the fix script it should complete now.

SQL> @xla_mulent_fix.sql;
-------------------------------------------------------------------------------
Please Note the Following information. Kindly open the Out file to proceed
further
Out file Location would be  /usr/tmp/9651687-fix-15:23:56.html
Log file Location would be  /usr/tmp/9651687-fix.log
-------------------------------------------------------------------------------

PL/SQL procedure successfully completed.

SQL> exit

You have insufficient privileges for the current operation. Please contact your System Administrator

Issue: "You have insufficient privileges for the current operation. Please contact your System Administrator" ,the URL changed from https (SSL) to http (non-SSL) . 

Cause: The issue occurred after configuring the SSL on R12 applications, this caused in the WORKFLOW notification approvals, whenever they click the approve button in the notification the issue occurred,

-in the $CONTEXT_FILE we did not enable the flag for ssl terminated environments (ie. SSL is enabled on the load balancer/SSL accelerator): 
<sslterminator oa_var="s_enable_sslterminator">#</sslterminator> 


Solution:

As per the document for SSL : 

Enabling SSL or TLS in Oracle E-Business Suite Release 12 
( Doc ID 376700.1 ) --------->Section 3--->step 8->table "Changes when using an SSL Accelerator" 

1. Check the httpd.conf to confirm that ssl_terminator.conf is included as this is required for SSL Accelerators. 

grep ssl_terminator.conf $INST_TOP/ora/10.1.3/Apache/Apache/conf/httpd.conf 

2. Check that the "s_enable_sslterminator" autoconfig parameter is not disable with #. 

grep s_enable_sslterminator $INST_TOP/appl/admin/$CONTEXT_FILE 

If this shows # as value, it must be removed and Autoconfig must be run to update the httpd.conf to include the ssl_terminator.conf. 

3. start the applications so the changes to the httpd.conf is seen by the web server. 

ORA-39213: Metadata processing is not available

Issue: Export: Release 11.2.0.3.0 - Production on Wed Sep 30 19:44:23 2015

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORA-39006: internal error
ORA-39213: Metadata processing is not available

Solution:

Verify if all the components in the database are valid.

SQL> SELECT comp_id, version, status FROM dba_registry;

COMP_ID                        VERSION                        STATUS

------------------------------ ------------------------------ -----------

OWB                            11.2.0.1.0                     VALID

APEX                           3.2.1.00.10                    VALID

EM                             11.2.0.3.0                     VALID

AMD                            11.2.0.3.0                     VALID

SDO                            11.2.0.3.0                     VALID

ORDIM                          11.2.0.3.0                     VALID

XDB                            11.2.0.3.0                     VALID

CONTEXT                        11.2.0.3.0                     VALID

EXF                            11.2.0.3.0                     VALID

RUL                            11.2.0.3.0                     VALID

OWM                            11.2.0.3.0                     VALID

CATALOG                        11.2.0.3.0                     VALID

CATPROC                        11.2.0.3.0                     VALID

JAVAVM                         11.2.0.3.0                     VALID

XML                            11.2.0.3.0                     VALID

CATJAVA                        11.2.0.3.0                     VALID

APS                            11.2.0.3.0                     VALID

XOQ                            11.2.0.3.0                     VALID

Run the below statement and retest the issue.

SQL> execute dbms_metadata_util.load_stylesheets;

PL/SQL procedure successfully completed.

SQL> exit

 expdp parfile=expdp_cltp_tables.par

Export: Release 11.2.0.3.0 - Production on Wed Sep 30 19:47:32 2015

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production

With the Partitioning, OLAP, Data Mining and Real Application Testing options

Starting "SYS"."CLTB_TABLES_EXP":  /******** AS SYSDBA parfile=expdp_cltp_tables.par

Estimate in progress using BLOCKS method...

Processing object type TABLE_EXPORT/TABLE/TABLE_DATA

Total estimation using BLOCKS method: 3.375 MB

Processing object type TABLE_EXPORT/TABLE/TABLE

. . exported "GFCPROD"."CLTB_EVENT_ENTRIES":"SYS_P369"   1.278 MB    4788 rows

. . exported "GFCPROD"."CLTP_ACCOUNT_SCHEDULES":"SYS_P378"  327.8 KB    2383 rows

. . exported "GFCPROD"."CLTB_ACCOUNT_APPS_MASTER":"FUNDED"  296.3 KB     281 rows

. . exported "GFCPROD"."CLTP_ACCOUNT_COMP_BALANCES":"SYS_P371"  53.75 KB     785 rows

. . exported "GFCPROD"."CLTP_ACCOUNT_COMP_SCH":"SYS_P374"  63.08 KB     418 rows

. . exported "GFCPROD"."CLTP_ACCOUNT_COMP_SCH":"SYS_P375"  61.03 KB     398 rows

The issue should resolve.

ORA-39065: unexpected master process exception in DISPATCH

Issue while running the export command:

Export: Release 11.2.0.3.0 - Production on Wed Sep 30 19:34:01 2015

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORA-39006: internal error
ORA-39065: unexpected master process exception in DISPATCH
ORA-00600: internal error code, arguments: [kokle_lob2lob13:input mismatch], [1], [], [], [], [], [], [], [], [], [], []

ORA-39097: Data Pump job encountered unexpected error -600

Cause: JOB_QUEUE_PROCESSES was  set to 0

show parameter JOB_QUEUE_PROCESSES

job_queue_processes                  integer

0

Solution: Modified the parameter and restarted the database.

alter system set JOB_QUEUE_PROCESSES=2 scope=spfile;

show parameter JOB_QUEUE_PROCESSES

job_queue_processes                  integer

2

SQL> shutdown immediate;

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL> startup;

ORACLE instance started.

Total System Global Area 4710043648 bytes

Fixed Size                  2236248 bytes

Variable Size            3758096552 bytes

Database Buffers          838860800 bytes

Redo Buffers              110850048 bytes

Database mounted.

Database opened.

SQL> 

Retest the issue it should resolve the expdp error.