Workflow emails not working for specific users

Issue: After the migration of the ERP from Solaris 10 to Solaris 11, the workflow notication Mails are sending to few users.

Cause: After the migration it was observed that the notification_preference value was changed to DISABLED.

Solution:

Step 1: Verify the value of the notification_preference in the workflow table wf_local_roles as below for all the users affected.

select notification_preference from the wf_local_roles where user_name in 'AFFECTED USER, ','user2'.......;

If the value is DISABLED proceed to the next step.

Step 2: Update the notification_preference value to MAILHTML using the below package.

exec FND_PREFERENCE.put('affected_user', 'WF', 'MAILTYPE', 'MAILHTML');

Step 3: Restart the Workflow Mailer from the Service Component window.

Note: Do not restart workflow mailer containers.

This should resolve the issue.

Unable to find an Output Post Processor service to post-process request 37175438.

Issue:

When any custom report is executed the report is not generated in PDF instead the output file returns in xml file.

Concurrent Request log reports below erro.

Unable to find an Output Post Processor service to post-process request 37175438.

Cause: AQTABLES contains huge number of records.

AQs tables are used to look for "subscriptions" by FNDSMs. That is, when ICM calls for FNDSM to start, they "subscribe" to this queue to identify its status. The time taken for the process cleanup prior to the internal manager start up with the regular CMs is correlated to the number of processes that were not stopped cleanly. In case of un-clean shutdown, the process to restart will be longer as manager spends extra cycles to perform housekeeping tasksRun the following as SYS:.

Solution:

1. Do a clean Shutdown of the concurrent managers.

2. Ensure no FNDLIBR processes are running.

3. Check the number of records in the table :

SQL> select count(*) from applsys.FND_CP_GSM_OPP_AQTBL;

  COUNT(*)

----------

     85596

4. Run the following as SYS:

SQL> DECLARE

po dbms_aqadm.aq$_purge_options_t;

BEGIN

po.block := FALSE;

DBMS_AQADM.PURGE_QUEUE_TABLE(

queue_table => 'APPLSYS.FND_CP_GSM_OPP_AQTBL',

purge_condition => NULL,

purge_options => po);

END;

/  2    3    4    5    6    7    8    9   10

PL/SQL procedure successfully completed.

SQL> commit;

Commit complete.

5. Re-check again the number of records in the table

SQL> select count(*) from applsys.FND_CP_GSM_OPP_AQTBL;

  COUNT(*)

----------

         0

5. Restart the concurrent managers

"unable to create shared memory segment" occurred in Solaris 11

Issue: While Starting the database "unable to create shared memory segment" occurred in Solaris 11

RMAN> startup mount pfile=initINTP.ora.plug;

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of startup command at 10/04/2017 13:04:47
RMAN-04014: startup failed: ORA-27125: unable to create shared memory segment
SVR4 Error: 22: Invalid argument
Additional information: 2640
Additional information: 33554432
Additional information: 8192
RMAN-04017: startup error description: ORA-32006: SEC_CASE_SENSITIVE_LOGON initialization parameter has been deprecated

RMAN> exit

Cause: Memory is not allocated in the OS level for the USER.

Solution:
Verify as root user if the user name is available in the projects file, if the user is not found, add the user with attributes as below.

Run as root user.

#projects -l
#projadd -p 104 -U intpora -G dba,oinstall -K 'project.max-shm-memory=(priv,161061273600,deny)' user.intpora

RMAN> startup mount pfile=initINTP.ora.plug;

Oracle instance started
database mounted

Total System Global Area   10737418240 bytes

Fixed Size                     2925344 bytes
Variable Size               6375349472 bytes
Database Buffers            4328521728 bytes
Redo Buffers                  30621696 bytes

RMAN>

Retest the issue.

PRCR-1079 : Failed to start resource ora.asm

Issue:

PRCR-1079 : Failed to start resource ora.asm
ORA-01017: invalid username/password; logon denied
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-01017: invalid username/password; logon denied
. For details refer to "(:CLSN00107:)" in "/ebspgrid/app/12.1.0.2/diag/crs/euxhoerpdbtn/crs/trace/ohasd_oraagent_grid.trc".

CRS-2674: Start of 'ora.asm' on 'euxhoerpdbtn' failed

Cause: The ASM OS Authentication was failing due to the config.o was corrupted under $GRID_HOME/rdbms/lib

Solution:

$ORACLE_HOME/rdbms/lib

$rm config.o

$ make -f ins_rdbms.mk $ORACLE_HOME/rdbms/lib/config.o

$ relink all

NOte: The solutions is applicable while connecting to the oracle database to an idle instance.

Oracle error 922: has been detected in alterpassword2.

Issue while changing the APPS password in R12.1.3

cat L6460833.log
+---------------------------------------------------------------------------+
Application Object Library: Version : 12.1

Copyright (c) 1998, 2013, Oracle and/or its affiliates. All rights reserved.

FNDCPASS:
+---------------------------------------------------------------------------+

Current system time is 04-OCT-2017 10:43:17

+---------------------------------------------------------------------------+

Working...
APP-FND-02704: Unable to alter user APPS to change password.
Oracle error 922:  has been detected in alterpassword2.

+---------------------------------------------------------------------------+
Concurrent request completed
Current system time is 04-OCT-2017 10:43:45

+------------------------

Cause: Password contains invalid character "$", password can contain only characters _, $, and #. 


Solution:

Remove Special Characters and retry.

R12.2 ONLINE PATCHING (ADOP) PARAMETERS/COMMANDS

ADOP PATCHING PARAMETERS DETAILS:

PARAMETER	DEFAULT VALUE	POSSIBLE VALUES	DESCRIPTION
phase	N/A	PHASE=PREPARE PHASE=APPLY PHASE=CUTOVER PHASE=CLEANUP PHASE=FINALIZE PHASE=ACTUALIZE_ALL PHASE=FS_CLONE PHASE=ABORT	These are the eight phases in which adop can run. It is most important and mandatory parameter that is used with adop. You can also club multiple phases in single command like ‘PHASE=PREPARE,APPLY’ although abort and fs_clone need to be run alone and can’t be clubbed.Standard phases: prepare – Prepare the instance for patch application. apply – Apply patches (to the patch edition). finalize – Ready the instance for cutover. It is run automatically. cutover – Make the patch edition the new run edition. cleanup – Drop obsolete objects and data from old editions. It is run automatically. There are also three special phases, used as per requirement. Special phases: abort – Abort the current patching cycle. The abort phase can be run after either the prepare or apply phases have been run, but not after the cutover phase. actualize_all – Create new copies of all code objects in the patch edition. fs_clone – Copy the run file system to the patch file system.
loglevel	LOGLEVEL=EVENT	LOGLEVEL=STATEMENT LOGLEVEL=PROCEDURE LOGLEVEL=EVENT LOGLEVEL=WARNING LOGLEVEL=ERROR LOGLEVEL=UNEXPECTED	STATEMENT > for debugging. PROCEDURE > for debugging high level procedures. EVENT > to capture informational messages in normal processing. (default) WARNING > to capture any internal error that is handled by the system and does not affect processing. ERROR > indicates action failed, need to be reviewed, but the system continue processing. UNEXPECTED > indicates an unrecoverable error, requires user intervention before processing can continue.
cleanup_mode	CLEANUP_MODE=STANDARD	CLEANUP_MODE=FULL CLEANUP_MODE=STANDARD CLEANUP_MODE=QUICK	Cleanup processing needs to happen after adop finishes the patching work.Quick mode > shortest execution time, skips non-essential actions Standard mode > All quick mode action + drops covered objects Full mode > All quick mode action +  remove all unused code, data, and old editions and takes much longer
finalize_mode	FINALIZE_MODE=QUICK	FINALIZE_MODE=QUICK FINALIZE_MODE=FULL	Quick mode > shortest execution, skips non-essential actions, no gather statistics. Full mode > Gather statistics, may improve performance after cutover, can take an hour extra to complete.
input_file	N/A	INPUT_FILE=<Absolute input_file path>	To specify the name of the input_file supplied to adop. (see details on input_file later in this post)
workers	N/A (depends on number of available CPU cores)	WORKERS=<User-specified-value>	Number of parallel workers used to execute tasks.In earlier released  adpatch used to prompt for number of workers. With adop in R12.2, if you want  to override the default formula that oracle uses now to calculation number of workers, use the WORKERS parameter. Take care that you don’t specify very high number of workers or else adop will fail.
maxworkers	N/A (depends on number of available CPU cores)	MAXWORKERS=<User-specified-value>	Maximum parallel workers that can be engaged. maxworkers should always be set to greater than the desired number of workers.
runcontextfile	$CONTEXT_FILE	RUNCONTEXTFILE=<Absolute context_file path>	To specify the non-default context file patch in RUN filesystem
patchcontextfile	Standard context file path in patch FS	PATCHCONTEXTFILE=<Absolute context_file path>	To specify the non-default context file patch in PATCH filesystem
patches	N/A	PATCHES=<User-specified-value>For Standard Patch:when>patch directory is  a 6- to 8-digit numberPATCHES=<patch_number>   For Non-Standard Patch when > patch directory is not a 6- to 8-digit number example NLS patches <patch_number>_<language_code>. > patch driver files are not named *<patchnum>.drv example merged patchesPATCHES=<patch_number>:<driver_file>.drv	This parameter specifies the patches adop needs to apply.Remember the numbered-only patches (standard) and containing-a-colon categories of patch (non-standard) can be mixed.Like:PATCHES= <patch_number1>,  <patch_number2>:<driver_file2>.drv
defaultsfile	$APPL_TOP/admin/<SID>_patch/adalldefaults.txt	DEFAULTSFILE=<Absolute defaults_file path>	Default file locations on both the run APPL_TOP and patch APPL_TOP is: $APPL_TOP/admin/<SID>_patch/adalldefaults.txtIn case you have created your own defaults file and want to use that instead, then use this parameter.
patchtop	$NE_BASE/EBSapps/patch	PATCHTOP=<Absolute patch_location_file path>	Default patch_top location is below.$NE_BASE/EBSapps/patchIf you want to keep your patches in some other lcoation, then you need to use this patrameter to let ADOP know where to search for patches pointed by ‘patches’ parameter.If you have a multi-node environment, you must download and unzip the patches (under $APPL_TOP_NE/EBSapps/patch) on the respective nodes.
merge	MERGE=NO	MERGE=NO MERGE=YES	In R12.2, oracle has integrated patch merging action in the patching command itself. In earlier releases we used to first merge patches using admrgpch command.By using MERGE=YES option ADOP will merge all the unified driver files into a single driver file.
abandon	ABANDON=NO	ABANDON=YES ABANDON=NO	If the patch you are applying went into error, you have two option when you start the adop utility again.1) you corrected error and want to continue with previous adop session:ABANDON=NO2) you decided that you don’t want to correct issue for now and want to abandon the previous adop session:ABANDON=YES
restart	RESTART=YES	RESTART=NO RESTART=YES	If the patch you are applying went into error and you corrected the issue and want to restart the previous patching session.It is just the reverse of ABANDON parameter.Remember ABANDON and RESTART will always have opposite value.
flags	N/A	FLAGS=AUTOSKIP	Use “flags=autoskip” in conjunction with the “abandon=no” parameter at the command-line to skip a failing patching step to “Continue as if a patch were successful”. You need to review the “autoskip” log that gets generated during the patching cycle in order to make sure that their were no errors and to take required actions in case of any errors
allnodes	ALLNODES=NO	ALLNODES=NO ALLNODES=YES	This parameter comes into picture when you have multi node setup. If you want to run adop on all nodes then use ALLNODES=YES.
action	ACTION=DB	ACTION=DB ACTION=NODB	Use this parameter to specify whether to perform database actions or skip. For example if you are in a multi-node environment and adop has already updated the database so when running on other node just use ACTION=NODB to save time.Remember when you are using ‘allnodes=yes’ in a multi-node ‘action=db’ must be specified.
apply	APPLY=YES	APPLY=YES APPLY=NO	To run adop in test mode (without applying any patches),specify apply=no
autoskip	AUTOSKIP=YES	AUTOSKIP=YES AUTOSKIP=NO	This parameter control whether the user is prompted about skipping actions in non-interactive patching. This is specifically useful when you are applying patches in multi node setup.
mtrestart	MTRESTART=YES	MTRESTART=YES MTRESTART=NO	This parameter specify whether to restart application tier services after cutover phase or not.
cm_wait	CM_WAIT=INFINITE (will wait forever)	CM_WAIT=<user_specified_number	Specifies the number of minutes to wait until the ICM will be forced down.
allowcoredump	ALLOWCOREDUMP=NO	ALLOWCOREDUMP=NO ALLOWCOREDUMP=YES	To specify that a core dump will be generated if adop crashes.
analytics	ANALYTICS=NO	ANALYTICS=NO ANALYTICS=YES	To specify that a report will be generated that can help debug certian adop issue.
preinstall	N/A	PREINSTALL=Y	This mode is used only if the patch readme instructs. Generally this mode is used during the upgrade process to update AD utilities, apply pre-upgrade patches, or work around other patching issues.It will Compares version numbers, Copies files, Relinks FND and AD executables, Saves patch information. It also runs autoconfig if required.The dual file system in Release 12.2 means that there is no need to shut down application tier services before running AutoConfig.
-help	N/A	N/A	Shows the help screen.
-status-status -detail	Latest Session	-STATUS (for latest session) -STATUS <SESSION_ID> (for specific session)	Display status of the latest adop session.Use ‘adop -status -detail’ for detailed info
-examples	N/A	N/A	Displays some commonly used adop ample commnads

adop cutover phase status showing failed in the cloned instance.

Run the following steps:


Node Name       Node Type       Phase           Status          Started                        Finished                       Elapsed
--------------- --------------- --------------- --------------- ------------------------------ ------------------------------ ------------
xxxx12          master          PREPARE         COMPLETED       22-NOV-16 04:00:33 +08:00      22-NOV-16 04:06:20 +08:00      0:05:47
                                APPLY           COMPLETED       22-NOV-16 04:44:02 +08:00      22-NOV-16 05:01:25 +08:00      0:17:23
                                FINALIZE        COMPLETED       22-NOV-16 01:11:06 +08:00      22-NOV-16 01:11:17 +08:00      0:00:11
                                CUTOVER         FAILED          22-NOV-16 01:26:09 +08:00      22-NOV-16 03:19:33 +08:00      1:53:24
                                CLEANUP         NOT STARTED

File System Synchronization Used in this Patching Cycle: Full

File System Information:
========================

File System: SINGLE NODE INSTANCE

Node Name   Node Type   Current Base                   Other Base                     Non Editioned Base             Inst Top
----------- ----------- ------------------------------ ------------------------------ ------------------------------ ------------------------------
xxxx12      master      /u01/appluat/appluat/fs2            /u01/appluat/appluat/fs1            /u01/appluat/appluat/fs_ne          /u01/appluat/appluat/fs2/inst/apps/
                                                                                                                     appluat_xxxx
Custom File System Synchronization script: /u01/appluat/appluat/fs_ne/EBSapps/appl/ad/custom/adop_sync.drv


Solution:
[appluat@XXXXXX PATCH]$ adop phase=cutover mtrestart=no

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

Please wait. Validating credentials...
RUN file system context file: /u01/appluat/UAT/fs1/inst/apps/UAT_XXXXXX/appl/admin/UAT_XXXXXX.xml

PATCH file system context file: /u01/appluat/UAT/fs2/inst/apps/UAT_XXXXXX/appl/admin/UAT_XXXXXX.xml
Execute SYSTEM command : df /u01/appluat/UAT/fs2

Worker count determination...

Validation successful. All expected nodes are listed in ADOP_VALID_NODES table.
[EVENT]     [START 2016/02/23 22:58:56] Performing database sanity checks
[EVENT]     [END   2016/02/23 22:58:57] Finished performing database sanity checks
  There is already a session which is incomplete. Details are:
        Session Id            :   12
        Prepare phase status  :   COMPLETED
        Apply phase status    :   COMPLETED
        Cutover  phase status :   RUNNING
        Abort phase status    :   NOT APPLICABLE
        Session status        :   FAILED
  Will continue with previous session
  [START 2016/02/23 22:58:58] adzdoptl.pl run
    ADOP Session ID: 12
    Phase: cutover
    Log file: /u01/appluat/UAT/fs_ne/EBSapps/log/adop/12/adop_20160223_225831.log
    [EVENT]     [START 2016/02/23 22:58:58] Calling task: "Executing txkADOPValidation script on XXXXXX"; script: "/u01/appluat/UAT/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPValidations.pl"; args: " -contextfile=/u01/appluat/UAT/fs1/inst/apps/UAT_XXXXXX/appl/admin/UAT_XXXXXX.xml -patchctxfile=/u01/appluat/UAT/fs2/inst/apps/UAT_XXXXXX/appl/admin/UAT_XXXXXX.xml -phase=cutover -logloc=/u01/appluat/UAT/fs_ne/EBSapps/log/adop/12/cutover_20160223_225831/UAT_XXXXXX -promptmsg=hide"
      [EVENT]     [START 2016/02/23 22:58:58] Executing txkADOPValidation script on XXXXXX
        [EVENT]     Calling: /u01/appluat/UAT/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPValidations.pl
Oracle E-Business Suite Online Patching
RUN edition environment variables have been set.


calling: /u01/appluat/UAT/fs1/FMW_Home/webtier/perl/bin/perl /u01/appluat/UAT/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkValidateSetup.pl -contextfile=/u01/appluat/UAT/fs1/inst/apps/UAT_XXXXXX/appl/admin/UAT_XXXXXX.xml -patchctxfile=/u01/appluat/UAT/fs2/inst/apps/UAT_XXXXXX/appl/admin/UAT_XXXXXX.xml -logloc=/u01/appluat/UAT/fs_ne/EBSapps/log/adop/12/cutover_20160223_225831/UAT_XXXXXX -phase=cutover -promptmsg=hide

Program : txkValidateSetup.pl started @ Tue Feb 23 22:59:01 2016

*** Log File = /u01/appluat/UAT/fs_ne/EBSapps/log/adop/12/cutover_20160223_225831/UAT_XXXXXX/ADOPValidations_Tue_Feb_23_22_59_01_2016.log

SETUP VALIDATION is in progress. This may take few minutes to complete.

Completed execution of all tests.

=========================== START OF VALIDATION REPORT ====================


Validation Results for Node: XXXXXX
---------------------------------------
All validations were successful, please check /u01/appluat/UAT/fs_ne/EBSapps/log/adop/12/cutover_20160223_225831/UAT_XXXXXX/ADOPValidations_Tue_Feb_23_22_59_01_2016.log

=========================== END OF VALIDATION REPORT ======================
      [EVENT]     [END   2016/02/23 22:59:05] Executing txkADOPValidation script on XXXXXX
    [EVENT]     [END   2016/02/23 22:59:05] Executing txkADOPValidation script on XXXXXX
    [EVENT]     [START 2016/02/23 22:59:05] Checking if all nodes were synchronized in the previous apply phase
    [EVENT]     [END   2016/02/23 22:59:05] Checking if all nodes were synchronized in the previous apply phase
    [START 2016/02/23 22:59:05] cutover phase
      [EVENT]     [START 2016/02/23 22:59:07] Performing Cutover Phase steps
        User specified mtrestart=no, hence middle tier service startup will be skipped.
        User would need to bring up the services manually.
      [EVENT]     [END   2016/02/23 22:59:08] Performing Cutover Phase steps
    [END   2016/02/23 22:59:08] cutover phase
    [START 2016/02/23 22:59:08] Generating Post Cutover Reports
    [END   2016/02/23 22:59:08] Generating Post Cutover Reports
    [EVENT]     [START 2016/02/23 22:59:09] Generating AD_ZD_LOGS Report
      [EVENT]     Report: /u01/appluat/UAT/fs1/EBSapps/appl/ad/12.0.0/sql/ADZDSHOWLOG.sql


      [EVENT]     Output: /u01/appluat/UAT/fs_ne/EBSapps/log/adop/12/cutover_20160223_225831/UAT_XXXXXX/adzdshowlog.out
    [EVENT]     [END   2016/02/23 22:59:09] Generating AD_ZD_LOGS Report

Broadcast message from appluat@XXXXXX.myoracle.com (Tue Feb 23 22:59:09 2016):

UAT environment has changed.
All users must re-source the environment using below command:
source /u01/appluat/UAT/EBSapps.env run|patch
  [END   2016/02/23 22:59:09] adzdoptl.pl run
  adop phase=cutover - Completed Successfully

  Log file: /u01/appluat/UAT/fs_ne/EBSapps/log/adop/12/adop_20160223_225831.log

adop exiting with status = 0 (Success)
==============================


Case2:

ADOP Cutover Failed 

Problem Description 
===========
We were trying to apply hrms patches on our test environment, where cutover phase status showing failed. 
However we noticed fs1 & fs2 got swapped but stauts showing FAILED. Please suggest. 

Solution:
=====

1. Ensure that you have taken a backup of your system before applying the recommended steps. 

2. Please take backup of context files from both run fs and patch fs from application tier. 

3. Edit both context files and for line: 

oacore_jvm_start_options oa_var="s_oacore_jvm_start_options">-d64 -Xms512m -Xmx512m -XX:CompileThreshold=8000 -XX:PermSize=128m 

replace it with: 

oacore_jvm_start_options oa_var="s_oacore_jvm_start_options">-d64 -Xms512m -Xmx2048m -XX:CompileThreshold=8000 -XX:PermSize=128m 

save. 

4. Reboot the server. When the server is rebooted, the changes will be activated at system level and the oacore server will take the new Xmx settings. 
 Obs: there is no need to run autoconfig -- as these are OS system settings for OACORE startup. 

5. Restart cutover phase with option: 

adop phase=cutover mtrestart=no 

Obs: this will continue the cutover phase that previously failed, until completion 

6. After the cutover completes, you can restart application tier services manually on the new run fs. 

cd $ADMIN_SCRIPTS_HOME 

adstrtal.sh