mod_oc4j: Failed to find a failover oc4j process for session request for destination: application://oacore (no island or jgroup).

mod_oc4j: Failed to find a failover oc4j process for session request for destination: application://oacore (no island or jgroup).

<<<<<<<<<<<<<<<Explanation>>>>>>>>>>>>>>>

Islands are no longer used within iAS version 10.1.3 in EBS R12, the new term is "cluster". A cluster is stated as "two or more OC4J server nodes hosting the same set of applications". A standard instance of Applications Release12 has some aspects of clustering enabled by default. There are many components to a cluster and all of them have not been configured in Oracle E-Business suite R12. So the error may change as following :

mod_oc4j: Failed to find a failover oc4j process for session request for destination: application://form (no cluster or jgroup).

<<<<<<<<<<<<<<<Reason>>>>>>>>>>>>>>>

??? Load: e.g. Too many clients connecting to the application, The number of requests are high enough that the process cannot respond to new requests prior to reaching the timeout period.

??? Performance: Heavily loaded JVM or Lack of enough memory for JVM

??? Code: Internal to Application Server, one of the components are not operating correctly or efficiently. External to ApplicationServer, E-Business code (seeded or custom code) causing memory leaks.

??? Lack of free ports for AJP communication

<<<<<<<<<<<<<<<Solution Approach#1: Resize oc4j oacore jvm heap size as well as number of oacore processes in context file>>>>>>>>>>>>>>>


The number of jvms (oc4j in R12) is configured by the autoconfig variables s_oacore_nprocs, s_disco_nprocs, s_forms_servlet_nprocs (11i), s_forms_nprocs (R12) and s_xmlsvcs_nprocs.

In 11i, those changes are made in the $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.conf file, while in R12 in the $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml

Thumb Rule: 1 active JVM/OC4J instance per CPU core.  So if you have 8 CUPs of dual core, then you can configure upto 16 JVM. 1 JVM handles 100 Users, so based on the user load configure JVMs accordingly.


<<<<<Solution Approach#2: Proper Heap Configuration >>>>>

The heap is configured by s_oacore_jvm_start_option, s_forms_jvm_start_options in R12. For 12.1 and higher start with the following and increase as per the user load:
-Xmx1024M -Xms512M -XX:MaxPermSize=256M
      -XX:NewRatio=2 -XX:+PrintGCTimeStamps

Also, Add the following parameter to the DBC file:
JDBC\:oracle.jdbc.maxCachedBufferSize=262144

In JDK 1.6, the JVM detects that you have a server class machine (2 or more CPUs with 2GB or more memory), and will automatically enable Parallel Throughput Garbage Collector. The number of GC threads defaults to the number of CPUs on the machine. If you are running multiple JVMs on the same machine, or if your machine has more than 2 CPUs, to avoid the GC threads to be overly active you should reduce the number of GC threads by using:

-XX:+UseParallelGC -XX:ParallelGCThreads=2

Note: Although the setup of those parameters can be temporarily done manually changing (jserv.conf and jserv.properties in 11i; opmn.xml in R12), the correct way to update those parameters is using the context editor, update the variables s_oacore_nprocs and s_forms_servlet_nprocs, and then run autoconfig.

<<<<<Solution Approach#3: Long Running JVM >>>>>

Make sure to have the following setting in the jserv.properties file:
wrapper.bin.parameters=-DLONG_RUNNING_JVM=true

<<<<< Solution Approach#4: Java Cache Port Value >>>>>

The port value for java cache (s_java_object_cache_port) in context file should match profile option value JTF_DIST_CACHE value. 

 + The value for java cache port can be check in Context File -> parameter "s_java_object_cache_port" 
 + The value of the profile option JTF_DIST_CACHE can be found using the sql : 
      select fnd_profile.value('JTF_DIST_CACHE_PORT') from dual

<<<<< Solution Approach#5: Same Server Multiple EBS Instances >>>>>>

If the E-Business suite environment has multiple application servers then ensure that the following Autoconfig variables point to a "local disk". Hence this must be checked and changed on each application tier context file. 
     s_lock_pid_dir 
     s_pids_dir 
     s_web_pid_file

If the above variables are reset then run autoconfig for the changes to be effective.

<<<<< Solution Approach#6: EBS Instances with Load Balancer >>>>>>

Refer Oracle Support Note: Using Load-Balancers with Oracle E-Business Suite Release 12 (Doc ID 380489.1)

1. Make sure that the load balancer always implements Session Stickiness (also named Session Binding). This ensures that every time a request with an existing session is received it will be sent to the HTTP Server which created the session and therefore the routingID will always be able to be correctly decoded.

<<<<<<<<<<<<<<<Solution Approach#7: AJP Ports>>>>>>>>>>>>>>>

Ensure the port range used by AJP protocol by various middle tier components are not occupied by any other service. Run the following command to find the AJP port range.

$grep -i ajp $CONTEXT_FILE 

<ajp_protocol oa_var="s_ajp_protocol">ajp</ajp_protocol> 
<oacore_ajp_portrange oa_var="s_oacore_ajp_portrange" oa_type="PORT" base="21500" step="5" range="5" 
label="OC4J AJP Port Range for Oacore">21500-21504</oacore_ajp_portrange> 
<forms_ajp_portrange oa_var="s_forms_ajp_portrange" oa_type="PORT" base="22000" step="5" range="5" 
label="OC4J AJP Port Range for Forms">22000-22004</forms_ajp_portrange> 
<oafm_ajp_portrange oa_var="s_oafm_ajp_portrange" oa_type="PORT" base="25000" step="5" range="5" 
label="OC4J AJP Port Range for Oafm">25000-25004</oafm_ajp_portrange>

<<<<< Solution Approach#8: Firewall Between EBS Application Servers >>>>>

If any firewall exists between application server (Like firewall between forms/web tier & Concurrent tier etc) then refer the following workaround.

To improve performance, the mod_oc4j component in Oracle HTTP Server(OHS) process maintains open TCP connections to the AJP port within each OC4J components. In situations where a firewall exists between OHS and OC4J, packages sent via AJP are rejected if the connections can be idle for periods in excess of the inactivity timeout of stateful firewalls. However, the AJP socket is not closed as long as the socket remains open, the worker thread is tied to it and is never returned to the thread pool. OC4J will continue to create more threads, and will eventually exhaust system resources. 

Set the following parameters in the mod_oc4j.conf (Present in the directory $IAS_ORACLE_HOME/Apache/Apache/conf) configuration file. The value of Oc4jConnTimeout sets the length of inactivity in seconds, before the session is considered inactive. 

Oc4jUserKeepalive on 
Oc4jConnTimeout 12000 (or a similar value)

Also read, how to monitor JDBC Connections in my blog.
http://mkkoracleapps.blogspot.co.uk/2013/07/how-to-monitor-jdbc-and-jvm-in.html

Cheers!!!
Malay Khawas
Oracle Apps/Fusion DBA