How to check AD to OID Synch is happening fine

How to check AD to OID Synch is happening fine

Run this command for 2-3 Hours intervals:

ldapsearch -h mkktestserver1.unixdomain.local -p 3060 -D cn=orcladmin -w MalayFalsePass -s sub -b "" -T -createtimestamp "objectclass=orcluserv2" createtimestamp > createlist1.txt

ldapsearch -h mkktestserver1.unixdomain.local -p 3060 -D cn=orcladmin -w MalayFalsePass -s sub -b "" -T -createtimestamp "objectclass=orcluserv2" createtimestamp > createlist2.txt

ldapsearch -h mkktestserver1.unixdomain.local -p 3060 -D cn=orcladmin -w MalayFalsePass -s sub -b "" -T -createtimestamp "objectclass=orcluserv2" createtimestamp > createlist3.txt

ldapsearch -h mkktestserver1.unixdomain.local -p 3060 -D cn=orcladmin -w MalayFalsePass -s sub -b "" -T -createtimestamp "objectclass=orcluserv2" createtimestamp > createlist4.txt

Open the createlist?.txt file and make sure at the end of the file you see new entries are getting created.

ASM Related steps for 11gR2 Grid Infrastructure

ASM Related steps for 11gR2 Grid Infrastructure

I am putting this ASM related steps only, on request of a friend. In two projects where I handled HP EVA Storage also, I setup 2 node RAC for eBusiness Suite. If you require any other part of the implementation experience, would be happy to share.

This was done for Oracle Enterprise Linux

Prepare Storage and Disks

1. Run fdisk for RDBMS and GRID Binaries Multipathing Disk(Both Node)
2. Run partprobe for RDBMS and GRID Binaries Multipathing Disk(Both Node)
3. Run fdisk for ASM Related Multipathing Disk(Database Node1)
4. Run partprobe for ASM Related Multipathing Disk(Both Node)
5. Run mkfs.ext3 for RDBMS and GRID Binaries Multipathing Disk(Database Node1)

Multipathing Disks Used for ASM:

mpath2 - PRODDATA
mpath6 - PRODFRA
mpath5 - PRODOCR1
mpath4 - PRODOCR2
mpath3 - PRODOCR3

As an example fdisk and partprobe is shown for mapth2 disk, same needs to be done for others:

[root@mkkracdbserver1 ~]# fdisk /dev/mapper/mpath2
[root@mkkracdbserver1 ~]# partprobe /dev/mapper/mpath2

[root@mkkracdbserver1 ~]# ls -ltr /dev/mapper/mpath2*
brw-rw---- 1 root disk 253,  0 Aug  9 00:02 /dev/mapper/mpath2
brw-rw---- 1 root disk 253, 10 Aug  9 00:02 /dev/mapper/mpath2p1

[root@mkkracdbserver2 ~]# partprobe /dev/mapper/mpath2

[root@mkkracdbserver2 ~]# ls -ltr /dev/mapper/mpath2*
brw-rw---- 1 root disk 253, 0 Aug  6 18:07 /dev/mapper/mpath2
brw-rw---- 1 root disk 253, 9 Aug  9 00:06 /dev/mapper/mpath2p1



Install Oracle ASM Libraries RPM

# rpm -ivh oracleasm-support-2.1.4-1.el5.x86_64.rpm
# rpm -ivh oracleasmlib-2.0.4-1.el5.x86_64.rpm
# rpm -ivh oracleasm-2.6.18-238.el5-2.0.5-1.el5.x86_64.rpm

# /etc/init.d/oracleasm init
# /etc/init.d/oracleasm exit
# /etc/init.d/oracleasm status

To Load and initialize the ASMLib driver issue init
To Stop the ASMLib driver issue exit
To Display the status of the Oracle ASMLib driver issue status




Configure ASMLib Driver

This needs to be done on both the Database Node

# /etc/init.d/oracleasm configure

[root@mkkracdbserver1 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

Default user to own the driver interface []: oraprod
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]

[root@mkkracdbserver2 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

Default user to own the driver interface []: oraprod
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]




Create ASM Disks

1. Use Oracleasm Createdisk Command on Database Node1
2. Use Oracleasm Listdisks Command on Database Node1
2. Use Oracleasm Scandisks Command on Database Node2
2. Use Oracleasm Listdisks Command on Database Node2

Important Note: Do not deviate from the above order, or else you may end up with non-visible ASM disks while doing the installation. You will have to change the diskstring parameter to asm_diskstring='/dev/oracleasm/disks/*' to make the disks visible. To overcome the issue, reboot both the database node and disks will be visible.


[root@mkkracdbserver1 ~]# oracleasm createdisk ASMDATA01 /dev/mapper/mpath2p1
Writing disk header: done
Instantiating disk: done

[root@mkkracdbserver1 ~]# oracleasm createdisk ASMFRA01 /dev/mapper/mpath6p1
Writing disk header: done
Instantiating disk: done

[root@mkkracdbserver1 ~]# oracleasm createdisk ASMOCR01 /dev/mapper/mpath5p1
Writing disk header: done
Instantiating disk: done

[root@mkkracdbserver1 ~]# oracleasm createdisk ASMOCR02 /dev/mapper/mpath4p1
Writing disk header: done
Instantiating disk: done

[root@mkkracdbserver1 ~]# oracleasm createdisk ASMOCR03 /dev/mapper/mpath3p1
Writing disk header: done
Instantiating disk: done

[root@mkkracdbserver1 ~]# oracleasm listdisks
ASMDATA01
ASMFRA01
ASMOCR01
ASMOCR02
ASMOCR03

[root@mkkracdbserver2 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "ASMDATA01"
Instantiating disk "ASMOCR03"
Instantiating disk "ASMOCR02"
Instantiating disk "ASMOCR01"
Instantiating disk "ASMFRA01"

[root@mkkracdbserver2 ~]# oracleasm listdisks
ASMDATA01
ASMFRA01
ASMOCR01
ASMOCR02
ASMOCR03


Implement Metalink Note 1059847.1 for Multipathing Disks

This needs to be done on both the Database Nodes

Important Note: If this is not done, then root.sh will fail in Database Node2.

11GR2 GRID INFRASTRUCTURE INSTALLATION FAILS WHEN RUNNING ROOT.SH ON NODE 2 OF RAC USING ASMLIB [ID 1059847.1]

1. Modify the /etc/sysconfig/oracleasm with:

ORACLEASM_SCANORDER="dm"
ORACLEASM_SCANEXCLUDE="sd"

# /etc/init.d/oracleasm restart

[root@mkkracdbserver1 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@mkkracdbserver1 ~]#

[root@mkkracdbserver2 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@mkkracdbserver2 ~]#

Cheers !!!!
Malay Khawas
Oracle Apps/Fusion DBA

How to do test of ar package, Installed on OS | Exception String: Error in invoking target 'nnfgt.o' of makefile

How to do test of ar package, Installed on OS | Exception String: Error in invoking target 'nnfgt.o' of makefile

a) create the following program called test.c :

main()
{
printf("hello\n");
}

b) compile it to a .o file :

gcc -c test.c

This generates an object file called test.o

c) Now test if basic ar functionality works;

ar cr myarch.a test.o

--> does the myarch.a file get created ok ?
--> if so can you query it with ar -t myarch.a ?

d) now test ar rv myarch2.a test.o 

Spool:

$ gcc -c test.c
$ ls -ltr
total 25970
-rw-r--r-- 1 apppoc dba 174 Apr 8 16:09 local.profile
-rw-r--r-- 1 apppoc dba 157 Apr 8 16:09 local.login
-rw-r--r-- 1 apppoc dba 136 Apr 8 16:09 local.cshrc
drwxr-xr-x 2 apppoc dba 96 May 2 12:18 bea
-rw------- 1 apppoc dba 13004722 May 5 02:03 core
-rw-r--r-- 1 apppoc dba 449 May 5 10:49 oamenv2.sh
-rw-r--r-- 1 apppoc dba 283726 May 6 08:07 core_strings.txt
-rw-r--r-- 1 apppoc dba 486 May 7 11:32 oamenv.sh
-rw-r--r-- 1 apppoc dba 30 May 8 12:07 test.c
-rw-r--r-- 1 apppoc dba 732 May 8 12:07 test.o
$ ar cr myarch.a test.o
Illegal Instruction(coredump)
$ ar rv myarch2.a test.o
a - test.o
Illegal Instruction(coredump)
$ cat test.c
main()
{
printf("hello\n");
}
$


Lesson Learned:

While doing OID Installation, config.sh was failing with following to execute the make command.
Exception String: Error in invoking target 'nnfgt.o' of makefile in OID Installation

Issue with opmnctl Startup or Status - Swap Issue

Issue with opmnctl Startup or Status - Swap Issue

applive WEBTIERLIVE /export/home/applive $ opmnctl status
Error occurred during initialization of VM
Could not reserve enough space for object heap
applive WEBTIERLIVE /export/home/applive $

Increase the OS Level Swap Space. It work fine for you.

Connecting to EM Fails With Error "503 Service Unavailable" after OID Installation

Connecting to EM Fails With Error "503 Service Unavailable" after OID Installation

After Installation of Oracle Internet Directory, I was unable to connect to EM and getting an error 503 Service Unavailable.

When I cheched the targets.xml file, I noticed OID EM Farm entry were not present.

$DOMAIN_HOME/sysman/state/targets.xml

I put the following entries immediately after the very first <Targets> line:

<Target TYPE="oracle_ias_farm" NAME="Farm_IDMDomain" DISPLAY_NAME="Farm_IDMDomain">
<Property NAME="MachineName" VALUE="mkkoidserver1.mkkdomain.local"/>
<Property NAME="Port" VALUE="7001"/>
<Property NAME="Protocol" VALUE="t3"/>
<Property NAME="isLocal" VALUE="true"/>
<Property NAME="serviceURL" VALUE="service:jmx:t3://mkkoidserver1.mkkdomain.local:7001/jndi/weblogic.management.mbeanservers.domainruntime"/>
<Property NAME="WebLogicHome" VALUE="/opt/oracle/IDMLIVE_MW_HOME/WebLogic/wlserver_10.3"/>
<Property NAME="DomainHome" VALUE="/opt/oracle/IDMLIVE_MW_HOME/WebLogic/user_projects/domains/IDMDomain"/>
</Target>

Restart the OID Stack.

ldap_bind: UnKnown Error Encountered for Java External Authentication Plugin

ldap_bind: UnKnown Error Encountered for Java External Authentication Plugin

applive IDMLIVE /export/home/applive $ ldapcompare -h mkkoidserver1 -p 3060 -D "cn=orcladmin" -w ************* -b "cn=adtooidsyncuser generic,ou=information technology,ou=india,cn=users,dc=mkk,dc=ad,dc=local" -a userPassword -v "*************"
ldap_compare_s: UnKnown Error Encountered

applive IDMLIVE /export/home/applive $

applive IDMLIVE /export/home/applive $ ldapbind -h mkkoidserver1 -p 3060 -D "cn=adtooidsyncuser generic,ou=information technology,ou=india,cn=users,dc=mkk,dc=ad,dc=local" -w "*************"
ldap_bind: UnKnown Error Encountered


There is a bug with OID 11.1.1.6.

Due to issues to login to ODSM, I changes orcljvmoptions from 64M to 512M using LDAPMODIFY command.

<Jun 18, 2013 4:58:33 PM BST> <Error> <oracle.adfinternal.view.faces.config.rich.RegistrationConfigurator> <BEA-000000> <ADF_FACES-60096:Server Exception during PPR, #8

If you face this issue you would have to increase orcljvm value to 512 or something more than 64M.

●● Use ldapmodify to update heap size for dsaconfig:

ldapmodify -h mkkoidserver1 -p 3060 -D cn=orcladmin -w ************* << eof
dn: cn=dsaconfig,cn=configsets,cn=oracle internet directory
changetype: modify
replace: orcljvmoptions
orcljvmoptions: -Xmx512M
eof

●● Restart Complete OID Stack.

If you try to change this using /em, instead of -Xmx512M, it would be saved as -xmx512m. Even LDAPMODIFY also saved the entry in small letter only. Note x and m are in small letter.

applive IDMLIVE /export/home/applive $ ldapsearch -h mkkoidserver1 -p 3060 -D cn=orcladmin -w ************* -b "cn=dsaconfig,cn=configsets,cn=oracle internet directory" -s base "objectclass=*"
cn=dsaconfig,cn=configsets,cn=oracle internet directory
orclallattrstodn=NOT ASCII
orclecachemaxentries=100000
orclecachemaxsize=209715200
orclecacheenabled=1
orclautocatalog=1
orcljvmoptions=-xmx512m
orclrscacheattr=uid
orclrscacheattr=mail
orclrscacheattr=cn

For two of my instances it saved correctly with capital X and M. But for live instance it created a hell for me.

As of now, I directly updated the database. Make sure you take a proper backup. Hope issues from /em or ldapmodify would be resolved soon by Oracle.

select attrval from ds_attrstore where attrname = 'orcljvmoptions';

update ds_attrstore  set attrval='-Xmx64M' where attrname = 'orcljvmoptions';
commit;

I reverted back the value to 64M which comes as default during installation.


Cheers !!!!!
Malay Khawas
Oracle Apps/Fusion DBA

How to Monitor JDBC and JVM in eBusiness Suite

How to Monitor JDBC and JVM in eBusiness Suite

Use Oracle Support Note for complete script:
monitor_jdbc_conn.sql - Script to monitor JDBC connections in Apps eBusiness Suite (Doc ID 557194.1)

JDBC Connection Usage Per JVM Process:
select machine, process, count(*) from gv$session
where program like '%JDBC%'
group by machine, process
order by 1 asc;

Connection Usage Per Module
select count(*), module
from gv$session
where program like '%JDBC%'
group by module
order by 1 asc;

Idle connections for more than 3 hours:
select count(*),machine, program
from gv$session
where program like '%JDBC%'
and  last_call_et > 3600 *3
group by machine, program;

Inactive connections which last ran fnd_security_pkg.fnd_encrypted_pwd:
select s.sql_hash_value, t.sql_text, s.last_call_et
from gv$session s , gv$sqltext t
where s.username = 'APPLSYSPUB'
and s.sql_hash_value= t.hash_value
and t.sql_text like  '%fnd_security_pkg.fnd_encrypted_pwd(:1,:2,:3%';

mod_oc4j: Failed to find a failover oc4j process for session request for destination: application://oacore (no island or jgroup).

mod_oc4j: Failed to find a failover oc4j process for session request for destination: application://oacore (no island or jgroup).

<<<<<<<<<<<<<<<Explanation>>>>>>>>>>>>>>>

Islands are no longer used within iAS version 10.1.3 in EBS R12, the new term is "cluster". A cluster is stated as "two or more OC4J server nodes hosting the same set of applications". A standard instance of Applications Release12 has some aspects of clustering enabled by default. There are many components to a cluster and all of them have not been configured in Oracle E-Business suite R12. So the error may change as following :

mod_oc4j: Failed to find a failover oc4j process for session request for destination: application://form (no cluster or jgroup).

<<<<<<<<<<<<<<<Reason>>>>>>>>>>>>>>>

??? Load: e.g. Too many clients connecting to the application, The number of requests are high enough that the process cannot respond to new requests prior to reaching the timeout period.

??? Performance: Heavily loaded JVM or Lack of enough memory for JVM

??? Code: Internal to Application Server, one of the components are not operating correctly or efficiently. External to ApplicationServer, E-Business code (seeded or custom code) causing memory leaks.

??? Lack of free ports for AJP communication

<<<<<<<<<<<<<<<Solution Approach#1: Resize oc4j oacore jvm heap size as well as number of oacore processes in context file>>>>>>>>>>>>>>>


The number of jvms (oc4j in R12) is configured by the autoconfig variables s_oacore_nprocs, s_disco_nprocs, s_forms_servlet_nprocs (11i), s_forms_nprocs (R12) and s_xmlsvcs_nprocs.

In 11i, those changes are made in the $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.conf file, while in R12 in the $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml

Thumb Rule: 1 active JVM/OC4J instance per CPU core.  So if you have 8 CUPs of dual core, then you can configure upto 16 JVM. 1 JVM handles 100 Users, so based on the user load configure JVMs accordingly.


<<<<<Solution Approach#2: Proper Heap Configuration >>>>>

The heap is configured by s_oacore_jvm_start_option, s_forms_jvm_start_options in R12. For 12.1 and higher start with the following and increase as per the user load:
-Xmx1024M -Xms512M -XX:MaxPermSize=256M
      -XX:NewRatio=2 -XX:+PrintGCTimeStamps

Also, Add the following parameter to the DBC file:
JDBC\:oracle.jdbc.maxCachedBufferSize=262144

In JDK 1.6, the JVM detects that you have a server class machine (2 or more CPUs with 2GB or more memory), and will automatically enable Parallel Throughput Garbage Collector. The number of GC threads defaults to the number of CPUs on the machine. If you are running multiple JVMs on the same machine, or if your machine has more than 2 CPUs, to avoid the GC threads to be overly active you should reduce the number of GC threads by using:

-XX:+UseParallelGC -XX:ParallelGCThreads=2

Note: Although the setup of those parameters can be temporarily done manually changing (jserv.conf and jserv.properties in 11i; opmn.xml in R12), the correct way to update those parameters is using the context editor, update the variables s_oacore_nprocs and s_forms_servlet_nprocs, and then run autoconfig.

<<<<<Solution Approach#3: Long Running JVM >>>>>

Make sure to have the following setting in the jserv.properties file:
wrapper.bin.parameters=-DLONG_RUNNING_JVM=true

<<<<< Solution Approach#4: Java Cache Port Value >>>>>

The port value for java cache (s_java_object_cache_port) in context file should match profile option value JTF_DIST_CACHE value. 

 + The value for java cache port can be check in Context File -> parameter "s_java_object_cache_port" 
 + The value of the profile option JTF_DIST_CACHE can be found using the sql : 
      select fnd_profile.value('JTF_DIST_CACHE_PORT') from dual

<<<<< Solution Approach#5: Same Server Multiple EBS Instances >>>>>>

If the E-Business suite environment has multiple application servers then ensure that the following Autoconfig variables point to a "local disk". Hence this must be checked and changed on each application tier context file. 
     s_lock_pid_dir 
     s_pids_dir 
     s_web_pid_file

If the above variables are reset then run autoconfig for the changes to be effective.

<<<<< Solution Approach#6: EBS Instances with Load Balancer >>>>>>

Refer Oracle Support Note: Using Load-Balancers with Oracle E-Business Suite Release 12 (Doc ID 380489.1)

1. Make sure that the load balancer always implements Session Stickiness (also named Session Binding). This ensures that every time a request with an existing session is received it will be sent to the HTTP Server which created the session and therefore the routingID will always be able to be correctly decoded.

<<<<<<<<<<<<<<<Solution Approach#7: AJP Ports>>>>>>>>>>>>>>>

Ensure the port range used by AJP protocol by various middle tier components are not occupied by any other service. Run the following command to find the AJP port range.

$grep -i ajp $CONTEXT_FILE 

<ajp_protocol oa_var="s_ajp_protocol">ajp</ajp_protocol> 
<oacore_ajp_portrange oa_var="s_oacore_ajp_portrange" oa_type="PORT" base="21500" step="5" range="5" 
label="OC4J AJP Port Range for Oacore">21500-21504</oacore_ajp_portrange> 
<forms_ajp_portrange oa_var="s_forms_ajp_portrange" oa_type="PORT" base="22000" step="5" range="5" 
label="OC4J AJP Port Range for Forms">22000-22004</forms_ajp_portrange> 
<oafm_ajp_portrange oa_var="s_oafm_ajp_portrange" oa_type="PORT" base="25000" step="5" range="5" 
label="OC4J AJP Port Range for Oafm">25000-25004</oafm_ajp_portrange>

<<<<< Solution Approach#8: Firewall Between EBS Application Servers >>>>>

If any firewall exists between application server (Like firewall between forms/web tier & Concurrent tier etc) then refer the following workaround.

To improve performance, the mod_oc4j component in Oracle HTTP Server(OHS) process maintains open TCP connections to the AJP port within each OC4J components. In situations where a firewall exists between OHS and OC4J, packages sent via AJP are rejected if the connections can be idle for periods in excess of the inactivity timeout of stateful firewalls. However, the AJP socket is not closed as long as the socket remains open, the worker thread is tied to it and is never returned to the thread pool. OC4J will continue to create more threads, and will eventually exhaust system resources. 

Set the following parameters in the mod_oc4j.conf (Present in the directory $IAS_ORACLE_HOME/Apache/Apache/conf) configuration file. The value of Oc4jConnTimeout sets the length of inactivity in seconds, before the session is considered inactive. 

Oc4jUserKeepalive on 
Oc4jConnTimeout 12000 (or a similar value)

Also read, how to monitor JDBC Connections in my blog.
http://mkkoracleapps.blogspot.co.uk/2013/07/how-to-monitor-jdbc-and-jvm-in.html

Cheers!!!
Malay Khawas
Oracle Apps/Fusion DBA

Oracle Database 12c: Interactive Quick Reference

Oracle Database 12c: Interactive Quick Reference is now available from the Oracle Learning Library! 

With this interactive poster, you can find descriptions of database architectural components, DBA view information, performance view information, background process information, as well as references to relevant documentation.

Enjoy!!! reading the following link:
http://pub.vitrue.com/Rmh2

New features of 12c Database: 
http://docs.oracle.com/cd/E24628_01/doc.121/e25353/whats_new.htm