Performance tuning on OID - Consolidated Details

Performance tuning on OID - Consolidated Details

OID_Perf_Reco_1: Set DSA config to skip referrals

This is applicable when there are no referrals setup in OID.

By default this capability IS ENABLED and severely impacts performance when large groups (>200K) or large number of nested groups are involved.

Definition: A referral is a special type of entry that when obtained in a search, it contains the location of the actual entry, which could be in another part of the directory tree or even in another ldap server altogether. Unless you have specifically set-up referrals you most likely do not have any.

Confirm, there is no "Referrals" in place in OID:
ldapsearch -h mkkoidserver1 -p XXXX -D "cn=orcladmin" -w "xxxxxxxxxxxxx" -s sub -b "" objectclass=referral

If this doesn't return any rows then, there are no referrals in place.

Set the value of orclskiprefinsql in DSA config, to 1. This would make DSA config to skip referrals.

ldapmodify -h mkkoidserver1 -p XXXX -D cn=orcladmin -w xxxxxxxxxxxxx << eof
dn: cn=dsaconfig,cn=configsets,cn=oracle internet directory
changetype: modify 
replace: orclskiprefinsql 
orclskiprefinsql: 1 
eof



OID_Perf_Reco_2: orclinmemfiltprocess is very expensive on Oracle Database

This attribute can help significantly with the performance of certain types of search operations. It has been identified to be particularly useful with OAM, as some of the searches OAM performs can be especially expensive in the database without the use of "orclinmemfiltprocess".

ldapmodify -h mkkoidserver1 -p XXXX -D "cn=orcladmin" -w "xxxxxxxxxxxxx" -v <EOF 
dn: cn=dsaconfig,cn=configsets,cn=oracle internet directory 
changetype: modify 
replace: orclinmemfiltprocess 
orclinmemfiltprocess:(|(!(obuseraccountcontrol=*))(obuseraccountcontrol=activated)) 
orclinmemfiltprocess:(|(obuseraccountcontrol=activated)(!(obuseraccountcontrol=*))) 
orclinmemfiltprocess:(obapp=groupservcenter)(!(obdynamicparticipantsset=*)) 
orclinmemfiltprocess:(objectclass=oblixworkflowinstance) 
orclinmemfiltprocess:(objectclass=inetorgperson) 
orclinmemfiltprocess:(objectclass=oblixorgperson) 
orclinmemfiltprocess:(objectclass=oblixworkflowstepinstance) 
EOF

For OID 11g it should come as default, cross check and apply, it it applicable for you.

OID_Perf_Reco_3: Run oidstats.sql Regularly as part of daily housekeeping/maintenance

Run oidstats.sql any-time large updates are made to the OID. For large Active Directories, where changes are very frequent, and AD to OID to synchronisation is enabled, this is a very good option to perform on regular basis.

. $HOME/oidenv.sh
cd $MW_HOME/Oracle_IDM1/ldap/admin
sqlplus ods/xxxxxxxxxxxxx@OIDDB
START oidstats.sql;

Remove Dangling DNs: http://mkkoracleapps.blogspot.co.uk/2013/10/remove-dangling-dns-from-oid.html

Removing Configured WebLogic Server Domain

How to remove a domain from a WebLogic Server Installation... As it is just a domain deployment, no tool is required, just remove the appropriate content as given below.....

Make sure you are keeping a proper backup before performing the steps mentioned below:

1. Remove the domain directory $MW_HOME/user_projects/domains/<Domain_Name>
Here in this example I am removing: eag_domain, so issue the following command,

rm -rf $MW_HOME/user_projects/domains/eag_domain

2. Remove the line for eag_domain from domain-registry.xml file

$ cat domain-registry.xml
<?xml version="1.0" encoding="UTF-8"?>
<domain-registry xmlns="http://xmlns.oracle.com/weblogic/domain-registry">
  <domain location="/opt/oracle/OAMLIVE_MW_HOME/WebLogic/user_projects/domains/OAMDomain"/>
  <domain location="/opt/oracle/OAMLIVE_MW_HOME/WebLogic/user_projects/domains/eag_domain"/>


3. Remove the line for eag_domain from nodemanager.domains

cd $MW_HOME/wlserver_10.3/common/nodemanager

$ cat nodemanager.domains
#Domains and directories created by Configuration Wizard
#Fri Jul 12 11:44:27 BST 2013
eag_domain=/opt/oracle/OAMLIVE_MW_HOME/WebLogic/user_projects/domains/eag_domain
OAMDomain=/opt/oracle/OAMLIVE_MW_HOME/WebLogic/user_projects/domains/OAMDomain

TABLESPACE Monitoring

Useful SQLs to monitor tablespace

Find the Tablespace which having less than 5% space left

This query will give the output for APPS_TS_TX_DATA & APPS_TS_TX_IDX tablespaces:
select trunc(sysdate), c.name, b.tablespace_name, tbs_size, tbs_size - a.free_space, a.free_space, round(a.free_space / tbs_size * 100, 0) per_free
from  (select tablespace_name, round(sum(bytes)/1024/1024/1024 ,2) as free_space from dba_free_space group by tablespace_name) a,
(select tablespace_name, sum(bytes)/1024/1024/1024 as tbs_size from dba_data_files group by tablespace_name) b, (select name from v$database) c
where a.TABLESPACE_NAME(+)=B.TABLESPACE_NAME and a.TABLESPACE_NAME like 'APPS_TS_TX%' and round(a.free_space / tbs_size * 100, 0) <= 5 order by 6;

This query will give the output for all tablespaces:
select trunc(sysdate), c.name, b.tablespace_name, tbs_size, tbs_size - a.free_space, a.free_space, round(a.free_space / tbs_size * 100, 0) per_free
from  (select tablespace_name, round(sum(bytes)/1024/1024/1024 ,2) as free_space from dba_free_space group by tablespace_name) a,
(select tablespace_name, sum(bytes)/1024/1024/1024 as tbs_size from dba_data_files group by tablespace_name) b, (select name from v$database) c
where a.TABLESPACE_NAME(+)=B.TABLESPACE_NAME and round(a.free_space / tbs_size * 100, 0) <= 5 order by 6;

Tablespace Percentage(%) Used

It will give How much Percentage(%) is already used for a Particular Tablespace

SET LINESIZE 200

SELECT A.TABLESPACE_NAME, ROUND(A.BYTES/1024/1024) "TOTAL", ROUND(B.BYTES/1024/1024) "USED", ROUND(C.BYTES/1024/1024) "FREE",

ROUND((B.BYTES*100)/A.BYTES) "% USED", ROUND((C.BYTES*100)/A.BYTES) "% FREE"

FROM SYS.SM$TS_AVAIL A, SYS.SM$TS_USED B, SYS.SM$TS_FREE C

WHERE A.TABLESPACE_NAME=B.TABLESPACE_NAME AND A.TABLESPACE_NAME=C.TABLESPACE_NAME AND A.TABLESPACE_NAME='&PLEASE_PROVIDE_TABLESPACE_NAME';

Add Datafile to Tablespace

SELECT * FROM dba_tablespace_usage_metrics ORDER BY used_percent;

SELECT max(bytes) FROM dba_free_space WHERE tablespace_name = 'APPS_TS_TX_DATA'; 

SELECT FILE_NAME FROM DBA_DATA_FILES WHERE TABLESPACE_NAME='<TS_NAME>' ORDER BY FILE_NAME;

ALTER TABLESPACE <TABLE_SPACE_NAME> ADD DATAFILE '<DBF_NAME_WITH_LOCATION>' SIZE 4096M;

ALTER TABLESPACE <TABLE_SPACE_NAME> ADD DATAFILE '<DBF_NAME_WITH_LOCATION>' SIZE 4096M autoextend on;

Alter database datafile '/xxxxx/xxxx/zpbdxx.dbf' autoextend on;

select ts.name||'|'||df.name||'|'||bytes/1024/1024||'|'||CREATE_BYTES/1024/1024 from v$tablespace ts, v$datafile df where ts.ts#=df.ts# and ts.name ='&TABLESPACE_NAME';

ALTER TABLESPACE <TS_NAME> ADD DATAFILE '<DBF_NAME_WITH_PATH>' SIZE 100M AUTOEXTEND ON NEXT 10M MAXSIZE 4096M;

ALTER TABLESPACE APPS_TS_TX_IDX ADD DATAFILE '/u01/UAT4/oracle/db/apps_st/data/APPS_TS_TX_IDX_002.dbf' SIZE 100M AUTOEXTEND ON NEXT 10M MAXSIZE 4096M;

Add Datafile to Tablespace Using Autoextent ON

ALTER TABLESPACE <TS_NAME> ADD DATAFILE '<DBF_NAME_WITH_PATH>' SIZE 100M AUTOEXTEND ON NEXT 10M MAXSIZE 4096M;

ALTER TABLESPACE APPS_TS_TX_IDX ADD DATAFILE '/u01/UAT4/oracle/db/apps_st/data/APPS_TS_TX_IDX_002.dbf' SIZE 100M AUTOEXTEND ON NEXT 10M MAXSIZE 4096M;

Resize Datafile

SQL> alter database datafile '/prod/oradata/custom01/prod/ts_custom_data13.dbf' resize 4096m;

SQL> alter database datafile '/prod/oradata/custom01/prod/ts_custom_data13.dbf' modify autoextend by 50m;

Using staticports.ini for Oracle HTTP Server during OAM/EBS Integration

Using staticports.ini for Oracle HTTP Server during OAM/EBS Integration

Instead of 7777 of HTTP Server port, we had a requirement for 7778... 

HTTP WebTier Version 11.1.1.6

Create a file staticports.ini before starting the installation, and choose manual port configuration:

[OPMN]
OPMN Local Port = 6706
OPMN Remote Port = 6707

[OHS]
OHS Port = 7778
OHS Proxy Port = 9998
OHS SSL Port = 4444

[WEBCACHE]
Web Cache Listen Port = 7790
Web Cache Admin Port = 7791
Web Cache Statistics Port = 7792
Web Cache Invalidation Port = 7793
Web Cache SSL Port = 7794

appdev WEBTIERDEV /export/home/appdev $ opmnctl status -l

Processes in Instance: instance1
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
ias-component                    | process-type       |     pid | status   |        uid |  memused |    uptime | ports
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
webcache1                        | WebCache-admin     |   11620 | Alive    | 1319634190 |    12896 |  90:03:21 | http_admin:7791
webcache1                        | WebCache           |   11619 | Alive    | 1319634189 |    35176 |  90:03:21 | http_stat:7792,http_invalidation:7793,https_listen:7794,http_listen:7790
ohs1                             | OHS                |   11618 | Alive    | 1319634188 |     4680 |  90:03:21 | https:9998,https:4444,http:7778

appdev WEBTIERDEV /export/home/appdev $

Data Source/JDBC Connection Pool tuning for eBusiness AccessGate

Data Source/JDBC Connection Pool tuning for eBusiness AccessGate

I noticed, suddenly, eBusiness Access through Oracle Access Manager stopped working. When I looked into AccessGate Managed server log, I found the following informations.

Jul 30, 2013 10:16:52 AM oracle.apps.fnd.ext.sso.AppsHttpServletRequestWrapper <init>
SEVERE: Cannot retrieve LDAP information for guid='NOT_FOUND'
Jul 30, 2013 10:17:35 AM oracle.apps.fnd.ext.sso.EbsServer$2 load
SEVERE: Failed to get server connection server=113
weblogic.jdbc.extensions.PoolLimitSQLException: weblogic.common.resourcepool.ResourceLimitException: No resources currently available in pool EBSLIVE to allocate to applications, please increase the size of the pool and retry..
at weblogic.jdbc.common.internal.JDBCUtil.wrapAndThrowResourceException(JDBCUtil.java:252)
at weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:456)

Solution

This is purely data source related issue. When you deploy AccessGate for eBusiness Suite, the ANT deployment create the data source also.

Login to WebLogic Console >>> Services >>> Data Source >>> Click on the Data Source EBSLIVE >>> Connection Pool Tab >>> Advanced

Set Initial Capacity = Max capacity of JDBC connection pool
Set Inactive Connection Timeout to 5 Minutes
Increase Maximum Capacity Value from Default Value of 15 to 30.

Restart AccessGate Managed and Admin Server....

How to Start Complete OAM Fusion Middleware Stack

How to Start Complete OAM Fusion Middleware Stack

Start OID Components:

Start the Oracle WebLogic Administration Server for OID:
. $HOME/oidenv.sh
echo $DOMAIN_HOME
nohup $DOMAIN_HOME/bin/startWebLogic.sh -Dweblogic.management.username=weblogic -Dweblogic.management.password={{{WebLogicPass}}} > $HOME/oidweblogic.log 2>&1 &

Start OID OPMN Component:
. $HOME/oidenv.sh
opmnctl startall
opmnctl status
opmnctl status -l

Start WebLogic Managed Server for OID:
. $HOME/oidenv.sh
nohup $DOMAIN_HOME/bin/startManagedWebLogic.sh wls_ods1 http://mkkoamoidserver1:7001 -Dweblogic.management.username=weblogic -Dweblogic.management.password={{{WebLogicPass}}} -Dweblogic.system.StoreBootIdentity=true > $HOME/oidmanaged.log 2>&1 &

**************************

Start OAM Components:

Start the Oracle WebLogic Administration Server for OAM:
. $HOME/oamenv.sh
echo $DOMAIN_HOME
nohup $DOMAIN_HOME/bin/startWebLogic.sh -Dweblogic.management.username=weblogic -Dweblogic.management.password={{{WebLogicPass}}} > $HOME/oamweblogic.log 2>&1 &

Start WebLogic Managed Server for OAM:
. $HOME/oamenv.sh
echo $DOMAIN_HOME
nohup $DOMAIN_HOME/bin/startManagedWebLogic.sh oam_server1 http://mkkoamoidserver1:7002 \
-Dweblogic.management.username=weblogic -Dweblogic.management.password={{{WebLogicPass}}} \
-Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true -Dweblogic.system.StoreBootIdentity=true > $HOME/oammanaged.log 2>&1 &

****************************


Start HTTP WebTier:

. $HOME/webtierenv.sh
opmnctl startall
opmnctl status
opmnctl status -l

***************************


Start EBS Access Gate:

Start WebLogic AdminServer:
. $HOME/eagenv.sh
echo $DOMAIN_HOME

nohup $DOMAIN_HOME/bin/startWebLogic.sh -Dweblogic.http.isWLProxyHeadersAccessible=true \
-Dweblogic.management.username=weblogic -Dweblogic.management.password={{{WebLogicPass}}} > $HOME/eagweblogic.log 2>&1 &

Start WebLogic Managed Server:
. $HOME/eagenv.sh
echo $DOMAIN_HOME

nohup $DOMAIN_HOME/bin/startManagedWebLogic.sh eag_server1 http://mkkoamoidserver1:7041 \
-Dweblogic.http.isWLProxyHeadersAccessible=true \
-Dweblogic.management.username=weblogic -Dweblogic.management.password={{{WebLogicPass}}} \
-Dweblogic.system.StoreBootIdentity=true > $HOME/eagmanaged.log 2>&1 &

******************************

You have encountered an unexpected PLSQL Error, Please contact System Administrator

You have encountered an unexpected PLSQL Error, Please contact System Administrator

Justification: Cloning has de-registered the EBS instance from OID, and that is the reason, Autolink of user from EBStoOID is not happening.

If you manually update the user with the GUID information, single sign-on is working fine.

When enabled FND Debug, in fnd_log_messages I noticed the following:

106500486|fnd.plsql.oid.fnd_ldap_util.get_oid_session: |ORA-31202: DBMS_LDAP: LDAP client/server error: Invalid credentials|

106500487|fnd.plsql.oid.fnd_ldap_util.c_get_oid_session: |ORA-31202: DBMS_LDAP: LDAP client/server error: Invalid credentials|

106500488|fnd.plsql.oid.fnd_oid_plug.get_realm_dn: |END ->ORA-31202: DBMS_LDAP: LDAP client/server error: Invalid credentials|

106500489|fnd.plsql.oid.fnd_ldap_wrapper.create_user: |ORA-31202: DBMS_LDAP: LDAP client/server error: Invalid credentials|

**** This happened during one of my GOLIVE and during the cutover we window we had the cloning of production is also involved...

**** Solution is straight forward... OID component has been deregistered during the cloning process...

Register using following command:

$FND_TOP/bin/txkrun.pl -script=SetSSOReg -registeroid=yes -provisiontype=4

How to Turn Archiving OFF/ON

How to Turn Archiving OFF/ON

1. Shut down the database instance
SQL> shutdown immediate

2. Start up a new instance and mount, but do not open the database
SQL> startup mount

3. Put the database into archivelog mode/noarchivelog mode
SQL> alter database noarchivelog;
SQL> alter database archivelog;                                  >>>>>>>>>>> IF YOU NEED TO TURN ON ARCHIVING

4. Open the database
SQL> alter database open; 

5. Verify your database is now in noarchivelog/archivelog mode.
SQL> ARCHIVE LOG LIST
Database log mode                   Archive Mode
Automatic archival                    Disabled
Archive destination                   USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence         22
Next log sequence to archive      24
Current log sequence                 24

SQL> alter system switch logfile;

RMAN Archive Log Delete - Ready Reference

RMAN Archive Log Delete - Ready Reference

Archive Log Location: 


select name from SYS.V_$ARCHIVED_LOG;

I use the following archive log delete command, and that solve my purpose:

RMAN> delete noprompt archivelog all completed before 'SYSDATE-3' backed up 01 times to device type disk;
RMAN> DELETE FORCE NOPROMPT OBSOLETE RECOVERY WINDOW OF 2 DAYS DEVICE TYPE DISK;
RMAN> DELETE FORCE NOPROMPT OBSOLETE REDUNDANCY = 2 DEVICE TYPE DISK;
RMAN> DELETE NOPROMPT FORCE ARCHIVELOG UNTIL TIME 'SYSDATE-1' DEVICE TYPE DISK;
RMAN> DELETE NOPROMPT FORCE ARCHIVELOG UNTIL TIME 'SYSDATE-1/2' DEVICE TYPE DISK;
RMAN> DELETE NOPROMPT FORCE ARCHIVELOG UNTIL SEQUENCE 260 DEVICE TYPE DISK;


Archive Log List:

RMAN> list archivelog all;
RMAN> list copy of archivelog until time ‘SYSDATE-10′;
RMAN> list copy of archivelog from time ‘SYSDATE-10′
RMAN> list copy of archivelog from time ‘SYSDATE-10′ until time ‘SYSDATE-2′;
RMAN> list copy of archivelog from sequence 1000;
RMAN> list copy of archivelog until sequence 1500;
RMAN> list copy of archivelog from sequence 1000 until sequence 1500;

Archive Log Delete:

RMAN> delete archivelog all;
RMAN> delete archivelog until time ‘SYSDATE-10′;
RMAN> delete archivelog from time ‘SYSDATE-10′
RMAN> delete archivelog from time ‘SYSDATE-10′ until time ‘SYSDATE-2′;
RMAN> delete archivelog from sequence 1000;
RMAN> delete archivelog until sequence 1500;
RMAN> delete archivelog from sequence 1000 until sequence 1500;

Oracle Access Manager 11gR2 for eBusiness R12.1.3

Oracle Access Manager 11gR2 for eBusiness R12 12.1.3

Two weeks back, I completed another Access Manager Implementation to provide zero sign to eBusiness Suite 12.1.3.


I have used following versions for this implementation:

Oracle Access Manager: 11.1.2.0
Oracle Identity Management: 11.1.1.6
Oracle Access Manager WebGate: 11.1.2.0
Oracle E-Business Suite AccessGate: 1.2
Oracle Fusion Middleware WebTier 11.1.1.7
Oracle Database for RCU: 11.2.0.3
Oracle E-Business Suite Release 12: 12.1.3

• For Windows Native Authentication we had Windows Server 2003 AD.
• Most of the client Machines are Windows 7, also tested from Mac OS.

Another level of complexity was to integrate Oracle Access Manager with Microsoft UAG. For external users UAG is delegating the Kerberos Authentication with WNA CredCollector on behalf of user.

• AD to OID Synchronisation with, of more than 70K records.
• External Authentication Plugin Set-up AD ldapbind and AD ldapcompare) in case, WNA is not operational.

• In front of OAM, F5 Load balancer and similarly for access gate another F5 Load balancer  SSL termination is happening at F5 Level. As UAG is in picture, Proxy Header SSL is also implemented.

Also, in one of the Development Server 2 Instances of Test OAM Installation is really a good experience. Still I am struggling to find the reason, let me call it as Installation#1(first installation) and Installation#2.

If I start Installation#2 after server reboot, then I am unable to start Installation#1. Reverse is not true.

Another good experience is Playing with KVNO. If you set password for a user, used for keytab file, multiple time, KVNO goes into three digit and kerberos would not work, until and unless you mention -kvno in the command line argument.

Though in terms of Identity and Access Management it was 5th Implementation Project, but for various complexities it was pleasant experience to explore.

How to check AD to OID Synch is happening fine

How to check AD to OID Synch is happening fine

Run this command for 2-3 Hours intervals:

ldapsearch -h mkktestserver1.unixdomain.local -p 3060 -D cn=orcladmin -w MalayFalsePass -s sub -b "" -T -createtimestamp "objectclass=orcluserv2" createtimestamp > createlist1.txt

ldapsearch -h mkktestserver1.unixdomain.local -p 3060 -D cn=orcladmin -w MalayFalsePass -s sub -b "" -T -createtimestamp "objectclass=orcluserv2" createtimestamp > createlist2.txt

ldapsearch -h mkktestserver1.unixdomain.local -p 3060 -D cn=orcladmin -w MalayFalsePass -s sub -b "" -T -createtimestamp "objectclass=orcluserv2" createtimestamp > createlist3.txt

ldapsearch -h mkktestserver1.unixdomain.local -p 3060 -D cn=orcladmin -w MalayFalsePass -s sub -b "" -T -createtimestamp "objectclass=orcluserv2" createtimestamp > createlist4.txt

Open the createlist?.txt file and make sure at the end of the file you see new entries are getting created.

ASM Related steps for 11gR2 Grid Infrastructure

ASM Related steps for 11gR2 Grid Infrastructure

I am putting this ASM related steps only, on request of a friend. In two projects where I handled HP EVA Storage also, I setup 2 node RAC for eBusiness Suite. If you require any other part of the implementation experience, would be happy to share.

This was done for Oracle Enterprise Linux

Prepare Storage and Disks

1. Run fdisk for RDBMS and GRID Binaries Multipathing Disk(Both Node)
2. Run partprobe for RDBMS and GRID Binaries Multipathing Disk(Both Node)
3. Run fdisk for ASM Related Multipathing Disk(Database Node1)
4. Run partprobe for ASM Related Multipathing Disk(Both Node)
5. Run mkfs.ext3 for RDBMS and GRID Binaries Multipathing Disk(Database Node1)

Multipathing Disks Used for ASM:

mpath2 - PRODDATA
mpath6 - PRODFRA
mpath5 - PRODOCR1
mpath4 - PRODOCR2
mpath3 - PRODOCR3

As an example fdisk and partprobe is shown for mapth2 disk, same needs to be done for others:

[root@mkkracdbserver1 ~]# fdisk /dev/mapper/mpath2
[root@mkkracdbserver1 ~]# partprobe /dev/mapper/mpath2

[root@mkkracdbserver1 ~]# ls -ltr /dev/mapper/mpath2*
brw-rw---- 1 root disk 253,  0 Aug  9 00:02 /dev/mapper/mpath2
brw-rw---- 1 root disk 253, 10 Aug  9 00:02 /dev/mapper/mpath2p1

[root@mkkracdbserver2 ~]# partprobe /dev/mapper/mpath2

[root@mkkracdbserver2 ~]# ls -ltr /dev/mapper/mpath2*
brw-rw---- 1 root disk 253, 0 Aug  6 18:07 /dev/mapper/mpath2
brw-rw---- 1 root disk 253, 9 Aug  9 00:06 /dev/mapper/mpath2p1



Install Oracle ASM Libraries RPM

# rpm -ivh oracleasm-support-2.1.4-1.el5.x86_64.rpm
# rpm -ivh oracleasmlib-2.0.4-1.el5.x86_64.rpm
# rpm -ivh oracleasm-2.6.18-238.el5-2.0.5-1.el5.x86_64.rpm

# /etc/init.d/oracleasm init
# /etc/init.d/oracleasm exit
# /etc/init.d/oracleasm status

To Load and initialize the ASMLib driver issue init
To Stop the ASMLib driver issue exit
To Display the status of the Oracle ASMLib driver issue status




Configure ASMLib Driver

This needs to be done on both the Database Node

# /etc/init.d/oracleasm configure

[root@mkkracdbserver1 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

Default user to own the driver interface []: oraprod
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]

[root@mkkracdbserver2 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

Default user to own the driver interface []: oraprod
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]




Create ASM Disks

1. Use Oracleasm Createdisk Command on Database Node1
2. Use Oracleasm Listdisks Command on Database Node1
2. Use Oracleasm Scandisks Command on Database Node2
2. Use Oracleasm Listdisks Command on Database Node2

Important Note: Do not deviate from the above order, or else you may end up with non-visible ASM disks while doing the installation. You will have to change the diskstring parameter to asm_diskstring='/dev/oracleasm/disks/*' to make the disks visible. To overcome the issue, reboot both the database node and disks will be visible.


[root@mkkracdbserver1 ~]# oracleasm createdisk ASMDATA01 /dev/mapper/mpath2p1
Writing disk header: done
Instantiating disk: done

[root@mkkracdbserver1 ~]# oracleasm createdisk ASMFRA01 /dev/mapper/mpath6p1
Writing disk header: done
Instantiating disk: done

[root@mkkracdbserver1 ~]# oracleasm createdisk ASMOCR01 /dev/mapper/mpath5p1
Writing disk header: done
Instantiating disk: done

[root@mkkracdbserver1 ~]# oracleasm createdisk ASMOCR02 /dev/mapper/mpath4p1
Writing disk header: done
Instantiating disk: done

[root@mkkracdbserver1 ~]# oracleasm createdisk ASMOCR03 /dev/mapper/mpath3p1
Writing disk header: done
Instantiating disk: done

[root@mkkracdbserver1 ~]# oracleasm listdisks
ASMDATA01
ASMFRA01
ASMOCR01
ASMOCR02
ASMOCR03

[root@mkkracdbserver2 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "ASMDATA01"
Instantiating disk "ASMOCR03"
Instantiating disk "ASMOCR02"
Instantiating disk "ASMOCR01"
Instantiating disk "ASMFRA01"

[root@mkkracdbserver2 ~]# oracleasm listdisks
ASMDATA01
ASMFRA01
ASMOCR01
ASMOCR02
ASMOCR03


Implement Metalink Note 1059847.1 for Multipathing Disks

This needs to be done on both the Database Nodes

Important Note: If this is not done, then root.sh will fail in Database Node2.

11GR2 GRID INFRASTRUCTURE INSTALLATION FAILS WHEN RUNNING ROOT.SH ON NODE 2 OF RAC USING ASMLIB [ID 1059847.1]

1. Modify the /etc/sysconfig/oracleasm with:

ORACLEASM_SCANORDER="dm"
ORACLEASM_SCANEXCLUDE="sd"

# /etc/init.d/oracleasm restart

[root@mkkracdbserver1 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@mkkracdbserver1 ~]#

[root@mkkracdbserver2 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@mkkracdbserver2 ~]#

Cheers !!!!
Malay Khawas
Oracle Apps/Fusion DBA

How to do test of ar package, Installed on OS | Exception String: Error in invoking target 'nnfgt.o' of makefile

How to do test of ar package, Installed on OS | Exception String: Error in invoking target 'nnfgt.o' of makefile

a) create the following program called test.c :

main()
{
printf("hello\n");
}

b) compile it to a .o file :

gcc -c test.c

This generates an object file called test.o

c) Now test if basic ar functionality works;

ar cr myarch.a test.o

--> does the myarch.a file get created ok ?
--> if so can you query it with ar -t myarch.a ?

d) now test ar rv myarch2.a test.o 

Spool:

$ gcc -c test.c
$ ls -ltr
total 25970
-rw-r--r-- 1 apppoc dba 174 Apr 8 16:09 local.profile
-rw-r--r-- 1 apppoc dba 157 Apr 8 16:09 local.login
-rw-r--r-- 1 apppoc dba 136 Apr 8 16:09 local.cshrc
drwxr-xr-x 2 apppoc dba 96 May 2 12:18 bea
-rw------- 1 apppoc dba 13004722 May 5 02:03 core
-rw-r--r-- 1 apppoc dba 449 May 5 10:49 oamenv2.sh
-rw-r--r-- 1 apppoc dba 283726 May 6 08:07 core_strings.txt
-rw-r--r-- 1 apppoc dba 486 May 7 11:32 oamenv.sh
-rw-r--r-- 1 apppoc dba 30 May 8 12:07 test.c
-rw-r--r-- 1 apppoc dba 732 May 8 12:07 test.o
$ ar cr myarch.a test.o
Illegal Instruction(coredump)
$ ar rv myarch2.a test.o
a - test.o
Illegal Instruction(coredump)
$ cat test.c
main()
{
printf("hello\n");
}
$


Lesson Learned:

While doing OID Installation, config.sh was failing with following to execute the make command.
Exception String: Error in invoking target 'nnfgt.o' of makefile in OID Installation

Issue with opmnctl Startup or Status - Swap Issue

Issue with opmnctl Startup or Status - Swap Issue

applive WEBTIERLIVE /export/home/applive $ opmnctl status
Error occurred during initialization of VM
Could not reserve enough space for object heap
applive WEBTIERLIVE /export/home/applive $

Increase the OS Level Swap Space. It work fine for you.

Connecting to EM Fails With Error "503 Service Unavailable" after OID Installation

Connecting to EM Fails With Error "503 Service Unavailable" after OID Installation

After Installation of Oracle Internet Directory, I was unable to connect to EM and getting an error 503 Service Unavailable.

When I cheched the targets.xml file, I noticed OID EM Farm entry were not present.

$DOMAIN_HOME/sysman/state/targets.xml

I put the following entries immediately after the very first <Targets> line:

<Target TYPE="oracle_ias_farm" NAME="Farm_IDMDomain" DISPLAY_NAME="Farm_IDMDomain">
<Property NAME="MachineName" VALUE="mkkoidserver1.mkkdomain.local"/>
<Property NAME="Port" VALUE="7001"/>
<Property NAME="Protocol" VALUE="t3"/>
<Property NAME="isLocal" VALUE="true"/>
<Property NAME="serviceURL" VALUE="service:jmx:t3://mkkoidserver1.mkkdomain.local:7001/jndi/weblogic.management.mbeanservers.domainruntime"/>
<Property NAME="WebLogicHome" VALUE="/opt/oracle/IDMLIVE_MW_HOME/WebLogic/wlserver_10.3"/>
<Property NAME="DomainHome" VALUE="/opt/oracle/IDMLIVE_MW_HOME/WebLogic/user_projects/domains/IDMDomain"/>
</Target>

Restart the OID Stack.

ldap_bind: UnKnown Error Encountered for Java External Authentication Plugin

ldap_bind: UnKnown Error Encountered for Java External Authentication Plugin

applive IDMLIVE /export/home/applive $ ldapcompare -h mkkoidserver1 -p 3060 -D "cn=orcladmin" -w ************* -b "cn=adtooidsyncuser generic,ou=information technology,ou=india,cn=users,dc=mkk,dc=ad,dc=local" -a userPassword -v "*************"
ldap_compare_s: UnKnown Error Encountered

applive IDMLIVE /export/home/applive $

applive IDMLIVE /export/home/applive $ ldapbind -h mkkoidserver1 -p 3060 -D "cn=adtooidsyncuser generic,ou=information technology,ou=india,cn=users,dc=mkk,dc=ad,dc=local" -w "*************"
ldap_bind: UnKnown Error Encountered


There is a bug with OID 11.1.1.6.

Due to issues to login to ODSM, I changes orcljvmoptions from 64M to 512M using LDAPMODIFY command.

<Jun 18, 2013 4:58:33 PM BST> <Error> <oracle.adfinternal.view.faces.config.rich.RegistrationConfigurator> <BEA-000000> <ADF_FACES-60096:Server Exception during PPR, #8

If you face this issue you would have to increase orcljvm value to 512 or something more than 64M.

●● Use ldapmodify to update heap size for dsaconfig:

ldapmodify -h mkkoidserver1 -p 3060 -D cn=orcladmin -w ************* << eof
dn: cn=dsaconfig,cn=configsets,cn=oracle internet directory
changetype: modify
replace: orcljvmoptions
orcljvmoptions: -Xmx512M
eof

●● Restart Complete OID Stack.

If you try to change this using /em, instead of -Xmx512M, it would be saved as -xmx512m. Even LDAPMODIFY also saved the entry in small letter only. Note x and m are in small letter.

applive IDMLIVE /export/home/applive $ ldapsearch -h mkkoidserver1 -p 3060 -D cn=orcladmin -w ************* -b "cn=dsaconfig,cn=configsets,cn=oracle internet directory" -s base "objectclass=*"
cn=dsaconfig,cn=configsets,cn=oracle internet directory
orclallattrstodn=NOT ASCII
orclecachemaxentries=100000
orclecachemaxsize=209715200
orclecacheenabled=1
orclautocatalog=1
orcljvmoptions=-xmx512m
orclrscacheattr=uid
orclrscacheattr=mail
orclrscacheattr=cn

For two of my instances it saved correctly with capital X and M. But for live instance it created a hell for me.

As of now, I directly updated the database. Make sure you take a proper backup. Hope issues from /em or ldapmodify would be resolved soon by Oracle.

select attrval from ds_attrstore where attrname = 'orcljvmoptions';

update ds_attrstore  set attrval='-Xmx64M' where attrname = 'orcljvmoptions';
commit;

I reverted back the value to 64M which comes as default during installation.


Cheers !!!!!
Malay Khawas
Oracle Apps/Fusion DBA

How to Monitor JDBC and JVM in eBusiness Suite

How to Monitor JDBC and JVM in eBusiness Suite

Use Oracle Support Note for complete script:
monitor_jdbc_conn.sql - Script to monitor JDBC connections in Apps eBusiness Suite (Doc ID 557194.1)

JDBC Connection Usage Per JVM Process:
select machine, process, count(*) from gv$session
where program like '%JDBC%'
group by machine, process
order by 1 asc;

Connection Usage Per Module
select count(*), module
from gv$session
where program like '%JDBC%'
group by module
order by 1 asc;

Idle connections for more than 3 hours:
select count(*),machine, program
from gv$session
where program like '%JDBC%'
and  last_call_et > 3600 *3
group by machine, program;

Inactive connections which last ran fnd_security_pkg.fnd_encrypted_pwd:
select s.sql_hash_value, t.sql_text, s.last_call_et
from gv$session s , gv$sqltext t
where s.username = 'APPLSYSPUB'
and s.sql_hash_value= t.hash_value
and t.sql_text like  '%fnd_security_pkg.fnd_encrypted_pwd(:1,:2,:3%';

mod_oc4j: Failed to find a failover oc4j process for session request for destination: application://oacore (no island or jgroup).

mod_oc4j: Failed to find a failover oc4j process for session request for destination: application://oacore (no island or jgroup).

<<<<<<<<<<<<<<<Explanation>>>>>>>>>>>>>>>

Islands are no longer used within iAS version 10.1.3 in EBS R12, the new term is "cluster". A cluster is stated as "two or more OC4J server nodes hosting the same set of applications". A standard instance of Applications Release12 has some aspects of clustering enabled by default. There are many components to a cluster and all of them have not been configured in Oracle E-Business suite R12. So the error may change as following :

mod_oc4j: Failed to find a failover oc4j process for session request for destination: application://form (no cluster or jgroup).

<<<<<<<<<<<<<<<Reason>>>>>>>>>>>>>>>

??? Load: e.g. Too many clients connecting to the application, The number of requests are high enough that the process cannot respond to new requests prior to reaching the timeout period.

??? Performance: Heavily loaded JVM or Lack of enough memory for JVM

??? Code: Internal to Application Server, one of the components are not operating correctly or efficiently. External to ApplicationServer, E-Business code (seeded or custom code) causing memory leaks.

??? Lack of free ports for AJP communication

<<<<<<<<<<<<<<<Solution Approach#1: Resize oc4j oacore jvm heap size as well as number of oacore processes in context file>>>>>>>>>>>>>>>


The number of jvms (oc4j in R12) is configured by the autoconfig variables s_oacore_nprocs, s_disco_nprocs, s_forms_servlet_nprocs (11i), s_forms_nprocs (R12) and s_xmlsvcs_nprocs.

In 11i, those changes are made in the $IAS_ORACLE_HOME/Apache/Jserv/etc/jserv.conf file, while in R12 in the $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml

Thumb Rule: 1 active JVM/OC4J instance per CPU core.  So if you have 8 CUPs of dual core, then you can configure upto 16 JVM. 1 JVM handles 100 Users, so based on the user load configure JVMs accordingly.


<<<<<Solution Approach#2: Proper Heap Configuration >>>>>

The heap is configured by s_oacore_jvm_start_option, s_forms_jvm_start_options in R12. For 12.1 and higher start with the following and increase as per the user load:
-Xmx1024M -Xms512M -XX:MaxPermSize=256M
      -XX:NewRatio=2 -XX:+PrintGCTimeStamps

Also, Add the following parameter to the DBC file:
JDBC\:oracle.jdbc.maxCachedBufferSize=262144

In JDK 1.6, the JVM detects that you have a server class machine (2 or more CPUs with 2GB or more memory), and will automatically enable Parallel Throughput Garbage Collector. The number of GC threads defaults to the number of CPUs on the machine. If you are running multiple JVMs on the same machine, or if your machine has more than 2 CPUs, to avoid the GC threads to be overly active you should reduce the number of GC threads by using:

-XX:+UseParallelGC -XX:ParallelGCThreads=2

Note: Although the setup of those parameters can be temporarily done manually changing (jserv.conf and jserv.properties in 11i; opmn.xml in R12), the correct way to update those parameters is using the context editor, update the variables s_oacore_nprocs and s_forms_servlet_nprocs, and then run autoconfig.

<<<<<Solution Approach#3: Long Running JVM >>>>>

Make sure to have the following setting in the jserv.properties file:
wrapper.bin.parameters=-DLONG_RUNNING_JVM=true

<<<<< Solution Approach#4: Java Cache Port Value >>>>>

The port value for java cache (s_java_object_cache_port) in context file should match profile option value JTF_DIST_CACHE value. 

 + The value for java cache port can be check in Context File -> parameter "s_java_object_cache_port" 
 + The value of the profile option JTF_DIST_CACHE can be found using the sql : 
      select fnd_profile.value('JTF_DIST_CACHE_PORT') from dual

<<<<< Solution Approach#5: Same Server Multiple EBS Instances >>>>>>

If the E-Business suite environment has multiple application servers then ensure that the following Autoconfig variables point to a "local disk". Hence this must be checked and changed on each application tier context file. 
     s_lock_pid_dir 
     s_pids_dir 
     s_web_pid_file

If the above variables are reset then run autoconfig for the changes to be effective.

<<<<< Solution Approach#6: EBS Instances with Load Balancer >>>>>>

Refer Oracle Support Note: Using Load-Balancers with Oracle E-Business Suite Release 12 (Doc ID 380489.1)

1. Make sure that the load balancer always implements Session Stickiness (also named Session Binding). This ensures that every time a request with an existing session is received it will be sent to the HTTP Server which created the session and therefore the routingID will always be able to be correctly decoded.

<<<<<<<<<<<<<<<Solution Approach#7: AJP Ports>>>>>>>>>>>>>>>

Ensure the port range used by AJP protocol by various middle tier components are not occupied by any other service. Run the following command to find the AJP port range.

$grep -i ajp $CONTEXT_FILE 

<ajp_protocol oa_var="s_ajp_protocol">ajp</ajp_protocol> 
<oacore_ajp_portrange oa_var="s_oacore_ajp_portrange" oa_type="PORT" base="21500" step="5" range="5" 
label="OC4J AJP Port Range for Oacore">21500-21504</oacore_ajp_portrange> 
<forms_ajp_portrange oa_var="s_forms_ajp_portrange" oa_type="PORT" base="22000" step="5" range="5" 
label="OC4J AJP Port Range for Forms">22000-22004</forms_ajp_portrange> 
<oafm_ajp_portrange oa_var="s_oafm_ajp_portrange" oa_type="PORT" base="25000" step="5" range="5" 
label="OC4J AJP Port Range for Oafm">25000-25004</oafm_ajp_portrange>

<<<<< Solution Approach#8: Firewall Between EBS Application Servers >>>>>

If any firewall exists between application server (Like firewall between forms/web tier & Concurrent tier etc) then refer the following workaround.

To improve performance, the mod_oc4j component in Oracle HTTP Server(OHS) process maintains open TCP connections to the AJP port within each OC4J components. In situations where a firewall exists between OHS and OC4J, packages sent via AJP are rejected if the connections can be idle for periods in excess of the inactivity timeout of stateful firewalls. However, the AJP socket is not closed as long as the socket remains open, the worker thread is tied to it and is never returned to the thread pool. OC4J will continue to create more threads, and will eventually exhaust system resources. 

Set the following parameters in the mod_oc4j.conf (Present in the directory $IAS_ORACLE_HOME/Apache/Apache/conf) configuration file. The value of Oc4jConnTimeout sets the length of inactivity in seconds, before the session is considered inactive. 

Oc4jUserKeepalive on 
Oc4jConnTimeout 12000 (or a similar value)

Also read, how to monitor JDBC Connections in my blog.
http://mkkoracleapps.blogspot.co.uk/2013/07/how-to-monitor-jdbc-and-jvm-in.html

Cheers!!!
Malay Khawas
Oracle Apps/Fusion DBA

Oracle Database 12c: Interactive Quick Reference

Oracle Database 12c: Interactive Quick Reference is now available from the Oracle Learning Library! 

With this interactive poster, you can find descriptions of database architectural components, DBA view information, performance view information, background process information, as well as references to relevant documentation.

Enjoy!!! reading the following link:
http://pub.vitrue.com/Rmh2

New features of 12c Database: 
http://docs.oracle.com/cd/E24628_01/doc.121/e25353/whats_new.htm