Tuning of HTTP WebTier for Access Manager SSO Implementation

Tuning of HTTP WebTier for Access Manager SSO Implementation

. $HOME/webtierenv.sh
opmnctl status -l

. $HOME/webtierenv.sh
cd $MW_HOME/Oracle_WT1/instances/instance1/diagnostics/logs/OPMN/opmn
grep 'Process Unreachable' opmn.log

. $HOME/webtierenv.sh
cd $MW_HOME/Oracle_WT1/instances/instance1/diagnostics/logs/OHS/ohs1
grep 'still did not exit, sending a SIGKILL' ohs1.log

Make sure no latest OHS Restart has happened.

$MW_HOME/Oracle_WT1/instances/instance1/config/OHS/ohs1/httpd.conf
Look for mpm_worker_module and change MaxClients to 300, ThreadsPerChild to 50

There are multiple tuning recommedtation for HTTP WebTier in the following link, and should be carried out as recommended performance tuning activities:
http://docs.oracle.com/cd/E23943_01/core.1111/e10108/http.htm

Find the complete Fusion Middleware tuning guide at:
http://docs.oracle.com/cd/E23943_01/core.1111/e10108/toc.htm

How to Dump Provisioning and Synchronisation Profile from OID

How to Dump Provisioning and Synchronisation Profile from OID

How to Dump Provisioning Profile from OID

This would help you understand what are the Provisioning Profile exists in OID, configuration and status of all those.

ldapsearch -h mkktestserver1.unixdomain.local -p 3060 -D "cn=orcladmin" \
-w MalayFalsePass -L -s sub -b "cn=provisioning profiles,cn=changelog subscriber,cn=oracle internet directory" objectclass=*

Accordingly you can take a call on either enabling or disabling those, or changing the interval or troubleshooting purpose.
orclstatus: ENABLED
orclodipprovisioningappname: ebusl
orclodipprovisioningappname: ebsgold

How to Dump Synchronisation Profile from OID

manageSyncProfiles list -h mkktestserver1.unixdomain.local -p 7005 -D weblogic

Output of this would be something like:
Following are the registered profile(s):
ADtoOIDSynch >>> This is the synch profile that is in place.....

manageSyncProfiles get -h mkktestserver1.unixdomain.local -p 7005 -D weblogic -pf ADtoOIDSynch

Output of this would be something like(I have given only the important fields and removed the attribute part)

If these are configured in OID DIP Synch Profile AD to OID Synch should be issueless.

Profile ADtoOIDSynch details are :
odip.profile.condirfilter = searchfilter=(|(objectclass=group)(objectclass=organizationalUnit)(&(objectclass=user)(!(objectclass=computer))))
odip.profile.status = ENABLE

odip.profile.syncmode = IMPORT
odip.profile.version = 4.0
odip.profile.configfile = [INTERFACEDETAILS]
Reader: oracle.ldap.odip.gsi.ActiveChgReader
CheckAllEntries: false
SkipErrorToSyncNextChange: true
UpdateSearchCount: 100
SearchDeltaSize: 500

odip.profile.lastchgnum = 122685657
odip.profile.debuglevel = 0

You can Validate Synch Profile using following Command:

manageSyncProfiles validateProfile -h mkktestserver1.unixdomain.local -p 7005 -D weblogic -pf ADtoOIDSynch

Sample output for validateProfile
Map rules "orclodipattributemappingrules" have the following warnings:
Attribute rule "2" has warning: Source attribute 'cn' is optional for a required destination attribute 'cn'
Attribute rule "7" has warning: Source attribute 'samaccountname' is optional for a required destination attribute 'orclsamaccountname'
Attribute rule "22" has warning: Source attribute 'employeeid' is optional for a required destination attribute 'sn'
Attribute rule "25" has warning: Source attribute 'samaccountname' is optional for a required destination attribute 'orclsamaccountname'
Attribute rule "25" has warning: Source attribute 'userprincipalname' is optional for a required destination attribute 'orclsamaccountname'.

You can test Synch Profile using following Command:

You need to disable the profile temporarily to do this testProfile.
manageSyncProfiles testProfile -h mkktestserver1.unixdomain.local -p 7005 -D weblogic -pf ADtoOIDSynch

SynchronizationStatus : Synchronization Successful
SynchronizationErrors :
ECID : 39cb4812cad10e7e:-123fc09f:1401eaa2992:-8000-0000000000005981
View the related logs using the above ECID with the wlst command :
displaylogs(target=<ManagedServer_Name>, ecid='39cb4812cad10e7e:-123fc09f:1401eaa2992:-8000-0000000000005981')

Now this ECID you can look in wls_ods1-diagnostic.log, in my case it was warning only. This report that "SynchronizationStatus : Synchronization Successful", this means, synch is going okay...

In the wls_ods1-diagnostic.log, you can see this ECID in warning as reported in validateProfile.

Following Commands are also very useful to troubleshoot Synch Profile Issues:

ldapsearch -p 389 -h mkktestADserver1.addomain.local -D "AdtoOidSyncUser@vaa.vtg.local" -w "MalayFalsePass" -b "" -s base "objectclass=*" highestCommittedUSN
ldapsearch -h mkktestOIDserver1.unixdomain.local -p 3060 -D "cn=orcladmin" -w MalayFalsePass -b "" -s base "objectclass=*" lastchangenumber
ldapsearch -h mkktestOIDserver1.unixdomain.local -p 3060 -D "cn=orcladmin" -w MalayFalsePass \

-b "cn=subscriber profile,cn=changelog subscriber,cn=oracle internet directory" -s sub "objectclass=*"

Ad to OID Synch Issue

Ad to OID Synch Issue

By default, Microsoft Active Directory Connector retrieves changes to all objects in the container configured for synchronization. If you are interested in retrieving only a certain type of change, for example only changes to users and groups, then you should configure an LDAP search filter. This filter screens out changes that are not required when Microsoft Active Directory Connector queries Microsoft Active Directory. The filter is stored in the searchfilter attribute in the synchronization profile.

To troubleshout this kind of DIP issue enable TRACE:32 Logging for the following components:
oracle.dip.config
oracle.dip.mbean.prov
oracle.dip.mbean.sync
oracle.dip.util

You can achive this using enterprise manager console:
Navigation: http://mkktestserver1.unixdomain.local:7003/em >>> Click on wls_ods1 >>> Drop down WebLogic Server >>> Logs >>> Log Configuration

Monitor the managed server log to capture the issue.

Users are not getting synchronised from AD to OID.

Put a searchfilter in ADtoOIDSynch profile definition.
searchfilter=(|(objectclass=group)(objectclass=organizationalUnit)(&(objectclass=user)(!(objectclass=computer))))

In wls_ods1 log you would see a message like following after this change:
[2013-07-28T13:34:15.178+01:00] [wls_ods1] [NOTIFICATION] [DIP-10252] [oracle.dip] [tid: ADtoOIDSynch] [userId: <anonymous>] [ecid: 0000K0Uebzv03zD_R9c9yd1Hwpv2000002,0] [APP: DIP#11.1.1.2.0] Found Search Filter : ((|(objectclass=group)(objectclass=organizationalUnit)(&(objectclass=user)(!(objectclass=computer))))).

To verify if synchronisation is properly happening or not you can follow this link:
http://mkkoracleapps.blogspot.com/2013/07/how-to-check-ad-to-oid-synch-is.html

Remove Dangling DNs from OID

Remove Dangling DNs from OID

. $HOME/oidenv.sh
oiddiag collect_all=true
cd $MW_HOME/asinst_1/diagnostics/logs/OID/tools

In this directory, a LDIF File with Dangling DNs entries would be created, run ldapmodify to delete these. This would delete the entries, and is a recommended steps to improve the OID Performance.

ldapmodify -h mkktestserver1.unixdomain.local -p 3060 -D cn=orcladmin -w MalayFalsePass -v -f oiddiag20130824212502_FixMembrAttr.ldif

Putting one sample entry from the LDIF File below:
dn: cn=gs_apps_essbase_live_businesuser,ou=groups,ou=united kingdom,cn=users,dc=mkkaddomain,dc=root,dc=local
changetype: modify
delete: uniquemember
uniquemember: cn=amit joggarish,ou=itadmin,ou=united kingdom,dc=mkkaddomain,dc=root,dc=local

>>> This means that the groups would be deleted from the user's unique member group. This is noted as Dangling DN as the same group doesnot exist in OID, but this has come from Microsoft AD.

Remote Diagnostic Agent(RDA) Analysis for WebLogic Admin/Managed Server

Remote Diagnostic Agent(RDA) Analysis for WebLogic Admin/Managed Server

RDA is a powerful tool to gather various important information about the setup in place. In this post I am discussing RDA for WebLogic Server.

1. Set Environmental Variables related to the domain to be profiled, by running: 
cd $DOMAIN_HOME/bin
. ./setDomainEnv.sh

2. Configure RDA to profile WebLogic Server related files:
cd $MW_HOME/oracle_common/rda
./rda.sh -S -p WebLogicServer

This will configure the RDA to include the relevant data for WLS. It will open a text wizard, will ask you some general questions, and set the environment for running a collection with WLS data.

3. Run RDA (with no parameters) to trigger the actual collection
cd $MW_HOME/oracle_common/rda
./rda.sh 

4. You can check then the output, on a browser. In the OUTPUT folder, look for "<YOUR_PREFIX>_start.htm".

This is very useful info for Oracle also to work on Service Request.

Performance tuning on OID - Consolidated Details

Performance tuning on OID - Consolidated Details

OID_Perf_Reco_1: Set DSA config to skip referrals

This is applicable when there are no referrals setup in OID.

By default this capability IS ENABLED and severely impacts performance when large groups (>200K) or large number of nested groups are involved.

Definition: A referral is a special type of entry that when obtained in a search, it contains the location of the actual entry, which could be in another part of the directory tree or even in another ldap server altogether. Unless you have specifically set-up referrals you most likely do not have any.

Confirm, there is no "Referrals" in place in OID:
ldapsearch -h mkkoidserver1 -p XXXX -D "cn=orcladmin" -w "xxxxxxxxxxxxx" -s sub -b "" objectclass=referral

If this doesn't return any rows then, there are no referrals in place.

Set the value of orclskiprefinsql in DSA config, to 1. This would make DSA config to skip referrals.

ldapmodify -h mkkoidserver1 -p XXXX -D cn=orcladmin -w xxxxxxxxxxxxx << eof
dn: cn=dsaconfig,cn=configsets,cn=oracle internet directory
changetype: modify 
replace: orclskiprefinsql 
orclskiprefinsql: 1 
eof



OID_Perf_Reco_2: orclinmemfiltprocess is very expensive on Oracle Database

This attribute can help significantly with the performance of certain types of search operations. It has been identified to be particularly useful with OAM, as some of the searches OAM performs can be especially expensive in the database without the use of "orclinmemfiltprocess".

ldapmodify -h mkkoidserver1 -p XXXX -D "cn=orcladmin" -w "xxxxxxxxxxxxx" -v <EOF 
dn: cn=dsaconfig,cn=configsets,cn=oracle internet directory 
changetype: modify 
replace: orclinmemfiltprocess 
orclinmemfiltprocess:(|(!(obuseraccountcontrol=*))(obuseraccountcontrol=activated)) 
orclinmemfiltprocess:(|(obuseraccountcontrol=activated)(!(obuseraccountcontrol=*))) 
orclinmemfiltprocess:(obapp=groupservcenter)(!(obdynamicparticipantsset=*)) 
orclinmemfiltprocess:(objectclass=oblixworkflowinstance) 
orclinmemfiltprocess:(objectclass=inetorgperson) 
orclinmemfiltprocess:(objectclass=oblixorgperson) 
orclinmemfiltprocess:(objectclass=oblixworkflowstepinstance) 
EOF

For OID 11g it should come as default, cross check and apply, it it applicable for you.

OID_Perf_Reco_3: Run oidstats.sql Regularly as part of daily housekeeping/maintenance

Run oidstats.sql any-time large updates are made to the OID. For large Active Directories, where changes are very frequent, and AD to OID to synchronisation is enabled, this is a very good option to perform on regular basis.

. $HOME/oidenv.sh
cd $MW_HOME/Oracle_IDM1/ldap/admin
sqlplus ods/xxxxxxxxxxxxx@OIDDB
START oidstats.sql;

Remove Dangling DNs: http://mkkoracleapps.blogspot.co.uk/2013/10/remove-dangling-dns-from-oid.html

Data Source/JDBC Connection Pool tuning for eBusiness AccessGate

Data Source/JDBC Connection Pool tuning for eBusiness AccessGate

I noticed, suddenly, eBusiness Access through Oracle Access Manager stopped working. When I looked into AccessGate Managed server log, I found the following informations.

Jul 30, 2013 10:16:52 AM oracle.apps.fnd.ext.sso.AppsHttpServletRequestWrapper <init>
SEVERE: Cannot retrieve LDAP information for guid='NOT_FOUND'
Jul 30, 2013 10:17:35 AM oracle.apps.fnd.ext.sso.EbsServer$2 load
SEVERE: Failed to get server connection server=113
weblogic.jdbc.extensions.PoolLimitSQLException: weblogic.common.resourcepool.ResourceLimitException: No resources currently available in pool EBSLIVE to allocate to applications, please increase the size of the pool and retry..
at weblogic.jdbc.common.internal.JDBCUtil.wrapAndThrowResourceException(JDBCUtil.java:252)
at weblogic.jdbc.common.internal.RmiDataSource.getPoolConnection(RmiDataSource.java:456)

Solution

This is purely data source related issue. When you deploy AccessGate for eBusiness Suite, the ANT deployment create the data source also.

Login to WebLogic Console >>> Services >>> Data Source >>> Click on the Data Source EBSLIVE >>> Connection Pool Tab >>> Advanced

Set Initial Capacity = Max capacity of JDBC connection pool
Set Inactive Connection Timeout to 5 Minutes
Increase Maximum Capacity Value from Default Value of 15 to 30.

Restart AccessGate Managed and Admin Server....