Showing posts with label WebLogic. Show all posts
Showing posts with label WebLogic. Show all posts

Thursday, 17 October 2013

Set the DataSource Connection Continuity to avoid Admin/Managed Service Restart

Set the DataSource Connection Continuity to avoid Admin/Managed Service Restart

In a IDM domain, if AdminServer is started before the DB is up and running and subsequently DB is brought up, OPSS data source does not refresh and hence prevents access to application on Admin Server

Connection Creation Retry Frequency >>> This is needed if the Datasource will be down before the Admin server starts
Test Connections on Reserve >>> This is required if the datasource goes down after successful start

Navigation: WebLogic Console >>> Services >>> Data Sources >>> Click on the Data Sources >>> Connection Pool >>> Advanced

This should be done for OAM, OID and eBusiness AccessGate datasources.


Understanding "Managed Server Independence" in WebLogic Configuration

Understanding "Managed Server Independence" in WebLogic Configuration

"Managed Server Independence" specifies whether this Managed Server can be started when the Administration Server is unavailable.

Check that all managed servers have "Managed Server Independence" Enabled (by default it is)
Navigation: WebLogic Console >>> Environment >>> Servers >>> Click on the Name of the Managed Server >>> Configuration >>> Tuning >>> Advanced
Check if Managed Server Independence is Enabled

If you have "Managed Server Independence" Enabled on all managed servers, 
You can restart AdminServer without any problem (the managed servers will continue to work)

I did work for this for OAM Domain, IDM Domain, and eBusiness AccessGate Domain. So if any of the admin server is down, you eBusiness SSO Login would continue to work.

Thursday, 10 October 2013

Enterprise Manager is not showing Proper Status of OID Components

Enterprise Manager is not showing Proper Status of OID Components

In /em console the status of OID components are showing down, specifically DIP and Admin server.

AdminServer-diagnostic.log shows following error:
[2013-07-26T19:18:58.183+01:00] [AdminServer] [ERROR] [J2EE JMX-46335] [] [tid: DmsThread-15] [userId: <anonymous>] [ecid: 0000K06ty6w03zD_R9c9yd1HvKaN000002,0] [APP: NonJ2EEManagement#11.1.1] MBean attribute access denied. [[
MBean: oracle.as.management.mbeans.register:type=opmnInfoCenter
Getter for attribute InstanceNames
Detail: Access denied. Required roles: Admin, Operator, Monitor, executing subject: principals=[] java.lang.SecurityException: Access denied. Required roles: Admin, Operator, Monitor, executing subject: principals=[]
at oracle.as.jmx.framework.wls.spi.security.WLSMBeanSecurityHelper.isInWlsGlobalSecurityRoles(WLSMBeanSecurityHelper.java:245)

Solution

1. Stop OID Admin and Managed Server

2. Edit $DOMAIN_HOME/config/fmwconfig/servers/AdminServer/dms_config.xml using following contents:

Old Value
 <dumpConfiguration>
 <dump intervalSeconds="10800" maxSizeMBytes="75" enabled="true"/>
 </dumpConfiguration>

New Value
 <dumpConfiguration>
 <dump intervalSeconds="10800" maxSizeMBytes="75" enabled="false"/>
 </dumpConfiguration>

3. Stop OID Admin and Managed Server

Also apply the patch mentioned in FM 11g Control Console Shows Webcache or other Opmn-Managed Target As DOWN But It Is UP. (Doc ID 1501246.1 )

Error In Fusion Middleware Control Console After Patchset 11.1.1.6 - JMX-46335 (Doc ID 1477246.1)
OBIEE 11g: Error: "[ERROR] [J2EE JMX-46335]...[APP: NonJ2EEManagement] MBean attribute access denied." after Upgrade from 11.1.1.5 To 11.1.1.6 (Doc ID 1439413.1)
FM 11g Control Console Shows Webcache or other Opmn-Managed Target As DOWN But It Is UP. (Doc ID 1501246.1 )

Remote Diagnostic Agent(RDA) Analysis for WebLogic Admin/Managed Server

Remote Diagnostic Agent(RDA) Analysis for WebLogic Admin/Managed Server

RDA is a powerful tool to gather various important information about the setup in place. In this post I am discussing RDA for WebLogic Server.

1. Set Environmental Variables related to the domain to be profiled, by running: 
cd $DOMAIN_HOME/bin
. ./setDomainEnv.sh

2. Configure RDA to profile WebLogic Server related files:
cd $MW_HOME/oracle_common/rda
./rda.sh -S -p WebLogicServer

This will configure the RDA to include the relevant data for WLS. It will open a text wizard, will ask you some general questions, and set the environment for running a collection with WLS data.

3. Run RDA (with no parameters) to trigger the actual collection
cd $MW_HOME/oracle_common/rda
./rda.sh 

4. You can check then the output, on a browser. In the OUTPUT folder, look for "<YOUR_PREFIX>_start.htm".

This is very useful info for Oracle also to work on Service Request.

JVM Heap Tuning for OID Managed Server/WebLogic Server

JVM Heap Tuning for OID Managed Server/WebLogic Server

This is applicable for any weblogic Admin and Managed Server.
In $DOMAIN_HOME/servers/wls_ods1/logs/wls_ods1.log you would notice an error like this and OID managed server would go to UNKNOW status.

####<Jul 26, 2013 5:57:00 AM BST> <Error> <Kernel> <mkktestserver1.unixdomain.local> <wls_ods1> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <39cb4812cad10e7e:-5dbeb3ba:14006f69c2f:-8000-0000000000005522> <1374814620626> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>

Solution

From nodemanager.properties I can observe that we are using start up scripts to bring up OID servers. From start up script I can see we are using a maximum of 1024M of heap which might not be sufficient.

JVM Heap is configured in the following file:
$DOMAIN_HOME/bin/setDomainEnv.sh

Go to the following line:
EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES} -Didm.oracle.home=${IDM_ORACLE_HOME} -Xms512m -Xmx1024m -Xss512K -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Dweblogic.management.discover=false"
export EXTRA_JAVA_PROPERTIES

Since we are using a 64 bit JDK, it is well worth to try update memory settings to:
EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES} -Didm.oracle.home=${IDM_ORACLE_HOME} -Xms2048m -Xmx2048m -Xss512K -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Dweblogic.management.discover=false"
export EXTRA_JAVA_PROPERTIES

Keep monitoring wls_ods1.log for Java Heap Space Issue.

applive IDMLIVE /opt/oracle/IDMLIVE_MW_HOME/WebLogic/user_projects/domains/IDMDomain/bin $ diff setDomainEnv.sh.43 setDomainEnv.sh
372c372
< EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES} -Didm.oracle.home=${IDM_ORACLE_HOME} -Xms512m -Xmx1024m -Xss512K -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Dweblogic.management.discover=false"
---
> EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES} -Didm.oracle.home=${IDM_ORACLE_HOME} -Xms2048m -Xmx2048m -Xss512K -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Dweblogic.management.discover=false"
applive IDMLIVE /opt/oracle/IDMLIVE_MW_HOME/WebLogic/user_projects/domains/IDMDomain/bin $

Sunday, 11 August 2013

Removing Configured WebLogic Server Domain

How to remove a domain from a WebLogic Server Installation... As it is just a domain deployment, no tool is required, just remove the appropriate content as given below.....

Make sure you are keeping a proper backup before performing the steps mentioned below:

1. Remove the domain directory $MW_HOME/user_projects/domains/<Domain_Name>
Here in this example I am removing: eag_domain, so issue the following command,

rm -rf $MW_HOME/user_projects/domains/eag_domain

2. Remove the line for eag_domain from domain-registry.xml file

$ cat domain-registry.xml
<?xml version="1.0" encoding="UTF-8"?>
<domain-registry xmlns="http://xmlns.oracle.com/weblogic/domain-registry">
  <domain location="/opt/oracle/OAMLIVE_MW_HOME/WebLogic/user_projects/domains/OAMDomain"/>
  <domain location="/opt/oracle/OAMLIVE_MW_HOME/WebLogic/user_projects/domains/eag_domain"/>


3. Remove the line for eag_domain from nodemanager.domains

cd $MW_HOME/wlserver_10.3/common/nodemanager

$ cat nodemanager.domains
#Domains and directories created by Configuration Wizard
#Fri Jul 12 11:44:27 BST 2013
eag_domain=/opt/oracle/OAMLIVE_MW_HOME/WebLogic/user_projects/domains/eag_domain
OAMDomain=/opt/oracle/OAMLIVE_MW_HOME/WebLogic/user_projects/domains/OAMDomain

Saturday, 20 July 2013

How to do test of ar package, Installed on OS | Exception String: Error in invoking target 'nnfgt.o' of makefile

How to do test of ar package, Installed on OS | Exception String: Error in invoking target 'nnfgt.o' of makefile

a) create the following program called test.c :

main()
{
printf("hello\n");
}

b) compile it to a .o file :

gcc -c test.c

This generates an object file called test.o

c) Now test if basic ar functionality works;

ar cr myarch.a test.o

--> does the myarch.a file get created ok ?
--> if so can you query it with ar -t myarch.a ?

d) now test ar rv myarch2.a test.o 

Spool:

$ gcc -c test.c
$ ls -ltr
total 25970
-rw-r--r-- 1 apppoc dba 174 Apr 8 16:09 local.profile
-rw-r--r-- 1 apppoc dba 157 Apr 8 16:09 local.login
-rw-r--r-- 1 apppoc dba 136 Apr 8 16:09 local.cshrc
drwxr-xr-x 2 apppoc dba 96 May 2 12:18 bea
-rw------- 1 apppoc dba 13004722 May 5 02:03 core
-rw-r--r-- 1 apppoc dba 449 May 5 10:49 oamenv2.sh
-rw-r--r-- 1 apppoc dba 283726 May 6 08:07 core_strings.txt
-rw-r--r-- 1 apppoc dba 486 May 7 11:32 oamenv.sh
-rw-r--r-- 1 apppoc dba 30 May 8 12:07 test.c
-rw-r--r-- 1 apppoc dba 732 May 8 12:07 test.o
$ ar cr myarch.a test.o
Illegal Instruction(coredump)
$ ar rv myarch2.a test.o
a - test.o
Illegal Instruction(coredump)
$ cat test.c
main()
{
printf("hello\n");
}
$


Lesson Learned:

While doing OID Installation, config.sh was failing with following to execute the make command.
Exception String: Error in invoking target 'nnfgt.o' of makefile in OID Installation

Connecting to EM Fails With Error "503 Service Unavailable" after OID Installation

Connecting to EM Fails With Error "503 Service Unavailable" after OID Installation

After Installation of Oracle Internet Directory, I was unable to connect to EM and getting an error 503 Service Unavailable.

When I cheched the targets.xml file, I noticed OID EM Farm entry were not present.

$DOMAIN_HOME/sysman/state/targets.xml

I put the following entries immediately after the very first <Targets> line:

<Target TYPE="oracle_ias_farm" NAME="Farm_IDMDomain" DISPLAY_NAME="Farm_IDMDomain">
<Property NAME="MachineName" VALUE="mkkoidserver1.mkkdomain.local"/>
<Property NAME="Port" VALUE="7001"/>
<Property NAME="Protocol" VALUE="t3"/>
<Property NAME="isLocal" VALUE="true"/>
<Property NAME="serviceURL" VALUE="service:jmx:t3://mkkoidserver1.mkkdomain.local:7001/jndi/weblogic.management.mbeanservers.domainruntime"/>
<Property NAME="WebLogicHome" VALUE="/opt/oracle/IDMLIVE_MW_HOME/WebLogic/wlserver_10.3"/>
<Property NAME="DomainHome" VALUE="/opt/oracle/IDMLIVE_MW_HOME/WebLogic/user_projects/domains/IDMDomain"/>
</Target>

Restart the OID Stack.



Sunday, 11 November 2012

After Changing LDAP Providers, Users Are Not Picking Up Correct Rights or Permissions When Logging In


After Changing LDAP Providers, Users Are Not Picking Up Correct Rights or Permissions When Logging In

Cause Justification:

This issue is caused because the user logging in does not have administrative rights, or Imaging is not picking up the correct administrative rights for the user, so the user is being presented with basic options without any of the configuration menus.
If using a new or changed LDAP provider, the latter can happen because the user's or group's GUIDs from the LDAP provider are cached from the initial startup of Imaging, and so may not match the new GUIDs. This means that the users are not receiving the proper rights, and so are being logged in as normal users with basic rights.


To resolve this issue, you must ensure that the user logging in is an administrator within WebLogic Server. If the issue occurred after adding or changing LDAP providers, you can refresh/reset the cached GUID values in the Imaging security tables, by performing one of the following solutions:

Method 1, via the WLST tool:

Step#1: Connect to WLST tool using following steps:
. ${HOME}/db_wls_env.sh
cd ${MW_HOME}/Oracle_ECM1/common/bin
./wlst.sh
wls:/offline> connect()
Supply username, password, server URL

wls:/offline> connect()
Please enter your username :weblogic
Please enter your password :
Please enter your server URL [t3://localhost:7001] :
Connecting to t3://localhost:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'mkkbase_domain'.

Step#2: Execute the following command to refresh the GUIDs:
wls:/mkkbase_domain/serverConfig> refreshIPMSecurity()
wls:/mkkbase_domain/serverConfig> exit()

Step#3: Login to IPM Database using PL/SQL Developer and Check the following three tables, to ensure that their GUID columns are refreshed:
SYSTEM_SECURITY
DEFINITION_SECURITY
DOCUMENT_SECURITY

Method 2, via the Enterprise Manager front end:

1. Login into the EM Console(http://mkkaixserver1:7001/em
2. Navigate down to the Imaging server under the Weblogic Domain Folder.
3. Once the right hand pane refreshes, click on the 'WebLogic Server' drop down menu and select 'System MBean Browser'.
4. On the MBean Browser tree go to Application Defined MBeans --> oracle.imaging --> Server: IPM_server1 --> cmd --> cmd
5. Click on the 'refreshIPMSecurity' link on the right hand pane.
6. Press Invoke button.

Step#3: Login to IPM Database using PL/SQL Developer and Check the following three tables, to ensure that their GUID columns are refreshed:
SYSTEM_SECURITY
DEFINITION_SECURITY
DOCUMENT_SECURITY

If the above step fails then manually truncate the IPM Security tables and changing the admin user would be next solution:

1. Clear the Imaging System_Security table with either a truncate table or delete * from System_Security command
create table DEV1_IPM.SYSTEM_SECURITY_20120207 as select * from DEV1_IPM.SYSTEM_SECURITY;
create table DEV1_IPM.DEFINITION_SECURITY_20120207 as select * from DEV1_IPM.DEFINITION_SECURITY;
create table DEV1_IPM.DOCUMENT_SECURITY_20120207 as select * from DEV1_IPM.DOCUMENT_SECURITY;
truncate table DEV1_IPM.SYSTEM_SECURITY;
truncate table DEV1_IPM.DEFINITION_SECURITY
truncate table DEV1_IPM.DOCUMENT_SECURITY

2. Restart the IPM Server Only

3. Login as MKKIPMADMIN as the first user to IPM. Now, MKKIPMADMIN will be the new administrator for IPM.

How to do the FINEST Logging for IPM?


1. Login to WebLogic EM Console: http://mkkaixserver1.mkkdomain.intranet:7001/EM
2. Expand the WebLogic Domain.
3. Expand the Domain Name.
4. Click on IPM_SERVER (default install name = IPM_server1).
5. Click on the WebLogic Server control Fly out menu and select Logs -> Log Configuration.
6. Select the Log levels tab.
7. Set the View section to Runtime Loggers.
8. Expand the Root Logger. (if there is no expansion available, see below for steps on adding it)
9. Expand oracle.
10. Expand oracle.imaging.
11. Expand oracle.imaging.service.
12. At this point, you should see the oracle.imaging.service.exceptions.
13. Set this to FINEST.

If you do not see oracle.imaging.service.exceptions, this can be manually added by following these steps:
1. Collapse root loggers completely (else you will receive the error "javax.faces.model.NoRowAvailableException")
2. On the Log Configuration screen, set the View section to "Loggers With Persistent Log Level State".
3. On the bottom of the screen, expand Specify Loggers.
4. In the Name section, type "oracle.imaging.service.exceptions".
5. Set the Oracle Diagnostic Logging level to "FINEST".

IPM Diagnostic Logfile Location would be:
${MW_HOME}/user_projects/domains/mkkbase_domain/servers/IPM_server1/logs/IPM_server1-diagnostic.log

Various Start/Stop Options for WebCentre Content Management


To stop/start Oracle Content Server with the Oracle WebLogic Server Administration Console:

1. On the Administration Console Domain Structure navigation bar, select Environment, then Servers.
2. On the Conversion tab for the Summary of Servers section, select the name of the Oracle UCM server for the Oracle Content Server instance.
3. In the Settings for server_name section, click the Control tab.
4. In the Server Status area, click Shutdown.

To stop Oracle Content Server with Fusion Middleware Control:

1. In the Fusion Middleware Control navigation tree, expand the appropriate domain name (for example, UCM_ucm_domain).
2. Expand Content Management, then Universal Content Management, then Content Server.
3. Select the Oracle Content Server instance name (for example, Oracle Content Server (UCM_server1)). The home page for your Oracle Content Server instance displays.
4. From the UCM menu on the Oracle Content Server page, select Control, then Shut Down.... The Oracle Content Server instance is shut down.

To stop Oracle Content Server Using Script:

/d12/oraecm/Oracle/Middleware/user_projects/domains/mkkbase_domain/bin/stopManagedWebLogic.sh UCM_server1
/d12/oraecm/Oracle/Middleware/user_projects/domains/mkkbase_domain/stopWebLogic.sh
/d12/oraecm/Oracle/Middleware/user_projects/domains/mkkbase_domain/bin/stopManagedWebLogic.sh IPM_server1 http://mkkaixserver1.mkkdomain.intranet:7001

To start Oracle Content Server Using Script:

Start WebLogic Admin Services:
${MW_HOME}/user_projects/domains/mkkbase_domain/startWebLogic.sh

Start UCM Managed Services:
${MW_HOME}/user_projects/domains/mkkbase_domain/bin/startManagedWebLogic.sh UCM_server1

Start IPM Managed Services:
${MW_HOME}/user_projects/domains/mkkbase_domain/bin/startManagedWebLogic.sh IPM_server1 http://mkkaixserver1.mkkdomain.intranet:7001

Starting Weblogic Administration Server without Giving Password:

$ cat start_weblogic.sh
/d12/oraecm/Oracle/Middleware/user_projects/domains/mkkbase_domain/startWebLogic.sh \
-Dweblogic.management.username=weblogic \
-Dweblogic.management.password=***********

Enabling WebLogic/Managed Servers to Start Without Supplying Credentials


You can enable the Administration Server and Managed Servers to start without prompting you for the administrator username and password.

1. For the Administration Server, create a boot.properties file:

Create the following directory:
${MW_HOME}/user_projects/domains/mkkbase_domain/servers/AdminServer/security
mkkbase_domain >>> is the domain name

Use a text editor to create a file called boot.properties in the security directory created in the previous step, and enter the following lines in the file:
username=weblogic
password=*********

For each Managed Server:

Create the following directory:
${MW_HOME}/user_projects/domains/mkkbase_domain/servers/UCM_server1/security
${MW_HOME}/user_projects/domains/mkkbase_domain/servers/IPM_server1/security

Copy the boot.properties file you created for the Administration Server to the security directory of Managed Server you created in the previous step.

Restart the Administration Server and Managed Servers, now it shouldnot ask for the weblogic username and password.

Monday, 25 June 2012

Changing Password of Java Keystores(WebLogic 11g)


Changing Password of Java Keystores(WebLogic 11g)


Set the environment by running setDomainEnv.cmd
${MW_HOME}/user_projects/domains/<domain name>/bin


cd ${JAVA_HOME}/lib/security/cacerts
keytool -list -v -keystore cacerts
It will prompt for password: enter the password ("changeit" is the default cacert password).


Take a backup of the cacerts files and run keytool store password:
keytool -storepasswd -keystore cacerts


Confirm that the Keystore Password is changed:
keytool -list -v -keystore cacerts


Now in the console you can apply the new password for the java keystore passphrase.


Use the below command for changing the password of a custom keystore:
keytool -storepasswd -new new_storepass -keystore <your_custom_keystore.jks>


Cheers !!!!!!!!!!!!!!!!!!!!

Tuesday, 19 June 2012

java thread dump (stack trace) in WebLogic


java thread dump (stack trace) in WebLogic


Why java thread dumps are required?
Sometimes we see a particular thread of the application deployed in Weblogic gets STUCK indifinetely and may require a restart to come out of it. This can happen for a number of reasons. To find out what the application was doing or waiting for which lead it to STUCK state, a thread dump is required to investigate the cause of this situation. The thread dump provides Java stack trace. A Java stack trace is a snapshot of the thread in a Java1 Virtual Machine (JVM). A stack trace can range from few lines to thousands of lines of diagnostics (this depends on how complex the application is).


1. Using kill -3


●● Unix operating systems have a command called 'kill' which sends different signals to a target running process or a daemon. A signal is a form of IPC (inter-process communication) that is used in the Unix operating systems to notify a process (its signal handler) that an event has occurred. Therefore, there are many uses for the kill command. For example, to kill (end / terminate) a process, signal SIGKILL (signal number 9) can be used. This cuases the process to end immediately.
●● Likewise, we can use kill -3 (signal SIGQUIT, quit and dump core) command to send a signal to the process whose dump we require. This signal causes the Java Virtual Machine to generate a stack trace of the process. This signal does terminate the process rather produces a dump.


kill -3 <PID>


●● where PID is the Java process ID in the operating system. Process ID can be found out using ps -ef command which lists all the processes running, and which can be further filetred using grep command with process name (e.g. java) or username as argument etc.


If you have started the Weblogic server using nohup, then dump is redirected to nohup.out.


2. Using jstack utility


By default, the output is redirected to stdout. This can be changed in Weblogic start up script using JAVA_OPTION.
jstack <PID>
To get a forced thread dump (when a process is not responding to jstack), use -F option.


3. Using WebLogic Admin console


Login to AdminConsole >>> Admin Server >>> Monitoring >>> Threads >>> Dump Thread Stacks


4. Using Weblogic Admin utility


java weblogic.Admin -url t3://WLS_hostname:Port -username <user> -password <password> GET -pretty -type JVMRuntime
Before running the above utility, you may have to source the setDomainEnv.sh from the Weblogic_domain_home/bin directory.




Cheers !!!!!


>>>> If you feel that you would like to read on some topics, please feel free to put a comment in the blog, I would definitely help. If you need any help for any Oracle Apps or Database related issues, put a comment.