Security on Oracle Data Dictionary | O7_DICTIONARY_ACCESSIBILITY

Security on Oracle Data Dictionary | O7_DICTIONARY_ACCESSIBILITY

The data dictionary tables and views for a given database are stored in the SYSTEM tablespace for that database. All the data dictionary tables and views for a given database are owned by the user SYS. Connecting to the database with the SYSDBA privilege gives full access to the data dictionary

SQL> SELECT TABLE_NAME FROM DICTIONARY;

INIT Parameter O7_DICTIONARY_ACCESSIBILITY (set as FALSE) enables to secure Oracle Data Dictionary.

Oracle Database provides highly granular privileges. One such privilege, commonly referred to as the ANY privilege, like DROP ANY TABLE. It is possible to protect the Oracle data dictionary from accidental or malicious use of the ANY privilege by setting 07_DICTIONARY_ACCESSIBILITY initialization parameter to FALSE.

For changes in O7_DICTIONARY_ACCESSIBILITY to get reflected Database restart is required.

Default Password Security Settings and Various Audits in 11g

Default Password Security Settings  and Various Audits in 11g

If applications use the default password security settings from Oracle Database 10g Release 2 (10.2), then you can revert to these settings until you modify them to use the Release 11g password security settings. To do so, run the undopwd.sql script.

undopwd.sql: This script is called by DBCA to undo the 11g secure configuration changes to the password portion of the default profile. It reverts to the default 10gR2 settings. It is not intended to be run during upgrade, since that would undo all customer settings as well.

secconf.sql: This script would enable the 11g default password security related settings and enable various auditing parts,

select resource_name, limit from dba_profiles where profile='DEFAULT' and resource_type='PASSWORD';

Oracle Database 10gR2 Settings:

ALTER PROFILE DEFAULT LIMIT
FAILED_LOGIN_ATTEMPTS 10
PASSWORD_LIFE_TIME UNLIMITED
PASSWORD_GRACE_TIME UNLIMITED
PASSWORD_LOCK_TIME UNLIMITED
PASSWORD_REUSE_TIME UNLIMITED
PASSWORD_REUSE_MAX UNLIMITED
;

Oracle Database 11g Settings:

ALTER PROFILE DEFAULT LIMIT
PASSWORD_LIFE_TIME 180
PASSWORD_GRACE_TIME 7
PASSWORD_REUSE_TIME UNLIMITED
PASSWORD_REUSE_MAX UNLIMITED
FAILED_LOGIN_ATTEMPTS 10
PASSWORD_LOCK_TIME 1
;

Audit alter any table by access;
Audit create any table by access;
Audit drop any table by access;
Audit Create any procedure by access;
Audit Drop any procedure by access;
Audit Alter any procedure by access;
Audit Grant any privilege by access;
Audit grant any object privilege by access;
Audit grant any role by access;
Audit audit system by access;
Audit create external job by access;
Audit create any job by access;
Audit create any library by access;
Audit create public database link by access;
Audit exempt access policy by access;
Audit alter user by access;
Audit create user by access;
Audit role by access;
Audit create session by access;
Audit drop user by access;
Audit alter database by access;
Audit alter system by access;
Audit alter profile by access;
Audit drop profile by access;
Audit database link by access;
Audit system audit by access;
Audit profile by access;
Audit public synonym by access;
Audit system grant by access;
Audit directory by access;

Oracle Database Security - Steps to achieve high Security

Oracle Database Security - Steps to achieve high Security

Recently, I was engaged with an Oracle Database Security related tasks, would provide the details in multiple post. As of now just posting the road map, I used for this.

How to Disable Oracle Label Security | OLS

How to Disable Oracle Label Security | OLS

OLS and the Audit table AUD$:

●● Installation of Label Security causes the audit table SYS.AUD$ to be dropped and recreated in the SYSTEM schema. Its existing contents are copied into the new SYSTEM.AUD$ table.
●● If you deinstall Label Security, AUD$ is recreated in the SYS schema and dropped from the SYSTEM schema. Again the contents are copied from one to the other before dropping.

Starting as of version 11.2.0.1, when you install the Enterprise Edition, all options that belong to it are always installed, the available Options selections in the installer only decides if they should be enabled or not, to enable or disable OLS afterwards, you can use chopt.

chopt enable lbac
chopt disable lbac

This works on both Unix/Linux and Windows type Operating system.

If you want to disable OLS for a particular policy in 11gR2 you can use following command:
SA_AUDIT_ADMIN.NOAUDIT ('AROLS', 'XXCTO', 'APPLY, REMOVE');

Syntax:
PROCEDURE AUDIT (
 policy_name IN VARCHAR2,
 users IN VARCHAR2 DEFAULT NULL,
 option IN VARCHAR2 DEFAULT NULL,
 type IN VARCHAR2 DEFAULT NULL,
 success IN VARCHAR2 DEFAULT NULL);

Disabling Oracle Label Security for 12c

If Oracle Database Vault has been enabled, then do not disable Oracle Label Security.

SELECT VALUE FROM V$OPTION WHERE PARAMETER = 'Oracle Label Security';
SELECT PARAMETER, VALUE FROM V$OPTION WHERE PARAMETER = 'Oracle Database Vault';

sqlplus '/as sysdba'
EXEC LBACSYS.OLS_ENFORCEMENT.DISABLE_OLS;
SHUTDOWN IMMEDIATE
STARTUP

For Oracle Real Application Cluster (Oracle RAC) environment or a multitenant environment, repeat these steps for each Oracle RAC node or PDB on which you enabled Oracle Label Security.

EXEC LBACSYS.OLS_ENFORCEMENT.ENABLE_OLS;

Removal of OLS Data Dictionary:

This 9i method still works for higher versions.

cd $ORACLE_HOME/rdbms/admin/
sqlplus "/ as sysdba"
START catnools.sql


For 11gR2 it doesn't require downtime.

For 12c it requires downtime.

Thumb rule for sizing UNDO Tablespace Size

Thumb rule for sizing UNDO Tablespace Size

Sizing an UNDO tablespace requires three pieces of data.
(UR) UNDO_RETENTION in seconds
(UPS) Number of undo data blocks generated per second
(DBS) Overhead varies based on extent and file size (db_block_size)

The undo space needed is calculated as:
UndoSpace = UR * (UPS * DBS)

This query would give you the required minimum size in MB:

SELECT (UR * (UPS * DBS)/1024/1024) AS "Bytes" FROM (SELECT value AS UR FROM v$parameter WHERE name = 'undo_retention'),
(SELECT undoblks/((end_time-begin_time)*86400) AS UPS
FROM v$undostat
WHERE undoblks = (SELECT MAX(undoblks) FROM v$undostat)),
(SELECT block_size AS DBS
FROM dba_tablespaces
WHERE tablespace_name = (SELECT UPPER(value) FROM v$parameter WHERE name = 'undo_tablespace'));

Thumb rule from past experience: OEM 12c Cloud control suggests you set the size to 10 times of this value. I have tested for multiple clients it works well in 5 times in most of the cases, need observations afterwards.

Now, suppose your current undo_retention is 900 and you are increasing that to 9000. The required UNDO tablespace size would be(in MB):
Result_Of_Above_Query*5*10
5 is required as per thumb rule.
10 is required as you are increasing the UNDO_RETENTION value to 10 times.

ASM Related steps for 11gR2 Grid Infrastructure

ASM Related steps for 11gR2 Grid Infrastructure

I am putting this ASM related steps only, on request of a friend. In two projects where I handled HP EVA Storage also, I setup 2 node RAC for eBusiness Suite. If you require any other part of the implementation experience, would be happy to share.

This was done for Oracle Enterprise Linux

Prepare Storage and Disks

1. Run fdisk for RDBMS and GRID Binaries Multipathing Disk(Both Node)
2. Run partprobe for RDBMS and GRID Binaries Multipathing Disk(Both Node)
3. Run fdisk for ASM Related Multipathing Disk(Database Node1)
4. Run partprobe for ASM Related Multipathing Disk(Both Node)
5. Run mkfs.ext3 for RDBMS and GRID Binaries Multipathing Disk(Database Node1)

Multipathing Disks Used for ASM:

mpath2 - PRODDATA
mpath6 - PRODFRA
mpath5 - PRODOCR1
mpath4 - PRODOCR2
mpath3 - PRODOCR3

As an example fdisk and partprobe is shown for mapth2 disk, same needs to be done for others:

[root@mkkracdbserver1 ~]# fdisk /dev/mapper/mpath2
[root@mkkracdbserver1 ~]# partprobe /dev/mapper/mpath2

[root@mkkracdbserver1 ~]# ls -ltr /dev/mapper/mpath2*
brw-rw---- 1 root disk 253,  0 Aug  9 00:02 /dev/mapper/mpath2
brw-rw---- 1 root disk 253, 10 Aug  9 00:02 /dev/mapper/mpath2p1

[root@mkkracdbserver2 ~]# partprobe /dev/mapper/mpath2

[root@mkkracdbserver2 ~]# ls -ltr /dev/mapper/mpath2*
brw-rw---- 1 root disk 253, 0 Aug  6 18:07 /dev/mapper/mpath2
brw-rw---- 1 root disk 253, 9 Aug  9 00:06 /dev/mapper/mpath2p1



Install Oracle ASM Libraries RPM

# rpm -ivh oracleasm-support-2.1.4-1.el5.x86_64.rpm
# rpm -ivh oracleasmlib-2.0.4-1.el5.x86_64.rpm
# rpm -ivh oracleasm-2.6.18-238.el5-2.0.5-1.el5.x86_64.rpm

# /etc/init.d/oracleasm init
# /etc/init.d/oracleasm exit
# /etc/init.d/oracleasm status

To Load and initialize the ASMLib driver issue init
To Stop the ASMLib driver issue exit
To Display the status of the Oracle ASMLib driver issue status




Configure ASMLib Driver

This needs to be done on both the Database Node

# /etc/init.d/oracleasm configure

[root@mkkracdbserver1 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

Default user to own the driver interface []: oraprod
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]

[root@mkkracdbserver2 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

Default user to own the driver interface []: oraprod
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]




Create ASM Disks

1. Use Oracleasm Createdisk Command on Database Node1
2. Use Oracleasm Listdisks Command on Database Node1
2. Use Oracleasm Scandisks Command on Database Node2
2. Use Oracleasm Listdisks Command on Database Node2

Important Note: Do not deviate from the above order, or else you may end up with non-visible ASM disks while doing the installation. You will have to change the diskstring parameter to asm_diskstring='/dev/oracleasm/disks/*' to make the disks visible. To overcome the issue, reboot both the database node and disks will be visible.


[root@mkkracdbserver1 ~]# oracleasm createdisk ASMDATA01 /dev/mapper/mpath2p1
Writing disk header: done
Instantiating disk: done

[root@mkkracdbserver1 ~]# oracleasm createdisk ASMFRA01 /dev/mapper/mpath6p1
Writing disk header: done
Instantiating disk: done

[root@mkkracdbserver1 ~]# oracleasm createdisk ASMOCR01 /dev/mapper/mpath5p1
Writing disk header: done
Instantiating disk: done

[root@mkkracdbserver1 ~]# oracleasm createdisk ASMOCR02 /dev/mapper/mpath4p1
Writing disk header: done
Instantiating disk: done

[root@mkkracdbserver1 ~]# oracleasm createdisk ASMOCR03 /dev/mapper/mpath3p1
Writing disk header: done
Instantiating disk: done

[root@mkkracdbserver1 ~]# oracleasm listdisks
ASMDATA01
ASMFRA01
ASMOCR01
ASMOCR02
ASMOCR03

[root@mkkracdbserver2 ~]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Instantiating disk "ASMDATA01"
Instantiating disk "ASMOCR03"
Instantiating disk "ASMOCR02"
Instantiating disk "ASMOCR01"
Instantiating disk "ASMFRA01"

[root@mkkracdbserver2 ~]# oracleasm listdisks
ASMDATA01
ASMFRA01
ASMOCR01
ASMOCR02
ASMOCR03


Implement Metalink Note 1059847.1 for Multipathing Disks

This needs to be done on both the Database Nodes

Important Note: If this is not done, then root.sh will fail in Database Node2.

11GR2 GRID INFRASTRUCTURE INSTALLATION FAILS WHEN RUNNING ROOT.SH ON NODE 2 OF RAC USING ASMLIB [ID 1059847.1]

1. Modify the /etc/sysconfig/oracleasm with:

ORACLEASM_SCANORDER="dm"
ORACLEASM_SCANEXCLUDE="sd"

# /etc/init.d/oracleasm restart

[root@mkkracdbserver1 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@mkkracdbserver1 ~]#

[root@mkkracdbserver2 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@mkkracdbserver2 ~]#

Cheers !!!!
Malay Khawas
Oracle Apps/Fusion DBA

How to setup Passwordless SSH

Suppose I want to login from oradev1 to orapsup, without giving password, so I need to generate the key in oradev1 node.

In oradev1 Node:
ssh-keygen -t rsa
cat $HOME/.ssh/id_rsa.pub
Path of ssh-keygen: /usr/local/bin/ssh-keygen
/usr/local/bin/ssh-keygen -t rsa

In orapsup Node:
vi $HOME/.ssh/authorized_keys
And, put the contents of id_rsa.pub taken from oradev1 in the file
If .ssh directory doesnot exist, then create it by running the following

command:
ssh-keygen -t rsa

[oradev1@mkkdevserver1 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oradev1/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oradev1/.ssh/id_rsa.
Your public key has been saved in /home/oradev1/.ssh/id_rsa.pub.
The key fingerprint is:
74:b3:cd:c9:8d:1c:40:ca:3c:ce:82:ab:04:7f:25:7f oradev1@mkkdevserver1.mkkdomain.local
[oradev1@mkkdevserver1 ~]$
[oradev1@mkkdevserver1 ~]$ cd .ssh/
[oradev1@mkkdevserver1 .ssh]$ ls
id_rsa  id_rsa.pub  known_hosts
[oradev1@mkkdevserver1 .ssh]$ cat id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAtBBAsSSfzZC7Zu3fFlO8xCUaO1jzuXnPe+Z38nNboZLspjFkNaEkdYd9Ej6ZS+uBfTjrCVgCTBEdb0wLzAQlvLVLVro/M9gwjhoNGQyBKJyDQYmkUy7RSFuUdFupioT+YIOGfdQBbpn8YFbfeTWd23Khcaxkpdt/n4hlBY/PdAYUPxNvypnp49jtbideF6yiSqXF1gBDhWefh9Zd9KlIInKY6k0MNqTfn6WY57/PuSiWq+i4WKs/whZWOqnTKlulHVhBSSEjFaYKyPtPvgZbl13cbIQsvcB3TUMnYJT1JylmwMCw6ZGJIATLJMTA4WpNhHDO07e+6a7RtroyY63z5w== oradev1@mkkdevserver1.mkkdomain.local
[oradev1@mkkdevserver1 .ssh]$

Now whatever contents you have on file id_rsa.pub, you have to put that in authorized_keys file of orapsup.

[orapsup@mkkdevserver2 .ssh]$ ls -ltr /home/orapsup/.ssh/authorized_keys
-rw-r--r-- 1 orapsup dba 1374 Apr  5 13:52 /home/orapsup/.ssh/authorized_keys
[orapsup@mkkdevserver2 .ssh]$
View the /home/orapsup/.ssh/authorized_keys file and paste at the end of the file the above public key content.
Now you are ready to login from oradev1 to orapsup,  without giving any password.

Cheers !!!!!!!!!!!

After 11gR2 Grid Infrastructure Cluster Installation, run Cluster Verification Utility to Verify Everything is Up and Running

After 11gR2 Grid Infrastructure Cluster Installation, run Cluster Verification Utility to Verify Everything is Up and Running

[oraprod@mkkprodracdb1 grid]$ ./runcluvfy.sh stage -post crsinst -n mkkprodracdb1,mkkprodracdb2

Performing post-checks for cluster services setup

Checking node reachability...
Node reachability check passed from node "mkkprodracdb1"


Checking user equivalence...
User equivalence check passed for user "oraprod"
Checking time zone consistency...
Time zone consistency check passed.


Checking Cluster manager integrity...

Checking CSS daemon...
Oracle Cluster Synchronization Services appear to be online.

Cluster manager integrity check passed


UDev attributes check for OCR locations started...
UDev attributes check passed for OCR locations


UDev attributes check for Voting Disk locations started...
UDev attributes check passed for Voting Disk locations

Default user file creation mask check passed

Checking cluster integrity...


Cluster integrity check passed


Checking OCR integrity...

Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations


ASM Running check passed. ASM is running on all cluster nodes

Checking OCR config file "/etc/oracle/ocr.loc"...

OCR config file "/etc/oracle/ocr.loc" check successful


Disk group for ocr location "+PRODOCR1" available on all the nodes


Checking size of the OCR location "+PRODOCR1" ...

Size check for OCR location "+PRODOCR1" successful...
Size check for OCR location "+PRODOCR1" successful...

WARNING:
This check does not verify the integrity of the OCR contents. Execute 'ocrcheck' as a privileged user to verify the contents of OCR.

OCR integrity check passed

Checking CRS integrity...

CRS integrity check passed

Checking node application existence...

Checking existence of VIP node application (required)
Check failed.
Check failed on nodes:
        mkkprodracdb2

Checking existence of ONS node application (optional)
Check passed.

Checking existence of GSD node application (optional)
Check ignored.

Checking existence of EONS node application (optional)
Check passed.

Checking existence of NETWORK node application (optional)
Check passed.


Checking Single Client Access Name (SCAN)...

Checking name resolution setup for "mkkprodscan.mkkdomain.intranet"...

Verification of SCAN VIP and Listener setup passed
OCR detected on ASM. Running ACFS Integrity checks...

Starting check to see if ASM is running on all cluster nodes...

ASM Running check passed. ASM is running on all cluster nodes

Starting Disk Groups check to see if at least one Disk Group configured...
Disk Group Check passed. At least one Disk Group configured

Task ACFS Integrity check passed

Checking Oracle Cluster Voting Disk configuration...

Oracle Cluster Voting Disk configuration check passed

User "oraprod" is not part of "root" group. Check passed

Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed

Checking if CTSS Resource is running on all nodes...
CTSS resource check passed


Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed

Check CTSS state started...
CTSS is in Observer state. Switching over to clock synchronization checks using NTP


Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
NTP Configuration file check passed

Checking daemon liveness...
Liveness check passed for "ntpd"

NTP daemon slewing option check passed

NTP daemon's boot time configuration check for slewing option passed

NTP common Time Server Check started...
PRVF-5408 : NTP Time Server "192.5.41.41" is common only to the following nodes "mkkprodracdb1"
PRVF-5408 : NTP Time Server "193.204.114.232" is common only to the following nodes "mkkprodracdb1"
PRVF-5408 : NTP Time Server "69.36.224.15" is common only to the following nodes "mkkprodracdb1"
Check of common NTP Time Server passed

Clock time offset check from NTP Time Server started...
Clock time offset check passed

Clock synchronization check using Network Time Protocol(NTP) passed


Oracle Cluster Time Synchronization Services check passed

Post-check for cluster services setup was unsuccessful.
Checks did not pass for the following node(s):
        mkkprodracdb2
[oraprod@mkkprodracdb1 grid]$

[oraprod@mkkprodracdb1 grid]$ ./runcluvfy.sh stage -post crsinst -n all -verbose

Performing post-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "mkkprodracdb1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  mkkprodracdb2                           yes
  mkkprodracdb1                           yes
Result: Node reachability check passed from node "mkkprodracdb1"


Checking user equivalence...

Check: User equivalence for user "oraprod"
  Node Name                             Comment
  ------------------------------------  ------------------------
  mkkprodracdb2                           passed
  mkkprodracdb1                           passed
Result: User equivalence check passed for user "oraprod"
Checking time zone consistency...
Time zone consistency check passed.


Checking Cluster manager integrity...

Checking CSS daemon...

  Node Name                             Status
  ------------------------------------  ------------------------
  mkkprodracdb2                           running
  mkkprodracdb1                           running

Oracle Cluster Synchronization Services appear to be online.

Cluster manager integrity check passed


UDev attributes check for OCR locations started...
Result: UDev attributes check passed for OCR locations


UDev attributes check for Voting Disk locations started...
Result: UDev attributes check passed for Voting Disk locations


Check default user file creation mask
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  mkkprodracdb2   0022                      0022                      passed
  mkkprodracdb1   0022                      0022                      passed
Result: Default user file creation mask check passed

Checking cluster integrity...

  Node Name
  ------------------------------------
  mkkprodracdb1
  mkkprodracdb2

Cluster integrity check passed


Checking OCR integrity...

Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations


ASM Running check passed. ASM is running on all cluster nodes

Checking OCR config file "/etc/oracle/ocr.loc"...

OCR config file "/etc/oracle/ocr.loc" check successful


Disk group for ocr location "+PRODOCR1" available on all the nodes


Checking size of the OCR location "+PRODOCR1" ...

Size check for OCR location "+PRODOCR1" successful...
Size check for OCR location "+PRODOCR1" successful...

WARNING:
This check does not verify the integrity of the OCR contents. Execute 'ocrcheck' as a privileged user to verify the contents of OCR.

OCR integrity check passed

Checking CRS integrity...
The Oracle clusterware is healthy on node "mkkprodracdb2"
The Oracle clusterware is healthy on node "mkkprodracdb1"

CRS integrity check passed

Checking node application existence...

Checking existence of VIP node application
  Node Name     Required                  Status                    Comment
  ------------  ------------------------  ------------------------  ----------
  mkkprodracdb2   yes                       unknown                   failed
  mkkprodracdb1   yes                       online                    passed
Result: Check failed.

Checking existence of ONS node application
  Node Name     Required                  Status                    Comment
  ------------  ------------------------  ------------------------  ----------
  mkkprodracdb2   no                        online                    passed
  mkkprodracdb1   no                        online                    passed
Result: Check passed.

Checking existence of GSD node application
  Node Name     Required                  Status                    Comment
  ------------  ------------------------  ------------------------  ----------
  mkkprodracdb2   no                        does not exist            ignored
  mkkprodracdb1   no                        does not exist            ignored
Result: Check ignored.

Checking existence of EONS node application
  Node Name     Required                  Status                    Comment
  ------------  ------------------------  ------------------------  ----------
  mkkprodracdb2   no                        online                    passed
  mkkprodracdb1   no                        online                    passed
Result: Check passed.

Checking existence of NETWORK node application
  Node Name     Required                  Status                    Comment
  ------------  ------------------------  ------------------------  ----------
  mkkprodracdb2   no                        online                    passed
  mkkprodracdb1   no                        online                    passed
Result: Check passed.


Checking Single Client Access Name (SCAN)...
  SCAN VIP name     Node          Running?      ListenerName  Port          Running?
  ----------------  ------------  ------------  ------------  ------------  ------------
  mkkprodscan.mkkdomain.intranet  mkkprodracdb1   true          LISTENER      1581          true

Checking name resolution setup for "mkkprodscan.mkkdomain.intranet"...
  SCAN Name     IP Address                Status                    Comment
  ------------  ------------------------  ------------------------  ----------
  mkkprodscan.mkkdomain.intranet  192.19.1.130              passed
  mkkprodscan.mkkdomain.intranet  192.19.1.129              passed
  mkkprodscan.mkkdomain.intranet  192.19.1.128              passed

Verification of SCAN VIP and Listener setup passed
OCR detected on ASM. Running ACFS Integrity checks...

Starting check to see if ASM is running on all cluster nodes...

ASM Running check passed. ASM is running on all cluster nodes

Starting Disk Groups check to see if at least one Disk Group configured...
Disk Group Check passed. At least one Disk Group configured

Task ACFS Integrity check passed

Checking Oracle Cluster Voting Disk configuration...

Oracle Cluster Voting Disk configuration check passed

Checking to make sure user "oraprod" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  mkkprodracdb2   does not exist            passed
  mkkprodracdb1   does not exist            passed
Result: User "oraprod" is not part of "root" group. Check passed

Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed

Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
  Node Name                             Status
  ------------------------------------  ------------------------
  mkkprodracdb2                           passed
  mkkprodracdb1                           passed
Result: CTSS resource check passed


Querying CTSS for time offset on all nodes...
Result: Query of CTSS for time offset passed

Check CTSS state started...
Check: CTSS state
  Node Name                             State
  ------------------------------------  ------------------------
  mkkprodracdb2                           Observer
  mkkprodracdb1                           Observer
CTSS is in Observer state. Switching over to clock synchronization checks using NTP


Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP Configuration file check passed

Checking daemon liveness...

Check: Liveness for "ntpd"
  Node Name                             Running?
  ------------------------------------  ------------------------
  mkkprodracdb2                           yes
  mkkprodracdb1                           yes
Result: Liveness check passed for "ntpd"

Checking NTP daemon command line for slewing option "-x"
Check: NTP daemon command line
  Node Name                             Slewing Option Set?
  ------------------------------------  ------------------------
  mkkprodracdb2                           yes
  mkkprodracdb1                           yes
Result:
NTP daemon slewing option check passed

Checking NTP daemon's boot time configuration, in file "/etc/sysconfig/ntpd", for slewing option "-x"

Check: NTP daemon's boot time configuration
  Node Name                             Slewing Option Set?
  ------------------------------------  ------------------------
  mkkprodracdb2                           yes
  mkkprodracdb1                           yes
Result:
NTP daemon's boot time configuration check for slewing option passed

NTP common Time Server Check started...
PRVF-5408 : NTP Time Server "192.5.41.41" is common only to the following nodes "mkkprodracdb1"
NTP Time Server "182.23.102.198" is common to all nodes on which the NTP daemon is running
PRVF-5408 : NTP Time Server "193.204.114.232" is common only to the following nodes "mkkprodracdb1"
PRVF-5408 : NTP Time Server "69.36.224.15" is common only to the following nodes "mkkprodracdb1"
NTP Time Server ".LOCL." is common to all nodes on which the NTP daemon is running
Check of common NTP Time Server passed

Clock time offset check from NTP Time Server started...
Checking on nodes "[mkkprodracdb2, mkkprodracdb1]"...
Check: Clock time offset from NTP Time Server

Time Server: 182.23.102.198
Time Offset Limit: 1000.0 msecs
  Node Name     Time Offset               Status
  ------------  ------------------------  ------------------------
  mkkprodracdb2   -25.438                   passed
  mkkprodracdb1   11.585                    passed
Time Server "182.23.102.198" has time offsets that are within permissible limits for nodes "[mkkprodracdb2, mkkprodracdb1]".

Time Server: .LOCL.
Time Offset Limit: 1000.0 msecs
  Node Name     Time Offset               Status
  ------------  ------------------------  ------------------------
  mkkprodracdb2   0.0                       passed
  mkkprodracdb1   0.0                       passed
Time Server ".LOCL." has time offsets that are within permissible limits for nodes "[mkkprodracdb2, mkkprodracdb1]".
Clock time offset check passed

Result: Clock synchronization check using Network Time Protocol(NTP) passed


Oracle Cluster Time Synchronization Services check passed

Post-check for cluster services setup was unsuccessful.
Checks did not pass for the following node(s):
        mkkprodracdb2
[oraprod@mkkprodracdb1 grid]$

Highlighted issue was due to IP Conflict in DNS, after the IP Conflict is resolved in DNS, and root.sh run in both Database Server Nodes, performed the following steps to add the VIP Manually:

Aftre the IP Conflicts resolved manually run the following command from db02:

# ./srvctl add vip -n mkkprodracdb2 -k 1 -A mkkprodracdb2-vip/255.255.255.0/eth0
# ./srvctl start vip -n mkkprodracdb2
# ./srvctl start listener -n mkkprodracdb2

Cluster Verification Utility Before 11gR2 Grid Infrastructure Installation

Before Proceeding for GRID Installation Make Sure all following errors are corrected:

[oraprod@mkkprodracdb1 grid]$ ./runcluvfy.sh stage -pre crsinst -n mkkprodracdb1,mkkprodracdb2

Performing pre-checks for cluster services setup

Checking node reachability...
Node reachability check passed from node "mkkprodracdb1"


Checking user equivalence...
User equivalence check passed for user "oraprod"

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Node connectivity passed for subnet "172.16.1.0" with node(s) mkkprodracdb2,mkkprodracdb1
TCP connectivity check passed for subnet "172.16.1.0"


Interfaces found on subnet "172.16.1.0" that are likely candidates for VIP are:
mkkprodracdb2 eth0:172.16.1.155
mkkprodracdb1 eth0:172.16.1.154

WARNING:
Could not find a suitable set of interfaces for the private interconnect

Node connectivity check passed

Total memory check passed
Available memory check passed
Swap space check passed
Free disk space check passed for "mkkprodracdb2:/tmp"
Free disk space check passed for "mkkprodracdb1:/tmp"
User existence check passed for "oraprod"
Group existence check failed for "oinstall"
Check failed on nodes:
        mkkprodracdb2,mkkprodracdb1
Group existence check passed for "dba"
Membership check for user "oraprod" in group "oinstall" [as Primary] failed
Check failed on nodes:
        mkkprodracdb2,mkkprodracdb1
Membership check for user "oraprod" in group "dba" passed
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
System architecture check passed
Kernel version check passed
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check failed for "file-max"
Check failed on nodes:
        mkkprodracdb2,mkkprodracdb1
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
Kernel parameter check failed for "aio-max-nr"
Check failed on nodes:
        mkkprodracdb2,mkkprodracdb1
Package existence check passed for "ocfs2-tools-1.2.7"
Package existence check passed for "make-3.81"
Package existence check passed for "binutils-2.17.50.0.6"
Package existence check passed for "gcc-4.1"
Package existence check passed for "libaio-0.3.106 (i386)"
Package existence check passed for "libaio-0.3.106 (x86_64)"
Package existence check passed for "glibc-2.5-24 (i686)"
Package existence check passed for "glibc-2.5-24 (x86_64)"
Package existence check passed for "compat-libstdc++-33-3.2.3 (i386)"
Package existence check passed for "compat-libstdc++-33-3.2.3 (x86_64)"
Package existence check passed for "elfutils-libelf-0.125 (x86_64)"
Package existence check passed for "elfutils-libelf-devel-0.125"
Package existence check passed for "glibc-common-2.5"
Package existence check passed for "glibc-devel-2.5 (i386)"
Package existence check passed for "glibc-devel-2.5 (x86_64)"
Package existence check passed for "glibc-headers-2.5"
Package existence check passed for "gcc-c++-4.1.2"
Package existence check failed for "libaio-devel-0.3.106 (i386)"
Check failed on nodes:
        mkkprodracdb2
Package existence check failed for "libaio-devel-0.3.106 (x86_64)"
Check failed on nodes:
        mkkprodracdb2,mkkprodracdb1
Package existence check passed for "libgcc-4.1.2 (i386)"
Package existence check passed for "libgcc-4.1.2 (x86_64)"
Package existence check passed for "libstdc++-4.1.2 (i386)"
Package existence check passed for "libstdc++-4.1.2 (x86_64)"
Package existence check passed for "libstdc++-devel-4.1.2 (x86_64)"
Package existence check passed for "sysstat-7.0.2"
Package existence check passed for "unixODBC-2.2.11 (i386)"
Package existence check failed for "unixODBC-2.2.11 (x86_64)"
Check failed on nodes:
        mkkprodracdb2,mkkprodracdb1
Package existence check passed for "unixODBC-devel-2.2.11 (i386)"
Package existence check failed for "unixODBC-devel-2.2.11 (x86_64)"
Check failed on nodes:
        mkkprodracdb2,mkkprodracdb1
Package existence check passed for "ksh-20060214"
Check for multiple users with UID value 0 passed
Current group ID check passed
Core file name pattern consistency check passed.

User "oraprod" is not part of "root" group. Check passed
Default user file creation mask check passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
NTP Configuration file check passed

Checking daemon liveness...
Liveness check passed for "ntpd"

NTP daemon slewing option check passed

NTP daemon's boot time configuration check for slewing option passed

NTP common Time Server Check started...
PRVF-5408 : NTP Time Server "128.252.19.1" is common only to the following nodes "mkkprodracdb1"
PRVF-5408 : NTP Time Server "93.84.115.2" is common only to the following nodes "mkkprodracdb1"
PRVF-5408 : NTP Time Server "69.164.222.108" is common only to the following nodes "mkkprodracdb1"
Check of common NTP Time Server passed

Clock time offset check from NTP Time Server started...
PRVF-5413 : Node "mkkprodracdb1" has a time offset of 188828.0 that is beyond permissible limit of 1000.0 from NTP Time Server "172.16.1.11"
Clock time offset check passed

Clock synchronization check using Network Time Protocol(NTP) passed


Pre-check for cluster services setup was unsuccessful on all the nodes.
[oraprod@mkkprodracdb1 grid]$


Use the following command to check networking set up with CVU:
./runcluvfy.sh comp nodecon -n mkkprodracdb1,mkkprodracdb2


Reference:
10gR2, 11gR1 and 11gR2 Oracle Clusterware (CRS / Grid Infrastructure) & RAC Command (crsctl, srvctl, cluvfy etc) Syntax and Reference [ID 1332452.1]