Sunday 11 November 2012

After 11gR2 Grid Infrastructure Cluster Installation, run Cluster Verification Utility to Verify Everything is Up and Running



After 11gR2 Grid Infrastructure Cluster Installation, run Cluster Verification Utility to Verify Everything is Up and Running

[oraprod@mkkprodracdb1 grid]$ ./runcluvfy.sh stage -post crsinst -n mkkprodracdb1,mkkprodracdb2

Performing post-checks for cluster services setup

Checking node reachability...
Node reachability check passed from node "mkkprodracdb1"


Checking user equivalence...
User equivalence check passed for user "oraprod"
Checking time zone consistency...
Time zone consistency check passed.


Checking Cluster manager integrity...

Checking CSS daemon...
Oracle Cluster Synchronization Services appear to be online.

Cluster manager integrity check passed


UDev attributes check for OCR locations started...
UDev attributes check passed for OCR locations


UDev attributes check for Voting Disk locations started...
UDev attributes check passed for Voting Disk locations

Default user file creation mask check passed

Checking cluster integrity...


Cluster integrity check passed


Checking OCR integrity...

Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations


ASM Running check passed. ASM is running on all cluster nodes

Checking OCR config file "/etc/oracle/ocr.loc"...

OCR config file "/etc/oracle/ocr.loc" check successful


Disk group for ocr location "+PRODOCR1" available on all the nodes


Checking size of the OCR location "+PRODOCR1" ...

Size check for OCR location "+PRODOCR1" successful...
Size check for OCR location "+PRODOCR1" successful...

WARNING:
This check does not verify the integrity of the OCR contents. Execute 'ocrcheck' as a privileged user to verify the contents of OCR.

OCR integrity check passed

Checking CRS integrity...

CRS integrity check passed

Checking node application existence...

Checking existence of VIP node application (required)
Check failed.
Check failed on nodes:
        mkkprodracdb2

Checking existence of ONS node application (optional)
Check passed.

Checking existence of GSD node application (optional)
Check ignored.

Checking existence of EONS node application (optional)
Check passed.

Checking existence of NETWORK node application (optional)
Check passed.


Checking Single Client Access Name (SCAN)...

Checking name resolution setup for "mkkprodscan.mkkdomain.intranet"...

Verification of SCAN VIP and Listener setup passed
OCR detected on ASM. Running ACFS Integrity checks...

Starting check to see if ASM is running on all cluster nodes...

ASM Running check passed. ASM is running on all cluster nodes

Starting Disk Groups check to see if at least one Disk Group configured...
Disk Group Check passed. At least one Disk Group configured

Task ACFS Integrity check passed

Checking Oracle Cluster Voting Disk configuration...

Oracle Cluster Voting Disk configuration check passed

User "oraprod" is not part of "root" group. Check passed

Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed

Checking if CTSS Resource is running on all nodes...
CTSS resource check passed


Querying CTSS for time offset on all nodes...
Query of CTSS for time offset passed

Check CTSS state started...
CTSS is in Observer state. Switching over to clock synchronization checks using NTP


Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
NTP Configuration file check passed

Checking daemon liveness...
Liveness check passed for "ntpd"

NTP daemon slewing option check passed

NTP daemon's boot time configuration check for slewing option passed

NTP common Time Server Check started...
PRVF-5408 : NTP Time Server "192.5.41.41" is common only to the following nodes "mkkprodracdb1"
PRVF-5408 : NTP Time Server "193.204.114.232" is common only to the following nodes "mkkprodracdb1"
PRVF-5408 : NTP Time Server "69.36.224.15" is common only to the following nodes "mkkprodracdb1"
Check of common NTP Time Server passed

Clock time offset check from NTP Time Server started...
Clock time offset check passed

Clock synchronization check using Network Time Protocol(NTP) passed


Oracle Cluster Time Synchronization Services check passed

Post-check for cluster services setup was unsuccessful.
Checks did not pass for the following node(s):
        mkkprodracdb2
[oraprod@mkkprodracdb1 grid]$

[oraprod@mkkprodracdb1 grid]$ ./runcluvfy.sh stage -post crsinst -n all -verbose

Performing post-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "mkkprodracdb1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  mkkprodracdb2                           yes
  mkkprodracdb1                           yes
Result: Node reachability check passed from node "mkkprodracdb1"


Checking user equivalence...

Check: User equivalence for user "oraprod"
  Node Name                             Comment
  ------------------------------------  ------------------------
  mkkprodracdb2                           passed
  mkkprodracdb1                           passed
Result: User equivalence check passed for user "oraprod"
Checking time zone consistency...
Time zone consistency check passed.


Checking Cluster manager integrity...

Checking CSS daemon...

  Node Name                             Status
  ------------------------------------  ------------------------
  mkkprodracdb2                           running
  mkkprodracdb1                           running

Oracle Cluster Synchronization Services appear to be online.

Cluster manager integrity check passed


UDev attributes check for OCR locations started...
Result: UDev attributes check passed for OCR locations


UDev attributes check for Voting Disk locations started...
Result: UDev attributes check passed for Voting Disk locations


Check default user file creation mask
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  mkkprodracdb2   0022                      0022                      passed
  mkkprodracdb1   0022                      0022                      passed
Result: Default user file creation mask check passed

Checking cluster integrity...

  Node Name
  ------------------------------------
  mkkprodracdb1
  mkkprodracdb2

Cluster integrity check passed


Checking OCR integrity...

Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations


ASM Running check passed. ASM is running on all cluster nodes

Checking OCR config file "/etc/oracle/ocr.loc"...

OCR config file "/etc/oracle/ocr.loc" check successful


Disk group for ocr location "+PRODOCR1" available on all the nodes


Checking size of the OCR location "+PRODOCR1" ...

Size check for OCR location "+PRODOCR1" successful...
Size check for OCR location "+PRODOCR1" successful...

WARNING:
This check does not verify the integrity of the OCR contents. Execute 'ocrcheck' as a privileged user to verify the contents of OCR.

OCR integrity check passed

Checking CRS integrity...
The Oracle clusterware is healthy on node "mkkprodracdb2"
The Oracle clusterware is healthy on node "mkkprodracdb1"

CRS integrity check passed

Checking node application existence...

Checking existence of VIP node application
  Node Name     Required                  Status                    Comment
  ------------  ------------------------  ------------------------  ----------
  mkkprodracdb2   yes                       unknown                   failed
  mkkprodracdb1   yes                       online                    passed
Result: Check failed.

Checking existence of ONS node application
  Node Name     Required                  Status                    Comment
  ------------  ------------------------  ------------------------  ----------
  mkkprodracdb2   no                        online                    passed
  mkkprodracdb1   no                        online                    passed
Result: Check passed.

Checking existence of GSD node application
  Node Name     Required                  Status                    Comment
  ------------  ------------------------  ------------------------  ----------
  mkkprodracdb2   no                        does not exist            ignored
  mkkprodracdb1   no                        does not exist            ignored
Result: Check ignored.

Checking existence of EONS node application
  Node Name     Required                  Status                    Comment
  ------------  ------------------------  ------------------------  ----------
  mkkprodracdb2   no                        online                    passed
  mkkprodracdb1   no                        online                    passed
Result: Check passed.

Checking existence of NETWORK node application
  Node Name     Required                  Status                    Comment
  ------------  ------------------------  ------------------------  ----------
  mkkprodracdb2   no                        online                    passed
  mkkprodracdb1   no                        online                    passed
Result: Check passed.


Checking Single Client Access Name (SCAN)...
  SCAN VIP name     Node          Running?      ListenerName  Port          Running?
  ----------------  ------------  ------------  ------------  ------------  ------------
  mkkprodscan.mkkdomain.intranet  mkkprodracdb1   true          LISTENER      1581          true

Checking name resolution setup for "mkkprodscan.mkkdomain.intranet"...
  SCAN Name     IP Address                Status                    Comment
  ------------  ------------------------  ------------------------  ----------
  mkkprodscan.mkkdomain.intranet  192.19.1.130              passed
  mkkprodscan.mkkdomain.intranet  192.19.1.129              passed
  mkkprodscan.mkkdomain.intranet  192.19.1.128              passed

Verification of SCAN VIP and Listener setup passed
OCR detected on ASM. Running ACFS Integrity checks...

Starting check to see if ASM is running on all cluster nodes...

ASM Running check passed. ASM is running on all cluster nodes

Starting Disk Groups check to see if at least one Disk Group configured...
Disk Group Check passed. At least one Disk Group configured

Task ACFS Integrity check passed

Checking Oracle Cluster Voting Disk configuration...

Oracle Cluster Voting Disk configuration check passed

Checking to make sure user "oraprod" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  mkkprodracdb2   does not exist            passed
  mkkprodracdb1   does not exist            passed
Result: User "oraprod" is not part of "root" group. Check passed

Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed

Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes
  Node Name                             Status
  ------------------------------------  ------------------------
  mkkprodracdb2                           passed
  mkkprodracdb1                           passed
Result: CTSS resource check passed


Querying CTSS for time offset on all nodes...
Result: Query of CTSS for time offset passed

Check CTSS state started...
Check: CTSS state
  Node Name                             State
  ------------------------------------  ------------------------
  mkkprodracdb2                           Observer
  mkkprodracdb1                           Observer
CTSS is in Observer state. Switching over to clock synchronization checks using NTP


Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP Configuration file check passed

Checking daemon liveness...

Check: Liveness for "ntpd"
  Node Name                             Running?
  ------------------------------------  ------------------------
  mkkprodracdb2                           yes
  mkkprodracdb1                           yes
Result: Liveness check passed for "ntpd"

Checking NTP daemon command line for slewing option "-x"
Check: NTP daemon command line
  Node Name                             Slewing Option Set?
  ------------------------------------  ------------------------
  mkkprodracdb2                           yes
  mkkprodracdb1                           yes
Result:
NTP daemon slewing option check passed

Checking NTP daemon's boot time configuration, in file "/etc/sysconfig/ntpd", for slewing option "-x"

Check: NTP daemon's boot time configuration
  Node Name                             Slewing Option Set?
  ------------------------------------  ------------------------
  mkkprodracdb2                           yes
  mkkprodracdb1                           yes
Result:
NTP daemon's boot time configuration check for slewing option passed

NTP common Time Server Check started...
PRVF-5408 : NTP Time Server "192.5.41.41" is common only to the following nodes "mkkprodracdb1"
NTP Time Server "182.23.102.198" is common to all nodes on which the NTP daemon is running
PRVF-5408 : NTP Time Server "193.204.114.232" is common only to the following nodes "mkkprodracdb1"
PRVF-5408 : NTP Time Server "69.36.224.15" is common only to the following nodes "mkkprodracdb1"
NTP Time Server ".LOCL." is common to all nodes on which the NTP daemon is running
Check of common NTP Time Server passed

Clock time offset check from NTP Time Server started...
Checking on nodes "[mkkprodracdb2, mkkprodracdb1]"...
Check: Clock time offset from NTP Time Server

Time Server: 182.23.102.198
Time Offset Limit: 1000.0 msecs
  Node Name     Time Offset               Status
  ------------  ------------------------  ------------------------
  mkkprodracdb2   -25.438                   passed
  mkkprodracdb1   11.585                    passed
Time Server "182.23.102.198" has time offsets that are within permissible limits for nodes "[mkkprodracdb2, mkkprodracdb1]".

Time Server: .LOCL.
Time Offset Limit: 1000.0 msecs
  Node Name     Time Offset               Status
  ------------  ------------------------  ------------------------
  mkkprodracdb2   0.0                       passed
  mkkprodracdb1   0.0                       passed
Time Server ".LOCL." has time offsets that are within permissible limits for nodes "[mkkprodracdb2, mkkprodracdb1]".
Clock time offset check passed

Result: Clock synchronization check using Network Time Protocol(NTP) passed


Oracle Cluster Time Synchronization Services check passed

Post-check for cluster services setup was unsuccessful.
Checks did not pass for the following node(s):
        mkkprodracdb2
[oraprod@mkkprodracdb1 grid]$

Highlighted issue was due to IP Conflict in DNS, after the IP Conflict is resolved in DNS, and root.sh run in both Database Server Nodes, performed the following steps to add the VIP Manually:

Aftre the IP Conflicts resolved manually run the following command from db02:

# ./srvctl add vip -n mkkprodracdb2 -k 1 -A mkkprodracdb2-vip/255.255.255.0/eth0
# ./srvctl start vip -n mkkprodracdb2
# ./srvctl start listener -n mkkprodracdb2




No comments:

Post a Comment