How to test Multicast CLUSTER INTERCONNECT on Oracle Database 11g R2

This note apply to Oracle Database 11.2.0.2.0 .

 

Sometimes the Interconnect latency is high.

If one instance is down and when try to restart this instance, the CSS daemon can’t start, and when ran the “crsctl stat res -t -init” the status showed forever “starting”.

The error of CSSD :

2010-09-16 23:13:15.839: [ CSSD][1087465792]clssnmvDHBValidateNCopy: node 1, node1, has a disk HB, but no network HB, DHB has rcfg 180134562, wrtcnt, 8627, LATS 9564064, lastSeqNo 8624, uniqueness 1284701023, timestamp 1284703995/10564774

 

How to know if you have a problem on Multicast Cluster Interconnect ?

 

1. Download the tool :

The mcasttest.pl-tool give you an indication whether or not multicasting has been disabled on the network switches, if it fails to use multicast on either the 230.0.1.0 or 224.0.0.251 multicast address for example. In general, work with your network administrator, if you suspect multicasting has been disabled on the network switches, affecting the private interconnect communication accordingly.

Download on the Oracle WebSite: mcasttest.pl-tool

 

2. Extract mcasttest :

tar -xzvf mcasttest.tgz

OR

gunzip mcasttest.tgz
tar xvf mcasttest.tar

 

3. Verify your hostname:

[oracle@server-01 mcasttest]$ hostname
server-01.commerce.com

 

4. Identify the network interface that run interconnect:

run with grid user :

[grid@server-01 ~]$ $GRID_HOME/bin/oifcfg getif
bond0  10.253.10.0  global  public
bond1  192.168.200.0  global  cluster_interconnect

The network interface is bond1

 

5. Run mcasttest:

The mcasttest.pl program requires two arguments:
The node list (specified with -n)
The list of interfaces to be used for the private interconnect (specified with -i).

Put the hostname first in the list.

See bellow when the multicast is ok :

[oracle@server-01 mcasttest]$ perl mcasttest.pl -n server01.commerce.com,server-02.commerce.com -i bond1
########### Setup for node server01.commerce.com ##########
Checking node access 'server01.commerce.com'
Checking node login 'server01.commerce.com'
Checking/Creating Directory /tmp/mcasttest for binary on node 'server01.commerce.com'
Distributing mcast2 binary to node 'server01.commerce.com'
########### Setup for node server02.commerce.com ##########
Checking node access 'server02.commerce.com'
Checking node login 'server02.commerce.com'
Checking/Creating Directory /tmp/mcasttest for binary on node 'server02.commerce.com'
Distributing mcast2 binary to node 'server02.commerce.com'
########### testing Multicast on all nodes ##########

Test for Multicast address 230.0.1.0

Apr 10 09:17:20 | Multicast Succeeded for bond1 using address 230.0.1.0:42000

Test for Multicast address 224.0.0.251

Apr 10 09:17:21 | Multicast Succeeded for bond1 using address 224.0.0.251:42001

 

6. Problem solution :

If test has failed for the 230.0.1.0 address, but succeeded for the 224.0.0.251 multicast address, restart the interface bond and try again:

 ifconfig bond1 down
 ifconfig bond1 up

In case not solve the patch: 9974223 must be applied to enable Oracle Grid Infrastructure to use the 224.0.0.251 multicast address.

 

Good Luck !

 


1 Comment to "How to test Multicast CLUSTER INTERCONNECT on Oracle Database 11g R2"

  1. 05/03/2012 - 4:20 AM | Permalink

    This post is priceless. When can I find out more?

Leave a Reply