Now since you have a cluster with separated file system space. It is now time to link 2 FE together. Note that, we will use Heartbeat to control the HA cluster after this, but at first we will manually set-up the cluster to work without Heartbeat. Some step will be undone in Heartbeat setup step. We will start doing everything to fe1 first, then synchronize everything to fe2.

Frontend1 (FE1) Set-up

IP Aliasing

First, you need to add IP aliases for fe1 IP addresses to your eth0 and eth1. Or the better way is to move your central private and public IP addresses into IP alias.

• Edit /etc/sysconfig/ifcfg-eth0 and ifcfg-eth1, change the IP address to fe1 IP instead of central one (172.16.0.2 and 203.123.123.1)
• Copy /etc/sysconfig/ifcfg-eth0 to /etc/sysconfig/ifcfg-eth0:0, do the similar to eth1 (make ifcfg-eth1:0), then add the central IP to these files. I would recommend to remove any HWADDR entry in any files.
• Restart your fe1 network, to let any change take effect.

service network restart

• Make sure that everything is still working after this step (checking ganglia, try ssh connection to any compute nodes, checking SGE, etc…)

Network setup

Now it’s time to fix the database. Make change to the app_globals table in your database. The main idea here is to fix the database and let’s ROCKS re-create the correct /etc/hosts and named configuration. Run the following command to fix the database variable.

• Fix cluster name and change it into central one

At the time when I did all these, I manipulate the database directly. These commands below should yield equivalent (or better) effect.

rocks set var Kickstart PrivateHostname cluster (without dns domain name)

rocks set var Kickstart PublicHostname cluster.public

rocks set var Kickstart PublicAddress 203.123.123.123

• Now fix the entries in network table and change the hostname. The idea is to trick ROCKS database to have it make a correct /etc/hosts and DNS entries for us.

rocks set host interface name fe1 eth0 cluster

• Next, add entries for fe2 into the database. Replace the mac address and kernel module name to the correct one. If you don’t know what are those, try “rocks list host interface” on FE2.

rocks add host fe2 membership=Frontend rack=0 rank=0

rocks add host interface fe2 eth0 ip=172.16.0.3 mac=set-mac-here module=set-module-here name=fe2 subnet=private

• Create/Edit /etc/hosts.local, add FE1 private IP and Cluster public IP into the hosts file

172.16.0.2 fe1.local fe1

203.123.123.1 cluster.mydomain.name cluster

• Edit /etc/sysconfig/static-routes, add more route to all FE into the file

…..

any host 255.255.255.255 dev eth0

any host 203.123.123.2 gw 172.16.0.2

any host 203.123.123.3 gw 172.16.0.3

any host 203.123.123.1 gw 172.16.0.1

…

• Edit /var/named/rocks.domain.local, add the private local IP of fe1 to the file

fe1 A 172.16.0.2

• Edit /var/named/reverse.rocks.domain.16.172.local

2.0 PTR fe1.local.

• Now it is time to try regenerate /etc/hosts and all named configuration. Run “rocks sync config” once

Don’t forget to cross your finger here :P rocks sync config

• If the command above success, you should have a working /etc/hosts and named configuration check

  • /etc/hosts - See if you have all valid entries here. You should have fe1 (private, public), fe2 (private, public), cluster (private, public), and all other nodes listed here
  • /var/named/rocks.domain - same as above except that you will not have any public address listed here
  • /var/named/reverse.rocks.domain - same as above

• If you still need NFS export on this Frontend, check /etc/exports and modify it accordingly

Synchronize data between your Frontend

SSH key copy

We need to make both Frontend accessible to each other for the sake of simplicity.

• Copy root private key and public key from /root/.ssh to 2nd Frontend.

scp /root/.ssh/id_rsa* fe2:/root/.ssh/

• Test SSH from your FE2 to any compute nodes, it should work without password prompt now.

• Copy /etc/ssh/ssh_host_* to FE2, so SSH client will not complain for mismatch host key when fail-over occured.

scp /etc/ssh/ssh_host_* fe2:/etc/ssh/

Ganglia gmond

• We need to edit Ganglia multicast IP on FE2 to match with FE1 in order to make the machine appear in Ganglia monitoring network. Edit /etc/gmond.conf on FE2, look for mcast_join line and fix it to match FE1.
• Reboot FE2, or restart greceptor, gmond, and gmetad

Synchronize database & hosts & named

It is time to synchronize database on both FE. There are various way to do this, the easiest way is

mysqldump -u apache -p cluster > cluster.sql

scp cluster.sql fe2:/tmp/

ssh fe2

mysql -u root -p cluster < /tmp/cluster.sql

The next thing to synchronize is /etc/hosts.local and /var/named/*.domain.local to fe2

scp /etc/hosts.local fe2:/etc/

scp /var/named/*.domain.local fe2:/var/named/

Everything is finished!. It’s time to synchronize configuration once

rocks sync config

Now /etc/hosts and named configuration should be updated to match FE1.

Compute nodes set-up

Network configuration

I did all this manually in each node for this part.May need to have more testing

• Edit /etc/hosts, adding all other Frontend into /etc/hosts (by default). I modified /export/rocks/install/site-profiles/5.1/nodes/extend-client.xml with the following

<file name=”/etc/hosts” mode=”append” >

203.123.123.2 fe1

203.123.123.3 fe2

</file>

• Edit /export/rocks/install/site-profiles/extend-routes-client.xml, modify /etc/sysconfig/static-routes to add static route to each frontend.

<file name=”/etc/sysconfig/static-routes” mode=”append” >

any host 203.123.123.1 gw 172.16.0.1

any host 203.123.123.3 gw 172.16.0.3

</file>

• Rebuild rocks distro and try re-install compute nodes

cd /export/rocks/install && rocks create distro

tentakel -g compute /boot/kickstart/cluster-kickstart-pxe

That should be all for now. Note again that all above are meant to make all the network configuration correct on all nodes.

Testing

If all above are working (which is the case for me). You should be able to

• SSH password less from BOTH FE to any compute nodes
• Compute nodes can communicate with any other compute nodes and both FE, especially with any public IP
• Name resolving mechanism are working. You can use “getent hosts compute-0-1.local” or “getent hosts 172.16.255.253” from any machine.
• From both FE, issue “rocks list host” return correct list of nodes
• Try access Ganglia on both FE, it should displayed correct list of nodes along with node statistics

Again, I wrote this after I finish the set-up so some information might gone amiss.