Purescale is a fascinating feature of DB2 and with the new Cancun Release, it is officially possible to install it on virtual machines. While others do not support virtualization except if it is the same vendor’s solution, DB2 supports almost all hypervisors (VMware, Hyper-V, XEN and so on).
Brief knowledge about Purescale
Purescale is the first and only application agnostic + active-active database cluster without replication lag (except mainframe).
Let’s have a look about purescale architecture:
A purescale member is simply engine of DB2 database which works coordinated with CF. A member is a server which accepts database access requests, runs transactions and returns results. Every purescale cluster needs at least one member and can have up to 128 members.
A CF ( Cluster Cashing Facility) CF provides a scalable and centralized locking mechanism to ensure data coherency. CF is a server just like a member, but their resource requirements differ from members. Typically the sum of core across all purescale members is 6x-12x more than the CF (6x for write-heavy workloads and 12x for read-heavy workloads). The memory of a CF should be at least equal to %40 of sum of all member’s memory. Purescale clusters should have 2 CFs, one as Primary CF and the second as Secondary CF. Although clusters with just one CF also works, it will be the single point of failure for the cluster. So I highly recommend that you have both primary and secondary CFs for your production environment.
Some features of Purescale :
- Purescale uses shared storage thus there is no replication, thanks to GPFS. This eliminates replication lag.
- Purescale has its own load balancer, so you don’t have to distribute your workload. Purescale WLB(Workload Balancer) will do the job based on existing workload on members. This is not just a round-robin distribution of requests to bunch of IP adresses. WLB actually looks for existing workload on members and distribute the incoming requests to idle ones.
- You can use any member as a standalone DB2 database. Nobody will notice the difference between single DB2 installation and Purescale.
- Application is not aware of cluster, that means you don’t have to change anything in application. You also don’t need to tweak your application to have the benefits of cluster. Just changing the connection string will be enough.
- Purescale doesn’t have SPOF. Every single software component has a backup instance running either on same server or other server and that backup software intervenes and replaces itself with the failed software.
- Purescale supports replication between datacenters with HADR. Even if your existing datacenter becomes unusable because of disasters, you will have a consistent server running on another datacenter.
- When a running transaction is lost on a members, that transaction is rolled back by other members and guarantees database consistency.
- For better performance, cluster interconnect should be an InfiniBand connection. InfiniBand provides fast communication(10-15 microseconds) between members and CFs.
- You just install purescale on one of the members, installation itself distributes the binary files to other members and CFs. No need to do anything else on other members of CFs.
For this tutorial, I will install purescale on SLES 11 SP3 which runs on VMware ESX 5. But You can installa it on VMware Workstation as well. Both are pretty much the same since both are VMware products. If you have a little knowledge about VirtualBox, it should work for VirtualBox too.
Prerequisites for this installation:
1. 5 VMs on ESX.
Storage VM(Storage.PureScale): 1 vCPU, 2 GB Memory, 100 GB disk.
Member VM(M0.PureScale and M1.PureScale): 2 vCPU, 4 GB Memory, 16 GB disk.
CF VM(CFP.PureScale and CFS.PureScale):2 vCPU, 4 GB Memory, 16 GB disk (same as members).
2. DB2 10.5 Cancun Release IBM.
- Make a google search for “DB2 fix pack download”.
- Choose DB2 10.5.0.4 Cancun Release
- Choose Server Fix Pack Download (direct link to download page)
- Enter you IBM ID user name and password if you have one, if you don’t, create one.
3. Suse Linux Enterprise Server 11 SP3 for AMD64/Intel64 (direct link to download page)
4. (If you are going to install it on VMware Workstation) Workstation 10 Trial for Windows (direct link to download page). You can also use VirtualBox instead of VMware Workstation.
If you are willing to use VMware Workstation, you have to make some configuration.
Click the “Edit” menu then the “Virtual Network Editor”. Select “VMnet8”, click “DHCP Settings…”, enter 63-23-59 for default lease time and 0-0-0 for max lease time.
This is for making sure that your virtual servers IP addresses won’t change. This also makes installation easier since you won’t need to provide static IP addresses.
Installation
- The storage VM will serve 2 disks, one for data and the other for tiebreaker. Tiebreaker is used to determine which cluster component is alive and prevent split-brain. Tiebreaker disk size needs to be greater than 50 MB. So we will configure a 100 MB disk as a tiebreaker.
- Both the tiebreaker and the shared disk will be served through iSCSI. Other servers will see them as SCSI disks but they actually resides on the iSCSI server and will be accessed with TCP/IP protocol.
Now let’s create our VMs
We will need 5 VMs for this installation. One will act as storage, others will be members and CFs.
1. Create Storage VM
- Create a VM which has 1 vCPU, 2 GB RAM, 100 GB disk space.
- Name it as Storage.PureScale.
2. Create member and CF VMs
- All VMs will have 2 vCPU, at least 4 GB RAM and 16 GB disk space.
- Add a second NIC for these VMs. Second NICs will be used for cluster interconnect. Second NIC must use another network. That means if you are using 10.1.1.0 network for first NIC, then you need another notwork for the second NIC, something like 10.1.2.0.
- Name them as CFP.PureScale, CFS.PureScale, M0.PureScale and M1.PureScale.
Install SLES on Storage VM
Mount your SLES iso file into storage VM and boot.
If you are using VMware ESX, it will be hard to mount the SLES iso file. Because mounting an iso file is supported only if the server is running. So server already will pass the boot order and find nothing in CD/DVD-ROM. In order to mount a bootable CD into a ESX VM, first power on VM, it will look for a boot device over network, mount the .iso file then press CTRL + ALT + INSERT. The VM will boot into CD.
- Welcome screen: Click “I Agree to Licence Terms” and “Next”
- Media check: Click “Next”
- Installation Mode: Choose “New Installation” and click “Next”
- Clock and Time Zone: Uncheck “Hardware Clock Set to UTC”
- Server Base Scenario: Choose “Physical Machine” and click “Next”
- Installation Settings:
Click on Partitioning
Choose “Custom Partitioning (for experts)” and click “Next”
Click “sda” then click “add”
Choose “Primary Partition” then click “Next”, choose “Custom Size” and enter “4.00 GB” as size
Choose “swap” as file system and “swap” for mount point, then click “Finish”
Add another “Primary Partition”, enter “16.00 GB” as custom size, choose “ext3” as file system and “/” as mount point.
Add third “Primary Partition”, enter “100.00 MB” as custom size (note that it is MB not GB! Because 100 MB is enough for tiebreaker disk, no need to be wasteful), choose “Do not format partition” and “Do not mount partition”.
Add forth “Primary partition”, leave the custom size as it is (79.90 GB), again choose “Do not format partition” and “Do not mount partition”.
Now in the “Expert Partitioner” screen, you have to see something like this:
If everything is OK then click “Accept”
On “Installation Settings” screen, click “Software”
Click “Details…”, search for “gcc”, then check “gcc” and “gcc++”
Search for “kernel-source” and check “kernel-source”
Search again for “iscsi” and check “open-iscsi” and “iscsitarget”
Click “Accept” and return to “Installation Settings” screen, then click “Install”
Now SLES will be installed. After installation is done, we will make configuration.
In configuration step, it will ask for root password. Enter a password, click “Next”.
On “Hostname and Domain Name” screen, enter “storage” as hosname and “ps.local” as domain name, uncheck all check boxes.
On “Network Configuration” screen, disable IPv6, disable firewall.
On “Network Settings” screen, click “Edit”, check “Statically assigned IP address”.
Now this is really important, if you are using ESX, then you should provide an IP address which is compatible with your network adapters address range. In this case, my VM needs an IP address needs to be in the range of 10.254.20-23.xx. You will also need to enter appropriate subnet mask and default gateway. You can get these IP addresses from your network administrator.
On the other hand, if you are using VMware workstation or something like that, you should click “Cancel” and leave this screen. Then the IP adress will be assigned by VMware workstation itself. Since this IP adress may change for your installation, I will refer to this IP address as “storage_ip”.
On the “Test Internet Connection” screen, choose “No, Skip This Step” and click “Next”
On “Network Services Configuration” click “Next”
On “User Authentication Method” screen, choose “local (/etc/passwd)”, click “Next”
On “New Local User” screen, enter a username and a password, click “Next”
Click “Next”, until you see the “Installation Completed” screen.
On “Installation Completed” screen, uncheck “Clone This System for AutoYaST” and click “Finish”
Now storage node installation is complete.
Let’s move to other VMs
Install SLES on Member and CF VM
Installing SLES on other VMs is pretty much same, except a few differences.
- On “Installation Settings”, DO NOT click “partitioning”
- On “Hostname and Domain Name” screen, enter “m0” and “m1” as hostname for members, “cfp” and “cfs” for CFs, other options are same. Enter “ps.local” as “Domain Name” and uncheck “Change Hostname via DHCP”
If you are using VMware Workstation, IP addresses configuration for these servers will be left as it is. On ESX, you need to provide an IP address (If you don’t have a DHCP server running in your network).
Installing VMware Tools
Go to VM menu and click “Install/Update VMware Tools”
Open a terminal window and enter this commands:
cd /tmp; tar -zxvf "/media/VMware Tools/VM*.tar.gz" ./vmware-tools-distrib/vmware-install.pl
Press ENTER for all questions.
Now VMware Tools is installed.
Configuring NTP
You should configure NTP service for all VMs. This will make sure that all VMs have clocks synced.
If you have a private NTP server that’s fine, you can use it. If you are using VMware Workstation, you can user public NTP servers. Alternatively, you can use storage VM as NTP server and others as NTP Client.
On all VMs, run YaST and do proper NTP configuration:
Click, “Delete” then click “Add”, choose “Server”,
Enter either you private NTP server address or a public address then click OK.
Choose “Now and on Boot” and click OK.
Now NTP is configured. All VMs will have same clock. This is a “must have” for clusters.
Configuring IP Addresses
Remember you have two NICs on members and CFs and they must belong to different networks.
This is my ESX configuration:
You see while the eth0 belongs to 10.254.20.0 network, eth1 belongs to 10.254.7.0 network.
You should enter you first networks default gateway IP into “Routing -> Default Gateway”:
Now let’s have an overview of IP addresses.
We will need 2 separate networks. The first network which is used by eth0, will be used for accessing VMs from outside the cluster, while eth1 will be used for cluster interconnect. That means, DB2 PureScale cluster will user eth1 for communication. You won’t use it!
The IP configuration will be something like this:
- Storage.PureScale: network1.IP1 (eth0)
- CFP.PureScale: network1.IP2 (eth0), network2.IP1 (eth1)
- CFS.PureScale: network1.IP3 (eth0), network2.IP2 (eth1)
- M0.PureScale: network1.IP4 (eth0), network2.IP3 (eth1)
- M1.PureScale: network1.IP5 (eth0), network2.IP4 (eth1)
Some examples:
- Network1 might be 10.1.1.0 and IP1 might be 1 thus network1.IP1 = 10.1.1.1
- Network2 might be 10.1.2.0 and IP3 might be 3 thus network2.IP3 = 10.1.2.3
Use ping to validate that all network1 IP addresses are accessible for all VMs and all network2 IP addresses are accessible for all members and CFs (CFs and CFS have different meanings, since CFS means CF Secondary while CFs means plural form of CF servers).
Have a hostname for each IP address.
This is my configuration of /etc/hosts for all VMs except Storage.PureScale:
127.0.0.1 localhost
# eth0 interfaces
10.254.20.203 storage.ps.local storage
10.254.20.204 cfp.ps.local cfp
10.254.20.205 cfs.ps.local cfs
10.254.20.206 m0.ps.local m0
10.254.20.207 m1.ps.local m1
# eth1 interfaces
10.254.7.84 cfpeth1.ps.local cfpeth1
10.254.7.85 cfseth1.ps.local cfseth1
10.254.7.86 m0eth1.ps.local m0eth1
10.254.7.87 m1eth1.ps.local m1eth1
On all members and CFs, use vi or gedit to edit your /etc/hosts file, write all IP addresses and hostnames next to them (something like above).
If everything is fine, then this commands should work for all of them (CTRL+C to end ping):
# echo "pinging eth0 and eth1 hostnames" # ping storage # ping cfp # ping cfpeth1 # ping cfs # ping cfseth1 # ping m0 # ping m0eth1 # ping m1 # ping m1eth1 # echo "pinging eth0, fully qualified hostnames" # ping cfp.ps.local # ping cfs.ps.local # ping m0.ps.local # ping m1.ps.local # echo "pinging eth1, fully qualified hostnames" # ping cfpeth1.ps.local # ping cfseth1.ps.local # ping m0eth1.ps.local # ping m1eth1.ps.local
Now if all ping commands are OK, then network configuration is successful.
Configuring Password-less SSH
In order to make all members and CFs accessible from purescale cluster services, all members must be able to access others without any password. To do that, we have to configure passwordless SSH for all members and CFs.
On all members and CFs, run these commands:
# ssh-keygen -t rsa (press enter for all questions) # cp /root/.ssh/id_rsa.pub /root/id_rsa.$(hostname)
This will create id_rsa.pub.hostname (cfp, cfs, m0, m1) on all members and CFs.
Now on all members and CFs run this command :
# scp /root/id_rsa.pub.* root@cfp:/root/
On CFP, there will be all id_rsa.pub files belonging to other VMs.
Now on CFP, run these commands:
# for file in /root/id_rsa.*; do cat $file >> /root/.ssh/authorized_keys; done # scp /root/authorized_keys root@cfs:/root/.ssh/ # scp /root/authorized_keys root@m0:/root/.ssh/ # scp /root/authorized_keys root@m1:/root/.ssh/
If everything is done correctly, then you should be able to SSH between all members and CFs without any password. Just to be sure, run these commands on all members and CFs:
# ssh cfp # logout # ssh cfs # logout # ssh m0 # logout # ssh m1 # logout
Configuring iSCSI Target on the Storage VM
On Storage.PureScale VM, configure iSCSI Target. Do this by opening iSCSI Target in YaST Control Center:
- Choose “When booting”
- Click “Target” tab
- Click “Add” to add a new target
- Click “Add” again to add a disk to that target
- Type “/dev/sda3”, click “OK”
- Click “Next”, choose “No Authentication” and click “Next” again
Now you will see the added target:
- Click “Add” to add a second target
- Click “Add” again to add a disk to that target
- Type “/dev/sda4”, click “OK”
- Click “Next”, choose “No Authentication” and click “Next” again
Make sure that “When booting” in the “Service” tab is selected and click “Finish”.
Now iSCSI Target configuration is done.
Configuring iSCSI Initiator on the Members and CFs
Next, we will configure iSCSI Initiator on all members and CFs.
- Open CFP console and start iSCSI Initiator.
- Choose “When Booting”
- Click “Discovered Targets”
- Click “Discovery” button at bottom
Enter IP address of Storage VM (network1.IP1)
Check “No Authentication” and click “Next”
Now you will see two iSCSI targets in the “Discovered Targets” tab.
Choose first target and click “Login”
- Choose “Automatic” for startup
- Check “No Authentication”
- Click “Next”
Do same thing to login second target. You will see both targets show “automatic” in “Start-Up” column.
Now we connected the CFP to Storage node.
Do this on all other members (CFS, M0 and M1).
You can validate iSCSI disks with lscssi command:
Installing PureScale
From your computer, copy the DB2 10.5 Cancun Release (v10.5fp4_linuxx64_server_t.tar.gz) to M0 with WinSCP. Then go to M0 to extract files and start installation:
# cd /setup # tar zxf v10.5fp4_linuxx64_server_t.tar.gz # rm v10.5fp4_linuxx64_server_t.tar.gz
# cd server_t # export IBM_RDAC=NO # ./db2setup
Now we start the installation.
Click “Install a Product”
Click “Install New” under the “DB2 10.5 Fix Pack 4 Advanced Editions for DB2 Purescale”
Click “Next”
Read all text and if you agree, choose “I agree terms in the license aggrements”
Click “Next”
Leave the installation directory as it is (/opt/ibm/db2/V10.5) and click “Next” until you get to “Set user information for DB2 instance owner”
Enter a password in the required boxes and click “Next”
Do same thing for fenced user and click “Next”
On “Setup a DB2 Cluster File System” screen
Enter “/dev/sdc” as shared disk and “/dev/sdc” for tiebreaker disk. Click “Next”
Now we got to “Host List” screen. Click “Add” and enter “m1eth1.ps.local”.
Add all hosts to the list one by one (cfpeth1.ps.local, cfseth1.ps.local).
Choose eth1 interfaces for all hosts.
Make sure “member” column is checked for both m0eth1 and m1eth1.
Make sure “Cluster Caching Facility” column is checked for both cfpeth1 and cfseth1.
Make sure “Primary Cluster Caching Facility” is CFP, NOT CFS! If it isn’t, use “Avanced” button do make CFP as primary and CFS as secondary.
Click “Next”, click “Continue” for the warning
Click “Validate” and when everything is validated click “Finish”
Congratz, now DB2 should be installed with a minor error. We will fix this by changing the notmon.cf file.
Changing netmon.cf file
Locate the netmon.cf file with this command:
# find / -name netmon.cf
It must be located at /var/ct/cfg/netmon.cf
Make a backup of it by:
# cp /var/ct/cfg/netmon.cf /var/ct/cfg/netmon.cf.backup
Edit the /var/ct/cfg/netmon.cf file with gedit or vi:
# gedit /var/ct/cfg/netmon.cf &
We have to add the default gateway of eth1 interface. It is most likely the first IP address of a network. For example, if your eth1 network is 10.1.2.0 then 10.1.2.1 is the default gateway but if you are not sure, contact your network administrator and have it.
If you are using VMware Workstation, check your virtual network adapter’s default gateway. It is exactly the IP address you need to add into netmon.cf file.
Add the IP address of the eth1 network gateway to netmon.cf like this and save it:
Now it’s time to get DB2 started.
Issue these commands:
# su – db2sdin1
# db2cluster -cm -clear -alert
# db2 start