Installing and configuring VCS/VXvM two node cluster

The application example below uses IBM Informix DB server

Hardware System installation

Note: The Below example uses 2 Sun fire M5000 servers for the hardware.
For the full XSCF configuration click here.

IP and FC port Layout

Below is an example of the Network and Fiber Channel layout.

xscf login information

Next will login to the physical domain VCS will be installed (assuming xscf install was completed described here).

login: admin
password: *****

Install the server with jumpstart

Note: The install below still uses Solaris 10 (which uses a jumpstart server for the OS install), a similar OS install can be done in Solaris 11.x by using an AI server.

The below add_install_client options where used to install the client

/rpool/opsproxy2/root/jumpstart/installs/sol10-1009/Solaris_10/Tools/add_install_client -e 0:21:28:71:4b:1a \
-c global:/rpool/opsproxy2/root/jumpstart/installs/sol10-1009/conf \
-s global:/rpool/opsproxy2/root/jumpstart/installs/sol10-1009 \
-p global:/rpool/opsproxy2/root/jumpstart/installs/sol10-1009/conf informix1 sun4u
 
/rpool/opsproxy2/root/jumpstart/installs/sol10-1009/Solaris_10/Tools/add_install_client -e 0:21:28:67:83:96 \
-c global:/rpool/opsproxy2/root/jumpstart/installs/sol10-1009/conf \
-s global:/rpool/opsproxy2/root/jumpstart/installs/sol10-1009 \
-p global:/rpool/opsproxy2/root/jumpstart/installs/sol10-1009/conf informix2 sun4u

After configuration was done, the fowling commend was issued to jumpstart the servers

# On informix1-xscf
{2c} ok show-nets
a) /pci@13,700000/network@0,3
[...] snip
d) /pci@13,700000/network@0 --- This was used nxge8
[...] snip
boot /pci@13,700000/network@0 - install
 
# On informix2-xscf
{29} ok show-nets
a) /pci@13,700000/network@0,3
[...] snip
j) /pci@3,700000/network@0 --- This was used nxge0
[...] snip
boot /pci@3,700000/network@0 - install

grep informix /etc/hosts

10.10.12.11     informix1 informix1.domain.com
10.10.12.12     informix2 informix2.domain.com

cat rules

hostname informix1 - m5000_zfs post_install.sh
hostname informix2 - m5000_zfs post_install.sh

cat m5000_zfs

install_type initial_install
cluster SUNWCall
pool rpool auto 16g auto mirror any any
bootenv installbe bename s10s_u9wos_14a dataset /var

cat sysidcfg

name_service=NONE
system_locale=en_US
system_locale=C
timeserver=localhost
terminal=vt100
security_policy=NONE
root_password=/Xx/xXxxXxXX
network_interface=PRIMARY {netmask=255.255.252.0 protocol_ipv6=no default_route=10.10.12.1}
timezone=US/Eastern
nfs4_domain=dynamic
keyboard=US-English
service_profile=limited_net
auto_reg=disable

Preparing the systems for VCS

Setup a ZFS mirrored pool system & VCS data

zpool create -f informix_data mirror c1t0d0 c1t1d0

Create informix user and group

/etc/passwd - informix:x:100:100::/usr/informix:/bin/bash
/etc/shadow - informix:XXXXXXXXXXX:13119::::::
/etc/group - informix::100:

Create an informix directory

zfs create -o mountpoint=/usr/informix informix_data/informix
chown informix:informix /usr/informix

Installing & configuring VCS

ssh-keygen -t dsa/rsa/rs1
informix1 - cat /.ssh/*.pub >/tmp/a
informix2 - cat /.ssh/*.pub >/tmp/b
 
scp a/b informix1:/tmp & informix2:/tmp
cat /tmp/a or /tmp/b >> authorized_keys

Make available the web installer

cd /informix_data/installs/dvd1-sol_sparc
./webinstaller

Install and select VCS options

Note: Connect your browser to server:14161, enter root login/password

# Example 
https://informix1.domain.com:14161/

Note: Make sure to specify both nodes, the install will then auto install all pkgs on both nodes

• Run first a pre-install check, if successful
• Select Storage Foundation and High Availability, and follow install steps below

When done, Select Storage Foundation for Cluster file system/HA, and run the installer

Now we are ready to run the Cluster Configuration

• The cluster Name was set to informix-cluster
• the private interfaces was selected as nxge3 and nxge7
• Create a cluster admin account with password when prompted
• Add mx.domain.com ass mail server
• Follow the rest of the install till completed

Install the Acc library on both nodes

/informix_data/installs/solaris/sol_sparc/vcs/application/acc_library/5.2.3.0_library/pkgs
pkgadd -d VRTSacclib.pkg

Install the informix agent on both nodes

cd /informix_data/installs/solaris/sol_sparc/vcs/database/informix_agent/5.0/5.1.1.0_agent/pkgs
pkgadd -d . VRTSinformix

Add the agent to the cluster framework – on one node only

cd /tmp
mkdir ./temp
cd ./temp
cp /etc/VRTSagents/ha/conf/Informix/InformixTypes51.cf .
echo 'include "InformixTypes51.cf"' > main.cf
/opt/VRTS/bin/hacf -verify .
/opt/VRTS/bin/haconf -makerw
sh main.cmd
/opt/VRTS/bin/haconf -dump

Modify the agent configuration

The Informix agent configuration file will need to be modified with the correct values

/etc/VRTSagents/ha/conf/Informix/InformixTypes51.cf

Informix instance name

• make sure to use the informix instance virtual name in the INFORMIXSQLHOSTS file.

Note: Use a virtual host name it ensures that if the application needs to be migrated, you are not tied down by the physical IP address given to the IBM Informix Dynamic Server.

demo_on ontlitcp infxsrvr.vxindia.com sqlexec

System tuning

Next lets do some system tuning before we move on to informix VXvM configuration.

System changes

/etc/system

** limit ZFS cache to 10GB
set zfs:zfs_arc_max = 10737418240
*
** Systems may panic/trap during normal operation.
set heaplp_use_stlb=0
*
** A large number of spurious PCIe correctable errors can be recorded in the FMA error log
set pcie:pcie_aer_ce_mask=0x31c1

/etc/fm/fmd/fmd.conf

setprop client.buflim 40m
 
setprop client.memlim 40m

Informix volume groups – VxVM Configuration

Initialize disk groups

vxdisksetup -i tagmastore-usp0_0100
vxdisksetup -i tagmastore-usp0_0200
vxdisksetup -i tagmastore-usp0_0300
vxdisksetup -i tagmastore-usp0_0400

Create informix database disk group

vxdg -s init ifxdb ifxdb01=tagmastore-usp0_0100
vxdg -g ifxdb adddisk ifxdb02=tagmastore-usp0_0200
vxdg -g ifxdb adddisk ifxdb03=tagmastore-usp0_0300
vxdg -g ifxdb adddisk ifxdb04=tagmastore-usp0_0400

# Verify
vxdisk list

Initialize Cluster fencing disk group

vxdisksetup -i tagmastore-usp0_000a
vxdisksetup -i tagmastore-usp0_000b
vxdisksetup -i tagmastore-usp0_000c

Create Cluster I/O fencing disk group

vxdg -s init votingdg votingdg01=tagmastore-usp0_000a
vxdg -g votingdg adddisk votingdg02=tagmastore-usp0_000b
vxdg -g votingdg adddisk votingdg03=tagmastore-usp0_000c

Configure I/O fencing

Select votingdg disk group

/opt/VRTS/install/installsfcfs -fencing

Configure Sub Disks

Get the the disk block count – free disk space

vxdg -g ifxdb free

Create the sub disk (get the size from the last col in the free output)

# My example
vxmake -g ifxdb sd ifxdb01-01 ifxdb01,0,841377920
vxmake -g ifxdb sd ifxdb02-01 ifxdb02,0,841377920
vxmake -g ifxdb sd ifxdb03-01 ifxdb03,0,841377920
vxmake -g ifxdb sd ifxdb04-01 ifxdb04,0,561480448

Note: I used 4 sets, which will be describe below.

Create Set 1

Create sub disks and plex

# Create sub disks
name=1
count=0
while [ $count -lt 266 ]; do vxmake -g ifxdb sd db-s1-d${name}_1 ifxdb01,${count}g,2g ;count=`expr $count + 2`;name=`expr $name + 1`; done

# Create plex
name=1
while [ $name -lt 134 ]; do vxmake -g ifxdb plex db-s1-p${name}_1 sd=db-s1-d${name}_1 ;name=`expr $name + 1`; done

# Create volume and set permission
name=1
while [ $name -lt 134 ]; do vxmake -g ifxdb vol s1-vol${name} user=informix group=informix mode=660 plex=db-s1-p${name}_1 ;name=`expr $name + 1`; done

# Start the volumes (startall didn't work, possibly a bug)
count=1
while [ $count -lt 134 ]; do vxvol -g ifxdb -f start s1-vol${count} ;count=`expr $count + 1`; done

Create Set 2

Create sub disks and plex

# Create sub disks
name=1
count=0
while [ $count -lt 266 ]; do vxmake -g ifxdb sd db-s2-d${name}_1 ifxdb02,${count}g,2g ;count=`expr $count + 2`;name=`expr $name + 1`; done

# Create plex
name=1
while [ $name -lt 134 ]; do vxmake -g ifxdb plex db-s2-p${name}_1 sd=db-s2-d${name}_1 ;name=`expr $name + 1`; done

# Create volume and set permission
name=1
while [ $name -lt 134 ]; do vxmake -g ifxdb vol s2-vol${name} user=informix group=informix mode=660 plex=db-s2-p${name}_1 ;name=`expr $name + 1`; done

# Start the volumes (startall didn't work, possibly a bug)
count=1
while [ $count -lt 134 ]; do vxvol -g ifxdb -f start s2-vol${count} ;count=`expr $count + 1`; done

Create Set 3

Create sub disks and plex

# Create sub disks
name=1
count=0
while [ $count -lt 266 ]; do vxmake -g ifxdb sd db-s3-d${name}_1 ifxdb03,${count}g,2g ;count=`expr $count + 2`;name=`expr $name + 1`; done

# Create plex
name=1
while [ $name -lt 134 ]; do vxmake -g ifxdb plex db-s3-p${name}_1 sd=db-s3-d${name}_1 ;name=`expr $name + 1`; done

# Create volume and set permission
name=1
while [ $name -lt 134 ]; do vxmake -g ifxdb vol s3-vol${name} user=informix group=informix mode=660 plex=db-s3-p${name}_1 ;name=`expr $name + 1`; done

# Start the volumes (startall didn't work, possibly a bug)
count=1
while [ $count -lt 134 ]; do vxvol -g ifxdb -f start s3-vol${count} ;count=`expr $count + 1`; done

Create Set 4

Create sub disks and plex

# Create sub disks
name=1
count=0
while [ $count -lt 266 ]; do vxmake -g ifxdb sd db-s4-d${name}_1 ifxdb04,${count}g,2g ;count=`expr $count + 2`;name=`expr $name + 1`; done

# Create plex
name=1
while [ $name -lt 134 ]; do vxmake -g ifxdb plex db-s4-p${name}_1 sd=db-s4-d${name}_1 ;name=`expr $name + 1`; done

# Create volume and set permission
name=1
while [ $name -lt 134 ]; do vxmake -g ifxdb vol s4-vol${name} user=informix group=informix mode=660 plex=db-s4-p${name}_1 ;name=`expr $name + 1`; done

# Start the volumes (startall didn't work, possibly a bug)
count=1
while [ $count -lt 134 ]; do vxvol -g ifxdb -f start s4-vol${count} ;count=`expr $count + 1`; done

Configure cache volumes

Initialize and add extra disk as cache used for snapshots.

vxdisksetup -i tagmastore-usp0_05e0
 
vxdg -g ifxdb adddisk cachevol01=tagmastore-usp0_05e0
vxassist -g ifxdb make cachevol 1023232000b layout=nostripe init=active cachevol01

Make sure the volumes are prepared

# Before prepare
vxprint -g ifxdb -F%instant s1-vol1 
off
vxprint -g ifxdb -F%fastresync s1-vol1 
off
 
# After prepare
vxprint -g ifxdb -F%instant s1-vol1
on
vxprint -g ifxdb -F%fastresync s1-vol1
on

Prepare the volumes

vxsnap -g ifxdb -f prepare s1-vol1
vxsnap -g ifxdb -f prepare s1-vol2
...
 
# Prepare all volumes
for i in `vxprint -v |awk '{print $2}'|egrep -v "^\$|group|NAME|dcl"`; do vxsnap -g ifxdb -f prepare $i drl=on; done

Create a space-optimized snapshot from s1-vol133 available as s1-vol133-snap1

Note: The size for changes are set as 775MB (meaning total change size)

vxsnap -g ifxdb make source=s1-vol133/newvol=s1-vol133-snap1/cachesize=775m/autogrow=yes

Create all snaps at once

echo 'vxsnap -g ifxdb make \' > /tmp/create_snap
vxprint -v |egrep -v "^\$|Disk|NAME|_dcl" |awk '{print "source="$2"/newvol="$2"-snap1/cachesize=775m/autogrow=yes \\"}' >>/tmp/create_snap
sh /tmp/create_snap

Note: If the above doesn’t work

for i in `vxprint -v |egrep -v "^\$|Disk|NAME|_dcl" |awk '{print $2}'`; do vxsnap -g ifxdb make source="$i"/newvol="$i"-snap1/cachesize=775m/autogrow=yes ;done

Check volume status

# Example output
vxsnap -g ifxdb print
NAME    SNAPOBJECT    TYPE    PARENT    SNAPSHOT    %DIRTY    %VALID
 
s1-vol133 --           volume   --       --           --       100.00
        s1-vol133-snap1_snp  volume   --       s1-vol133-snap1 0.00     --    
 
s1-vol133-snap1 s1-vol133_snp volume   s1-vol133 --           0.00     0.00

Refresh all volumes

To resync/refresh all volumes, making the snaps like the source
Get a list of all the volumes to a file, then run it

vxsnap -g ifxdb print |grep -v snap1_snp|grep "snap1 " |awk '{print "vxsnap -g ifxdb refresh",$1}' >/tmp/refresh_snap.sh

Restore(rollback) from target snapshot s1-vol133-snap1 to s1-vol133

vxsnap -g ifxdb restore s1-vol133 source=s1-vol133-snap1
 
echo 'vxsnap -g ifxdb -o iosize=10m restore \' >/tmp/restore_chunks
vxprint -v |egrep -v "^\$|Disk|NAME|_dcl|CV01|snap" |awk '{print $2,"source="$2"-snap1 \\"}' >>/tmp/restore_chunks
sh /tmp/restore_chunks
 
## Don't use
# Make sure to stop the db first
#vxprint -v |egrep -v "^\$|Disk|NAME|_dcl|snap" |awk '{print "vxsnap -g ifxdb restore",$2,"source="$2"-snap1"}' >/tmp/restore_chunks
#sh /tmp/restore_chunks

Destroy the snapshot

# Destroy the top volume of the snapshot (will leave dcl/drl)
vxedit -g ifxdb -rf rm s1-vol133-snap1
 
# will completely remove the snapshot drl/dcl etc... 
vxsnap -g ifxdb -f unprepare s1-vol133
 
# To remove just the snap
for i in {1..133}; do vxedit -g ifxdb -rf rm s1-vol${i}-snap1; done
for i in {1..133}; do vxedit -g ifxdb -rf rm s2-vol${i}-snap1; done
for i in {1..133}; do vxedit -g ifxdb -rf rm s3-vol${i}-snap1; done
for i in {1..133}; do vxedit -g ifxdb -rf rm s4-vol${i}-snap1; done
 
# To remove the snap and prepare in the mess just run this loop
for i in {1..133}; do vxedit -g ifxdb -rf rm s1-vol${i}-snap1; vxsnap -g ifxdb -f unprepare s1-vol${i};done
for i in {1..133}; do vxedit -g ifxdb -rf rm s2-vol${i}-snap1; vxsnap -g ifxdb -f unprepare s2-vol${i};done
for i in {1..133}; do vxedit -g ifxdb -rf rm s3-vol${i}-snap1; vxsnap -g ifxdb -f unprepare s3-vol${i};done
for i in {1..133}; do vxedit -g ifxdb -rf rm s4-vol${i}-snap1; vxsnap -g ifxdb -f unprepare s4-vol${i};done

To create a snapshot for multiple volumes at once

vxsnap -g ifxdb make \
source=s1-vol1/newvol=s1-vol1-snap1/cachesize=2g/autogrow=yes \
source=s1-vol2/newvol=s1-vol2-snap1/cachesize=2g/autogrow=yes \
source=s1-vol3/newvol=s1-vol3-snap1/cachesize=2g/autogrow=yes

Experiment helpful

# Refreshing an instant snapshot
vxsnap [-g diskgroup] refresh snapvol [source=vol] \
[[snapvol2 source=vol2]...] [sync=yes|no]
 
# Restoring a volume from an instant snapshot
vxsnap [-g diskgroup] restore [nmirrors=number] vol \
[source=snapvol] [destroy=yes|no]
 
# Displaying instant snapshot information
vxsnap [-g diskgroup] print [vol]
vxsnap -g dg -vx list

Get all VxVM permeates

vxprint -g ifxdb -m |less
 
# Set fast sync on
vxvol -g  set fastresync=on s1-vol1_dcl

Problems / solutions

Problem
Cannot complete the snapshot operation.
Error
VxVM vxassist ERROR V-5-1-6197 volume is busy in another operation
Solution
Verify from the ‘vxprint -g ‘ command if “TUTIL0 or PUTIL0” flags is set. If set, then perform the following command and then continue with the SNAP operation

# vxmend -g  clear all 

Remove disk plex or volumes

removes subdik with plex

vxedit -g ifxdb -rf rm ifxdb01-01

Removing a volume

vxedit -g ifxdb rm -f vol1

To remove a disk group with the disks

vxdg -g votingdg destroy votingdg
vxdisk rm tagmastore-usp0_000a

Additional commends

Disable/Enable VxVM DMP

vxdmpadm -f enable ctlr=c6
vxdmpadm list dmpnode all

Display Veritas fencing

vxfenadm -s all -f /etc/vxfentab
vxfenadm -x -k tagmastore-usp0_0100
vxfenadm -G all -f /etc/vxfentab

How to disable VCS auto monitor

Note: Once the preonline trigger is configured, you may see unexpected behavior
while manually switching or performing online operations on the Informix service
group. This behavior is a result of the control logic within the preonline trigger
that protects the Informix SDS Primary Secondary relationship. For system
maintenance, if you prefer to perform manual operations on the service groups,
you can do so by disabling the preonline trigger as follows:

# hagrp -modify service_group PreOnline 0

Switch and check primary secondary

switchprim Promotes the Informix SD Secondary Server to SD Primary
           Server. This action expects the name of the current/failed SD
           Primary Informix Instance as actionargs.
           This action is executed by preonline trigger in case of SD
           Primary failure.
           For example,
           # hares –action infx_inst2_res switchprim
           –actionargs infx_inst1 –sys infx_sys1
isprimary  Retrieves the Shared disk configuration of current informix
           instance i.e. it is configured as SD Primary or SD Secondary.
           For example,
           # hares –action infx_inst2_res isprimary –sys
           infx_sys1
Examples in our enverment
## Run on the primary to switch to secondary (in this example informix2 is the primary, and this is run on informix2)
hares -action informix-rs switchprim -actionargs infomrix1 -sys informix1
 
## Run on the secondary to make this primary (in this example informix2 is the primary, and we run this on infomrix1 to make it primary)
hares –action  isprimary informix1  –sys informix1 isprimary

VXS config files
To change VCS configuration

Modify /etc/VRTSvcs/conf/config/main.cf 

To re-read the config files restart VCS with

Run hastop -all -force 
Run hastart on each node 

VCS logs
vcs logs are located in /var/VRTSvcs/log, you could tail the engine_A.log or the Informix_A.log log. 

How to start Informix agent
haagent -start Informix -sys SystemName

VCS/VxVM disable procedure
Run the below commends on all cluster nodes to disable VCS and VxVM.
hastop -all -force
 
touch /etc/vx/reconfig.d/state.d/install-db 
svcadm disable svc:/system/llt:default svc:/system/gab:default svc:/system/vcs:default
 
reboot

Helpful links
VCS configuration

VxVM Cheat sheet

VxVM layers

VxVM volume Administration

Oracle Cluster and zone HA

VxVM snapshots / checkpoints

            

            

            

                

                    0
                    0
                    votes
                
Article Rating
                

            
Share this:
Click to share on Twitter (Opens in new window)
Click to share on Facebook (Opens in new window)
Click to share on LinkedIn (Opens in new window)
Click to share on Reddit (Opens in new window)
Click to share on Pinterest (Opens in new window)
Click to email a link to a friend (Opens in new window)
Like this:
Like Loading...


	
Related