Contents
- 1 Oracle CRS installation
- 2 Sun Cluster and QFS configuration
- 3 Complete the Sun Cluster and Oracle RAC configuration
- 4 Sun Cluster 3.2 tweeks
- 4.1 Increase Rac FramWork timeout
- 4.2 Remove wrongly set dependencies
- 4.3 Add QFS Resource restart dependencies
- 4.4 Fix Cluster cron jobs
- 4.5 Setup Cluster NTP to our servers
- 4.6 Enable dynamic pools
- 4.7 Chnage cluster to reboot if all disks fail
- 4.8 UnSet the cluster monitor for local disks
- 4.9 For extra protection add the clusterpriv address
- 5 Sun Cluster and GW Gateway configuration
- 6 Sun Cluster Help
- 7 Issues and resolution doing Sun Cluster and Oracle RAC configuration
- 8 Rman And NetBackup San Backup
- 9 References and help
Oracle CRS installation
Install oracle CRS 11G from CD
Oracle CRS Notes
- Make sure /etc/issue is empty other wise Oracle OCR wont install.
- Make sure ssh works without a password even local mentioned in part 1
Sun Cluster and QFS configuration
Create an mcf file
create mcf file on both servers (each file system needs an entry in the mcf file)
cat /etc/opt/SUNWsamfs/mcf # device ord type familyset on other parameters #-------------------------------------------------------------------------------- archivelogs 10 ma archivelogs on shared /dev/md/oralogs/dsk/d0 11 mm archivelogs on /dev/md/oralogs/dsk/d10 15 mr archivelogs on # # device ord type familyset on other parameters #-------------------------------------------------------------------------------- flashback 20 ma flashback on shared /dev/md/oralogs/dsk/d20 21 mm flashback on /dev/md/oralogs/dsk/d30 25 mr flashback on
Create QFS file system hosts file
Create hosts.[familyset name]
Note: server should be the same server as the metaset master
cat /etc/opt/SUNWsamfs/hosts.archivelogs # Host Host IP Server Not Server # Name Addresses Priority Used Host # ----- --------- -------- ---- ---- db1 clusternode1-priv 1 - server db2 clusternode2-priv 2 - cat /etc/opt/SUNWsamfs/hosts.flashback # Host Host IP Server Not Server # Name Addresses Priority Used Host # ----- --------- -------- ---- ---- db1 clusternode1-priv 1 - server db2 clusternode2-priv 2 -
Create samfs mount option file
Create /etc/opt/SUNWsamfs/samfs.cmd for each file system
cat /etc/opt/SUNWsamfs/samfs.cmd fs = archivelogs stripe=1 sync_meta=1 qwrite forcedirectio notrace rdlease=600 wrlease=600 aplease=600 nstreams=1024 dmr abr fs = flashback stripe=1 sync_meta=1 qwrite forcedirectio notrace rdlease=600 wrlease=600 aplease=600 dmr abr
Create the QFS file system
First run sam-fsd to check for errors (the errors column should show 0)
If no errors run
QFS to refresh all his config files with the new configuration:
samd config
Create QFS file system:
sammkfs -S archivelogs (run on one node only) sammkfs -S flashback (run on one node only) mkdir /archive_logs mkdir /flash_back
Add the mounts to vfstab
archivelogs - /archive_logs samfs - no shared,notrace flashback - /flash_back samfs - no shared,notrace
Note: Run first on QFS Master then on client
mount /archive_logs mount /flash_back
Note: To force a meta data server take over and re-create of the file system you could run
sammkfs [-R]-S db1 archivelogs sammkfs [-R]-S db1 flashback
Sun Cluster QFS configuration
To add QFS to Sun Cluster and metasets master failover
ln -s /opt/SUNWsamfs/sc/etc/SUNW.qfs /usr/cluster/lib/rgm/rtreg/SUNW.qfs
Register the QFS resource with Sun cluster:
scrgadm -a -t SUNW.qfs
Setup the QFS resource:
scrgadm -a -g QFS-rg -h db1,db2 -y RG_DEPENDENCIES="rac-fmwk-rg" scrgadm -a -g QFS-rg -t SUNW.qfs -j QFS-fs-rs -x QFSFileSystem=/archive_logs,/flash_back
Turn the QFS resource online:
scswitch -Z -g QFS-rg
If you add another file system at a latter time the cluster needs to know about it run:
clrs set -p QFSFileSystem=/archive_logs,/flash_back QFS-fs-rs
Complete the Sun Cluster and Oracle RAC configuration
Note:
Before you use this wizard, ensure that the following prerequistes are met: * The RAC framework resource group is created and is online. * Storage resources for Oracle files are configured. * The Oracle Real Application Clusters software is installed. * The installation of the Oracle Real Application Clusters software is verified. * The Oracle database is created. * Node-specific files and directories for a shared file system are created.
To complete the install run clsetup select the options below
- 3) Data Services – then
- 4) Oracle Real Application Clusters – then
- Answer yes for the next question
- 1) Oracle RAC Create Configuration – then
- 3) Oracle RAC Database Resources – then continue
Answer all the reset of questions
For a complete example Click Here
Sun Cluster and crs references is available Here
To manuel complete the Sun cluster and CRS setup run use the process below
/usr/cluster/bin/clresourcegroup online -emM rac-fmwk-rg
/usr/cluster/bin/clresourcetype register SUNW.crs_framework
/usr/cluster/bin/clresource create -t SUNW.crs_framework -g rac-fmwk-rg -p \
Resource_dependencies=rac-fmwk-rs -p \
resource_dependencies_offline_restart=scalocrdg-rs{local_node},scalvotingdg-rs{local_node} crs_framework-rs
/usr/cluster/bin/clresourcegroup online -emM rac-fmwk-rg
/usr/cluster/bin/clresourcegroup create -p Desired_primaries=2 -p \
nodelist=db1,db2 -p RG_mode=Scalable -p RG_affinities=++rac-fmwk-rg,++scaldg-rg -p \
Maximum_primaries=2 rac_server_proxy-rg
/usr/cluster/bin/clresource create -t SUNW.scalable_rac_server_proxy -g \
rac_server_proxy-rg -p Resource_dependencies=rac-fmwk-rs -p \
resource_dependencies_offline_restart=scaloradg-rs,scalvotingdg-rs,scalocrdg-rs,crs_framework-rs -p \
CRS_HOME=/oracle/product/11.1.0/crs_1 -p \
DB_NAME=DB -p ORACLE_HOME=/oracle/product/11.1.0/db_1 -p ORACLE_SID{db2}=DB2 -p \
ORACLE_SID{db1}=DB1 rac_server_proxy-rs
/usr/cluster/bin/clresourcegroup online -emM rac_server_proxy-rg
Attempting to launch command:
/opt/SUNWscor/dsconfig/bin/scproxy_crs NO /oracle/product/11.1.0/crs_1 scaloradg-rs
DB Proxy resource sun.db1.scaloradg-rs:
Creating Oracle CRS profile in /var/cluster/ucmm/profile.
/oracle/product/11.1.0/crs_1/bin/crs_profile -create sun.db1.scaloradg-rs \ -t application -d "Proxy for Sun Cluster resource scaloradg-rs" \ -dir /var/cluster/ucmm/profile -a /opt/SUNWscor/dsconfig/bin/scproxy_crs_action -p \ restricted -h db1 -f -o ra=60,st=20
Proxy resource sun.db1.scaloradg-rs: Registering with Oracle CRS.
/oracle/product/11.1.0/crs_1/bin/crs_register sun.db1.scaloradg-rs -dir /var/cluster/ucmm/profile
Proxy resource sun.db1.scaloradg-rs: Registered with Oracle CRS.
Adding dependency of ora.DB.DB1.inst on Sun Cluster proxy resource sun. db1.scaloradg-rs.
/oracle/product/11.1.0/crs_1/bin/crs_register ora.DB.DB1.inst -update -r "sun.db1.scaloradg-rs"
Dependency of ora.DB.DB1.inst now includes sun.db1.scaloradg-rs.
Proxy resource sun.db2.scaloradg-rs: Creating Oracle CRS profile in /var/cluster/ucmm/profile.
/oracle/product/11.1.0/crs_1/bin/crs_profile -create sun.db2.scaloradg-rs \ -t application -d "Proxy for Sun Cluster resource scaloradg-rs" -dir /var/cluster/ucmm/profile \ -a /opt/SUNWscor/dsconfig/bin/scproxy_crs_action -p \ restricted -h db2 -f -o ra=60,st=20
Proxy resource sun.db2.scaloradg-rs: Registering with Oracle CRS.
/oracle/product/11.1.0/crs_1/bin/crs_register sun.db2.scaloradg-rs -dir /var/cluster/ucmm/profile
Proxy resource sun.db2.scaloradg-rs: Registered with Oracle CRS.
Adding dependency of ora.DB.DB2.inst on Sun Cluster proxy resource sun. db2.scaloradg-rs.
/oracle/product/11.1.0/crs_1/bin/crs_register ora.DB.BDB2.inst -update -r "sun.db2.scaloradg-rs"
Dependency of ora.DB.DB2.inst now includes sun.db2.scaloradg-rs.
Attempting to launch command:
/opt/SUNWscor/dsconfig/bin/scproxy_crs NO /oracle/product/11.1.0/crs_1 scalvotingdg-rs DB
Proxy resource sun.db1.scalvotingdg-rs:
Creating Oracle CRS profile in /var/cluster/ucmm/profile.
/oracle/product/11.1.0/crs_1/bin/crs_profile -create sun.db1.scalvotingdg-rs -t application \ -d "Proxy for Sun Cluster resource scalvotingdg-rs" -dir /var/cluster/ucmm/profile -a \ /opt/SUNWscor/dsconfig/bin/scproxy_crs_action -p restricted -h db1 -f -o ra=60,st=20
Proxy resource sun.db1.scalvotingdg-rs: Registering with Oracle CRS.
/oracle/product/11.1.0/crs_1/bin/crs_register sun.db1.scalvotingdg-rs -dir /var/cluster/ucmm/profile
Proxy resource sun.db1.scalvotingdg-rs: Registered with Oracle CRS.
Starting proxy resource sun.db1.scalvotingdg-rs.
/oracle/product/11.1.0/crs_1/bin/crs_start sun.db1.scalvotingdg-rs
Attempting to start `sun.db1.scalvotingdg-rs` on member `db1` Start of `sun.db1.scalvotingdg-rs` on member `db1` succeeded.
Proxy resource sun.db1.scalvotingdg-rs started successfully.
Adding dependency of ora.DB.DB1.inst on Sun Cluster proxy resource sun.db1.scalvotingdg-rs.
/oracle/product/11.1.0/crs_1/bin/crs_register ora.DB.DB1.inst -update -r "sun.v.scaloradg-rs sun.db1.scalvotingdg-rs"
Dependency of ora.DB.DB1.inst now includes sun.db1.scalvotingdg-rs.
Proxy resource sun.db2.scalvotingdg-rs: Creating Oracle CRS profile in /var/cluster/ucmm/profile.
/oracle/product/11.1.0/crs_1/bin/crs_profile -create sun.db2.scalvotingdg-rs \ -t application -d "Proxy for Sun Cluster resource scalvotingdg-rs" \ -dir /var/cluster/ucmm/profile -a /opt/SUNWscor/dsconfig/bin/scproxy_crs_action -p \ restricted -h db2 -f -o ra=60,st=20
Proxy resource sun.db2.scalvotingdg-rs: Registering with Oracle CRS.
/oracle/product/11.1.0/crs_1/bin/crs_register sun.v.scalvotingdg-rs -dir /var/cluster/ucmm/profile
Proxy resource sun.db2.scalvotingdg-rs: Registered with Oracle CRS.
Starting proxy resource sun.db2.scalvotingdg-rs.
/oracle/product/11.1.0/crs_1/bin/crs_start sun.v.scalvotingdg-rs
Attempting to start `sun.db2.scalvotingdg-rs` on member `db2` Start of `sun.v.scalvotingdg-rs` on member `v` succeeded.
Proxy resource sun.db2.scalvotingdg-rs started successfully.
Adding dependency of ora.DB.DB2.inst on Sun Cluster proxy resource sun.
db2.scalvotingdg-rs.
/oracle/product/11.1.0/crs_1/bin/crs_register ora.DB.DB2.inst -update -r \ "sun.db2.scaloradg-rs sun.db2.scalvotingdg-rs"
Dependency of ora.DB.DB2.inst now includes sun.db2.scalvotingdg-rs.
Attempting to launch command:
/opt/SUNWscor/dsconfig/bin/scproxy_crs NO /oracle/product/11.1.0/crs_1 scalocrdg-rs DB
Proxy resource sun.db1.scalocrdg-rs: Creating Oracle CRS profile in /var/cluster/ucmm/profile.
/oracle/product/11.1.0/crs_1/bin/crs_profile -create sun.db1.scalocrdg-rs \ -t application -d "Proxy for Sun Cluster resource scalocrdg-rs" -dir /var/cluster/ucmm/profile \ -a /opt/SUNWscor/dsconfig/bin/scproxy_crs_action -p restricted -h db1 -f -o ra=60,st=20
Proxy resource sun.db1.scalocrdg-rs: Registering with Oracle CRS.
/oracle/product/11.1.0/crs_1/bin/crs_register sun.db1.scalocrdg-rs -dir /var/cluster/ucmm/profile
Proxy resource sun.db1.scalocrdg-rs: Registered with Oracle CRS.
Starting proxy resource sun.db1.scalocrdg-rs.
/oracle/product/11.1.0/crs_1/bin/crs_start sun.db1.scalocrdg-rs
Attempting to start `sun.db1.scalocrdg-rs` on member `db1` Start of `sun.db1.scalocrdg-rs` on member `db1` succeeded.
Proxy resource sun.db1.scalocrdg-rs started successfully.
Adding db1 of ora.DB.DB1.inst on Sun Cluster proxy resource sun. db1.scalocrdg-rs.
/oracle/product/11.1.0/crs_1/bin/crs_register ora.DB.DB1.inst -update \ -r "sun.db1.scaloradg-rs sun.db1.scalvotingdg-rs sun.db1.scalocrdg-rs"
Dependency of ora.DB.DB1.inst now includes sun.db1.scalocrdg-rs.
Proxy resource sun.db2.scalocrdg-rs: Creating Oracle CRS profile in /var/cluster/ucmm/profile.
/oracle/product/11.1.0/crs_1/bin/crs_profile -create sun.db2.scalocrdg-rs \ -t application -d "Proxy for Sun Cluster resource scalocrdg-rs" -dir /var/cluster/ucmm/profile \ -a /opt/SUNWscor/dsconfig/bin/scproxy_crs_action -p restricted -h db2 -f -o ra=60,st=20
Proxy resource sun.db2.scalocrdg-rs: Registering with Oracle CRS.
/oracle/product/11.1.0/crs_1/bin/crs_register sun.db2.scalocrdg-rs -dir /var/cluster/ucmm/profile
Proxy resource sun.db2.scalocrdg-rs: Registered with Oracle CRS.
Starting proxy resource sun.db2.scalocrdg-rs.
/oracle/product/11.1.0/crs_1/bin/crs_start sun.db2.scalocrdg-rs
Attempting to start `sun.db2.scalocrdg-rs` on member `db2` Start of `sun.db2.scalocrdg-rs` on member `db2` succeeded.
Proxy resource sun.db2.scalocrdg-rs started successfully.
Adding dependency of ora.DB.DB2.inst on Sun Cluster proxy resource sun.db2.scalocrdg-rs.
/oracle/product/11.1.0/crs_1/bin/crs_register ora.DB.DB2.inst -update \ -r "sun.db2.scaloradg-rs sun.db2.scalvotingdg-rs sun.db2.scalocrdg-rs"
Dependency of ora.DB.DB2.inst now includes sun.db2.scalocrdg-rs.
Sun Cluster 3.2 tweeks
Increase Rac FramWork timeout
Its recommended to increase the RAC FramWork Time Out
/usr/cluster/bin/clresource set -p Svm_step4_timeout=360 rac-svm-rs
Remove wrongly set dependencies
Fix the QFS-rg dependence that was wrongly set
Remove QFS-rg dependence:
/usr/cluster/bin/clresourcegroup set -p RG_dependencies= QFS-rg
ReAdd the correct dependencies:
/usr/cluster/bin/clresourcegroup set -p RG_affinities=++scaldg-rg QFS-rg
Add QFS Resource restart dependencies
Add a restart Resource dependence on the QFS-fs-rs resource
/usr/cluster/bin/clresource set -p Resource_dependencies_offline_restart=scaloradg-rs,scalocrdg-rs,scalvotingdg-rs QFS-fs-rs
Fix Cluster cron jobs
The cluster crontab jobs are times are wrong and needs to change
crontab -e Change From: 20 4 * * 0 /usr/cluster/lib/sc/newcleventlog /var/cluster/logs/eventlog 20 4 * * 0 /usr/cluster/lib/sc/newcleventlog /var/cluster/logs/DS 20 4 * * 0 /usr/cluster/lib/sc/newcleventlog /var/cluster/logs/commandlog Change To: 20 4 * * 0 /usr/cluster/lib/sc/newcleventlog /var/cluster/logs/eventlog 25 4 * * 0 /usr/cluster/lib/sc/newcleventlog /var/cluster/logs/DS 30 4 * * 0 /usr/cluster/lib/sc/newcleventlog /var/cluster/logs/commandlog
Setup Cluster NTP to our servers
Modify the /etc/inet/ntp.conf.cluster like this
#server 127.127.1.0 server ntp.pool.org
Enable dynamic pools
Make sure dynamic pools are enabled (for dynamic hardware configuration)
svcadm enable /system/pools/dynamic
Chnage cluster to reboot if all disks fail
By default Sun Cluster 3.2 will not reboot if all disk failed, to change that run
Get device listing: scdpm -p all:all Set the reboot flag to on: clnode set -p reboot_on_path_failure=enabled db1 db2 Verify the change: scdpm -p all:all
UnSet the cluster monitor for local disks
First get the local disk did, then unset the cluster monitor on this did’s (run this on all cluster nodes)
scdidadm -l cldevice unmonitor d1 d2 d3 Verify the change: scdpm -p all:all
Set and change the monitor to local only
Check the setting: cldg show dsk/d3 Device Groups === Device Group Name: dsk/d3 Type: Disk failback: false Node List: db1 preferenced: false localonly: false autogen: true numsecondaries: 1 device names: /dev/did/rdsk/d3s2 Change localonly to true for all local disks: cldg set -p localonly=true dsk/d3
For extra protection add the clusterpriv address
Note: This step is not needed but could be done for extra protection
Add all clusternode1-priv to the /etc/hosts,/etc/netmasks file
/etc/hosts Cluster private IP's: 10.10.5.129 clusternode1-priv 10.10.5.9 clusternode1-priv-physical1 10.10.5.17 clusternode1-priv-physical2 10.10.5.130 clusternode2-priv 10.10.5.10 clusternode2-priv-physical1 10.10.5.18 clusternode2-priv-physical2
Sun Cluster and GW Gateway configuration
Add GW VIP address to the local host files
cat /etc/hosts 10.10.10.12 gw-vip
Crated bea user and group accounts in /etc/password
Copy /usr/local/bea from joel/hosea to the new cluster
Configure GW Gateway to be managed by Sun Cluster
clresourcetype register SUNW.gds /usr/cluster/bin/clresourcegroup create -p nodelist=db1,db2 gw-vip-rg /usr/cluster/bin/clressharedaddress create -g gw-vip-rg -h gw-vip -N db1_gp@db1,wedb2bdb2_gp@db2 gw-vip-rs /usr/cluster/bin/clresourcegroup online -eM gw-vip-rg /usr/cluster/bin/clresourcegroup create -p Desired_primaries=2 -p Maximum_primaries=2 -p Nodelist=db1,db2 -p RG_SLM_type=manual \ -p Global_resources_used=false GW-rg /usr/cluster/bin/clresource create -t SUNW.gds:6 -g GW-rg -p Resource_project_name= \ -p Resource_dependencies=gw-vip-rs \ -p Load_balancing_policy=Lb_weighted -p Network_resources_used=gw-vip-rs -p Scalable=TRUE -p Load_balancing_weights=1@db1,1@db2 \ -p R_description="Scalable data service resource for SUNW.gds:6" -p Port_list=9999/tcp -p Validate_command= \ -p Start_command=/usr/cluster/GW/Start.sh -p Stop_command=/usr/cluster/GW/Stop.sh \ -p Probe_command=/usr/cluster/GW/Probe.sh GW-rs /usr/cluster/bin/clresourcegroup online -eM GW-rg
DR Configuration:
clresourcetype register SUNW.gds /usr/cluster/bin/clresourcegroup create -p nodelist=dbdr1,dbdr2 gwdr-vip-rg /usr/cluster/bin/clressharedaddress create -g gwdr-vip-rg -h gwdr-vip -N dbdr1_gp@dbdr1,dbdr2_gp@dbdr2 gwedr-vip-rs /usr/cluster/bin/clresourcegroup online -eM gwdr-vip-rg /usr/cluster/bin/clresourcegroup create -p Desired_primaries=2 -p Maximum_primaries=2 -p Nodelist=dbdr1,dbdr2 -p RG_SLM_type=manual \ -p Global_resources_used=false GW-rg /usr/cluster/bin/clresource create -t SUNW.gds:6 -g GW-rg -p Resource_project_name= \ -p Resource_dependencies=gwdr-vip-rs \ -p Load_balancing_policy=Lb_weighted -p Network_resources_used=gwdr-vip-rs -p Scalable=TRUE -p Load_balancing_weights=1@dbdr1,1@dbdr2 \ -p R_description="Scalable data service resource for SUNW.gds:6" -p Port_list=9999/tcp -p Validate_command= \ -p Start_command=/usr/cluster/GW/Start.sh -p Stop_command=/usr/cluster/GW/Stop.sh \ -p Probe_command=/usr/cluster/GW/Probe.sh GW-rs /usr/cluster/bin/clresourcegroup online -eM GW-rg
Sun Cluster Help
Sun Cluster commands
To show all resource groups and list there status or just resources
List resource groups status clrg status List resources status: clrs status
To stop or start a resource groups or multiple resource groups
To turn on: clrg switch -n db1,db2 resource-group1 resource-group2 To turn off: clrg ofline -n db1,db2 resource-group1 resource-group2
To disable or enable a resource or multiple resources
To enable a resource: clrs disable resource-1 resource-2 To disable a resource: clrs enable resource-1 resource-2
Oracle Commands
To check the crs status
Check to Oracle Clusterware and DB health: crs_stat -t Check CRS health: crsctl check crs
To restart the Oracle crs clean
If crs_stop -all doesn’t work then run this on each node that has a problem.
cd /oracle/product/11.1.0/crs_1/bin # ./crsctl stop crs Stopping resources. This could take several minutes. Error while stopping resources. Possible cause: CRSD is down. Stopping Cluster Synchronization Services. Unable to communicate with the Cluster Synchronization Services daemon. # To force a crs stop just run crsctl stop crs -force # ./crsctl start crs Attempting to start Oracle Clusterware stack The CRS stack will be started shortly # ./crsctl check crs Cluster Synchronization Services appears healthy Cluster Ready Services appears healthy Event Manager appears healthy
Issues and resolution doing Sun Cluster and Oracle RAC configuration
Problem 1
Problem: Cluster panic after reboot
- Problem: Cluster panic after reboot
Aug 28 08:59:40 svc.startd[8]: svc:/system/cluster/rpc-pmf:default: Method "/usr/cluster/lib/svc/method/svc_rpc_pmf start" failed with exit status 96. Aug 28 08:59:40 svc.startd[8]: system/cluster/rpc-pmf:default misconfigured: transitioned to maintenance (see 'svcs -xv' for details) Aug 28 08:59:43 Cluster.RGMPMF.lib: Unable to open door descriptor /var/run/rgmd_receptionist_door Aug 28 08:59:43 Cluster.RGMPMF.lib: Unable to open door descriptor /var/run/rgmd_receptionist_door Aug 28 08:59:44 Cluster.RGMPMF.lib: Unable to open door descriptor /var/run/rgmd_receptionist_door Aug 28 08:59:44 Cluster.RGMPMF.lib: Unable to open door descriptor /var/run/rgmd_receptionist_door Aug 28 08:59:45 Cluster.RGMPMF.lib: Unable to open door descriptor /var/run/rgmd_receptionist_door Notifying cluster that this node is panicking panic[cpu0]/thread=2a100047ca0: Failfast: Aborting zone "global" (zone ID 0) because "sc_zonesd" died 35 seconds ago. 000002a1000473f0 cl_runtime:__1cZsc_syslog_msg_log_no_args6Fpviipkc0_nZsc_syslog_msg_status_enum__+30 (6002c0c0000, 3, 0, 46, 2a1000475f0, 706c3461) %l0-3: 0000000000000000 0000000000010000 0000000000000000 0000000000000000 %l4-7: 0000000000000000 0000000000000001 0000000000000000 000000000180c000 000002a1000474a0 cl_runtime:__1cCosNsc_syslog_msgDlog6MiipkcE_nZsc_syslog_msg_status_enum__+1c (6002c0a73b8, 3, 0, 706c3461, 60032da8d4a, 70400) %l0-3: 0000060032da8c40 0000000000070400 00000000000706be 0000000000070400 %l4-7: 00000000706c3452 00000000706c3000 00000000706c2000 0000000000000001 000002a100047550 cl_haci:__1cHff_implNunit_timedout6M_v_+70 (60032da8cd8, 193c400, 16, 60032da8c40, 180c000, 1) %l0-3: 000000007aa8585c 0000000000000000 000000000121a000 0000000000000001 %l4-7: 0000000000000001 0000000000000001 0000000000000000 00000000018b8f58
Solution: Install Sun Cluster patch
Solution: Install Sun Cluster patch 126106-15 fixed the problem
Note: If you run in to this bug after the install of this patch you might need to uninstall and reinstall the patch
Another problem related to uninstall / reinstall of the patch
cd /etc/opt/SUNWcluster/conf /etc/opt/SUNWcluster/conf # grep oracle.dba.gid * udlm.conf:oracle.dba.gid : y udlm.conf.template:oracle.dba.gid : dba Change in udlm.conf the oracle.dba.gid : y to oracle.dba.gid : dba reboot when done
Problem 2
Problem: metaset metainit will refuse to work
- Problem: After adding disks to a multi-owner metaset metainit will refuse to work with errors.
Solution: A reboot is needed
Solution: A reboot is needed for the metadb to work properly
Before a re-boot: [root@db1] / # metadb -s oradg -i flags first blk block count u 16 32768 /dev/did/dsk/d11s7 u 16 32768 /dev/did/dsk/d22s7 u 16 32768 /dev/did/dsk/d10s7 u 16 32768 /dev/did/dsk/d21s7 u 16 32768 /dev/did/dsk/d9s7 u 16 32768 /dev/did/dsk/d20s7 u 16 32768 /dev/did/dsk/d8s7 After a reboot: [root@db1] / # metadb -s oradg -i flags first blk block count a m luo r 16 32768 /dev/did/dsk/d11s7 a luo r 16 32768 /dev/did/dsk/d22s7 a luo r 16 32768 /dev/did/dsk/d10s7 a luo r 16 32768 /dev/did/dsk/d21s7 a luo r 16 32768 /dev/did/dsk/d9s7 a luo r 16 32768 /dev/did/dsk/d20s7 a luo r 16 32768 /dev/did/dsk/d8s7
Problem 3
Problem: Not All IO components work in a M5000
- Problem: All IO components add to the 2nd IO boat dose not show up on the system
Solution: Moving system boards is needed
Solution:
The IO Boards add to the 2nd IO boat will only show up if one CPU and one Memory board is moved to the other IO Path (Slot3/Slot4)
Before the board was moved: XSCF> showhardconf SPARC Enterprise M5000; + Serial:BEF08265F0; Operator_Panel_Switch:Locked; + Power_Supply_System:Single; SCF-ID:XSCF#0; + System_Power:On; System_Phase:Cabinet Power On; Domain#0 Domain_Status:Running; MBU_B Status:Normal; Ver:0201h; Serial:BE081813E2 ; + FRU-Part-Number:CF00541-0478 05 /541-0478-05 ; + Memory_Size:16 GB; CPUM#0-CHIP#0 Status:Normal; Ver:0201h; Serial:PP074003RV ; + FRU-Part-Number:CA06761-D105 A3 /375-3477-01 ; + Freq:2.150 GHz; Type:16; + Core:2; Strand:2; CPUM#0-CHIP#1 Status:Normal; Ver:0201h; Serial:PP074003RV ; + FRU-Part-Number:CA06761-D105 A3 /375-3477-01 ; + Freq:2.150 GHz; Type:16; + Core:2; Strand:2; * CPUM#2-CHIP#0 Status:Deconfigured; Ver:0201h; Serial:PP0740030W ; + FRU-Part-Number:CA06761-D105 A3 /375-3477-01 ; + Freq:2.150 GHz; Type:16; + Core:2; Strand:2; * CPUM#2-CHIP#1 Status:Deconfigured; Ver:0201h; Serial:PP0740030W ; + FRU-Part-Number:CA06761-D105 A3 /375-3477-01 ; + Freq:2.150 GHz; Type:16; + Core:2; Strand:2; MEMB#0 Status:Normal; Ver:0101h; Serial:BF080998WR ; + FRU-Part-Number:CF00541-0545 05 /541-0545-05 ; MEM#0A Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-dc19390c; + Type:1A; Size:1 GB; MEM#0B Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-dc19390a; + Type:1A; Size:1 GB; MEM#1A Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-dc19390b; + Type:1A; Size:1 GB; MEM#1B Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-dc19390d; + Type:1A; Size:1 GB; MEM#2A Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-dc193906; + Type:1A; Size:1 GB; MEM#2B Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-dc193907; + Type:1A; Size:1 GB; MEM#3A Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-dc193908; + Type:1A; Size:1 GB; MEM#3B Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-dc193909; + Type:1A; Size:1 GB; MEMB#1 Status:Normal; Ver:0101h; Serial:BF0813AH3C ; + FRU-Part-Number:CF00541-0545 05 /541-0545-05 ; MEM#0A Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-e709709b; + Type:1A; Size:1 GB; MEM#0B Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-d421b34e; + Type:1A; Size:1 GB; MEM#1A Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-d421b34f; + Type:1A; Size:1 GB; MEM#1B Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-d421b350; + Type:1A; Size:1 GB; MEM#2A Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-e70970a4; + Type:1A; Size:1 GB; MEM#2B Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-e70970a5; + Type:1A; Size:1 GB; MEM#3A Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-e70970a6; + Type:1A; Size:1 GB; MEM#3B Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-e70970a7; + Type:1A; Size:1 GB; DDC_A#0 Status:Normal; DDC_A#1 Status:Normal; DDC_A#2 Status:Normal; DDC_A#3 Status:Normal; DDC_B#0 Status:Normal; DDC_B#1 Status:Normal; IOU#0 Status:Normal; Ver:0101h; Serial:BF0815B2EK ; + FRU-Part-Number:CF00541-2240 03 /541-2240-03 ; DDC_A#0 Status:Normal; DDCR Status:Normal; DDC_B#0 Status:Normal; PCI#1 Name_Property:network; Card_Type:Other; PCI#2 Name_Property:SUNW,qlc; Card_Type:Other; IOU#1 Status:Normal; Ver:0101h; Serial:BF0816BFL3 ; + FRU-Part-Number:CF00541-2240 03 /541-2240-03 ; DDC_A#0 Status:Normal; DDCR Status:Normal; DDC_B#0 Status:Normal; XSCFU Status:Normal,Active; Ver:0101h; Serial:BF08109GQ1 ; + FRU-Part-Number:CF00541-0481 04 /541-0481-04 ; OPNL Status:Normal; Ver:0101h; Serial:BF0813AFKG ; + FRU-Part-Number:CF00541-0850 06 /541-0850-06 ; PSU#0 Status:Normal; Serial:0017527-0748016175; + FRU-Part-Number:CF00300-2011 0150 /300-2011-01-50; + Power_Status:On; AC:200 V; PSU#1 Status:Normal; Serial:0017527-0748016184; + FRU-Part-Number:CF00300-2011 0150 /300-2011-01-50; + Power_Status:On; AC:200 V; PSU#2 Status:Normal; Serial:0017527-0803010075; + FRU-Part-Number:CF00300-2011 0150 /300-2011-01-50; + Power_Status:On; AC:200 V; PSU#3 Status:Normal; Serial:0017527-0803010108; + FRU-Part-Number:CF00300-2011 0150 /300-2011-01-50; + Power_Status:On; AC:200 V; FANBP_C Status:Normal; Ver:0301h; Serial:BF0812A7E2 ; + FRU-Part-Number:CF00541-0848 05 /541-0848-05 ; FAN_A#0 Status:Normal; FAN_A#1 Status:Normal; FAN_A#2 Status:Normal; FAN_A#3 Status:Normal; Notice "IOU#1" Now shows network and qlc. After the board was moved: XSCF> showhardconf SPARC Enterprise M5000; + Serial:BEF0825530; Operator_Panel_Switch:Service; + Power_Supply_System:Single; SCF-ID:XSCF#0; + System_Power:On; System_Phase:Cabinet Power On; Domain#0 Domain_Status:Running; MBU_B Status:Normal; Ver:0201h; Serial:BE0812107C ; + FRU-Part-Number:CF00541-0478 05 /541-0478-05 ; + Memory_Size:16 GB; CPUM#0-CHIP#0 Status:Normal; Ver:0201h; Serial:PP0816015K ; + FRU-Part-Number:CA06761-D105 A4 /375-3477-01 ; + Freq:2.150 GHz; Type:16; + Core:2; Strand:2; CPUM#0-CHIP#1 Status:Normal; Ver:0201h; Serial:PP0816015K ; + FRU-Part-Number:CA06761-D105 A4 /375-3477-01 ; + Freq:2.150 GHz; Type:16; + Core:2; Strand:2; CPUM#2-CHIP#0 Status:Normal; Ver:0201h; Serial:PP08160159 ; + FRU-Part-Number:CA06761-D105 A4 /375-3477-01 ; + Freq:2.150 GHz; Type:16; + Core:2; Strand:2; CPUM#2-CHIP#1 Status:Normal; Ver:0201h; Serial:PP08160159 ; + FRU-Part-Number:CA06761-D105 A4 /375-3477-01 ; + Freq:2.150 GHz; Type:16; + Core:2; Strand:2; MEMB#0 Status:Normal; Ver:0101h; Serial:BF0814AQLP ; + FRU-Part-Number:CF00541-0545 05 /541-0545-05 ; MEM#0A Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-da189d92; + Type:1A; Size:1 GB; MEM#0B Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-da189d8a; + Type:1A; Size:1 GB; MEM#1A Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-da189d93; + Type:1A; Size:1 GB; MEM#1B Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-da189d94; + Type:1A; Size:1 GB; MEM#2A Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-da189d8e; + Type:1A; Size:1 GB; MEM#2B Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-da189d90; + Type:1A; Size:1 GB; MEM#3A Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-da189d8b; + Type:1A; Size:1 GB; MEM#3B Status:Normal; + Code:2c000000000000000818HTF12872PY-667D20200-da189d91; + Type:1A; Size:1 GB; MEMB#4 Status:Normal; Ver:0101h; Serial:BF08078HL7 ; + FRU-Part-Number:CF00541-0545 05 /541-0545-05 ; MEM#0A Status:Normal; + Code:2c000000000000000818HTF12872Y-53EB3 0300-d41e17d2; + Type:1A; Size:1 GB; MEM#0B Status:Normal; + Code:2c000000000000000818HTF12872Y-53EB3 0300-d41e1793; + Type:1A; Size:1 GB; MEM#1A Status:Normal; + Code:2c000000000000000818HTF12872Y-53EB3 0300-e00dfa51; + Type:1A; Size:1 GB; MEM#1B Status:Normal; + Code:2c000000000000000818HTF12872Y-53EB3 0300-d41e1736; + Type:1A; Size:1 GB; MEM#2A Status:Normal; + Code:2c000000000000000818HTF12872Y-53EB3 0300-e00dfa52; + Type:1A; Size:1 GB; MEM#2B Status:Normal; + Code:2c000000000000000818HTF12872Y-53EB3 0300-e00dfa54; + Type:1A; Size:1 GB; MEM#3A Status:Normal; + Code:2c000000000000000818HTF12872Y-53EB3 0300-e00dfa53; + Type:1A; Size:1 GB; MEM#3B Status:Normal; + Code:2c000000000000000818HTF12872Y-53EB3 0300-e00dfa56; + Type:1A; Size:1 GB; DDC_A#0 Status:Normal; DDC_A#1 Status:Normal; DDC_A#2 Status:Normal; DDC_A#3 Status:Normal; DDC_B#0 Status:Normal; DDC_B#1 Status:Normal; IOU#0 Status:Normal; Ver:0101h; Serial:BF0742530G ; + FRU-Part-Number:CF00541-2240 02 /541-2240-02 ; DDC_A#0 Status:Normal; DDCR Status:Normal; DDC_B#0 Status:Normal; PCI#1 Name_Property:network; Card_Type:Other; PCI#2 Name_Property:SUNW,qlc; Card_Type:Other; IOU#1 Status:Normal; Ver:0101h; Serial:BF0820CQD5 ; + FRU-Part-Number:CF00541-2240 03 /541-2240-03 ; DDC_A#0 Status:Normal; DDCR Status:Normal; DDC_B#0 Status:Normal; PCI#1 Name_Property:network; Card_Type:Other; PCI#2 Name_Property:SUNW,qlc; Card_Type:Other; XSCFU Status:Normal,Active; Ver:0101h; Serial:BF07455P4J ; + FRU-Part-Number:CF00541-0481 04 /541-0481-04 ; OPNL Status:Normal; Ver:0101h; Serial:BF073943C0 ; + FRU-Part-Number:CF00541-0850 06 /541-0850-06 ; PSU#0 Status:Normal; Serial:0017527-0811015311; + FRU-Part-Number:CF00300-2011 0150 /300-2011-01-50; + Power_Status:On; AC:200 V; PSU#1 Status:Normal; Serial:0017527-0811015504; + FRU-Part-Number:CF00300-2011 0150 /300-2011-01-50; + Power_Status:On; AC:200 V; PSU#2 Status:Normal; Serial:0017527-0811015503; + FRU-Part-Number:CF00300-2011 0150 /300-2011-01-50; + Power_Status:On; AC:200 V; PSU#3 Status:Normal; Serial:0017527-0811015511; + FRU-Part-Number:CF00300-2011 0150 /300-2011-01-50; + Power_Status:On; AC:200 V; FANBP_C Status:Normal; Ver:0301h; Serial:BF08088QAR ; + FRU-Part-Number:CF00541-0848 04 /541-0848-04 ; FAN_A#0 Status:Normal; FAN_A#1 Status:Normal; FAN_A#2 Status:Normal; FAN_A#3 Status:Normal;
Problem 4
Problem: SSH X-forwarding dose not work work
- Problem: SSH Xforwarding won’t work (you could see this with env|grep -i display)
Solution: You will have to create a hostname6.lo0
Solution: You will have to create a hostname6.lo0 and reboot to get ssh Xforwarding to work
touch hostname6.lo0 reboot After reboot the fowling error will come and may be ignored. t_optmgmt: System error: Cannot assign requested address
Problem 5
Problem: Sun cluster reports online faulted for rac proxy startup
- Problem: The Sun Cluster reports rac_server_proxy-rg “Online faulted” Becuse it takes to long to start
Solution: Extend the cluster start time to 900 seconds
Solution: You will have to Extend the cluster start time to 900 seconds
/usr/cluster/bin/clresource set -p START_TIMEOUT=900 rac_server_proxy-rs
Problem 6
Problem: Oracle Rac hangs / waits
- Problem: The Oracle Data Base will wait for the private network to recover if failed
Solution: The Oracle clusterwere should only use the Sun cluster virtual private interface
Solution: You will have to remove the physical interfaces from the Oralce CRS database
/oracle/product/11.1.0/crs_1/bin/oifcfg getif nxge0 10.10.20.0 global public nxge3 10.11.5.8 global cluster_interconnect <- to be removed nxge4 10.10.20.0 global public nxge7 10.11.5.16 global cluster_interconnect <- to be removed clprivnet0 10.11.5.128 global cluster_interconnect /oracle/product/11.1.0/crs_1/bin/oifcfg delif -global nxge3 /oracle/product/11.1.0/crs_1/bin/oifcfg delif -global nxge7 /oracle/product/11.1.0/crs_1/bin/oifcfg getif nxge0 10.10.20.0 global public nxge4 10.10.20.0 global public clprivnet0 10.10.5.128 global cluster_interconnect
Rman And NetBackup San Backup
NetBackup Rman SAN Client Installation and configuration
References and help
List of references on how to setup Sun Cluster and Oracle RAC configuration