环境如下：
Linux操作系统：Centos 6.5 64bit （这个版本的redhat 6内核等OS在安装grid最后执行root.sh时会出现crs-4124，是oracle11.2.0.1的bug）
VMware version：Workstation 8.0.3 build-703057
Oracle版本：11G R2 grid/oracle 64位 11.2.0.1.0 for linux
步骤规划：
1，配置虚拟机、安装Linux操作系统
2，配置OS
3，配置ASM
4，安装Oracle grid
5，安装Oracle database
6，创建ASM磁盘组
7，建立数据库实例
1，配置虚拟机、安装Linux操作系统
1.1 新建虚拟机：RAC1和RAC2，并安装OS
下载Centos的iso后新建虚拟机后安装即可，这里略。注意内存分配，每个虚拟机至少1.5G内存比较好，我的机器8G内存，给每个虚拟机分了2G，给每个虚拟机分配50GB硬盘容量，配置2个桥接网卡。
1.2 配置Share Disks
新建共享硬盘，在DOS VM安装目录下执行：
cd \D F:/Software/vm8/
vmware-vdiskmanager.exe -c -s 1000Mb -a lsilogic -t 2 K:\VMDisk\Sharedisk\ocr.vmdk
vmware-vdiskmanager.exe -c -s 1000Mb -a lsilogic -t 2 K:\VMDisk\Sharedisk\ocr2.vmdk
vmware-vdiskmanager.exe -c -s 1000Mb -a lsilogic -t 2 K:\VMDisk\Sharedisk\votingdisk.vmdk
vmware-vdiskmanager.exe -c -s 20000Mb -a lsilogic -t 2 K:\VMDisk\Sharedisk\data.vmdk
vmware-vdiskmanager.exe -c -s 10000Mb -a lsilogic -t 2 K:\VMDisk\Sharedisk\backup.vmdk
配置虚拟机RAC1 RAC2目录下的vmx虚拟机配置文件，在行最后添加如下：
scsi1.present = "TRUE"
scsi1.virtualDev = "lsilogic"
scsi1.sharedBus = "virtual"
scsi1:1.present = "TRUE"
scsi1:1.mode = "independent-persistent"
scsi1:1.filename = "K:\VMDisk\Sharedisk\ocr.vmdk"
scsi1:1.deviceType = "plainDisk"
scsi1:2.present = "TRUE"
scsi1:2.mode = "independent-persistent"
scsi1:2.filename = "K:\VMDisk\Sharedisk\votingdisk.vmdk"
scsi1:2.deviceType = "plainDisk"
scsi1:3.present = "TRUE"
scsi1:3.mode = "independent-persistent"
scsi1:3.filename = "K:\VMDisk\Sharedisk\data.vmdk"
scsi1:3.deviceType = "plainDisk"
scsi1:4.present = "TRUE"
scsi1:4.mode = "independent-persistent"
scsi1:4.filename = "K:\VMDisk\Sharedisk\backup.vmdk"
scsi1:4.deviceType = "plainDisk"
scsi1:5.present = "TRUE"
scsi1:5.mode = "independent-persistent"
scsi1:5.filename = "K:\VMDisk\Sharedisk\ocr2.vmdk"
scsi1:5.deviceType = "plainDisk"
disk.locking = "false"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0"
diskLib.DataCacheMinReadAheadSize = "0"
diskLib.dataCachePageSize = "4096"
diskLib.maxUnsyncedWrites = "0"
保存退出。
最后虚拟机图大致如下：
再次提醒注意内存、硬盘和网卡配置。
配置虚拟机、安装Linux操作系统完成。
2，配置OS
以下没有特别说明都需要同时在两个节点执行。
2.1 关闭防火墙、selinux(考虑到后面安装时会碰到一些情况，所以需要执行这步骤）
关闭selinuxsetenforce 0
[root@rac1 ~]# vim /etc/selinux/config
SELINUX=disabled
关闭防火墙
service iptables stop
chkconfig iptables off
2.2 配置网络
修改节点主机名为rac1，rac2
hostname rac1
hostname rac2
或修改/etc/sysconfig/network
RAC1
eth0: 192.168.1.130
eth1: 10.10.17.221
RAC2
eth0: 192.168.1.140
eth1: 10.10.17.222
host文件，两个节点都配置Vim /etc/hosts
192.168.1.130 rac1
192.168.1.140 rac2
192.168.1.131 rac1-vip
192.168.1.141 rac2-vip
10.10.17.221 rac1-priv
10.10.17.222 rac2-priv
192.168.1.200 scan-ip
保存退出。
配置完之后互相ping看看是否连通。
2.3 配置CTSS服务、节点SSH等效
CTSS
service ntpd stop
chkconfig ntpd off
rm -rf /etc/ntpd.conf
rm -rf /var/run/ntpd.pid
查看ctss服务资源
crsctl check ctss
SSH
在grid安装时会提供配置步骤，所以不需要手工配置。
ssh具体配置
rac1 rac2
mkdir ~/.ssh
chmod 700 ~/.ssh
ssh-keygen -t rsa
ssh-keygen -t dsa
rac1 rac2
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
测试SSH
ssh rac2
ssh rac1
2.4 配置Oracle软件安装用户、组、环境变量、安装目录
#组
groupadd -g 1300 dba
groupadd -g 1301 oper
groupadd -g 1000 oinstall
groupadd -g 1200 asmadmin
groupadd -g 1201 asmdba
groupadd -g 1202 asmoper
#用户
useradd -m -u 1100 -g oinstall -G asmadmin,asmdba,asmoper -s /bin/bash grid
useradd -m -u 1101 -g oinstall -G dba,oper,asmdba -s /bin/bash oracle
#用户环境变量修改home下的 .bash_profile
ORACLE
export TMP=/tmp
export TMPDIR=$TMP
export ORACLE_SID=RAC1 # RAC1
export ORACLE_SID=RAC2 # RAC2
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1
export TNS_ADMIN=$ORACLE_HOME/network/admin
export PATH=/usr/sbin:$PATH
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
umask 022
GRID
export TMP=/tmp
export TMPDIR=$TMP
export ORACLE_SID=+ASM1 # RAC1
export ORACLE_SID=+ASM2 # RAC2
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/11.2.0/grid
export PATH=/usr/sbin:$PATH
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
umask 022
配置目录
mkdir -p /u01/app/11.2.0/grid
mkdir -p /u01/app/grid
mkdir -p /u01/app/oracle
chown grid:oinstall /u01/app/11.2.0/grid
chown grid:oinstall /u01/app/grid
chown oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/
chown -R grid:oinstall /u01
2.5 配置系统文件
配置oracle、grid用户的shell限制
vim /etc/security/limits.conf
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
修改/etc/pam.d/login配置文件
Vim /etc/pam.d/login
添加 session required pam_limits.so
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 1054472192
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586
net.ipv4.tcp_wmem = 262144 262144 262144
net.ipv4.tcp_rmem = 4194304 4194304 4194304
保存退出后执行 sysctl -p 立即生效。
2.6 配置共享磁盘 （在一个节点执行即可）
查看分区
ls /dev/sd*
使用fdisk进行分区，其他区按照自己的规划划分就可以了，例：
fdisk /dev/sdb
[root@rac1 ~]# fdisk /dev/sdb
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1000,default 1):
Using default value 1
Last cylinder,+cylinders or +size{K,M,G} (1-1000,default 1000): +500M
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (502-1000,default 502):
Using default value 502
Last cylinder,G} (502-1000,default 1000):
Using default value 1000
Command (m for help): w
The partition table has been altered!
最后配置完后查看分区
fidks -l
[root@rac1 ~]# ls /dev/sd*
/dev/sda /dev/sda1 /dev/sda2 /dev/sdb /dev/sdb1 /dev/sdc /dev/sdc1 /dev/sdd /dev/sdd1 /dev/sdd2 /dev/sde /dev/sde1 /dev/sde2 /dev/sdf /dev/sdf1
2.7 安装Oracle grid等需要的包
binutils-2.20.51.0.2-5.11.el6 (x86_64)compat-libcap1-1.10-1 (x86_64)
compat-libstdc++-33-3.2.3-69.el6 (x86_64)
compat-libstdc++-33-3.2.3-69.el6.i686
gcc-4.4.4-13.el6 (x86_64)
gcc-c++-4.4.4-13.el6 (x86_64)
glibc-2.12-1.7.el6 (i686)
glibc-2.12-1.7.el6 (x86_64)
glibc-devel-2.12-1.7.el6 (x86_64)
glibc-devel-2.12-1.7.el6.i686
ksh
libgcc-4.4.4-13.el6 (i686)
libgcc-4.4.4-13.el6 (x86_64)
libstdc++-4.4.4-13.el6 (x86_64)
libstdc++-4.4.4-13.el6.i686
libstdc++-devel-4.4.4-13.el6 (x86_64)
libstdc++-devel-4.4.4-13.el6.i686
libaio-0.3.107-10.el6 (x86_64)
libaio-0.3.107-10.el6.i686
libaio-devel-0.3.107-10.el6 (x86_64)
libaio-devel-0.3.107-10.el6.i686
make-3.81-19.el6
sysstat-9.0.4-11.el6 (x86_64)
3，配置ASM
3.1 查看操作系统内核版本对应下载
uname -a
下载oracleasm http://www.oracle.com/technetwork/topics/linux/asmlib/index-101839.html
安装
yum -y install kmod-oracleasm.x86_64
rpm -ivh oracleasmlib-2.0.4-1.el6.x86_64.rpm
rpm -ivh oracleasm-support-2.1.8-1.el6.x86_64.rpm
3.2 配置ASM磁盘
在两个节点上执行
[root@rac1 oracle]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: grid
Default group to own the driver interface []: asmdba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@rac1 oracle]# /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@rac1 ~]# /etc/init.d/oracleasm listdisks
VOL1
VOL2
VOL3
VOL4
VOL5
VOL6
VOL7
注：如果没有查看到asm磁盘组需要查看之前的selinux是否有禁用。
4，安装Oracle grid
安装前可以执行Oracle提供的检查环境工具
./runcluvfy.sh stage -pre crsinst -n rac1,rac2 -fixup -verbose
4.1 在安装目录下执行./runinstaller，如果出现安装grid提示must be configured to display at least 256 colors Failed，就执行如下
root 下先执行#xhost +
su - grid
export DISPLAY=:0.0
注意在此建立SSH等效。执行setup后点test测试。如果不成功就手工配置SSH再进行测试。
这个地方忘记截图了，就是类似这个图。定义磁盘组名字，这里我选择normal冗余，需要选择3个VOL。
这里是把OCR和votingdisk安装在所选磁盘。后面还需要新建asm磁盘组来安装其他数据文件。
这里的software location应该是 /u01/app/11.2.0/grid
通过Oracle的系统环境检查，安装缺失的包和修改系统环境参数等。
安装65%的时候可能会出现卡顿的情况，这个时候需要检查防火墙和selinux是否关闭，详细情况可以点击detail查看安装进度，如出现拷贝文件到RAC2节点的信息则正常。
根据提示执行orainstRoot.sh和root.sh脚本，先执行rac1后再执行rac2。
在执行root .sh时可能在的时候会提示出现crs-4124，这个是oracle的bug，当在redhat 6等OS下安装11.2.0.1版本Oracle的时候会出现此错误。
解决如下：
在执行root.sh时，在出现Adding daemon to inittab时，需要不停地执行命令：
/bin/dd if=/var/tmp/.oracle/npohasd of=/dev/null bs=1024 count=1
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2015-01-06 23:46:35: Parsing the host name
2015-01-06 23:46:35: Checking for super user privileges
2015-01-06 23:46:35: User has super user privileges
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root',privgrp 'root'..
Operation successful.
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
ADVM/ACFS is not supported on centos-release-6-6.el6.centos.12.2.x86_64
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
ASM created and started successfully.
DiskGroup DATA created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root',privgrp 'root'..
Operation successful.
CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 60b5c5168a574f3fbfabf7d93a17c18f.
Successful addition of voting disk a59fab87e8634f00bf5b44656d6caaf1.
Successful addition of voting disk 228f0986e7c84f20bf66ed6905b69e43.
Successfully replaced voting disk group with +DATA.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 60b5c5168a574f3fbfabf7d93a17c18f (ORCL:VOL1) [DATA]
2. ONLINE a59fab87e8634f00bf5b44656d6caaf1 (ORCL:VOL2) [DATA]
3. ONLINE 228f0986e7c84f20bf66ed6905b69e43 (ORCL:VOL4) [DATA]
Located 3 voting disk(s).
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rac1'
CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'rac1'
CRS-2676: Start of 'ora.DATA.dg' on 'rac1' succeeded
rac1 2015/01/06 23:52:15 /u01/app/11.2.0/grid/cdata/rac1/backup_20150106_235215.olr
Preparing packages for installation...
cvuqdisk-1.0.7-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 3906 MB Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.
root.sh执行完毕。在rac2节点执行也需要此步骤。
检查grid组件启动情况
[grid@rac1 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[grid@rac1 ~]$ crs_stat -v -t
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora.DATA.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE rac1
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE rac1
ora.eons ora.eons.type 0/3 0/ ONLINE ONLINE rac1
ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE rac1
ora.oc4j ora.oc4j.type 0/5 0/0 OFFLINE OFFLINE
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE rac1
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE rac1
ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac1
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac2
ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2
ora.rac2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac2
ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE rac1
备注：
在重启节点后出现Oracle has不能通信的情况
1，[root@rac1 ~]# crsctl check crs
CRS-4639: Could not contact Oracle High Availability Services
需要在启动时执行 /bin/dd if=/var/tmp/.oracle/npohasd of=/dev/null bs=1024 count=1。
然后再执行 crsctl check crs查看服务启动情况。
2，如果出现以下情况，需要查看日志提示，有一次是因为内核参数配置值的问题导致了这个错误。
[root@rac1 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
5，安装Oracle database软件
需要用oracle用户安装
su - oracle
./runInstaller
仅安装数据库软件即可，这里就根据OUI的提示安装就可以了，最后按照提示执行相关脚本。
找时间补上截图
6，创建ASM磁盘组
需要新建其他磁盘组来安装数据文件等
su - grid
asmca
根据界面提示创建磁盘组。
找时间补上截图
7，建立数据库实例
su - oracle./runInstaller
