Oracle 12cR1 RAC集群安装（一）--环境准备

（2）修改IP地址
每台服务器的公网IP与虚拟IP均需要修改，这里以节点1的公网IP修改为例，公网使用的网卡是eth0,使用下面方法修改eth0的配置

#进入网卡配置目录
cd /etc/sysconfig/network-scripts/

#修改eth0的网卡配置
vim ifcfg-eth0

DEVICE=eth0
HWADDR=00:0c:29:f8:80:bb
TYPE=Ethernet
ONBOOT=yes
IPADDR=192.168.10.11
NETMASK=255.255.255.0

其他网卡类似，修改完成后重启网卡

[root@node1 ~]# service network restart

最终2台服务器的网卡配置信息如下图

node1:

node2：

（3）修改/etc/hosts文件，2个节点都做相同的修改

[root@node1 ~]# vim /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.10.11   node1
192.168.10.12   node2
192.168.10.13   node1-vip
192.168.10.14   node2-vip
192.168.10.10   node-scan

10.10.10.11     node1-priv
10.10.10.12     node2-priv

（3）关闭防火墙

#临时关闭，重启主机后恢复原来的状态
service iptables stop

#永久关闭，重启生效
chkconfig iptables off

（4）关闭seLinux

将参数SELINUX=enforcing改为SELINUX=disabled

[root@node1 ~]# vim /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

重启服务器生效。

（四）创建用户和用户组，创建软件安装目录，配置用户环境变量

（1）创建用户oracle和grid，以及相关的用户组

/usr/sbin/groupadd -g 1000 oinstall
/usr/sbin/groupadd -g 1020 asmadmin
/usr/sbin/groupadd -g 1021 asmdba
/usr/sbin/groupadd -g 1022 asmoper
/usr/sbin/groupadd -g 1031 dba
/usr/sbin/groupadd -g 1032 oper
useradd -u 1100 -g oinstall -G asmadmin,asmdba,asmoper,oper,dba grid
useradd -u 1101 -g oinstall -G dba,asmdba,oper oracle

（2）创建GI和Oracle软件的安装目录，并授权

mkdir -p /u01/app/12.1.0/grid
mkdir -p /u01/app/grid
mkdir /u01/app/oracle
chown -R grid:oinstall /u01
chown oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/

软件安装目录结构如下：

（3）配置grid的环境变量

[grid@node1 ~]$ vi .bash_profile

# 在文件结尾添加如下内容

export TMP=/tmp
export TMPDIR=$TMP 
export ORACLE_SID=+ASM1   
export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/12.1.0/grid
export PATH=/usr/sbin:$PATH 
export PATH=$ORACLE_HOME/bin:$PATH 
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib 
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib 
umask 022

执行命令“source .bash_profile”使环境变量生效。

注意：如果是节点2，黄色参数需要改成ORACLE_SID=+ASM2。

（4）配置oracle的环境变量

[oracle@node1 ~]$ vim .bash_profile
#在文件结尾添加一下内容

export TMP=/tmp 
export TMPDIR=$TMP 
export ORACLE_SID=testdb1     
export ORACLE_BASE=/u01/app/oracle 
export ORACLE_HOME=$ORACLE_BASE/product/12.1.0/db_1
export TNS_ADMIN=$ORACLE_HOME/network/admin 
export PATH=/usr/sbin:$PATH 
export PATH=$ORACLE_HOME/bin:$PATH 
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib 
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib 
umask 022

执行命令“source .bash_profile”使环境变量生效。

注意：如果是节点2，黄色参数需要改成ORACLE_SID=testdb2。

（五）配置内核参数和资源限制

（1）配置操作系统的内核参数

在/etc/sysctl.conf文件结尾添加参数

kernel.msgmnb = 65536 
kernel.msgmax = 65536 
kernel.shmmax = 68719476736 
kernel.shmall = 4294967296 
fs.aio-max-nr = 1048576 
fs.file-max = 6815744 
kernel.shmall = 2097152 
kernel.shmmax = 2002012160
kernel.shmmni = 4096 
kernel.sem = 250 32000 100 128 
net.ipv4.ip_local_port_range = 9000 65500 
net.core.rmem_default = 262144 
net.core.rmem_max = 4194304 
net.core.wmem_default = 262144 
net.core.wmem_max = 1048586 
net.ipv4.tcp_wmem = 262144 262144 262144 
net.ipv4.tcp_rmem = 4194304 4194304 4194304
kernel.panic_on_oops = 1

sysctl -p生效。

（2）配置oracle和grid用户的资源限制

在/etc/security/limits.conf结尾添加参数

grid soft nproc 2047 
grid hard nproc 16384 
grid soft nofile 1024 
grid hard nofile 65536 
oracle soft nproc 2047 
oracle hard nproc 16384 
oracle soft nofile 1024 
oracle hard nofile 65536

（3）配置/etc/pam.d/login文件，在结尾添加参数

session    required     pam_limits.so

（六）软件包安装

使用yum工具安装缺失的软件包，软件包信息如下

binutils-2.20.51.0.2-5.11.el6 (x86_64)
compat-libcap1-1.10-1 (x86_64)
compat-libstdc++-33-3.2.3-69.el6 (x86_64)
compat-libstdc++-33-3.2.3-69.el6 (i686)
gcc-4.4.4-13.el6 (x86_64)
gcc-c++-4.4.4-13.el6 (x86_64)
glibc-2.12-1.7.el6 (i686)
glibc-2.12-1.7.el6 (x86_64)
glibc-devel-2.12-1.7.el6 (x86_64)
glibc-devel-2.12-1.7.el6 (i686)
ksh
libgcc-4.4.4-13.el6 (i686)
libgcc-4.4.4-13.el6 (x86_64)
libstdc++-4.4.4-13.el6 (x86_64)
libstdc++-4.4.4-13.el6 (i686)
libstdc++-devel-4.4.4-13.el6 (x86_64)
libstdc++-devel-4.4.4-13.el6 (i686)
libaio-0.3.107-10.el6 (x86_64)
libaio-0.3.107-10.el6 (i686)
libaio-devel-0.3.107-10.el6 (x86_64)
libaio-devel-0.3.107-10.el6 (i686)
libXext-1.1 (x86_64)
libXext-1.1 (i686)
libXtst-1.0.99.2 (x86_64)
libXtst-1.0.99.2 (i686)
libX11-1.3 (x86_64)
libX11-1.3 (i686)
libXau-1.0.5 (x86_64)
libXau-1.0.5 (i686)
libxcb-1.5 (x86_64)
libxcb-1.5 (i686)
libXi-1.3 (x86_64)
libXi-1.3 (i686)
make-3.81-19.el6
sysstat-9.0.4-11.el6 (x86_64)

其中x86_64代表64位操作系统，i686代表32位操作系统，只需安装对应版本即可。

使用下面命令安装

yum install -y binutils*
yum install -y compat-libcap1*
yum install -y compat-libstdc++*
yum install -y gcc*
yum install -y gcc-c++*
yum install -y glibc*
yum install -y glibc-devel*
yum install -y ksh
yum install -y libgcc*
yum install -y libstdc++*
yum install -y libstdc++-devel*
yum install -y libaio*
yum install -y libaio-devel*
yum install -y libXext* 
yum install -y libXtst*
yum install -y libX11*  
yum install -y libXau*  
yum install -y libxcb*  
yum install -y libXi*  
yum install -y make*
yum install -y sysstat*

（七）配置共享磁盘

oracle对于存放OCR磁盘组的大小要求如下

在这次安装中，磁盘规划如下：

（1）配置共享磁盘的方法

使用udev配置磁盘有2种方法，第一种是直接fdisk格式化磁盘，拿到/dev/sd*1的磁盘，然后使用udev绑定raw，第二种是获取wwid来绑定设备，生产中太长使用第二种方法。

（2）方法1：直接使用raw

(2.1)格式化磁盘，在1个节点上执行

# 在节点1上格式化，以/dev/sdb为例：

[root@node1 ~]# fdisk /dev/sdb

The number of cylinders for this disk is set to 3824.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-3824, default 1): 
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-3824, default 3824): 
Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

（2.2）在2个节点上配置raw设备

[root@node1 ~]# vi /etc/udev/rules.d/60-raw.rules
 # 在后面添加

ACTION=="add", KERNEL=="sdb1", RUN+="/bin/raw /dev/raw/raw1 %N"
ACTION=="add", KERNEL=="sdc1", RUN+="/bin/raw /dev/raw/raw2 %N"
ACTION=="add", KERNEL=="sdd1", RUN+="/bin/raw /dev/raw/raw3 %N"
ACTION=="add", KERNEL=="sde1", RUN+="/bin/raw /dev/raw/raw4 %N"
ACTION=="add", KERNEL=="sdf1", RUN+="/bin/raw /dev/raw/raw5 %N"
ACTION=="add", KERNEL=="sdg1", RUN+="/bin/raw /dev/raw/raw6 %N"


KERNEL=="raw[1]", MODE="0660", OWNER="grid", GROUP="asmadmin"
KERNEL=="raw[2]", MODE="0660", OWNER="grid", GROUP="asmadmin"
KERNEL=="raw[3]", MODE="0660", OWNER="grid", GROUP="asmadmin"
KERNEL=="raw[4]", MODE="0660", OWNER="grid", GROUP="asmadmin"
KERNEL=="raw[5]", MODE="0660", OWNER="grid", GROUP="asmadmin"
KERNEL=="raw[6]", MODE="0660", OWNER="grid", GROUP="asmadmin"

(2.3)启动裸设备，2个节点都要执行

[root@node1 ~]# start_udev

(2.4)查看裸设备，2个节点都要查看，如果有节点不能看到下面的raw设备信息，重启节点

[root@node1 ~]# raw -qa
/dev/raw/raw1:    bound to major 8, minor 17
/dev/raw/raw2:    bound to major 8, minor 33
/dev/raw/raw3:    bound to major 8, minor 49
/dev/raw/raw4:    bound to major 8, minor 65
/dev/raw/raw5:    bound to major 8, minor 81
/dev/raw/raw6:    bound to major 8, minor 97

（3）方法2：使用wwid来绑定设备

（3.1）编辑/etc/scsi_id.config文件，2个节点都要编辑

[root@node1 ~]# echo "options=--whitelisted --replace-whitespace"  >> /etc/scsi_id.config

（3.2）将磁盘wwid信息写入99-oracle-asmdevices.rules文件，2个节点都要编辑

[root@node1 ~]# for i in b c d e f g ;
> do 
> echo "KERNEL==\"sd*\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/\$name\", RESULT==\"`/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/sd$i`\", NAME=\"asm-disk$i\", OWNER=\"grid\", GROUP=\"asmadmin\", MODE=\"0660\""      >> /etc/udev/rules.d/99-oracle-asmdevices.rules
> done

（3.3）查看99-oracle-asmdevices.rules文件，2个节点都要查看

[root@node1 ~]# cd /etc/udev/rules.d/
[root@node1 rules.d]# more 99-oracle-asmdevices.rules 
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=
="36000c293f718a0dcf1f7b410fb9fd1d9", NAME="asm-diskb", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=
="36000c296f46877bf6cff9febd7700fb9", NAME="asm-diskc", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=
="36000c2902c030ca8a0b0a4a32ab547c7", NAME="asm-diskd", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=
="36000c2982ad4757618bd0d06d54d04b8", NAME="asm-diske", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=
="36000c29872b79e70266a992e788836b6", NAME="asm-diskf", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=
="36000c29b1260d00b8faeb3786092143a", NAME="asm-diskg", OWNER="grid", GROUP="asmadmin", MODE="0660"

（3.4）启动设备，2个节点都要执行

[root@node1 rules.d]# start_udev
Starting udev:                                             [  OK  ]

（3.5）确认磁盘已经添加成功

[root@node1 rules.d]# cd /dev
[root@node1 dev]# ls -l asm*
brw-rw---- 1 grid asmadmin 8, 16 Aug 13 05:31 asm-diskb
brw-rw---- 1 grid asmadmin 8, 32 Aug 13 05:31 asm-diskc
brw-rw---- 1 grid asmadmin 8, 48 Aug 13 05:31 asm-diskd
brw-rw---- 1 grid asmadmin 8, 64 Aug 13 05:31 asm-diske
brw-rw---- 1 grid asmadmin 8, 80 Aug 13 05:31 asm-diskf
brw-rw---- 1 grid asmadmin 8, 96 Aug 13 05:31 asm-diskg

（八）用户等效性配置

oracle在安装包已经提供了grid和oracle ssh节点互信配置的工具，直接使用配置非常方便

（1）配置grid用户等效性

（1.1）解压grid安装包，在节点1执行

[grid@node1 ~]$ ls
linuxamd64_12102_grid_1of2.zip  linuxamd64_12102_grid_2of2.zip
[grid@node1 ~]$ unzip -q linuxamd64_12102_grid_1of2.zip 
[grid@node1 ~]$ unzip -q linuxamd64_12102_grid_2of2.zip 
[grid@node1 ~]$ ls
grid  linuxamd64_12102_grid_1of2.zip  linuxamd64_12102_grid_2of2.zip

（1.2）配置节点互信

[grid@node1 sshsetup]$ pwd
/home/grid/grid/sshsetup
[grid@node1 sshsetup]$ ls
sshUserSetup.sh
[grid@node1 sshsetup]$ ./sshUserSetup.sh -hosts "node1 node2" -user grid -advanced

配置记录如下：

  1 The output of this script is also logged into /tmp/sshUserSetup_2019-08-13-06-12-30.log
  2 Hosts are node1 node2
  3 user is grid
  4 Platform:- Linux 
  5 Checking if the remote hosts are reachable
  6 PING node1 (192.168.10.11) 56(84) bytes of data.
  7 64 bytes from node1 (192.168.10.11): icmp_seq=1 ttl=64 time=0.014 ms
  8 64 bytes from node1 (192.168.10.11): icmp_seq=2 ttl=64 time=0.043 ms
  9 64 bytes from node1 (192.168.10.11): icmp_seq=3 ttl=64 time=0.032 ms
 10 64 bytes from node1 (192.168.10.11): icmp_seq=4 ttl=64 time=0.040 ms
 11 64 bytes from node1 (192.168.10.11): icmp_seq=5 ttl=64 time=0.042 ms
 12 
 13 --- node1 ping statistics ---
 14 5 packets transmitted, 5 received, 0% packet loss, time 3999ms
 15 rtt min/avg/max/mdev = 0.014/0.034/0.043/0.011 ms
 16 PING node2 (192.168.10.12) 56(84) bytes of data.
 17 64 bytes from node2 (192.168.10.12): icmp_seq=1 ttl=64 time=3.05 ms
 18 64 bytes from node2 (192.168.10.12): icmp_seq=2 ttl=64 time=0.716 ms
 19 64 bytes from node2 (192.168.10.12): icmp_seq=3 ttl=64 time=0.807 ms
 20 64 bytes from node2 (192.168.10.12): icmp_seq=4 ttl=64 time=1.37 ms
 21 64 bytes from node2 (192.168.10.12): icmp_seq=5 ttl=64 time=0.704 ms
 22 
 23 --- node2 ping statistics ---
 24 5 packets transmitted, 5 received, 0% packet loss, time 4007ms
 25 rtt min/avg/max/mdev = 0.704/1.331/3.053/0.896 ms
 26 Remote host reachability check succeeded.
 27 The following hosts are reachable: node1 node2.
 28 The following hosts are not reachable: .
 29 All hosts are reachable. Proceeding further...
 30 firsthost node1
 31 numhosts 2
 32 The script will setup SSH connectivity from the host node1 to all
 33 the remote hosts. After the script is executed, the user can use SSH to run
 34 commands on the remote hosts or copy files between this host node1
 35 and the remote hosts without being prompted for passwords or confirmations.
 36 
 37 NOTE 1:
 38 As part of the setup procedure, this script will use ssh and scp to copy
 39 files between the local host and the remote hosts. Since the script does not
 40 store passwords, you may be prompted for the passwords during the execution of
 41 the script whenever ssh or scp is invoked.
 42 
 43 NOTE 2:
 44 AS PER SSH REQUIREMENTS, THIS SCRIPT WILL SECURE THE USER HOME DIRECTORY
 45 AND THE .ssh DIRECTORY BY REVOKING GROUP AND WORLD WRITE PRIVILEDGES TO THESE
 46 directories.
 47 
 48 Do you want to continue and let the script make the above mentioned changes (yes/no)?
 49 yes
 50 
 51 The user chose yes
 52 Please specify if you want to specify a passphrase for the private key this script will create for the local host. Passphrase is used to encrypt the private key and makes SSH much more secure. Type ‘yes‘ or ‘no‘ and then press enter. In case you press ‘yes‘, you would need to enter the passphrase whenever the script executes ssh or scp. 
 53 The estimated number of times the user would be prompted for a passphrase is 4. In addition, if the private-public files are also newly created, the user would have to specify the passphrase on one additional occasion. 
 54 Enter ‘yes‘ or ‘no‘.
 55 yes
 56 
 57 The user chose yes
 58 Creating .ssh directory on local host, if not present already
 59 Creating authorized_keys file on local host
 60 Changing permissions on authorized_keys to 644 on local host
 61 Creating known_hosts file on local host
 62 Changing permissions on known_hosts to 644 on local host
 63 Creating config file on local host
 64 If a config file exists already at /home/grid/.ssh/config, it would be backed up to /home/grid/.ssh/config.backup.
 65 Removing old private/public keys on local host
 66 Running SSH keygen on local host
 67 Enter passphrase (empty for no passphrase):         备注：输入回车 
 68 Enter same passphrase again:  备注：输入回车  
 69 Generating public/private rsa key pair.
 70 Your identification has been saved in /home/grid/.ssh/id_rsa.
 71 Your public key has been saved in /home/grid/.ssh/id_rsa.pub.
 72 The key fingerprint is:
 73 a0:48:eb:ab:7d:39:0d:cf:d2:29:49:cd:f0:a0:85:9d grid@node1
 74 The key‘s randomart image is:
 75 +--[ RSA 1024]----+
 76 |                 |
 77 |                 |
 78 |  .o ..          |
 79 | ..oE. .         |
 80 |  oo.*  S        |
 81 | .. o +          |
 82 |  .. O .         |
 83 | . .B *          |
 84 |..o. +           |
 85 +-----------------+
 86 Creating .ssh directory and setting permissions on remote host node1
 87 THE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR grid. THIS IS AN SSH REQUIREMENT.
 88 The script would create ~grid/.ssh/config file on remote host node1. If a config file exists already at ~grid/.ssh/config, it would be backed up to ~grid/.ssh/config.backup.
 89 The user may be prompted for a password here since the script would be running SSH on host node1.
 90 Warning: Permanently added ‘node1,192.168.10.11‘ (RSA) to the list of known hosts.
 91 grid@node1‘s password:   备注：输入节点1 grid的密码  
 92 Done with creating .ssh directory and setting permissions on remote host node1.
 93 Creating .ssh directory and setting permissions on remote host node2
 94 THE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR grid. THIS IS AN SSH REQUIREMENT.
 95 The script would create ~grid/.ssh/config file on remote host node2. If a config file exists already at ~grid/.ssh/config, it would be backed up to ~grid/.ssh/config.backup.
 96 The user may be prompted for a password here since the script would be running SSH on host node2.
 97 Warning: Permanently added ‘node2,192.168.10.12‘ (RSA) to the list of known hosts.
 98 grid@node2‘s password:   备注：输入节点2 grid的密码  
 99 Done with creating .ssh directory and setting permissions on remote host node2.
100 Copying local host public key to the remote host node1
101 The user may be prompted for a p