记一次打PSU遇到的Copyfailed的问题

好久没做过安装和打PSU的活了，心里还有点小期待，本以为2个小时就可以结束的事情，结果轮到真正的打的时候，打了4个多小时才弄完，遇到了一些小问题，我这次打的环境是11.2.0.3.9，这里做个记录。废话就不说了，我们来先看下日志： root@crmdb1rac1:/#/orac

好久没做过安装和打PSU的活了，心里还有点小期待，本以为2个小时就可以结束的事情，结果轮到真正的打的时候，打了4个多小时才弄完，遇到了一些小问题，我这次打的环境是11.2.0.3.9，这里做个记录。废话就不说了，我们来先看下日志：

root@crmdb1rac1:/#/oracle/app/11.2.0/grid/OPatch/opatch auto /oraclelog/opatch_soft -ocmrf /oracle/app/grid/ocm0709.rsp
Executing /oracle/app/11.2.0/grid/perl/bin/perl /oracle/app/11.2.0/grid/OPatch/crs/patch11203.pl -patchdir /oraclelog -patchn opatch_soft -ocmrf /oracle/app/grid/ocm0709.rsp -paramfile /oracle/app/11.2.0/grid/crs/install/crsconfig_params
/oracle/app/11.2.0/grid/crs/install/crsconfig_params
/oracle/app/11.2.0/grid/crs/install/s_crsconfig_defs
This is the main log file: /oracle/app/11.2.0/grid/cfgtoollogs/opatchauto2014-07-10_00-18-14.log
This file will show your detected configuration and all the steps that opatchauto attempted to do on your system: /oracle/app/11.2.0/grid/cfgtoollogs/opatchauto2014-07-10_00-18-14.report.log
2014-07-10 00:18:14: Starting Clusterware Patch Setup
Using configuration parameter file: /oracle/app/11.2.0/grid/crs/install/crsconfig_params
patch /oraclelog/opatch_soft/17592127/custom/server/17592127  apply successful for home  /oracle/app/oracle/product/11.2.0/db 
 patch /oraclelog/opatch_soft/18031683  apply successful for home  /oracle/app/oracle/product/11.2.0/db 
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'crmdb1rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'crmdb1rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'crmdb1rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'crmdb1rac1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'crmdb1rac1'
CRS-2673: Attempting to stop 'ora.cvu' on 'crmdb1rac1'
CRS-2673: Attempting to stop 'ora.crmdb1rac1.vip' on 'crmdb1rac1'
CRS-2677: Stop of 'ora.cvu' on 'crmdb1rac1' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'crmdb1rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'crmdb1rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'crmdb1rac1'
CRS-2676: Start of 'ora.cvu' on 'crmdb1rac2' succeeded
CRS-2677: Stop of 'ora.crmdb1rac1.vip' on 'crmdb1rac1' succeeded
CRS-2672: Attempting to start 'ora.crmdb1rac1.vip' on 'crmdb1rac2'
CRS-2677: Stop of 'ora.scan1.vip' on 'crmdb1rac1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'crmdb1rac2'
CRS-2676: Start of 'ora.crmdb1rac1.vip' on 'crmdb1rac2' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'crmdb1rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'crmdb1rac2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'crmdb1rac2' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'crmdb1rac1' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'crmdb1rac2'
CRS-2676: Start of 'ora.oc4j' on 'crmdb1rac2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'crmdb1rac1'
CRS-2677: Stop of 'ora.ons' on 'crmdb1rac1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'crmdb1rac1'
CRS-2677: Stop of 'ora.net1.network' on 'crmdb1rac1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'crmdb1rac1' has completed
CRS-2677: Stop of 'ora.crsd' on 'crmdb1rac1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'crmdb1rac1'
CRS-2673: Attempting to stop 'ora.crf' on 'crmdb1rac1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'crmdb1rac1'
CRS-2673: Attempting to stop 'ora.evmd' on 'crmdb1rac1'
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'crmdb1rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'crmdb1rac1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'crmdb1rac1' succeeded
CRS-2677: Stop of 'ora.crf' on 'crmdb1rac1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'crmdb1rac1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'crmdb1rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'crmdb1rac1'
CRS-2677: Stop of 'ora.cssd' on 'crmdb1rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'crmdb1rac1'
CRS-2677: Stop of 'ora.gipcd' on 'crmdb1rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'crmdb1rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'crmdb1rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'crmdb1rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /oracle/app/11.2.0/grid
patch /oraclelog/opatch_soft/17592127  apply failed  for home  /oracle/app/11.2.0/grid
CRS-4123: Oracle High Availability Services has been started.

在打PSU进行到Successfully unlock /oracle/app/11.2.0/grid这一步的时候。这个时候屏幕的输出会非常非常的慢，我们可以观察后台的日志输出，我们发现在后面进行一个copy操作的时候出现了问题。如下所示：

 Patching component oracle.usm, 11.2.0.3.0...
 The following actions have failed:
 Copy failed from '/oracle/app/11.2.0/grid/.patch_storage/16619898_Jun_26_2013_21_46_53/files/lib/libeons.so' to '/oracle/app/11.2.0/grid/lib/libeons.so'... 
 Copy failed from '/oracle/app/11.2.0/grid/.patch_storage/16619898_Jun_26_2013_21_46_53/files/lib/libhasgen11.so' to '/oracle/app/11.2.0/grid/lib/libhasgen11.so'... 
 Copy failed from '/oracle/app/11.2.0/grid/.patch_storage/16619898_Jun_26_2013_21_46_53/files/lib/libocr11.so' to '/oracle/app/11.2.0/grid/lib/libocr11.so'... 
 Copy failed from '/oracle/app/11.2.0/grid/.patch_storage/16619898_Jun_26_2013_21_46_53/files/lib/libocrb11.so' to '/oracle/app/11.2.0/grid/lib/libocrb11.so'... 
 Copy failed from '/oracle/app/11.2.0/grid/.patch_storage/16619898_Jun_26_2013_21_46_53/files/lib/libocrutl11.so' to '/oracle/app/11.2.0/grid/lib/libocrutl11.so'... 

我们可以看到在copy这些文件的时候是无法成功copy的，这是因为我们的Oracle的这些so文件在内存中被占用着，我们需要执行slibclean去清理这些缓存。于是我做了回退的操作，然后把crs完整的停下来，做了几次slibclean的操作，在正常的启动crs。做完这些操作后，我发现打PSU仍然在这一步失败了。于是我查了一些资料。我把这几个coyp失败的.so的文件做了一个备份，移动到了其他的目录里面，然后crs就无法成功启动了。crs无法启动还怎么打PSU呢？于是我又把文件拷贝回去，我仔细观察了下，在打PSU的时候，会自动把crs停掉，然后就会运行我们刚刚所说的Successfully unlock /oracle/app/11.2.0/grid。那我能不能在停掉CRS这个阶段执行slibclean去清理这些缓存呢？

这是在CRS还在启动状态下占用的lib文件:

root@crmdb1rac1:/oracle/app/11.2.0/grid/.patch_storage#genkld |grep oracle
 900000004d98000     b2c7 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libnio.a
 900000004d77000    203b1 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libnet.a
 900000004d5f000    17840 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libzip.a
 900000004d31000    21700 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9ute23.so
 900000004d1e000    12c7c /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libiverel23.so
 900000004cb6000    6792f /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libjclscar_23.so
 900000004c91000    2427d /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9vrb23.so
 900000004c5e000    32c3f /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9jvmti23.so
 900000004c2f000    2ed2c /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9dyn23.so
 900000004b97000    97a31 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9gc23.so
 900000004b7f000    179e1 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9bcv23.so
 9000000046cf000   4af269 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9jit23.so
 9000000046be000    10eef /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9trc23.so
 900000004681000    2ac84 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9dmp23.so
 9000000046ac000    11bab /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9zlib23.so
 900000004649000    37b9e /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libjava.a
 90000000091a000     1d41 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libwrappers.a
 900000004588000    6f0fe /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9vm23.so
 9000000045f8000    50434 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9prt23.so
 900000000918000     1d2f /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9hookable23.so
 90000000456c000    1b631 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/j9vm/libjvm.so
 900000004059000     bf6a /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libj9thr23.so
 900000000916000     1ff3 /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/libjsig.so
 900000004049000     f6aa /oracle/app/oracle/product/11.2.0/db/jdk/jre/bin/classic/libjvm.so
 90000000ff8e000  248bf1b /oracle/app/11.2.0/grid/lib/libclntsh.so
 900000004513000    5852a /oracle/app/11.2.0/grid/lib/libcell11.so
 90000000434a000   1b3f27 /oracle/app/11.2.0/grid/lib/libdb-4.6.so
 90000000428a000    bf3a1 /oracle/app/11.2.0/grid/lib/libcrf11.so
 9000000041d7000     ad7d /oracle/app/11.2.0/grid/lib/libeonsserver.so
 9000000041b1000    15c4e /oracle/app/11.2.0/grid/lib/libonsx.so
 900000004019000    2509a /oracle/app/11.2.0/grid/lib/libeons.so
 90000000123b000  2dd6039 /oracle/app/11.2.0/grid/lib/libttsh11.so
 900000004012000     6c16 /oracle/app/11.2.0/grid/lib/libskgxn2.so
 90000000406c000   1449e1 /oracle/app/11.2.0/grid/lib/libgns11.so
 90000000122e000     c5c5 /oracle/app/11.2.0/grid/lib/libocrutl11.so
 90000000103e000   1ef18b /oracle/app/11.2.0/grid/lib/libocrb11.so
 900000000f9f000    9e0a5 /oracle/app/11.2.0/grid/lib/libocr11.so
 900000000921000   67dc0c /oracle/app/11.2.0/grid/lib/libhasgen11.so
 90000000f7a9000   2adb17 /usr/openv/lib/libpioracle.so

这是在CRS关闭后，我运行了好几次slibclean之后，占用的lib文件.

 root@crmdb1rac1:/usr/sbin#genkld |grep oracle
 900000001160000     b2c7 /oracle/app/11.2.0/grid/jdk/jre/bin/libnio.a
 90000000113f000    203b1 /oracle/app/11.2.0/grid/jdk/jre/bin/libnet.a
 900000001127000    17840 /oracle/app/11.2.0/grid/jdk/jre/bin/libzip.a
 9000000010f9000    21700 /oracle/app/11.2.0/grid/jdk/jre/bin/libj9ute23.so
 9000000010e6000    12c7c /oracle/app/11.2.0/grid/jdk/jre/bin/libiverel23.so
 90000000107e000    6792f /oracle/app/11.2.0/grid/jdk/jre/bin/libjclscar_23.so
 900000001059000    2427d /oracle/app/11.2.0/grid/jdk/jre/bin/libj9vrb23.so
 900000001026000    32c3f /oracle/app/11.2.0/grid/jdk/jre/bin/libj9jvmti23.so
 900000000ff7000    2ed2c /oracle/app/11.2.0/grid/jdk/jre/bin/libj9dyn23.so
 900000000f5f000    97a31 /oracle/app/11.2.0/grid/jdk/jre/bin/libj9gc23.so
 900000000a97000   4af269 /oracle/app/11.2.0/grid/jdk/jre/bin/libj9jit23.so
 900000000a86000    10eef /oracle/app/11.2.0/grid/jdk/jre/bin/libj9trc23.so
 900000000a49000    2ac84 /oracle/app/11.2.0/grid/jdk/jre/bin/libj9dmp23.so
 900000000a74000    11bab /oracle/app/11.2.0/grid/jdk/jre/bin/libj9zlib23.so
 900000000a0f000    37b9e /oracle/app/11.2.0/grid/jdk/jre/bin/libjava.a
 900000000a47000     1d41 /oracle/app/11.2.0/grid/jdk/jre/bin/libwrappers.a
 90000000094e000    6f0fe /oracle/app/11.2.0/grid/jdk/jre/bin/libj9vm23.so
 9000000009be000    50434 /oracle/app/11.2.0/grid/jdk/jre/bin/libj9prt23.so
 900000000432000     1d2f /oracle/app/11.2.0/grid/jdk/jre/bin/libj9hookable23.so
 900000000926000    1b631 /oracle/app/11.2.0/grid/jdk/jre/bin/j9vm/libjvm.so
 900000000942000     bf6a /oracle/app/11.2.0/grid/jdk/jre/bin/libj9thr23.so
 900000000430000     1ff3 /oracle/app/11.2.0/grid/jdk/jre/bin/libjsig.so
 900000000916000     f6aa /oracle/app/11.2.0/grid/jdk/jre/bin/classic/libjvm.so
 90000000f7a9000   2adb17 /usr/openv/lib/libpioracle.so	

可以看到基本上全是和JDK相关的一些lib文件，没有我们需要copy的这些lib文件了。做完这些操作之后，PSU补丁才安装成功。不知道从何时开始起，非要折腾几下才能把事情做完做好。

原文地址：记一次打PSU遇到的Copy failed的问题, 感谢原作者分享。