Oracle RAC数据库实例启动异常问题分析IPC Send timeout

最近，一个用户正在重启RAC在节点数据库实例中，发现启动速度非常慢。同时，业务部门反馈连接RAC生存节点的业务也受到影响。

在启动数据库时，通过对日志的分析，Reconfiguration速度慢，Reconfiguration后报错IPC Send timeout detected. Sender: ospid 53884 [oracle@test2 (LMD0)]，数据库实例组的节点驱逐；Wed Apr 13 19:28:02 2022
Instance termination initiated by instance 2 with reason 1.
Instance 2 received a reconfig event from its cluster manager indicating that this instance is supposed to be down
Please check instance 2's alert log and LMON trace file for more details.
Please also examine the CSS log files.
LMON (ospid: 47523): terminating the instance due to error 481

因此，有必要出现调查过程IPC Send timeout原因；这方面有BUG也可能有RAC节点负荷的原因可以参考MOS文件中的调查步骤逐项调查系统信息：

Instance Evicted After LMON to LMON IPC Send timeout Due to Storage Issue (Doc ID 2080029.1)
"ipc send timeout" Precedes Database Instance Crash or Eviction (Doc ID 1951216.1)
While Evicting One of the Instance, the Remaining instances Terminated by LMON with "LMON is running too slowly and in the middle of reconfiguration" (Doc ID 1949505.1)

有关日志如下：

 1. 2022-04-13 18:57:29.215 人工重启节点1集群软件， 数据库实例也成功启动， Wed Apr 13 18:58:32 2022 QMNC started with pid=100, OS id=52025  Completed: ALTER DATABASE OPEN /* db agent *//* {1:49652:2} */   2.节点2 RECONFIG节点1异常 --节点2 Wed Apr 13 19:22:26 2022 Starting ORACLE instance (normal)  --节点1： Wed Apr 13 19:28:00 2022 IPC Send timeout detected. Receiver ospid 47526 [ Wed Apr 13 19:28:00 2022 Errors in file /oracle/app/diag/rdbms/testnew/test1/trace/test1_lmd0_47526.trc: Wed Apr 13 19:28:02 2022 Instance termination initiated by instance 2 with reason 1. Instance 2 received a reconfig event from its cluster manager indicating that this instance is supposed to be down Please check instance 2's alert log and LMON trace file for more details. Please also examine the CSS log files. LMON (ospid: 47523): terminating the instance due to error 481 System state dump requested by (instance=1, osid=47523 (LMON)), summary=[abnormal instance termination]. System State dumped to trace file /oracle/app/diag/rdbms/testnew/test1/trace/test1_diag_47507_20220413192802.trc Wed Apr 13 19:28:03 2022 ORA-1092 : opitsk aborting process Instance terminated by LMON, pid = 47523  --节点2： Wed Apr 13 19:28:00 2022 IPC Send timeout detected. Sender: ospid 53884 [oracle@test2 (LMD0)] Receiver: inst 1 binc 429458022 ospid 47526 IPC Send timeout to 1.0 inc 4 for msg type 65521 from opid 11 Wed Apr 13 19:28:02 2022 Communications reconfiguration: instance_number 1 Wed Apr 13 19:28:02 2022 Dumping diagnostic data in directory=[cdmp_20220413192802], requested by (instance=1, osid=47523 (LMON)), summary=[abnormal instance termination]. Reconfiguration started (old inc 4, new inc 8)     ######################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################## 3.检查节点2启动情况，Reconfiguration过程中，IPC Send timeout 这也是节点2手动启动时感觉很慢的原因 ；同时，节点1在19:34启动时报ORA-00240错误 ，综合检查当时的网络、存储和节点负荷，参考MOS上文档。  Wed Apr 13 19:34:49 2022 Errors in file /oracle/app/diag/rdbms/testnew/test1/trace/test1_dbw0_247773.trc  (incident=168173): ORA-00240: control file enqueue held for more than 120 seconds  Instance Evicted After LMON to LMON IPC Send timeout Due to Storage Issue (Doc ID 2080029.1)  "ipc send timeout" Precedes Database Instance Crash or Eviction (Doc ID 1951216.1)  While Evicting One of the Instance, the Remaining instances Terminated by LMON with "LMON is running too slowly and in the middle of reconfiguration" (Doc ID 1949505.1)