Database/Oracle
[Oracle] CRS-5828 : Could not start agent '.../oraagent_orasvc01', CRS-2758 : Resource 'ora.plgcns.db' is in an unknown state
이상한금요일
2024. 8. 2. 13:55
728x90
현상
최신 RU(19.24)로 업데이트 후 crsctl 리소스 상태 조회 시 DB 리소스가 UNKNOWN 상태인 것을 확인
$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER_PLGCNS.lsnr
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.chad
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.net1.network
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.ons
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.proxy_advm
OFFLINE OFFLINE rac1 STABLE
OFFLINE OFFLINE rac2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac1 STABLE
ora.OCRVOTE.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.RECO.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE rac1 Started,STABLE
2 ONLINE ONLINE rac2 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.cvu
1 ONLINE ONLINE rac1 STABLE
ora.plgcns.db
1 OFFLINE UNKNOWN rac1 STABLE
2 OFFLINE UNKNOWN rac2 STABLE
ora.qosmserver
1 ONLINE ONLINE rac1 STABLE
ora.rac1.vip
1 ONLINE ONLINE rac1 STABLE
ora.rac2.vip
1 ONLINE ONLINE rac2 STABLE
ora.scan1.vip
1 ONLINE ONLINE rac2 STABLE
ora.scan2.vip
1 ONLINE ONLINE rac1 STABLE
ora.scan3.vip
1 ONLINE ONLINE rac1 STABLE
--------------------------------------------------------------------------------
srvctl로 db 리소스 컨트롤 안됨
$ srvctl start/stop database -d PLGCNS -i PLGCNS1
PRCR-1013 : Failed to start resource ora.plgcns.db
PRCR-1064 : Failed to start resource ora.plgcns.db on node rac1
CRS-2680: Clean of 'ora.plgcns.db' on 'rac1' failed
CRS-5802: Unable to start the agent process
sqlplus / as sysdba를 통해 수동으로 startup / shutdown immediate는 잘 됨
[orasvc01@rac1 engn001]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Thu Aug 1 10:44:59 2024
Version 19.24.0.0.0
Copyright (c) 1982, 2024, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area 2483024944 bytes
Fixed Size 8942640 bytes
Variable Size 1342177280 bytes
Database Buffers 1124073472 bytes
Redo Buffers 7831552 bytes
Database mounted.
Database opened.
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.24.0.0.0
[orasvc01@rac1 engn001]$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Thu Aug 1 10:46:00 2024
Version 19.24.0.0.0
Copyright (c) 1982, 2024, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.24.0.0.0
SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>
CRS에 database 리소스 재등록
$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER_PLGCNS.lsnr
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.chad
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.net1.network
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.ons
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.proxy_advm
OFFLINE OFFLINE rac1 STABLE
OFFLINE OFFLINE rac2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac1 STABLE
ora.OCRVOTE.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.RECO.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE rac1 Started,STABLE
2 ONLINE ONLINE rac2 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.cvu
1 ONLINE ONLINE rac1 STABLE
ora.plgcns.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE STABLE
ora.qosmserver
1 ONLINE ONLINE rac1 STABLE
ora.rac1.vip
1 ONLINE ONLINE rac1 STABLE
ora.rac2.vip
1 ONLINE ONLINE rac2 STABLE
ora.scan1.vip
1 ONLINE ONLINE rac2 STABLE
ora.scan2.vip
1 ONLINE ONLINE rac1 STABLE
ora.scan3.vip
1 ONLINE ONLINE rac1 STABLE
--------------------------------------------------------------------------------
crsctl stat res -t 조회 후 정상상태로 된 것을 확인해서 될 줄 알았으나 다시 UNKNOWN 상태로 빠짐
$ srvctl start database -d PLGCNS
PRCR-1079 : Failed to start resource ora.plgcns.db
CRS-2674: Start of 'ora.plgcns.db' on 'rac1' failed
CRS-2674: Start of 'ora.plgcns.db' on 'rac2' failed
CRS-2678: 'ora.plgcns.db' on 'rac1' has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
CRS-5802: Unable to start the agent process
CRS-2678: 'ora.plgcns.db' on 'rac2' has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
CRS-5802: Unable to start the agent process
$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER_PLGCNS.lsnr
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.chad
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.net1.network
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.ons
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.proxy_advm
OFFLINE OFFLINE rac1 STABLE
OFFLINE OFFLINE rac2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac1 STABLE
ora.OCRVOTE.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.RECO.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE rac1 Started,STABLE
2 ONLINE ONLINE rac2 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.cvu
1 ONLINE ONLINE rac1 STABLE
ora.plgcns.db
1 ONLINE UNKNOWN rac1 STABLE
2 ONLINE UNKNOWN rac2 STABLE
ora.qosmserver
1 ONLINE ONLINE rac1 STABLE
ora.rac1.vip
1 ONLINE ONLINE rac1 STABLE
ora.rac2.vip
1 ONLINE ONLINE rac2 STABLE
ora.scan1.vip
1 ONLINE ONLINE rac2 STABLE
ora.scan2.vip
1 ONLINE ONLINE rac1 STABLE
ora.scan3.vip
1 ONLINE ONLINE rac1 STABLE
--------------------------------------------------------------------------------
crs 중지 실패
# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rac1'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'rac1'
CRS-2673: Attempting to stop 'ora.chad' on 'rac1'
CRS-2679: Attempting to clean 'ora.plgcns.db' on 'rac1'
CRS-2677: Stop of 'ora.chad' on 'rac1' succeeded
CRS-2677: Stop of 'ora.qosmserver' on 'rac1' succeeded
CRS-2680: Clean of 'ora.plgcns.db' on 'rac1' failed
CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server 'rac1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.OCRVOTE.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.RECO.dg' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_PLGCNS.lsnr' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rac1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rac1'
CRS-2673: Attempting to stop 'ora.cvu' on 'rac1'
CRS-2677: Stop of 'ora.LISTENER_PLGCNS.lsnr' on 'rac1' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.rac1.vip' on 'rac1'
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rac1'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rac1'
CRS-2677: Stop of 'ora.scan3.vip' on 'rac1' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'rac1' succeeded
CRS-2677: Stop of 'ora.rac1.vip' on 'rac1' succeeded
CRS-2677: Stop of 'ora.cvu' on 'rac1' succeeded
CRS-2677: Stop of 'ora.OCRVOTE.dg' on 'rac1' succeeded
CRS-2677: Stop of 'ora.RECO.dg' on 'rac1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac1'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on 'rac1'
CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on 'rac1' succeeded
CRS-33677: Stop of resource group 'ora.asmgroup' on server 'rac1' succeeded.
CRS-2672: Attempting to start 'ora.qosmserver' on 'rac2'
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rac2'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rac2'
CRS-2672: Attempting to start 'ora.cvu' on 'rac2'
CRS-2672: Attempting to start 'ora.rac1.vip' on 'rac2'
CRS-2676: Start of 'ora.cvu' on 'rac2' succeeded
CRS-2676: Start of 'ora.rac1.vip' on 'rac2' succeeded
CRS-2676: Start of 'ora.scan3.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rac2'
CRS-2676: Start of 'ora.scan2.vip' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rac2'
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rac2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rac2' succeeded
CRS-2676: Start of 'ora.qosmserver' on 'rac2' succeeded
CRS-2799: Failed to shut down resource 'ora.plgcns.db' on 'rac1'
CRS-2794: Shutdown of Cluster Ready Services-managed resources on 'rac1' has failed
CRS-2675: Stop of 'ora.crsd' on 'rac1' failed
CRS-2799: Failed to shut down resource 'ora.crsd' on 'rac1'
CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has failed
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.
crs 비정상 종료 상태
$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER_PLGCNS.lsnr
ONLINE OFFLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.chad
ONLINE OFFLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.net1.network
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.ons
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.proxy_advm
OFFLINE OFFLINE rac1 STABLE
OFFLINE OFFLINE rac2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE rac2 STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE rac2 STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac2 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac2 STABLE
ora.OCRVOTE.dg(ora.asmgroup)
1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE rac2 STABLE
ora.RECO.dg(ora.asmgroup)
1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE rac2 STABLE
ora.asm(ora.asmgroup)
1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE rac2 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE rac2 STABLE
ora.cvu
1 ONLINE ONLINE rac2 STABLE
ora.plgcns.db
1 ONLINE UNKNOWN rac1 STABLE
2 ONLINE UNKNOWN rac2 STABLE
ora.qosmserver
1 ONLINE ONLINE rac2 STABLE
ora.rac1.vip
1 ONLINE INTERMEDIATE rac2 FAILED OVER,STABLE
ora.rac2.vip
1 ONLINE ONLINE rac2 STABLE
ora.scan1.vip
1 ONLINE ONLINE rac2 STABLE
ora.scan2.vip
1 ONLINE ONLINE rac2 STABLE
ora.scan3.vip
1 ONLINE ONLINE rac2 STABLE
--------------------------------------------------------------------------------
# crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.
# ps -ef | grep d.bin
root 1366086 1 0 Jul31 ? 00:07:28 /engn001/grdsvc01/grid/19c/bin/ohasd.bin reboot _ORA_BLOCKING_STACK_LOCALE=AMERICAN_AMERICA.AL32UTF8
grdsvc01 1366417 1 0 Jul31 ? 00:02:43 /engn001/grdsvc01/grid/19c/bin/mdnsd.bin
grdsvc01 1366418 1 0 Jul31 ? 00:06:48 /engn001/grdsvc01/grid/19c/bin/evmd.bin
grdsvc01 1366450 1 0 Jul31 ? 00:03:05 /engn001/grdsvc01/grid/19c/bin/gpnpd.bin
grdsvc01 1366520 1 0 Jul31 ? 00:07:38 /engn001/grdsvc01/grid/19c/bin/gipcd.bin
root 1366557 1 0 Jul31 ? 00:21:52 /engn001/grdsvc01/grid/19c/bin/osysmond.bin
grdsvc01 1366613 1 0 Jul31 ? 00:16:06 /engn001/grdsvc01/grid/19c/bin/ocssd.bin -S 1
root 1367046 1 0 Jul31 ? 00:07:50 /engn001/grdsvc01/grid/19c/bin/octssd.bin reboot
root 1367086 1 0 Jul31 ? 00:08:57 /engn001/grdsvc01/grid/19c/bin/crsd.bin reboot
root 2910073 2710469 0 10:12 pts/1 00:00:00 grep --color=auto d.bin
로그 확인
crs log 조회 시 아래와 같이 나타남
(log 경로 : $GRID_BASE/diag/crs/<hostname>/crs/trace/alert.log)
$ vi alert.log
...
2024-08-02 09:26:10.700 [CRSD(1367086)]CRS-5828: Could not start agent '/engn001/grdsvc01/grid/19c/bin/oraagent_orasvc01'. Details at (:CRSAGF00130:) {1:14031:19667} in /engn001/grdsvc01/grid/base/diag/crs/rac1/crs/trace/crsd.trc.
2024-08-02 09:29:40.746 [CRSD(1367086)]CRS-5828: Could not start agent '/engn001/grdsvc01/grid/19c/bin/oraagent_orasvc01'. Details at (:CRSAGF00130:) {1:14031:19667} in /engn001/grdsvc01/grid/base/diag/crs/rac1/crs/trace/crsd.trc.
2024-08-02 09:29:40.748 [CRSD(1367086)]CRS-2758: Resource 'ora.plgcns.db' is in an unknown state.
2024-08-02 09:29:40.772 [CRSD(1367086)]CRS-2758: Resource 'ora.plgcns.db' is in an unknown state.
2024-08-02 09:31:05.035 [OHASD(1366086)]CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
2024-08-02 09:34:35.433 [CRSD(1367086)]CRS-5828: Could not start agent '/engn001/grdsvc01/grid/19c/bin/oraagent_orasvc01'. Details at (:CRSAGF00130:) {1:14031:19693} in /engn001/grdsvc01/grid/base/diag/crs/rac1/crs/trace/crsd.trc.
2024-08-02 09:34:53.410 [OHASD(1366086)]CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has failed
...
원인
crsd_oraagent 관련 파일 중 crsd_oraagent_orasvc01* 권한이 잘못되어 있었음
grdsvc01:dba → orasvc01:dba 로 변경해야 함
(파일 경로 : $GRID_BASE/crsdata/<hostname>/output)
# cd /engn001/grdsvc01/grid/base/crsdata/rac2/output
# ls -al
total 248
drwxrwxrwt. 2 grdsvc01 dba 4096 Aug 2 09:26 .
drwxrwxr-x. 27 grdsvc01 dba 4096 Jun 28 23:13 ..
-rw-r--r--. 1 grdsvc01 dba 1187 Aug 2 09:34 crsd_jagent_grdsvc01OUT.trc
-rw-r--r--. 1 grdsvc01 dba 7 Aug 2 09:34 crsd_jagent_grdsvc01.pid
-rw-r--r--. 1 grdsvc01 dba 2537 Jul 31 18:22 crsd_oraagent_grdsvc01OUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:22 crsd_oraagent_grdsvc01.pid
-rw-r--r--. 1 grdsvc01 dba 3424 Jul 31 18:01 crsd_oraagent_orasvc01OUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 14 17:13 crsd_oraagent_orasvc01.pid
-rw-r--r--. 1 grdsvc01 dba 1208 Jul 31 18:19 crsd_orarootagent_rootOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:19 crsd_orarootagent_root.pid
-rw-r--r--. 1 grdsvc01 dba 1129 Jul 31 18:19 crsdOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:19 crsd.pid
-rw-r--r--. 1 grdsvc01 dba 1217 Aug 2 09:34 crsd_scriptagent_grdsvc01OUT.trc
-rw-r--r--. 1 grdsvc01 dba 7 Aug 2 09:34 crsd_scriptagent_grdsvc01.pid
-rw-r--r--. 1 grdsvc01 dba 377 Jul 31 18:18 crswrapexece.log
-rw-r--r--. 1 grdsvc01 dba 1129 Jul 31 18:18 evmdOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 evmd.pid
-rw-r--r--. 1 grdsvc01 dba 1804 Jul 31 18:18 evmloggerOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 evmlogger.pid
-rw-r--r--. 1 grdsvc01 dba 1759 Jul 31 18:18 gipcdOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 gipcd.pid
-rw-r--r--. 1 grdsvc01 dba 1376 Jul 31 18:18 gpnpdOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 gpnpd.pid
-rw-r--r--. 1 grdsvc01 dba 3027 Jul 31 18:18 mdnsdOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 mdnsd.pid
-rw-r--r--. 1 grdsvc01 dba 856 Jul 31 18:36 ochadOUT.trc
-rw-r--r--. 1 grdsvc01 dba 17196 Jul 31 18:18 ocssdOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 ocssd.pid
-rw-r--r--. 1 grdsvc01 dba 1141 Jul 31 18:18 octssdOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 octssd.pid
-rw-r--r--. 1 grdsvc01 dba 8423 Jul 31 18:18 ohasd_cssdagent_rootOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 ohasd_cssdagent_root.pid
-rw-r--r--. 1 grdsvc01 dba 8465 Jul 31 18:18 ohasd_cssdmonitor_rootOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 ohasd_cssdmonitor_root.pid
-rw-r--r--. 1 grdsvc01 dba 2911 Jul 31 18:18 ohasd_oraagent_grdsvc01OUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 ohasd_oraagent_grdsvc01.pid
-rw-r--r--. 1 grdsvc01 dba 1613 Jul 31 18:18 ohasd_orarootagent_rootOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 ohasd_orarootagent_root.pid
-rw-r--r--. 1 grdsvc01 dba 1891 Jul 31 18:18 ohasdOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 ohasd.pid
-rw-r--r--. 1 grdsvc01 dba 772 Jul 31 17:19 ologgerdOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 17:19 ologgerd.pid
-rw-r--r--. 1 orasvc01 dba 548 Jul 31 18:19 oraagent_3303959.out
-rw-r--r--. 1 orasvc01 dba 548 Jul 31 18:23 oraagent_3305707.out
-rw-r--r--. 1 orasvc01 dba 548 Jul 31 18:26 oraagent_3307451.out
-rw-r--r--. 1 orasvc01 dba 548 Aug 1 10:10 oraagent_3843685.out
-rw-r--r--. 1 orasvc01 dba 548 Aug 1 10:30 oraagent_3858559.out
-rw-r--r--. 1 orasvc01 dba 548 Aug 1 12:01 oraagent_3915298.out
-rw-r--r--. 1 orasvc01 dba 548 Aug 1 12:05 oraagent_3917348.out
-rw-r--r--. 1 orasvc01 dba 546 Aug 2 09:12 oraagent_408365.out
-rw-r--r--. 1 orasvc01 dba 546 Aug 2 09:22 oraagent_414032.out
-rw-r--r--. 1 orasvc01 dba 546 Aug 2 09:26 oraagent_416079.out
-rw-r--r--. 1 grdsvc01 dba 2403 Jul 31 18:18 osysmondOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 osysmond.pid해결방법
orasvc01 유저가 사용할 수 있도록 orasvc01:dba로 권한 변경
# chown -R orasvc01:dba crsd_oraagent_orasvc01OUT.trc crsd_oraagent_orasvc01.pid
# ls -al
total 248
drwxrwxrwt. 2 grdsvc01 dba 4096 Aug 2 09:26 .
drwxrwxr-x. 27 grdsvc01 dba 4096 Jun 28 23:13 ..
-rw-r--r--. 1 grdsvc01 dba 1187 Aug 2 09:34 crsd_jagent_grdsvc01OUT.trc
-rw-r--r--. 1 grdsvc01 dba 7 Aug 2 09:34 crsd_jagent_grdsvc01.pid
-rw-r--r--. 1 grdsvc01 dba 2537 Jul 31 18:22 crsd_oraagent_grdsvc01OUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:22 crsd_oraagent_grdsvc01.pid
-rw-r--r--. 1 orasvc01 dba 3424 Jul 31 18:01 crsd_oraagent_orasvc01OUT.trc
-rw-r--r--. 1 orasvc01 dba 8 Jul 14 17:13 crsd_oraagent_orasvc01.pid
-rw-r--r--. 1 grdsvc01 dba 1208 Jul 31 18:19 crsd_orarootagent_rootOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:19 crsd_orarootagent_root.pid
-rw-r--r--. 1 grdsvc01 dba 1129 Jul 31 18:19 crsdOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:19 crsd.pid
-rw-r--r--. 1 grdsvc01 dba 1217 Aug 2 09:34 crsd_scriptagent_grdsvc01OUT.trc
-rw-r--r--. 1 grdsvc01 dba 7 Aug 2 09:34 crsd_scriptagent_grdsvc01.pid
-rw-r--r--. 1 grdsvc01 dba 377 Jul 31 18:18 crswrapexece.log
-rw-r--r--. 1 grdsvc01 dba 1129 Jul 31 18:18 evmdOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 evmd.pid
-rw-r--r--. 1 grdsvc01 dba 1804 Jul 31 18:18 evmloggerOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 evmlogger.pid
-rw-r--r--. 1 grdsvc01 dba 1759 Jul 31 18:18 gipcdOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 gipcd.pid
-rw-r--r--. 1 grdsvc01 dba 1376 Jul 31 18:18 gpnpdOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 gpnpd.pid
-rw-r--r--. 1 grdsvc01 dba 3027 Jul 31 18:18 mdnsdOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 mdnsd.pid
-rw-r--r--. 1 grdsvc01 dba 856 Jul 31 18:36 ochadOUT.trc
-rw-r--r--. 1 grdsvc01 dba 17757 Aug 2 10:50 ocssdOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 ocssd.pid
-rw-r--r--. 1 grdsvc01 dba 1141 Jul 31 18:18 octssdOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 octssd.pid
-rw-r--r--. 1 grdsvc01 dba 8423 Jul 31 18:18 ohasd_cssdagent_rootOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 ohasd_cssdagent_root.pid
-rw-r--r--. 1 grdsvc01 dba 8465 Jul 31 18:18 ohasd_cssdmonitor_rootOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 ohasd_cssdmonitor_root.pid
-rw-r--r--. 1 grdsvc01 dba 2911 Jul 31 18:18 ohasd_oraagent_grdsvc01OUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 ohasd_oraagent_grdsvc01.pid
-rw-r--r--. 1 grdsvc01 dba 1613 Jul 31 18:18 ohasd_orarootagent_rootOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 ohasd_orarootagent_root.pid
-rw-r--r--. 1 grdsvc01 dba 1891 Jul 31 18:18 ohasdOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 ohasd.pid
-rw-r--r--. 1 grdsvc01 dba 772 Jul 31 17:19 ologgerdOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 17:19 ologgerd.pid
-rw-r--r--. 1 orasvc01 dba 548 Jul 31 18:19 oraagent_3303959.out
-rw-r--r--. 1 orasvc01 dba 548 Jul 31 18:23 oraagent_3305707.out
-rw-r--r--. 1 orasvc01 dba 548 Jul 31 18:26 oraagent_3307451.out
-rw-r--r--. 1 orasvc01 dba 548 Aug 1 10:10 oraagent_3843685.out
-rw-r--r--. 1 orasvc01 dba 548 Aug 1 10:30 oraagent_3858559.out
-rw-r--r--. 1 orasvc01 dba 548 Aug 1 12:01 oraagent_3915298.out
-rw-r--r--. 1 orasvc01 dba 548 Aug 1 12:05 oraagent_3917348.out
-rw-r--r--. 1 orasvc01 dba 546 Aug 2 09:12 oraagent_408365.out
-rw-r--r--. 1 orasvc01 dba 546 Aug 2 09:22 oraagent_414032.out
-rw-r--r--. 1 orasvc01 dba 546 Aug 2 09:26 oraagent_416079.out
-rw-r--r--. 1 grdsvc01 dba 2403 Jul 31 18:18 osysmondOUT.trc
-rw-r--r--. 1 grdsvc01 dba 8 Jul 31 18:18 osysmond.pid- 권한 변경 후 정상적으로 CRS 중지됨
# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server 'rac2'
CRS-33673: Attempting to stop resource group 'ora.asmgroup' on server 'rac2'
CRS-2673: Attempting to stop 'ora.OCRVOTE.dg' on 'rac2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac2'
CRS-2673: Attempting to stop 'ora.RECO.dg' on 'rac2'
CRS-2673: Attempting to stop 'ora.qosmserver' on 'rac2'
CRS-2673: Attempting to stop 'ora.LISTENER_PLGCNS.lsnr' on 'rac2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rac2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rac2'
CRS-2673: Attempting to stop 'ora.chad' on 'rac2'
CRS-2673: Attempting to stop 'ora.cvu' on 'rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rac2'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rac2'
CRS-2677: Stop of 'ora.RECO.dg' on 'rac2' succeeded
CRS-2677: Stop of 'ora.cvu' on 'rac2' succeeded
CRS-2677: Stop of 'ora.OCRVOTE.dg' on 'rac2' succeeded
CRS-2677: Stop of 'ora.LISTENER_PLGCNS.lsnr' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.rac2.vip' on 'rac2'
CRS-2677: Stop of 'ora.DATA.dg' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
CRS-2677: Stop of 'ora.scan2.vip' on 'rac2' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'rac2' succeeded
CRS-2677: Stop of 'ora.rac2.vip' on 'rac2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac2'
CRS-2677: Stop of 'ora.chad' on 'rac2' succeeded
CRS-2677: Stop of 'ora.qosmserver' on 'rac2' succeeded
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.asmnet1.asmnetwork' on 'rac2'
CRS-2677: Stop of 'ora.asmnet1.asmnetwork' on 'rac2' succeeded
CRS-33677: Stop of resource group 'ora.asmgroup' on server 'rac2' succeeded.
CRS-2672: Attempting to start 'ora.qosmserver' on 'rac1'
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rac1'
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rac1'
CRS-2672: Attempting to start 'ora.cvu' on 'rac1'
CRS-2672: Attempting to start 'ora.rac2.vip' on 'rac1'
CRS-2676: Start of 'ora.cvu' on 'rac1' succeeded
CRS-2676: Start of 'ora.rac2.vip' on 'rac1' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rac1'
CRS-2676: Start of 'ora.scan3.vip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rac1'
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rac1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rac1' succeeded
CRS-2676: Start of 'ora.qosmserver' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'rac2'
CRS-2677: Stop of 'ora.ons' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac2'
CRS-2677: Stop of 'ora.net1.network' on 'rac2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac2' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.storage' on 'rac2'
CRS-2673: Attempting to stop 'ora.crf' on 'rac2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac2' succeeded
CRS-2677: Stop of 'ora.crf' on 'rac2' succeeded
CRS-2677: Stop of 'ora.storage' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac2'
CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac2'
CRS-2677: Stop of 'ora.ctssd' on 'rac2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac2'
CRS-2677: Stop of 'ora.cssd' on 'rac2' succeeded
CRS-2673: Attempting to stop 'ora.driver.afd' on 'rac2'
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2'
CRS-2677: Stop of 'ora.driver.afd' on 'rac2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS 기동 후 리소스 상태 정상 확인
# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
# crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER_PLGCNS.lsnr
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.chad
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.net1.network
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.ons
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.proxy_advm
OFFLINE OFFLINE rac1 STABLE
OFFLINE OFFLINE rac2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac1 STABLE
ora.OCRVOTE.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.RECO.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE rac1 Started,STABLE
2 ONLINE ONLINE rac2 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
ora.cvu
1 ONLINE ONLINE rac1 STABLE
ora.plgcns.db
1 ONLINE ONLINE rac1 Open,HOME=/home/engn
001/orasvc01/oracle/
product/19c,STABLE
2 ONLINE ONLINE rac2 Open,HOME=/home/engn
001/orasvc01/oracle/
product/19c,STABLE
ora.qosmserver
1 ONLINE ONLINE rac1 STABLE
ora.rac1.vip
1 ONLINE ONLINE rac1 STABLE
ora.rac2.vip
1 ONLINE ONLINE rac2 STABLE
ora.scan1.vip
1 ONLINE ONLINE rac2 STABLE
ora.scan2.vip
1 ONLINE ONLINE rac1 STABLE
ora.scan3.vip
1 ONLINE ONLINE rac1 STABLE
--------------------------------------------------------------------------------
참고 사이트
Troubleshooting 19c RAC CRS resource db show “UNKNOWN” state , srvctl start instance CRS-2680 | ANBOB
有套ORACLE 19c RAC在使用crsctl 查看db resource时显示“UNKNOWN”, 但是用sqlplus 可以启动db 实例，srvctl status instance显示not running. 手动启动instance 使用srvctl 显示如下错误 [oracle@~]$ srvctl start instance -d -i INTS
www.anbob.com
728x90