Introduction:-
In this post we going to cover the ‘How to fix ADOP Cutover failed PRE_CUTOVER_CHECKS_COMPLETED’ while applying the adop patch in our EBS. After applying a patch with adop: (prepare/apply/finalize), the cutover failed.
👉Why ADOP Cutover failed PRE_CUTOVER_CHECKS?.
The services on primary node possibly partially up, but services down on the secondary node due to ADOP Cutover failed PRE_CUTOVER_CHECKS_COMPLETED.
adop cutover failed pre cutover checks
Summary report for current adop session: –
Node acedisupplier: Failed
– Cleanup status: Failed
– Apply status: Completed successfully
– Finalize status: Completed successfully
– Cutover status: FORCE_SHUTDOWN_COMPLETED
Node prdsvr: Failed
– Cleanup status: Failed
– Apply status: Completed successfully
– Finalize status: Completed successfully
– Cutover status: PRE_CUTOVER_CHECKS_COMPLETED
For more details, run the command: adop -status -detail
If we see CUTOVER_STATUS column the below will determine the code and status.
–>=’Y’ ‘COMPLETED’
–> not in (‘N’,’Y’,’X’) and status=’F’ ‘FAILED’
–>=’0′ ‘CUTOVER STARTED’
–>=’1′ ‘SERVICES SHUTDOWN COMPLETED’
–>=’3′ ‘DB CUTOVER COMPLETED’
–>=’D’ ‘FLIP SNAPSHOTS COMPLETED’
–>=’4′ ‘FS CUTOVER COMPLETED’
–>=’5′ ‘ADMIN STARTUP COMPLETED’
–>=’6′ ‘SERVICES STARTUP COMPLETED’
–>=’N’ ‘NOT STARTED’
–>=’X’ ‘NOT APPLICABLE’
Summary report for current adop session:
Node acedisupplier: Failed
– Cleanup status: Failed
– Apply status: Completed successfully
– Finalize status: Completed successfully
– Cutover status: FORCE_SHUTDOWN_COMPLETED
Node prdsvr: Failed
– Cleanup status: Failed
– Apply status: Completed successfully
– Finalize status: Completed successfully
– Cutover status: PRE_CUTOVER_CHECKS_COMPLETED
For more details, run the command: adop -status -detail
To check the details information setup analyzer.
ADOP Analyzer: Please upload the ICM ADOP Application Tier Analyzer output zip-file
by using the LATEST version of the analyzer available in
EBS Installation Configuration Management (ICM) Application DBA Online Patching (ADOP)
Analyzer (Doc ID 2379656.1).
SQL>select ADOP_SESSION_ID, EDITION_NAME, NODE_TYPE, NODE_NAME, APPLTOP_ID, PREPARE_STATUS,APPLY_STATUS, CUTOVER_STATUS, CLEANUP_STATUS, ABORT_STATUS, STATUSÂ from AD_ADOP_SESSIONSÂ order by ADOP_SESSION_ID;
ODM Data Collection:-
35.00 V_20211201_2323 slave acedisupplier 193.00 Y Y 1 N N F
35.00 V_20211201_2323 master prdsvr     193.00 Y Y 0 N N F
For the slave completed the shutdown but for the master is just started. Checking for existing adop cutover failed sessions.Â
Continuing with existing session [Session ID: 35].
Session Id : 35
Prepare phase status : COMPLETED
Apply phase status : COMPLETED
Cutover phase status : NOT COMPLETED
Abort phase status : NOT COMPLETED
Session status : RUNNING
cat /oracle/aced/EBSFS/fs1/inst/apps/aced_prdsvr/logs/appl/admin/log
/adapcctl.txt
12/01/21-23:51:35 :: adapcctl.sh version 120.0.12020000.6
12/01/21-23:51:35 :: adapcctl.sh: stopping OPMN managed OHS instance
opmnctl stopproc: stopping opmn managed processes…
============================================================
opmn id=prdsvr.aced-cranes.com:6203
Response: 0 of 1 processes stopped.
ias-instance id=EBS_web_OHS1
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
——————————————————————————–
ias-component/process-type/process-set:
EBS_web/OHS/OHS/
Error during ADOP Cutover failed.
–> Process (index=1,uid=1243434072,pid=34408)
time out while waiting for a managed process to stop
second stop attempted
time out while waiting for a managed process to stop
Log:
/oracle/aced/EBSFS/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/
diagnostics/logs/OHS/EBS_web/console~OHS~1.log
12/01/21-23:53:39 :: adapcctl.sh: exiting with status 204
Checking console~OHS~1.log.
——–
21/12/01 23:55:43 Stop process
——–
/oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/
apachectl hardstop: httpd (no pid file) not running
——–
21/12/01 23:56:45 Stop process
——–
/oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/
apachectl hardstop: httpd (no pid file) not running
——–
21/12/01 23:57:47 Stop process
——–
/oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/
apachectl hardstop: httpd (no pid file) not running
FileName
—————-
Attachment: <console_OHS_1.log>
Diagnose the adop cutover failed pre cutover checks issue.
Okay, we can try to stop the service manually by running the
/oracle/aced/EBSFS/fs1/inst/apps/aced_prdsvr/admin/scripts/adapcctl.sh stop
If it fails check below:
1>Please confirm the httpd.pid file configuration in httpd.conf file.
2>Please switch to the directory /oracle/aced/EBSFS/fs1/FMW_Home
/webtier/ohs/bin/
After that and run the following command.
Check the output
$ grep PIDFILE apachectl
Below are output
[applmgr@prdsvr bin]$ grep PIDFILE apachectl
PIDFILE=${ORACLE_CONFIG_HOME}/logs/httpd.pid
PIDFILE=${COMPONENT_LOG_PATH}/httpd.pid
if [ -f $PIDFILE ]
PID=`cat $PIDFILE`
#PIDFILE=`grep PidFile $1 | grep -v “#” | cut -d” ” -f 2`
PIDFILE=`grep “^PidFile” $1| cut -d” ” -f 2`
PIDFILE=`eval echo $PIDFILE`
if [ “x$PIDFILE” = “x” ]
We will check if the PID does exist :
[applmgr@prdsvr scripts]$ cat /oracle/aced/EBSFS/fs1/FMW_Home/
webtier/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web
/httpd.pid
cat: /oracle/aced/EBSFS/fs1/FMW_Home/webtier/instances/
EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/httpd.pid: No such file or directory
Some processes are running as below.
[applmgr@prdsvr scripts]$ ps -ef|grep applmgr
applmgr 3580 1 0 Oct23 ? 00:00:05 frmweb server webfile=HTTP-0,0,1,default
applmgr 9514 34408 0 Dec01 ? 00:00:20 /oracle/aced/EBSFS/fs1/
FMW_Home/webtier/ohs/bin/httpd.worker -DSSL
applmgr 10168 34408 0 Dec01 ? 00:00:21 /oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/httpd.worker -DSSL
applmgr 16163 1 0 Nov16 ? 00:00:01 frmweb server webfile=HTTP-0,0,1,default
root 19601 1270 0 04:32 pts/1 00:00:00 su – applmgr
applmgr 19603 19601 0 04:32 pts/1 00:00:00 -bash
root 19781 19717 0 Dec01 pts/0 00:00:00 su – applmgr
applmgr 19783 19781 0 Dec01 pts/0 00:00:00 -bash
lp 21624 35206 0 04:38 ? 00:00:00 smb://192.168.11.209/P6SPDDM1 226243
applmgr COMM-P6SPD.57030008 1 cpi=12 finishings=3 fitplot lpi=6
media=na_letter_8.5x11in number-up=1 page-bottom=86 page-left=57
page-right=57 page-top=30 scaling=100 wrap job-uuid=urn:uuid:a7311f94-7621-
3e6e-5a1b-8e6322635064 job-originating-host-name=localhost
applmgr 21707 19603 0 04:40 pts/1 00:00:00 ps -ef|grep applmgr
applmgr 34408 1 0 Oct16 ? 00:03:04 /oracle/aced/EBSFS/fs1/FMW_Home
/webtier/ohs/bin/httpd.worker -DSSL
applmgr 34506 34408 0 Oct16 ? 00:00:02 /oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/odl_rotatelogs -l
/oracle/aced/EBSFS/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/
diagnostics/logs/OHS/EBS_web/EBS_web-%Y%m%d%H%M%S.log 10M 70M
applmgr 34507 34408 0 Oct16 ? 00:24:08 /oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/odl_rotatelogs
/oracle/aced/EBSFS/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/
diagnostics/logs/OHS/EBS_web/access_log 43200
applmgr 34508 34408 0 Oct16 ? 00:00:00 /oracle/aced/EBSFS/fs1/FMW_Home/
webtier/ohs/bin/rotatelogs
/oracle/aced/EBSFS/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/
diagnostics/logs/OHS/EBS_web/access_log 86400
applmgr 34509 34408 0 Oct16 ? 00:00:00 /oracle/aced/EBSFS/fs1/FMW_Home/
webtier/ohs/bin/rotatelogs
/oracle/aced/EBSFS/fs1/FMW_Home/webtier/instances/
EBS_web_OHS1/diagnostics
/logs/OHS/EBS_web/ssl_request_log 86400
applmgr 34519 34408 0 Oct16 ? 00:00:00 /oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/odl_rotatelogs -l -h:/oracle/aced/EBSFS/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/
config/OHS/EBS_web/component_events.xml_EBS_web
/oracle/aced/EBSFS/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/
auditlogs/OHS/EBS_web/audit-pid34408-%Y%m%d%H%M%S.log 1M 4M
applmgr 34520 34408 0 Oct16 ? 00:00:47 /oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/httpd.worker -DSSL
applmgr 39310 34408 0 Dec01 ? 00:00:23 /oracle/aced/EBSFS/fs1/FMW_Home/webtier/ohs/bin/httpd.worker -DSSL
Solution:-
It seems the opmn is already down. Confirm that :Then check and confirm is started and PID does exist :
$ps -ef | grep -i opm
[applmgr@prdsvr scripts]$Â ./adopmnctl.sh status
You are running adopmnctl.sh version 120.0.12020000.2
Checking status of OPMN managed processes…
Processes in Instance: EBS_web_OHS1
———————————+——————–+———+———
ias-component | process-type | pid | status
———————————+——————–+———+———
EBS_web | OHS | 34408 | Stop
After kill pid 34408. it started now.
$kill -9Â 34408
We will go to the master node and start the opmn ?
$./adopmnctl.sh start
If it’s down then run adop cutover again.
Now Executed adop phase=cutover.
In adopmon frequently showing as below.
-2021/12/02 05:05:35 acedisupplier EVENT DB Cutover did not complete on Admin node: <prdsvr>
2021/12/02 05:05:35 acedisupplier EVENT Will wait for another minute and check again.
-2021/12/02 05:06:35 acedisupplier EVENT DB Cutover did not complete on Admin node: <prdsvr>
2021/12/02 05:06:35 acedisupplier EVENT Will wait for another minute and check again.
-2021/12/02 05:07:35 acedisupplier EVENT DB Cutover did not complete on Admin node: <prdsvr>
-2021/12/02 05:07:35 acedisupplier EVENT Will wait for another minute and check again.
2021/12/02 05:08:35 acedisupplier EVENT DB Cutover did not complete on Admin node: <prdsvr>
-2021/12/02 05:08:35 acedisupplier EVENT Will wait for another minute and check again.
It seems cutover is still trying to stop it regardless the opmn is down already. Since PID
does not exist is constantly failing.
cat /oracle/aced/EBSFS/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/
diagnostics/logs/OHS/EBS_web/httpd.pid
Retry the adop cutover:-
Then if everything looks good run the adop cutover again.
-2021/12/02 05:33:44 prdsvr EVENT Cutover System
-2021/12/02 05:33:44 prdsvr EVENT Switching to Patch Edition
2021/12/02 05:33:44 prdsvr EVENT Killing old sessions
2021/12/02 05:33:46 prdsvr EVENT Flipping snapshots in run and patch editions.
2021/12/02 05:33:47 prdsvr EVENT Performing file system cutover.
2021/12/02 05:34:38 acedisupplier EVENT Waiting for flipping of snapshots to complete.
on Admin node.
=>2021/12/02 05:34:39 acedisupplier EVENT Performing file system cutover.
2021/12/02 05:34:43 prdsvr EVENT Checking if file system cutover completed.
2021/12/02 05:34:43 prdsvr EVENT File System Cutover did not complete on node(s):
<acedisupplier>
-2021/12/02 05:34:43 prdsvr EVENT Will wait for another minute and check again.
2021/12/02 05:35:43 prdsvr EVENT File System Cutover did not complete on node(s):
<acedisupplier>
2021/12/02 05:35:43 prdsvr EVENT Will wait for another minute and check again.
2021/12/02 05:36:43 prdsvr EVENT File System Cutover did not complete on node(s):
<acedisupplier>
2021/12/02 05:36:43 prdsvr EVENT Will wait for another minute and check again.
2021/12/02 05:36:55 acedisupplier EVENT Checking if file system cutover completed.
2021/12/02 05:36:55 acedisupplier EVENT Waiting for Admin Server to start.
2021/12/02 05:36:55 acedisupplier EVENT Starting Admin Server did not complete on
Admin node: <prdsvr>
2021/12/02 05:36:55 acedisupplier EVENT Will wait for another minute and check again.
Broadcast message from [email protected] (Thu Dec 2 05:37:43 2021):
aced environment has changed.
All users must re-source the environment.
2021/12/02 05:37:40 prdsvr EVENT Starting run edition Admin Server.
2021/12/02 05:37:43 prdsvrdb EVENT Compile Edition: V_20211201_2323, In a Loop
2021/12/02 05:37:44 prdsvrdb EVENT Compile Edition: V_20211201_ 2323
2021/12/02 05:37:52 prdsvrdb EVENT Compile Edition: V_20211201 _2323
2021/12/02 05:37:55 acedisupplier EVENT Starting Admin Server did not complete on
Admin node: <prdsvr>
2021/12/02 05:37:55 acedisupplier EVENT Will wait for another minute and check again.
2021/12/02 05:38:55 acedisupplier EVENT Starting Admin Server did not complete on
Admin node: <prdsvr>
2021/12/02 05:38:56 acedisupplier EVENT Will wait for another minute and check again.
2021/12/02 05:38:59 prdsvr EVENT Starting application tier services..
2021/12/02 05:39:57 acedisupplier EVENT Starting application tier services.
Summary report for current adop session:-
Node acedisupplier: Completed successfully
– Cutover status: Completed successfully
Node prdsvr: Completed successfully
– Cutover status: Completed successfully
Adop status:-Â
For more details, run the command: adop -status -detail
[applmgr@prdsvr scripts]$ adop -status -detail
Enter the APPS password:
Connected.
==============================================================
ADOP (C.Delta.12)
Session Id: 35
Command: status
Output: /oracle/aced/EBSFS/fs_ne/EBSapps/log/adop/35/20211202_054604/
adzdshowstatus.out
===============================================================
Node Name Node Type Phase Status Started Finished Elapsed
————— ———- ————— ————— ——————– ——————– ————
prdsvr master PREPARE COMPLETED 2021/12/01 23:23:23 2021/12/01 23:31:07 0:07:44
APPLY COMPLETED 2021/12/01 23:39:26 2021/12/01 23:41:34 0:02:08
FINALIZE COMPLETED 2021/12/01 23:42:49 2021/12/01 23:44:49 0:02:00
CUTOVER COMPLETED 2021/12/01 23:48:16 2021/12/02 05:44:14 5:55:58
CLEANUP NOT STARTED
acedisupplier slave PREPARE COMPLETED 2021/12/01 23:25:25 2021/12/01 23:38:21 0:12:56
APPLY COMPLETED 2021/12/01 23:44:14 2021/12/01 23:44:15 0:00:01
FINALIZE COMPLETED 2021/12/01 23:42:49 2021/12/01 23:44:49 0:02:00
CUTOVER COMPLETED 2021/12/01 23:50:19 2021/12/02 05:45:47 5:55:28
CLEANUP NOT STARTED
After successfully completed adop cutover on master and slave node. Now we will run
adop phase=cleanup.
I have checked that all services are started. Now cleanup is completed.
Summary report for current adop session:
Node acedisupplier: Completed successfully
– Cleanup status: Completed successfully
Node prdsvr: Completed successfully
– Cleanup status: Completed successfully
For more details, run the command: adop -status -detail
The cleanup phase completed successfully.
adop exiting with status = 0 (Success)
You have new mail in /var/spool/mail/applmgr
[applmgr@prdsvr scripts]$
You have new mail in /var/spool/mail/applmgr
Here will check the status to resolve the issue ‘ADOP Cutover failed PRE_CUTOVER_CHECKS_COMPLETED’.
[applmgr@prdsvr scripts]$ adop -status -detail
Enter the APPS password:
Connected.
==============================================================
ADOP (C.Delta.12)
Session Id: 35
Command: status
Output: /oracle/aced/EBSFS/fs_ne/EBSapps/log/adop/35/
20211202_060018
/adzdshowstatus.out
===============================================================
Node Name Node Type Phase Status Started Finished Elapsed
————— ———- ————— ————— ——————– ——————– ————
prdsvr master PREPARE COMPLETED 2021/12/01 23:23:23 2021/12/01 23:31:07 0:07:44
APPLY COMPLETED 2021/12/01 23:39:26 2021/12/01 23:41:34 0:02:08
FINALIZE COMPLETED 2021/12/01 23:42:49 2021/12/01 23:44:49 0:02:00
CUTOVER COMPLETED 2021/12/01 23:48:16 2021/12/02 05:44:14 5:55:58
CLEANUP COMPLETED 2021/12/02 05:51:51 2021/12/02 05:54:15 0:02:24
acedisupplier slave PREPARE COMPLETED 2021/12/01 23:25:25 2021/12/01 23:38:21 0:12:56
APPLY COMPLETED 2021/12/01 23:44:14 2021/12/01 23:44:15 0:00:01
FINALIZE COMPLETED 2021/12/01 23:42:49 2021/12/01 23:44:49 0:02:00
CUTOVER COMPLETED 2021/12/01 23:50:19 2021/12/02 05:45:47 5:55:28
CLEANUP COMPLETED 2021/12/02 05:51:51 2021/12/02 05:54:15 0:02:24
File System Synchronization Type: Full
Generating full adop status report: /oracle/aced/EBSFS/fs_ne/EBSapps/log/adop/35/20211202_060018/adzdshowstatus.out
Cutover failed to stop the web since the PID was already killed
As there was not PID to kill cutover failed trying to stop the web
Hope your issue ‘ADOP Cutover failed PRE_CUTOVER_CHECKS_COMPLETED‘ will resolve after doing the above steps.
👉Related Posts:-
How to fix adop phase=prepare ValidateInventory: [ERROR]: At least one Oracle inventory check has failed.
How to fix txkGenADOPWrapper.pl line 143 ERRORCODE = 1 ERRORCODE_END
How to fix adop phase=prepare error |ssh is not enabled for the following nodes|[UNEXPECTED]ssh is not enabled.
How to fix Adop Fails With Error: ssh is not enabled for the following nodes
adop phase=prepare ORA-20008: No Concurrent Manager is running that can run concurrent program