I can't figure out a problem which has happened recently. During maintenance windows, one of my colleagues stopped Oracle RAC Services with database, after that we are unable to start the database. We cant it start on both nodes.
A couple words 'bout conf: Oracle RAC(Grid Infrastructure Patch Set Update : 11.2.0.3.4) with 2 nodes and Shared directory location which mounted from storagebox.
Problem: crsctl stat res -t shows that instance is STARTING, and I see one process of oracle: oracle 16801 99.1 0.0 235416 15444 ? Rs 03:22 8:25 oracleCRMSDB1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
but nothing happens and no alerts/traces generates under ../trace dir and no other processes of oracle appear as well.
Needful information below (crsctl conf + opatch lsinventory): http://pastebin.com/gxFQ9aTt
+log of grid oraagent here: http://ge.tt/429aGTo1/v/0?c
+log of oracle oraagent: http://ge.tt/2kT6ITo1/v/0?c
oraagent though contain this error:
2014-07-16 03:32:49.269: [ AGFW][2643461888] {1:55831:2}
Created alert : (:CRSAGF00113 : Aborting the command: start for resource: ora.crmsdb.db 1 1
2014-07-16 03:32:49.269: [ora.crmsdb.db][2643461888] {1:55831:2} [start] (:CLSN00110 clsn_agent::abort { 2014-07-16 03:32:49.269: [ora.crmsdb.db][2643461888] {1:55831:2} [start] abort { 2014-07-16 03:32:49.269: [ora.crmsdb.db][2643461888] {1:55831:2} [start] abort command: start 2014-07-16 03:32:49.269: [ora.crmsdb.db][2643461888] {1:55831:2} [start] tryActionLock { 2014-07-16 03:32:51.270: [ora.crmsdb.db][2643461888] {1:55831:2} [start] got lock 2014-07-16 03:32:51.270: [ora.crmsdb.db][2643461888] {1:55831:2} [start] tryActionLock } 2014-07-16 03:32:51.270: [ora.crmsdb.db][2643461888] {1:55831:2} [start] abort } 2014-07-16 03:32:51.270: [ora.crmsdb.db][2643461888] {1:55831:2} [start] (:CLSN00110 clsn_agent::abort }
CRS services were restarted on both nodes a few times, Both servers were rebooted as well.
I will really appreciate any ideas or thoughts! If you need any other info, just ask. Thanks in advance.