23 Troubleshooting RMAN Operations

RMAN Error Message Numbers

Table 23-2 indicates the error ranges for common RMAN error messages, all of which are described in Oracle Database Error Messages.

ORA-19511: Media Manager Errors

If a media manager error occurs, ORA-19511 is signaled, and the media manager is expected to provide RMAN a descriptive error. RMAN will display the error passed back to it by the media manager. For example, you might see this:

ORA-19511: Error received from media manager layer, error text:
   sbtpvt_open_input: file .* does not exist or cannot be accessed, errno = 2

The message from the media manager should provide you with enough information to let you fix the root problem. If it does not, you should refer to the documentation for your media manager or contact your media management vendor support representative for further information. ORA-19511 errors originate with the media manager, not with Oracle Database. The database merely passes the message on from the media manager. The cause can be addressed only by the media management vendor.

If you are still using an SBT 1.1-compliant media management layer, you may see some additional error message text. Output from an SBT 1.1-compliant media management layer is similar to the following:

ORA-19507: failed to retrieve sequential file, handle="c-140148591-20031014-06", parms=""
ORA-27007: failed to open file
Additional information: 7000
Additional information: 2
ORA-19511: Error received from media manager layer, error text:
   SBT error = 7000, errno = 0, sbtopen: backup file not found

The "Additional information" provided uses error codes specific to SBT 1.1. The values displayed correspond to the media manager message numbers and error text listed in Table 23-3. RMAN again signals the error, as an ORA-19511 Error received from media manager layer error, and a general error message related to the error code returned from the media manager and including the SBT 1.1 error number is then displayed.

The SBT 1.1 error messages are listed here for your reference. Table 23-3 lists media manager message numbers and their corresponding error text. In the error codes, O/S stands for operating system. The errors marked with an asterisk (*) are internal and should not typically be seen during normal operation.

Interpreting RMAN Errors: Example

You attempt a backup of tablespace users and receive the following message:

Starting backup at 29-AUG-02
using channel ORA_DISK_1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup command at 08/29/2002 15:14:03
RMAN-20202: tablespace not found in the recovery catalog
RMAN-06019: could not translate tablespace name "USESR"

The RMAN-03002 error indicates that the BACKUP command failed. You read the last two messages in the stack first and immediately see the problem: no tablespace user appears in the recovery catalog because you mistyped the name.

Interpreting Server Errors: Example

Assume that you attempt to recover a tablespace and receive the following errors:

RMAN> RECOVER TABLESPACE users;

Starting recover at 29-AUG-07
using channel ORA_DISK_1

starting media recovery
media recovery failed
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 08/29/2007 15:18:43
RMAN-11003: failure during parse/execution of SQL statement: alter database recover if needed tablespace USERS
ORA-00283: recovery session canceled due to errors
ORA-01124: cannot recover data file 8 - file is in use or recovery
ORA-01110: data file 8: '/oracle/oradata/trgt/users01.dbf'

As suggested, you start reading from the bottom up. The ORA-01110 message explains there was a problem with the recovery of datafile users01.dbf. The second error indicates that the database cannot recover the datafile because it is in use or already being recovered. The remaining RMAN errors indicate that the recovery session was canceled due to the server errors. Hence, you conclude that because you were not already recovering this datafile, the problem must be that the datafile is online and you need to take it offline and restore a backup.

Interpreting SBT 2.0 Media Management Errors: Example

Assume that you use a tape drive and see the following output during a backup job:

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
ORA-19624: operation failed, retry possible
ORA-19507: failed to retrieve sequential file, handle="/tmp/mydir", parms=""
ORA-27029: skgfrtrv: sbtrestore returned error
ORA-19511: Error received from media manager layer, error text:
  sbtpvt_open_input:file /tmp/mydir does not exist or cannot be accessed, errno=2

The error text displayed following the ORA-19511 error is generated by the media manager and describes the real source of the failure. See the media manager documentation to interpret this error.

Interpreting SBT 1.1 Media Management Errors: Example

Assume that you use a tape drive and see the following output during a backup job:

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on c1 channel at 09/04/2007 13:18:19
ORA-19506: failed to create sequential file, name="07d36ecp_1_1", parms=""
ORA-27007: failed to open file
SVR4 Error: 2: No such file or directory
Additional information: 7005
Additional information: 1
ORA-19511: Error received from media manager layer, error text:
   SBT error = 7005, errno = 2, sbtopen: system error

The main information of interest returned by SBT 1.1 media managers is the error code in the "Additional information" line:

Additional information: 7005

Referring to Table 23-3, you discover that error 7005 means that the media management device is busy. So, the media management software is not able to write to the device because it is in use or there is a problem with it.

Matching Server Sessions with Channels When One RMAN Session Is Active

When only one RMAN session is active, the easiest method for determining the server session ID for an RMAN channel is to execute the following query on the target database while the RMAN job is executing:

COLUMN CLIENT_INFO FORMAT a30
COLUMN SID FORMAT 999
COLUMN SPID FORMAT 9999

SELECT s.SID, p.SPID, s.CLIENT_INFO
FROM   V$PROCESS p, V$SESSION s
WHERE  p.ADDR = s.PADDR
AND    CLIENT_INFO LIKE 'rman%';

The following shows sample output:

SID SPID         CLIENT_INFO
---- ------------ ------------------------------
  14 8374         rman channel=ORA_SBT_TAPE_1

If you set an ID using the RMAN SET COMMAND ID command instead of using the system-generated default ID, then search for that value in the CLIENT_INFO column instead of 'rman%'.

Matching Server Sessions with Channels in Multiple RMAN Sessions

If more than one RMAN session is active, then it is possible for the V$SESSION.CLIENT_INFO column to yield the same information for a channel in each session. For example:

SID SPID         CLIENT_INFO
---- ------------ ------------------------------
  14 8374         rman channel=ORA_SBT_TAPE_1
   9 8642         rman channel=ORA_SBT_TAPE_1

In this case, you have the following methods for determining which channel corresponds to which SID value.

Components of an RMAN Session

The nature of an RMAN session depends on the operating system. In UNIX, an RMAN session has the following processes associated with it:

• The RMAN client process itself

• The default channel, the initial connection to the target database

• One target connection to the target database corresponding to each allocated channel

• The catalog connection to the recovery catalog database, if you use a recovery catalog

• An auxiliary connection to an auxiliary instance, during DUPLICATE or TSPITR operations

• A polling connection to the target database, used for monitoring RMAN command execution on the various allocated channels. By default, RMAN makes one polling connection. RMAN makes additional polling connections if you use different connect strings in the ALLOCATE CHANNEL or CONFIGURE CHANNEL commands. One polling connection exists for each distinct connect string used in the ALLOCATE CHANNEL or CONFIGURE CHANNEL command.

Process Behavior During a Suspended Job

RMAN usually stops responding because one of the channel connections is waiting in the media manager code for a tape resource. The catalog connection and the default channel appear to suspend, because they are waiting for RMAN to tell them what to do. Polling connections seem to be in an infinite loop while polling the RPC under the control of the RMAN process.

If you terminate the RMAN process itself, then you also terminate the catalog connection, the auxiliary connection, the default channel, and the polling connections. If target and auxiliary connections are suspended but not while executing media manager code, they also terminate. If either the target connection or any of the auxiliary connections are executing in the media management layer, then they will not terminate until the processes are manually terminated at the operating system level.

Not all media managers can detect the termination of the Oracle Database process. Those which cannot may keep resources busy or continue processing. Consult your media manager documentation for details.

Terminating the catalog connection does not cause the RMAN process to terminate because RMAN is not performing catalog operations while the backup or restore is in progress. Removing default channel and polling connections causes the RMAN process to detect that one of the channels is no longer present and then exits. In this case, the connections to the unresponsive channels remain active as described previously.

Terminating an RMAN Session: Basic Steps

After the unresponsive channels in the media manager code are terminated, the RMAN process detects this termination and exits, removing all connections except target connections that are still operative in the media management layer. The warning about the media manager resources still applies in this case.

To terminate an Oracle Database process that is not responding in the media manager:

1. Query V$SESSION and V$SESSION_WAIT as described in "Using V$ Views for RMAN Troubleshooting". For example, execute the following query:

   COLUMN EVENT FORMAT a17
   COLUMN SECONDS_IN_WAIT FORMAT 999
   COLUMN STATE FORMAT a10
   COLUMN CLIENT_INFO FORMAT a30
   
   SELECT p.SPID, s.EVENT, s.SECONDS_IN_WAIT AS SEC_WAIT, 
          sw.STATE, s.CLIENT_INFO
   FROM   V$SESSION_WAIT sw, V$SESSION s, V$PROCESS p
   WHERE  sw.EVENT LIKE '%MML%'
   AND    s.SID=sw.SID
   AND    s.PADDR=p.ADDR;
   

   Examine the SQL output to determine which SBT functions are waiting. For example, the output may be as follows:

   SPID EVENT             SEC_WAIT   STATE      CLIENT_INFO
   ---- -----------------  ---------- ---------- -----------------------------
   8642 Backup:MML write   600        WAITING    rman channel=ORA_SBT_TAPE_1
   8374 Backup:MML write   600        WAITING    rman channel=ORA_SBT_TAPE_2
   

2. Using operating system-level tools appropriate to your platform, end the unresponsive sessions. For example, on Linux execute a kill -9 command:

   % kill -9 8642 8374
   

   Some platforms include a command-line utility called orakill that enables you to terminate a specific thread. From a command prompt, run the following command, where sid identifies the database instance to target, and the thread_id is the SPID value from the query in Step 1:

   orakill sid thread_id
   

3. Check that the media manager also clears its processes. If any remain, the next backup or restore operation may freeze again, due to the previous problems in the backup or restore operation. In some media managers, the only solution is to shut down and restart the media manager. If the documentation from the media manager does not provide the needed information, contact technical support for the media manager.

   See Also:

   Your operating system-specific documentation for the relevant commands