26 Ensuring Application Continuity

26.1.1 Key Concepts for Application Continuity

This section describes several terms and concepts that you must understand to use Application Continuity. These terms are used throughout this chapter.

request

A request is a unit of work submitted from the application. It typically corresponds to the SQL and PL/SQL, and other database calls, of a single web request on a single database connection, and it is generally demarcated by the calls made to check-out and check-in the database connection from a connection pool. For recoverable errors, Application Continuity reestablishes the conversation state for a database session and repeats the request.

recoverable error

A recoverable error is an error that arises due to an external system failure, independent of the application session logic that is executing. Recoverable errors occur following planned and unplanned outages of foregrounds, networks, nodes, storage, and databases. The application receives an error code that can leave the application not knowing the status of the last operation submitted. Application Continuity reestablishes database sessions and resubmits the pending work for the class of recoverable errors.

Application Continuity does not resubmit work following call failures due to nonrecoverable errors. An example of a nonrecoverable error that would not be replayed is submission of invalid data values.

commit outcome

A transaction is committed by updating its entry in the transaction table. Oracle Database generates a redo-log record corresponding to this update and writes out this redo-log record. Once this redo-log record is written out to the redo log on disk, the transaction is considered committed at the database. From the client perspective, the transaction is considered committed when an Oracle message (termed Commit Outcome), generated after that redo is written, is received by the client. However, the commit message is not durable. (Transaction Guard, described in Chapter 25, obtains the commit outcome available when it has been lost.)

mutable objects

Mutable objects are nondeterministic function that can obtain a new value every time it is called, and thus their results can change frequently. Mutable objects cause a problem for replay because the results can change at replay. Consider sequence.NEXTVAL and SYSDATE, often used in key values. If a primary key is built with values from these function calls, and is used in later foreign keys or other binds, at replay the same function result must be returned.

Application Continuity provides mutable object value replacement at replay for granted Oracle function calls to provide opaque bind-variable consistency. If the call uses database functions that are mutable, including sequence.NEXTVAL, SYSDATE, SYSTIMESTAMP, and SYSGUID, the original values returned from the function execution are saved and are reapplied at replay.

session state consistency

After a COMMIT statement has executed, if state was changed in that transaction, it is not possible to replay the transaction to reestablish that state if the session is lost. When configuring Application Continuity, the applications are categorized depending on whether the session state after the initial setup is static or dynamic, and thus whether it is correct to continue past a COMMIT operation within a request.

• A session has dynamic state if the session state changes are not fully encapsulated by the initialization, and cannot be fully captured in a callback at failover. After the first transaction completes, failover is internally disabled until the next request begins.

• A session has a static state if all session state changes (for example, NLS settings and PL/SQL package state) occur as part of initialization, and can be encapsulated in a callback at failover.

26.2.1 How Application Continuity Works for Database Sessions

If a recoverable error occurs and if replay is enabled, recovery of the database session is attempted. Application Continuity performs the major steps shown in Figure 26-1. The major steps apply for both unplanned and planned outages, although specific steps vary depending on the type of outage. (For example, for planned outages further optimizations are possible, as explained in Section 26.2.2.6, "Using Application Continuity for Planned Outages".)

Figure 26-1 How Application Continuity Works

Description of "Figure 26-1 How Application Continuity Works"

As shown in Figure 26-1:

1. The client application makes a request, which is passed to a middle tier (such as the JDBC Thin driver, Universal Connection Pool, or WebLogic Server or third party pool) or directly to the database using the JDBC replay driver.

2. The JDBC replay driver issues each call in the request.

3. A FAN unplanned or planned down interrupt or recoverable error occurs. FAN/FCF then aborts the dead physical session.

4. Application Continuity begins the replay and does the following:

   1. Replaces the dead physical session with a new clean session and rewires FAN in case a later error occurs during or after replay.

   2. Prepares for replay by using Transaction Guard to determine the outcome of the in-flight transaction if one was open.

   3. Optionally, calls back using a labeling callback or reconnect callback for the initial state.

   4. Rebuilds the database session, recovering the transactional and nontransactional states, and validating at each step that the data and messages seen by the client driver are the same as those that the client may have seen and used to make a decision.

   5. Ends the replay and returns to runtime mode.

   6. Submits the last queued call.

      This is the last call made when the outage was discovered. During replay, only this call can execute a COMMIT. A COMMIT midway through rebuilding the session aborts replay (excluding autonomous transactions).

5. The response is returned to the application.

   If replay succeeded, the application can continue with the problem masked. If not, the application must handle the original error.

The behavior of Application Continuity after a communication failure depends on the Oracle products and technologies involved. For example:

• If you use Oracle Real Application Clusters or an Active Data Guard farm, after the connection instance is reestablished on another running instance, Application Continuity attempts to rebuild the session and replay the last transaction if there is one in flight.

• If you use Oracle Data Guard and fail over to a standby site, Application Continuity connects to the failover instance and attempts to rebuild the session and replay the last transaction there if a transaction was in flight. (Application Continuity does not replay if the Data Guard switchover or failover has lost data, and if this is not Active Data Guard reader farm with approved lags.)

• If you are using Oracle RAC or Oracle RAC One and not using Data Guard, and if an outage causes a break in all public networks or causes the database or database session to shut down briefly, Application Continuity attempts to rebuild the session and replay the last transaction (if a transaction was in flight) against the database after connectivity is restored.

26.2.2 Actions for Using Application Continuity for Java

Application Continuity for Java is available for general use with the following Oracle technologies:

• JDBC Thin Oracle replay driver

• Universal Connection Pool

• WebLogic Server

A main appeal of Application Continuity for Java is its ability to mask many outages when using the Oracle stack with few or no application changes. Applications must be verified and tested to ensure that they are suitable for replay before releasing with Application Continuity for Java. If you must perform any actions, in most cases these do not involve core application source code changes, but rather configuration changes or providing a callback to disable replay for any code module.

The Application Continuity for Java solution is embedded in Oracle Universal Connection Pool (UCP), and in the Oracle WebLogic Server and Generic data sources. When you use the Oracle connection pools, request boundaries are implicitly marked at check-out and check-in delimiting the size of each replay. However, when you are using third-party connection pools with Oracle JDBC Thin, or when you are using UCP or WebLogic Server but not returning connections to the pools, you may need to perform actions to obtain the benefits of Application Continuity for Java.

Support for Application Continuity is integrated into many Oracle applications, so the features in such applications are used automatically if you set the Application Continuity-related service attributes. For your own applications, however, you also must follow the steps described in this section.

The main actions for ensuring automatic application continuity for an application are the following:

1. Determine whether the application uses Oracle JDBC concrete classes. For Application Continuity to be used, the deprecated concrete classes must be replaced. For information about the deprecation of concrete classes, including actions to take if an application uses them, see My Oracle Support Note 1364193.1 (https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1364193.1).

2. Ensure that you have the necessary CPU and memory resources.

   • CPU: Application Continuity is managed on the client and server sides and needs additional CPU to operate.

     At the client, CPU is used to build proxy objects and for garbage collection (GC).

     At the server, CPU is used for validation. CPU overhead is reduced for platforms with current Intel and Sparc chips where validation is assisted in the hardware.

   • Memory: The replay driver requires more memory than the base driver because the calls are retained until the end of a request. At the end of the request, the calls are released to the garbage collector. This action differs from the base driver that releases closed calls.

     The memory consumption of the replay driver depends on the number of calls per request. If this number is small, then the memory consumption of the replay driver is less, and comparable to the base driver.

     To obtain the best performance, you must set the same value for both the -Xmx and -Xms parameters on the server. For example, if there is sufficient memory, then allocate 4 to 8 GB (or more) of memory for the Virtual Machine (VM), for example, by setting -Xms4g for 4 GB. If the -Xms parameter has a lower value, then the VM also uses a lower value from the operating system, and performance might suffer and garbage collection operations increase.

3. Determine whether the application borrows and returns connections from the WebLogic Server Pool or Universal Connection Pool for each request, or whether to add beginRequest and endRequest APIs to the application's own connection pool to identify request boundaries.

   (Do not use beginRequest and endRequest anywhere other than at request boundaries (check-in). endRequest indicates that the request is complete, and that it is now stateless. Replay starts from the next beginRequest. If there is prior state, it must be reestablished in the callback.)

4. Decide whether to use an optional callback for initializing connections. When using Oracle WebLogic Server or the Universal Connection Pool, connection labeling is recommended. The labeling is used for both runtime and replay.

5. Determine whether the application requires, and therefore needs to configure keeping original values for, SYSDATE, SYSTIMESTAMP, and SYS_GUID and sequences during failover (see Section 26.2.3, "Mutable Objects and Application Continuity").

6. Assess the application style for the session_state_consistency value, and set the appropriate value on the service:

   • If session_state_consistency is Dynamic, the application changes the environment or settings during the request. Replay is disabled after the first COMMIT until the end of replay API is called. Dynamic is the default mode, appropriate for most applications.

   • If session_state_consistency is Static, the application never changes the session state after initial setup. This mode is typical for database agnostic applications that do not use PL/SQL state and do not use ALTER part-way through transactions. Use this mode with caution, and only for "static" applications.

   For more information, see Section 26.2.4, "Session State Consistency," and Section 26.2.4.2, "Static Session State Consistency."

7. Determine whether replay must be explicitly disabled for any code path.

   For example, replay may need to be disabled for requests using external PL/SQL actions (see Section 26.2.2.8, "Disabling Replay in Application Continuity for Java").

8. Follow these configuration guidelines:

   • Use Oracle Database 12c Release 1 (12.1.0.1) or later.

   • Use Universal Connection Pool 12.1 (or later) or WebLogic Server 12.1.2 (or later) configured with the JDBC Replay data source; or for third party applications, including third party JDBC pools, use JDBC replay driver.

     Custom Java pools and standalone Java applications can use the JDBC Replay data source directly. When using custom Java pools and standalone applications, add the beginRequest and endRequest calls.

   • If the application does not borrow and return from the Oracle connection pools, explicitly mark request boundaries. For example, if using custom JDBC pools, WebSphere, TomCat, JBOSS, or other pools, then call beginRequest at check-out and call endRequest at check-in. These APIs can also be used for standalone JDBC applications without a connection pool.

   • Use a single pool with FAN/FCF from the WebLogic Data Source or UCP or third-party pool.

   • Use a database service to connect; never use a SID or an instance name.

   • Use a connection string that sets retries for new incoming connections and a delay between these retries.

   • For the service, set FAILOVER_TYPE to TRANSACTION, COMMIT_OUTCOME to TRUE, and Notification to TRUE. Optionally to find the best connections to use, set GOAL to SERVICE_TIME and CLB_Goal to Short.

Topics:

• When Is Application Continuity Transparent?

• Configuring Oracle JDBC for Application Continuity for Java

• Configuring Oracle Database for Application Continuity for Java

• Registering a Connection Initialization Callback in Application Continuity for Java (optional)

• Delaying the Reconnection in Application Continuity for Java

• Using Application Continuity for Planned Outages

• Running Without Application Continuity

• Disabling Replay in Application Continuity for Java

• Killing or Disconnecting a Session Without Replay

srvctl add service -db codedb -service GOLD -serverpool ora.Srvpool -clbgoal SHORT -rlbgoal SERVICE_TIME -failoverretry 30 -failoverdelay 10 -commit_outcome TRUE -failovertype TRANSACTION -replay_init_time 1800 -retention 86400 -notification TRUE

srvctl add service -db codedb -service GOLD -preferred serv1 -available serv2  -clbgoal SHORT -rlbgoal SERVICE_TIME -failoverretry 30 -failoverdelay 10 -commit_outcome TRUE -failovertype TRANSACTION -replay_init_time 1800 -retention 86400 -notification TRUE

declare
params dbms_service.svc_parameter_array;
begin
params('FAILOVER_TYPE'):='TRANSACTION';
params('REPLAY_INITIATION_TIMEOUT'):=1800;
params('RETENTION_TIMEOUT'):=86400;
params('FAILOVER_DELAY'):=10;
params('FAILOVER_RETRIES'):=30;
params('commit_outcome'):='true';
params('aq_ha_notifications'):='true';
dbms_service.modify_service('[your service]',params);
end;
/

select 
    sys_context('USERENV','DB_NAME') 
    ,sys_context('USERENV','HOST') 
    ,sys_context('USERENV','INSTANCE') 
    ,sys_context('USERENV','IP_ADDRESS') 
    ,sys_context('USERENV','ISDBA')  
    ,sys_context('USERENV','SESSIONID') 
    ,sys_context('USERENV','TERMINAL') 
    ,sys_context('USERENV','SID') 
from dual;

alter system kill session 'sid, serial#, @inst' noreplay;
alter system disconnect session 'sid, serial#, @inst' noreplay

26.2.3 Mutable Objects and Application Continuity

When a request is replayed, the default and desired treatment of mutable objects can vary. A mutable object is a nondeterministic function that can obtain a new value every time it is called. An example of a mutable object use is a call to the SYSTIMESTAMP function. Client applications using Application Continuity can determine whether to keep the original value for mutable functions if the request is replayed.

Support for keeping mutable object values is currently provided for SYSDATE, SYSTIMESTAMP, SYS_GUID, and sequence.NEXTVAL. If the original values are not kept and if different values for these mutable objects are returned to the client, replay is rejected because the client sees different results. If the application can use original values, configure mutable objects using the KEEP clause for owned sequences and GRANT KEEP for other users. (Most applications need sequence values to be kept at replay, for bind variable consistency.)

Table 26-1 shows examples of the treatment of mutable objects by products during replay. (Actual implementation depends on specific products and releases.)

To allow Application Continuity to keep and use original function results at replay:

• The database user running the application might have the KEEP DATE TIME and KEEP SYSGUID privileges granted, and the KEEP SEQUENCE object privilege on each sequence whose value is to be kept. For example:

  grant KEEP DATE TIME to user2;
  grant KEEP SYSGUID to user2;
  grant KEEP SEQUENCE on sales.seq1 to user2;
  

  Note:

  GRANT ALL ON <object> does not include (that is, does not grant the access provided by) the KEEP DATE TIME and KEEP SYSGUID privileges and the KEEP SEQUENCE object privilege.

  Grant privileges related to mutable object support only to application users, and to each application user, grant only the necessary privileges.

  Do not grant DBA privileges to database users running applications for which you want replay to be enabled.

• Sequences in the application can use the KEEP attribute, which keeps the original values of sequence.NEXTVAL for the sequence owner, so that the keys match during replay. Most applications need sequence values to be kept at replay. The following example sets the KEEP attribute for a sequence (in this case, one owned by the user executing the statement; for others, use GRANT KEEP SEQUENCE):

  SQL> CREATE SEQUENCE my_seq KEEP;
  SQL> -- Or, if the sequence already exists but without KEEP:
  SQL> ALTER SEQUENCE my_seq KEEP;
  

  Note:

  Specifying ALTER SEQUENCE ... KEEP/NOKEEP applies to the owner of the sequence. It does not affect other users (not the owner) that have the KEEP SEQUENCE object privileges. If you want NOKEEP for all users, be sure not to grant the KEEP SEQUENCE object privilege to these users (or to revoke it from each if they have been granted it).

• To keep function results (for named functions) at replay, the DBA must grant KEEP privileges to the user invoking the function. This security restriction ensures that it is valid for replay to save and restore function results for code that is not owned by that user.

The following additional considerations apply to granting privileges on mutable objects:

• If a user has the KEEP privilege granted on mutable object values, then the objects inherit mutable access when the SYS_GUID, SYSDATE, and SYSTIMESTAMP functions are called.

• If the KEEP privilege is revoked on mutable values on a sequence object, then SQL or PL/SQL blocks using that object do not allow mutable collection or application for that sequence.

• If granted privileges are revoked between runtime and failover, then the mutable values that are collected are not applied for replay.

• If new privileges are granted between runtime and failover, then mutable values are not collected and these values are not applied for replay.

26.2.4 Session State Consistency

Session state consistency describes how nontransactional state is changed during a request. Examples of session state are NLS settings, optimizer preferences, event settings, PL/SQL global variables, temporary tables, advanced queues, LOBs, and result cache. If nontransactional values change after the request starts, use the default value, Dynamic.

After a COMMIT has executed, if the state was changed in that transaction, it is not possible to replay the transaction to reestablish that state if the session is lost. Applications can be categorized depending on whether the session state after the initial setup or is static or dynamic, and hence whether it is correct to continue past a COMMIT operation.

Dynamic mode is appropriate for almost all applications. If you are unsure, use Dynamic mode. If your customers or users can modify your application, you must use Dynamic mode.

Topics:

• Dynamic Session State Consistency

• Static Session State Consistency

26.2.4.1 Dynamic Session State Consistency

A session has dynamic state if the session state changes are not fully encapsulated by the initialization, and cannot be fully captured in a callback at failover. Once the first transaction completes, failover is internally disabled until the next request starts. In Dynamic session state consistency mode, state changes occur during the request and replay is enabled at the call to beginRequest.

Set the session state consistency mode to Dynamic if the nontransactional session state changes while transactions are executing. Examples of nontransactional session state that can change at runtime are ALTER SESSION, PL/SQL global variables, SYS_CONTEXT, and temporary table contents. If the application changes nontransactional state inside transactions and commits, this state cannot be replayed and the state setting must be Dynamic. When using Dynamic mode for Application Continuity, replay is disabled at COMMIT until the next request begins. Dynamic is the default value.

Figure 26-2 shows the nontransactional session state (NTSS) changes during a request when the session state consistency mode is Dynamic.

Figure 26-2 Dynamic Mode: Nontransaction State Can Change During Request

Description of "Figure 26-2 Dynamic Mode: Nontransaction State Can Change During Request"

As shown in Figure 26-2, replay (that is, Application Continuity) is enabled at the beginRequest call, and is disabled on a COMMIT or an endRequest call. (The use of a restricted call also disables Application Continuity, but this is not shown in the figure.) Also shown in Figure 26-2 is the step logic for three application scenarios: no transaction, a transaction with COMMIT as the last statement, and a transaction with an embedded COMMIT statement.

For the request with no transaction, the logic is as follows:

1. Check out.

2. Replay is disabled at endRequest, at a restricted call, and for an explicit disableReplay call.

3. Begin request and enable replay.

4. Issue one or more SELECT statements and perhaps other PL/SQL statements.

5. (Other actions, indicated by a gap.)

6. Check in.

7. End request and disable replay.

For the request with a transaction with COMMIT as the last statement, the logic is as follows:

1. Check out.

2. Replay is disabled at COMMIT, at endRequest, at a restricted call, and for an explicit disableReplay call.

3. Begin request and enable replay.

4. Issue one or more SELECT statements and perhaps other PL/SQL statements.

5. The transaction begins.

6. (Other actions, indicated by a gap.)

7. Commit (which disables replay).

8. Check in.

9. End request.

For the request with a transaction with an embedded COMMIT statement, the logic is as follows:

1. Check out.

2. Replay is disabled at COMMIT, at endRequest, at a restricted call, and for an explicit disableReplay call.

3. Begin request and enable replay.

4. Issue one or more SELECT statements and perhaps other PL/SQL statements.

5. The transaction begins.

6. (Other actions, indicated by a gap.)

7. Commit (which disables replay).

8. (Other actions, indicated by a gap during which Application Continuity is not covering the application.)

9. Check in.

10. End request.

26.2.4.2 Static Session State Consistency

Set the session state consistency mode to Static if all nontransactional state changes, such as NLS settings, SYS_CONTEXT, PL/SQL variables, and optimizer preferences, are set as part of the initialization once per request, and if this session state does not change during transactions. The settings can be established once per connection at connection establishment with UCP labeling, for example, or at each checkout from a pool. These settings must be repeated in the replay callback. When using Static mode for Application Continuity, transactional failover continues beyond the first transaction of a request.

Static mode is not supported for applications that use calls that change nontransactional state in requests. Specific examples of such calls include:

• PL/SQL subprograms

• SYS_CONTEXT

• Hints

• DDL operations

• Autocommit

• ALTER SESSION

• ALTER SYSTEM

Specify static mode with caution. Use static mode only when the application does not change the NTSS (nontransactional session state) inside transactions. Declaring the session state consistency mode as Static indicates that it is safe to continue beyond the first COMMIT in a request. Dynamic mode is appropriate for most applications. Do not use static mode if users or customers can modify or customize the application.

Figure 26-3 shows the NTSS (nontransactional session state) remaining constant (that is, not changing) during a request when the session state consistency mode is Static.

Figure 26-3 Static Mode: Nontransactional State Never Changes During Request

Description of "Figure 26-3 Static Mode: Nontransactional State Never Changes During Request"

As shown in Figure 26-3, replay (that is, Application Continuity) is enabled at the beginRequest call, and is disabled on a restricted call or on a disableReplay or endRequest call. Also shown in Figure 26-3 is the step logic for three application scenarios: no transaction, one or more transactions each ending with COMMIT as the last statement, and a transaction with a COMMIT statement followed by a transaction with a restricted call that disables Application Continuity.

For the request with no transaction, the logic is as follows:

1. Check out.

2. Replay is disabled at endRequest, at a restricted call, and for an explicit disableReplay call.

3. Begin request and enable replay.

4. Issue one or more SELECT statements and perhaps other PL/SQL statements.

5. (Other actions, indicated by a gap.)

6. Check in.

7. End request and disable replay.

For the request with one or more transactions (each with COMMIT as the last statement), the logic is as follows:

1. Check out.

2. Replay is disabled at COMMIT, at endRequest, at a restricted call, and for an explicit disableReplay call.

3. Begin request and enable replay.

4. Issue one or more SELECT statements and perhaps other PL/SQL statements.

5. The transaction begins.

6. The transaction commits.

7. The transaction is purged.

8. (For each additional transaction, steps 4 through 7 occur.)

9. (Other actions, indicated by a gap.)

10. Check in.

11. End request.

For the request with a transaction with a COMMIT followed by a transaction with a restricted call, the logic is as follows:

1. Check out.

2. Replay is disabled at COMMIT, at endRequest, at a restricted call, and for an explicit disableReplay call.

3. Begin request and enable replay.

4. Issue one or more SELECT statements and perhaps other PL/SQL statements.

5. The transaction begins.

6. The transaction commits.

7. The transaction is purged.

8. The second transaction begins.

9. The transaction makes a restricted call, which causes Application Continuity to be disabled.

10. The transaction is purged.

11. (Other actions, indicated by a gap.)

12. Check in.

13. End request.