D How to Upgrade to Oracle Grid Infrastructure 12c Release 1

D.6.1 Verify System Readiness for Patches and Upgrades

Use the Release Notes and Cluster Verification Utility to make sure you are ready to patch or upgrade your software.

1. If you are completing a patch update of Oracle Clusterware or Oracle ASM, then after you download the patch software and before you start to patch or upgrade your software installation, review the Patch Set Release Notes that accompany the patch to determine if your system meets the system requirements for the operating system and the hardware.

2. Use the Cluster Verification Utility (CVU) to assist you with system checks in preparation for patching or upgrading.

   You can run CVU before starting the upgrade, however, the installer runs the appropriate CVU checks automatically, and prompts you to fix problems before proceeding with the upgrade.

D.6.2 Gather the Necessary System Information

Ensure that you have the information you will need when installing Oracle Grid Infrastructure into a new home directory, including the following:

• The Oracle base location for Oracle Clusterware

• An Oracle Grid Infrastructure home location that is different from your existing Oracle Clusterware home (or Grid home) location

• SCAN name and addresses, and other network addresses, as described in Chapter 4

• Privileged user operating system groups, as described in Section 5.1, "Creating Groups and Users for Oracle Grid Infrastructure and Oracle RAC"

• Local Administrator user access, or access as the user who performed the previous Oracle Clusterware installation

D.6.3 Upgrade to the Minimum Required Oracle Clusterware Release

If your current Oracle Clusterware installation is not currently running at least the minimum supported release, then a prerequisite check failure is reported. You must upgrade your current Oracle Clusterware installation to at least the minimum supported release before upgrading.

To upgrade existing Oracle Clusterware installations to a standard configuration Oracle Grid Infrastructure 12c cluster, your release must be greater than or equal to Oracle Clusterware 10g Release 1 (10.1.0.5), Oracle Clusterware 10g Release 2 (10.2.0.3), Oracle Grid Infrastructure 11g Release 1 (11.1.0.6), or Oracle Grid Infrastructure 11g Release 2 (11.2.0.2) or later.

• Upgrade your current Oracle Clusterware installation to at least the minimum supported release before upgrading to Oracle Grid Infrastructure 12c Release 1.

  For example, if you plan to upgrade your Oracle Clusterware 11g Release 1 installation to Oracle Clusterware 12c Release 1 (12.1) and your current Oracle Clusterware installation is release 11.1.0.2, then you must upgrade your current Oracle Clusterware installation to release 11.1.0.6 or higher before starting the upgrade.

D.6.4 Unset Environment Variables Used By Oracle Software

If you have had an existing installation on your system, and you are using the same user account to install Oracle Grid Infrastructure 12c Release 1 (12.1), then for the Oracle installation user you must unset environment variables that are connected with Oracle software homes.

Before upgrading, you must unset the following environment variables:

• ORACLE_BASE

• ORACLE_HOME

• ORACLE_SID

• ORA_NLS10

• TNS_ADMIN

• ORA_CRS_HOME

If you have set ORA_CRS_HOME as an environment variable, following instructions from Oracle Support, then unset it before starting an installation or upgrade. You should never use ORA_CRS_HOME as an environment variable except under explicit direction from Oracle Support.

1. From the Start menu, right click My Computer and select Properties.

2. In the System Properties window, select Advanced, then click the Environment Variables button.

3. To unset an environment variable, select the variable then click Delete.

4. When finished unsetting the environment variables, click OK.

5. Check to ensure that the user profile for the Oracle Installation User does not set any of these environment variables.

D.7.1 About the CVU Grid Upgrade Validation Command Options

You can use the Cluster Verification Utility (CVU) to validate your system readiness before upgrading.

You can run upgrade validations in one of two ways:

• Run OUI, and allow the CVU validation built into OUI to perform system checks

• Run the CVU manual script cluvfy.bat script to perform system checks

To use OUI to perform pre-install checks, run the installation as you normally would. OUI starts CVU, and performs system checks as part of the installation process. Selecting OUI to perform these checks is particularly appropriate if you think you have completed preinstallation checks, and you want to confirm that your system configuration meets minimum requirements for installation.

Navigate to the staging area for the upgrade, where the runcluvfy.bat command is located, and run the following command to check the readiness of your Oracle Clusterware installation for upgrades:

runcluvfy.bat stage -pre crsinst -upgrade

Running runcluvfy.bat with the -pre crsinst -upgrade options performs system checks to confirm if the cluster is in a correct state for upgrading from an existing clusterware installation.

The runcluvfy command uses the following syntax, where variable content is indicated by italics:

runcluvfy.bat stage -pre crsinst -upgrade [-rolling] -src_crshome 
src_Gridhome -dest_crshome dest_Gridhome -dest_version dest_version
[-verbose]

The options are:

• -rolling

  Use this flag to verify readiness for rolling upgrades.

• -src_crshome src_Gridhome

  Use this flag to indicate the location of the source Oracle Clusterware or Grid home that you are upgrading, where src_Gridhome is the path to the home to upgrade.

• -dest_crshome dest_Gridhome

  Use this flag to indicate the location of the upgrade Grid home, where dest_ Gridhome is the path to the Grid home.

• -dest_version dest_version

  Use the dest_version flag to indicate the release number of the upgrade, including any patchset. The release number must include the five digits designating the release to the level of the platform-specific patch. For example: 12.1.0.1.0.

• -verbose

  Use the -verbose flag to produce detailed output of individual checks

D.7.2 Example of Verifying System Upgrade Readiness for Grid Infrastructure

You can use the runcluvfy.bat command to check your system before upgrading.

Verify that the permissions required for installing Oracle Clusterware have been configured by running the following command:

C:\> runcluvfy.bat stage -pre crsinst -upgrade -rolling 
-src_crshome C:\app\11.2.0.3\grid -dest_crshome C:\app\12.1.0\grid
-dest_version 12.1.0.2.0 -verbose

D.9.1 Performing a Standard Upgrade from an Earlier Release

Use the following procedure to upgrade Oracle Clusterware or Oracle Automatic Storage Management (Oracle ASM) from an earlier release to a later release.

1. If there are non-clustered, or single-instance, Oracle databases that use Oracle ASM running on any of the nodes in the cluster, they must be shut down before you start the upgrade. Listeners associated with those databases do not have to be shut down.

   Note:

   Oracle recommends that you leave Oracle Real Application Clusters (Oracle RAC) instances running during the Oracle Clusterware upgrade. During the upgrade process, the database instances on the node being upgraded are stopped and started automatically during the upgrade process.

2. Start the installer, and select the option to upgrade an existing Oracle Clusterware and Oracle ASM installation.

3. On the node selection page, select all nodes.

   Note:

   In contrast with releases prior to Oracle Clusterware 11g Release 2, all upgrades are rolling upgrades, even if you select all nodes for the upgrade. If you are upgrading from Oracle Clusterware 11g, then all nodes are selected by default. You cannot select or de-select the nodes.

4. Select installation options as prompted.

5. Confirm your selections, and then the upgrade scripts are run automatically.

   Note:

   If you receive an error indicating the server must be restarted before the installation completes, see Section A.12.2, "Completing Failed or Interrupted Upgrades."

6. Oracle recommends that you upgrade Oracle ASM at the same time that you upgrade Oracle Clusterware.

   Until Oracle ASM is upgraded, Oracle databases that use Oracle ASM cannot be created and the Oracle ASM management tools in the Oracle Grid Infrastructure 12c Release 1 (12.1) home (for example, srvctl) do not work.

   • After the Oracle Clusterware upgrade completes, if you are upgrading from a release earlier than Oracle ASM 11g Release 2 (11.2.0.2), and you left the check box for ASM Configuration Assistant (ASMCA) selected, as is the default, then the installer starts ASMCA to upgrade Oracle ASM. If you unchecked the box on the interview stage of the upgrade, then ASMCA is not run automatically.

   • If an earlier release of Oracle ASM is installed, then the installer starts ASMCA to upgrade Oracle ASM to 12c Release 1 (12.1). You can upgrade Oracle ASM at this time, or upgrade it later.

   Note:

   After the upgrade, if you set the Oracle Cluster Registry (OCR) backup location manually to the earlier release Oracle Clusterware home (CRS home), then you must change the OCR backup location to the Oracle Grid Infrastructure home (Grid home). If you did not set the OCR backup location manually, then this issue does not concern you.

   Because upgrades of Oracle Clusterware are out-of-place upgrades, the Oracle Clusterware home for the earlier release cannot be the location of the OCR backups. Backups in the old Oracle Clusterware home could be deleted.

7. Because the Oracle Grid Infrastructure home is in a different location than the former Oracle Clusterware and Oracle ASM homes, update any scripts or applications that use utilities or other files that reside in the Oracle Clusterware and Oracle ASM homes.

D.9.2 Completing an Oracle Clusterware Upgrade when Nodes Become Unreachable

If some nodes become unreachable in the middle of an upgrade, then you cannot complete the upgrade, because the upgrade did not complete successfully on the unreachable nodes. Because the upgrade is incomplete, Oracle Clusterware remains in the earlier release.

1. Confirm that the upgrade is incomplete by entering the command:

   crsctl query crs activeversion
   

2. To resolve the incomplete upgrade, run the rootupgrade.bat command with the -force flag on any of the nodes where the rootupgrade.bat script has already completed as follows:

   Grid_home/rootupgrade -force

   For example, as the Oracle Installation User for Oracle Grid Infrastructure, run the following command:

   C:\> C:\app\12.1.0\grid\rootupgrade -force
   

   The force cluster upgrade has the following limitations:

   • All active nodes must be upgraded to the newer release

   • All inactive nodes (accessible or inaccessible) may be either upgraded or not upgraded

   • For inaccessible nodes, after patch set upgrades, you can delete the node from the cluster. If the node becomes accessible later, and the patch version upgrade path is supported, then you can upgrade it to the new patch version.

   This command forces the upgrade to complete.

3. Verify that the upgrade has completed by using the command crsctl query crs activeversion.

   The active release should be the upgrade release.

D.9.3 Completing the Upgrade of Inaccessible Nodes After Forcing an Upgrade

Starting with Oracle Grid Infrastructure 12c, after you complete a force cluster upgrade command, you can join inaccessible nodes to the cluster as an alternative to deleting the nodes, which was required in earlier releases.

To use this option, Oracle Grid Infrastructure 12c Release 1 (12.1) software must already be installed on the nodes.

1. Log in as the Oracle Installation User for Oracle Grid Infrastructure on one of the nodes that is a member of the cluster.

2. Change directory to the crs\install directory in the Oracle Grid Infrastructure 12c Release 1 (12.1) Grid home.

   C:\> cd app\12.1.0\grid\crs\install
   

3. Run the following PERL command, where existing_node is the node in the cluster on which you run the command, and upgraded_node is the inaccessible or unreachable node that you want to join to the cluster:

   C:\> rootupgrade.bat -join -existing_node upgraded_node
   

   Note:

   The -join operation is not supported for Oracle Clusterware releases earlier than 11.2.0.2.0. In such cases, delete the node and add it to the cluster using the addNode command.

D.9.4 Changing the First Node for Install and Upgrade

If the first node becomes inaccessible, you can force another node to be the first node for installation or upgrade.

• Installation: If gridonfig.bat fails to complete on the first node, run the following command on another node using the -force option:

  Grid_home\crs\config\gridconfig.bat -force -first
  

• Upgrade: If gridonfig.bat fails to complete on the first node, run the following command on another node using the -force option:

  Grid_home\crs\config\gridconfig.bat -upgrade -force -first
  

D.10.1 Preparing to Complete a Separate Upgrade of Oracle ASM

If you must complete upgrade of Oracle ASM separately, then note the following requirements:

• The active release of Oracle Clusterware must be 12c Release 1 (12.1). To determine the active release, enter the following command from the Grid home:

  C:\..\bin> crsctl query crs activeversion
  

• You can upgrade a single instance Oracle ASM installation to a clustered Oracle ASM installation. However, you can only upgrade an existing single instance Oracle ASM installation if you run the installation from the node on which Oracle ASM is installed. You cannot upgrade a single instance Oracle ASM installation on a remote node.

• You must ensure that any rebalance operations on your existing Oracle ASM installation are completed before starting the upgrade process.

• During the upgrade process, you alter the state of the Oracle ASM instances to an upgrade state. Because this upgrade state limits Oracle ASM operations, you should complete the upgrade process soon after you begin. The following operations are allowed when an Oracle ASM instance is in the upgrade state:

  • Diskgroup mounts and dismounts

  • Opening, closing, resizing, or deleting database files

  • Recovering instances

  • Queries of fixed views and packages: Users are allowed to query fixed views and run anonymous PL/SQL blocks using fixed packages, such as DBMS_DISKGROUP)

• You do not need to shut down database clients unless they are on Oracle ACFS.

D.10.2 Upgrading Oracle ASM Separately from an Oracle Grid Infrastructure Upgrade

Complete the following tasks if you must upgrade from an Oracle ASM release where Oracle ASM was installed in a separate Oracle home, or if the Oracle ASM portion of Oracle Grid Infrastructure upgrade failed to complete.

1. On the node you plan to start the upgrade, set the environment variable ASMCA_ROLLING_UPGRADE to true. For example:

   C:\> set ASMCA_ROLLING_UPGRADE=true
   

2. From the Oracle Grid Infrastructure 12c Release 1 (12.1) home, start ASMCA. For example:

   C:\> cd app\12.1\grid\bin
   C:\..bin> asmca.bat
   

3. Select Upgrade.

   Oracle ASM Configuration Assistant (ASMCA) upgrades Oracle ASM in succession for all nodes in the cluster.

4. After you complete the upgrade, run the command to unset the ASMCA_ROLLING_UPGRADE environment variable.

   C:\> set ASMCA_ROLLING_UPGRADE=
   

D.11.1 About Individual (One-Off) Oracle ASM Patches

Individual patches are called one-off patches.

An Oracle ASM one-off patch is available for a specific release of Oracle ASM. If a patch you want is available, then you can download the patch and apply it to Oracle ASM using the OPatch Utility.

The OPatch inventory keeps track of the patches you have installed for your release of Oracle ASM. If there is a conflict between the patches you have installed and patches you want to apply, then the OPatch Utility advises you of these conflicts. See Section D.11.3, "Patching Oracle ASM to a Software Patch Level" for information about applying patches to Oracle ASM using the OPatch Utility.

D.11.2 About Oracle Grid Infrastructure Software Patch Levels

The software patch level for Oracle Grid Infrastructure represents the set of all one-off patches applied to the Oracle Grid Infrastructure software release, including Oracle ASM. The release is the release number, in the format of major, minor, and patch set release number.

For example, with the release number 12.1.0.1, the major release is 12, the minor release is 1, and 0.0 is the patch set number. With one-off patches, the major and minor release numbers remain the same, though the patch levels change each time you apply or roll back an interim patch.

As with standard upgrades to Oracle Grid Infrastructure, at any given point in time for normal operation of the cluster, all the nodes in the cluster must have the same software release and patch level. Because one-off patches can be applied as rolling upgrades, all possible patch levels on a particular software release are compatible with each other.

D.11.3 Patching Oracle ASM to a Software Patch Level

1. Download the patch you want to apply from My Oracle Support:

   1. Go to https://support.oracle.com.

   2. Select the Patches and Updates tab to locate the patch.

      To locate patch bundles, you can perform a Product or Family (Advanced) search for your platform and software release.

      Oracle recommends that you select Recommended Patch Advisor, and enter the product group, release, and platform for your software. My Oracle Support provides you with a list of the most recent patches and critical patch updates (CPUs).

   3. Place the patch in an accessible directory, such as C:\downloads.

2. Change directory to the opatch directory in the Grid home.

   For example:

   C:\> cd app\12.1.0\grid\opatch
   

3. Review the patch documentation for the patch you want to apply, and complete all required steps before starting the patch upgrade.

4. Follow the instructions in the patch documentation to apply the patch.

D.13.1 About Downgrading Oracle Clusterware After an Upgrade

Downgrading Oracle Clusterware restores the Oracle Clusterware configuration to the state it was in before the Oracle Clusterware 12c Release 1 (12.1) upgrade. Any configuration changes you performed during or after the Oracle Grid Infrastructure 12c Release 1 (12.1) upgrade are removed and cannot be recovered.

In the downgrade procedures, the following variables are used:

• local node is the first node on which you started the upgrade process.

• remote nodes are all other nodes where you started the upgrade process.

To restore Oracle Clusterware to the earlier release, use the downgrade procedure for the release to which you want to downgrade.

D.13.2 Downgrading to Oracle Clusterware 11g Releases Prior to Release 11.2.0.2

These steps describe how to restore Oracle Clusterware to the earlier release.

1. If the rootupgrade script failed on a node other than the node on which you started the upgrade (the first node), then downgrade the node where the upgrade failed:

   Grid_home\crs\install\rootcrs.bat -downgrade
   

2. On all other nodes where the rootupgrade script ran successfully, except the first node, as an Administrator user, use the following command:

   Grid_home\crs\install\rootcrs.bat -downgrade [-force]
   

   This command stops the Oracle Clusterware 12c Release 1 (12.1) resources, and shuts down the Oracle Grid Infrastructure 12c Release 1 (12.1) stack.

3. After the rootcrs.bat -downgrade script has completed on all non-first nodes, on the first node use the following command syntax:

   Grid_home\crs\install\rootcrs.bat -downgrade -lastnode [-force]
   

   For example:

   C:\app\12.1.0\grid\crs\install> rootcrs.bat -downgrade -lastnode 
   

   Note:

   With Oracle Grid Infrastructure 12c, you no longer need to provide the location of the earlier release Grid home or release number.

   This script downgrades the OCR. To stop a partial or failed Oracle Grid Infrastructure 12c Release 1 (12.1) installation and restore the earlier release Oracle Clusterware, use the -force flag with this command. Run this command from a directory that has write permissions for the Oracle Grid Infrastructure installation user.

4. On any of the cluster member nodes where the rootcrs script has run successfully:

   1. Log in as the Oracle Grid Infrastructure installation owner.

   2. Use the following command to start the installer, where C:\app\12.1.0\grid represents the location of the new (upgraded) Grid home:

      setup.exe -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList
        -silent CRS=false ORACLE_HOME=C:\app\12.1.0\grid
      

      Add the flag -cfs if the Grid home is a shared home.

5. On any of the cluster member nodes where the upgrade completed successfully:

   1. Log in as the Oracle Grid Infrastructure installation owner.

   2. Use the following command to start the installer, where the path you provide for the flag ORACLE_HOME is the location of the home directory from the earlier Oracle Clusterware installation.

      For example, if the location of the earlier Oracle Clusterware installation is C:\app\crs, the you would use a command similar to the following:

      C:\> cd C:\app\12.1.0\grid\oui\bin
      C:\..bin> setup.exe -nowait -waitforcompletion -ignoreSysPrereqs
         -updateNodeList -silent CRS=true ORACLE_HOME=C:\app\crs
      

   3. For downgrades to 11.1 and earlier releases

      If you are downgrading to Oracle Clusterware 11g Release 1 (11.1) or an earlier release, then you must run root.bat manually from the earlier release Oracle Clusterware home to complete the downgrade after you complete step b.

      OUI prompts you to run root.bat manually from the earlier release Oracle Clusterware installation home in sequence on each member node of the cluster to complete the downgrade. After you complete this task, downgrade is completed.

      Running root.bat from the earlier release Oracle Clusterware installation home restarts the Oracle Clusterware stack, starts all the resources previously registered with Oracle Clusterware in the older release, and configures the old initialization scripts to run the earlier release Oracle Clusterware stack.

6. The installer stops the Oracle Grid Infrastructure 12c software stack. The Windows services used by Oracle Grid Infrastructure are recreated to refer to the Grid home for the older software release by one of the scripts during the downgrade process.

D.13.3 Downgrading to Oracle Clusterware 11g Release 2 (11.2.0.2) or Later Release

Use these steps to restore Oracle Clusterware to the earlier release.

1. On all remote nodes, as an Administrator user, use the command syntax Grid_home\crs\install\rootcrs.bat -downgrade [-force] to stop the Oracle Clusterware 12c Release 1 (12.1) resources, and shut down the Oracle Grid Infrastructure 12c Release 1 (12.1) stack.

   For example:

   C:\app\12.1.0\grid\crs\install> rootcrs.bat -downgrade
   

   To stop a partial or failed Oracle Grid Infrastructure 12c Release 1 (12.1) installation and restore the earlier release of Oracle Clusterware, use the -force flag with this command.

2. After the rootcrs.bat -downgrade script has completed on all remote nodes, on the local node use the following command syntax:

   Grid_home\crs\install\rootcrs.bat -downgrade -lastnode [-force]
   

   For example:

   C:\app\12.1.0\grid\crs\install> rootcrs.bat -downgrade -lastnode 
   

   Note:

   Starting with Oracle Grid Infrastructure 12c, you no longer need to provide the location of the earlier release Grid home or release number.

   This script downgrades the OCR. To stop a partial or failed Oracle Grid Infrastructure 12c Release 1 (12.1) installation and restore the earlier release of Oracle Clusterware, use the -force flag with this command.

3. On each cluster member node where the upgrade to Oracle Grid Infrastructure 12c has completed successfully:

   1. Log in as an Administrator user or the Oracle Installation user.

   2. Use the following command to start the installer, where C:\app\12.1.0\grid is the location of the new (upgraded) Grid home (12.1):

      setup.exe -updateNodeList -silent CRS=false ORACLE_HOME=C:\app\12.1.0\grid
      

      Add the flag -cfs if the Grid home is a shared home.

   3. Use the following command to start the installer from the Grid home (12.1), where the path you provide for the flag ORACLE_HOME is the location of the home directory from the earlier Oracle Clusterware installation, for example C:\app\11.2.0.2\grid:

      cd C:\app\12.1.0\grid\bin
      setup.exe -updateNodeList -silent CRS=true ORACLE_HOME=C:\app\11.2.0.2\grid
      

   4. For downgrades to 11.2.0.2

      If you are downgrading to Oracle Clusterware 11g Release 1 (11.2.0.2), then you must start the Oracle Clusterware stack manually after you complete step c.

      On each node, start Oracle Clusterware from the earlier release Oracle Clusterware home using the command crsctl start crs. For example, where the earlier release home is C:\app\11.2.0.2\grid, use the following command on each node:

      C:\app\11.2.0.2\grid\bin\crsctl start crs
      

4. For downgrades to 12.1.0.1

   If you are downgrading to Oracle Grid Infrastructure 12c Release 1 (12.1.0.1), then run the following commands to configure the Grid Management Database:

   1. Start the 12.1.0.1 Oracle Clusterware stack on all nodes.

   2. On any node, remove the MGMTDB resource as follows:

      12101_Grid_home\bin\srvctl remove mgmtdb
      

   3. Run DBCA in the silent mode from the Oracle Grid Infrastructure 12c home and create the Management Database as follows, where ASM_DG_NAME is the name of a valid Oracle ASM disk group:

      12101_Grid_home\bin\dbca -silent -createDatabase -templateName
       MGMTSeed_Database.dbc -sid "-MGMTDB" -gdbName _mgmtdb -storageType ASM
       -diskGroupName ASM_DG_NAME -characterset AL32UTF8 
       -datafileJarLocation 12101_grid_home/assistants/dbca/templates
       -autoGeneratePasswords
      

   4. Configure the Management Database by running the Configuration Assistant from the location 12101_Grid_home\bin\mgmtca.

D.14.1 Updating the Enterprise Manager Cloud Control Target After Upgrades

1. Log in to Enterprise Manager Cloud Control

2. Navigate to the Targets menu, and then to the Cluster page.

3. Click a cluster target that was upgraded.

4. Click Cluster, then Target Setup, and then Monitoring Configuration from the menu.

5. Update the value for Oracle Home with the new Grid home path.

6. Save the updates.

D.14.2 Updating the Enterprise Manager Agent Base Directory After Upgrades

The Agent Base Directory is a directory where the Management Agent home is created.

The Management Agent home is in the path Agent_Base_directory\core\EMAgent_Version. For example, if the Agent Base Directory is C:\app\emagent, then Oracle creates the Management Agent home as C:\app\emagent\core\12.1.0.1.0.

1. Navigate to the bin directory in the Management Agent home.

2. In the C:\app\emagent\core\12.1.0.1.0\bin directory, open the file emctl with a text editor.

3. Locate the parameter CRS_HOME, and update the parameter to the new Grid home path.

4. Repeat steps 1-3 on each node of the cluster with an Enterprise Manager agent.