B How to Upgrade to Oracle Grid Infrastructure 12c Release 1

B.5.1 Checks to Complete Before Upgrade an Existing Oracle Clusterware Installation

Complete the following tasks before starting an upgrade:

1. For each node, use Cluster Verification Utility to ensure that you have completed preinstallation steps. It can generate Fixup scripts to help you to prepare servers. In addition, the installer will help you to ensure all required prerequisites are met.

   Ensure that you have information you will need during installation, including the following:

   • An Oracle base location for Oracle Clusterware.

   • An Oracle Grid Infrastructure home location that is different from your existing Oracle Clusterware location.

   • SCAN name and addresses, and other network addresses, as described in Chapter 4.

   • Privileged user operating system groups, as described in Chapter 6.

   • root user access, to run scripts as root during installation, using one of the options described in Section 7.1.1.

2. For the installation owner running the installation, if you have environment variables set for the existing installation, then unset the environment variables $ORACLE_HOME and $ORACLE_SID, as these environment variables are used during upgrade. For example:

   $ unset ORACLE_BASE
   $ unset ORACLE_HOME
   $ unset ORACLE_SID
   

   See Also:

   Section B.5.2, "Unset Oracle Environment Variables"

3. For AIX HACMP vendor clusterware on an Oracle Clusterware release 11.1 (11.1.0.7), do the following:

   1. When OUI prompts, do not run rootpre.sh.

   2. Select the Install type as Upgrade and complete the upgrade. Do not run rootupgrade.sh when prompted.

   3. On the first node, shut down Oracle Clusterware as root. Run rootpre.sh, ensure there are no errors, and then run rootupgrade.sh.

   4. On other nodes except last one, execute step c on each node as root.

   5. On the last node, execute step c sequentially as root.

   6. After rootupgrade.sh completes successfully on the last node, return to OUI and continue the remaining upgrade steps.

B.5.2 Unset Oracle Environment Variables

Unset Oracle environment variables.

If you have set ORA_CRS_HOME as an environment variable, following instructions from Oracle Support, then unset it before starting an installation or upgrade. You should never use ORA_CRS_HOME as an environment variable except under explicit direction from Oracle Support.

Check to ensure that installation owner login shell profiles (for example, .profile or .cshrc) do not have ORA_CRS_HOME set.

If you have had an existing installation on your system, and you are using the same user account to install this installation, then unset the following environment variables: ORA_CRS_HOME; ORACLE_HOME; ORA_NLS10; TNS_ADMIN; and any other environment variable set for the Oracle installation user that is connected with Oracle software homes.

Also, ensure that the $ORACLE_HOME/bin path is removed from your PATH environment variable.

B.5.3 Running the Oracle ORAchk Upgrade Readiness Assessment

ORAchk (Oracle RAC Configuration Audit Tool) Upgrade Readiness Assessment can be used to obtain an automated upgrade-specific health check for upgrades to Oracle Grid Infrastructure 11.2.0.3, 11.2.0.4, and 12.1.0.1. You can run the ORAchk Upgrade Readiness Assessment tool and automate many of the manual pre-upgrade and post upgrade checks.

Oracle recommends that you download and run the latest version of ORAchk from My Oracle Support. For information about downloading, configuring, and running ORAchk configuration audit tool, refer to My Oracle Support note 1457357.1, which is available at the following URL:

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1457357.1

B.6.1 About the CVU Grid Upgrade Validation Command Options

Navigate to the staging area for the upgrade, where the runcluvfy.sh command is located, and run the command runcluvfy.sh stage -pre crsinst -upgrade to check the readiness of your Oracle Clusterware installation for upgrades. Running runcluvfy.sh with the -pre crsinst -upgrade flags performs system checks to confirm if the cluster is in a correct state for upgrading from an existing clusterware installation.

The command uses the following syntax, where variable content is indicated by italics:

runcluvfy.sh stage -pre crsinst -upgrade [-rolling] 
-src_crshome src_Gridhome -dest_crshome dest_Gridhome -dest_version dest_release
[-fixup][-method {sudo|root} [-location dir_path] [-user user_name]] [-verbose]

The options are:

• -rolling

  Use this flag to verify readiness for rolling upgrades.

• -src_crshome src_Gridhome

  Use this flag to indicate the location of the source Oracle Clusterware or Grid home that you are upgrading, where src_Gridhome is the path to the home that you want to upgrade.

• -dest_crshome dest_Gridhome

  Use this flag to indicate the location of the upgrade Grid home, where dest_ Gridhome is the path to the Grid home.

• -dest_version dest_release

  Use the -dest_version flag to indicate the release number of the upgrade, including any patchset, and then enter the release number. The release number must include the five digits designating the release to the level of the platform-specific patch. For example: 11.2.0.2.0.

• -fixup [-method {sudo|root} [-location dir_path] [-user user_name]

  Use the -fixup flag to indicate that you want to generate instructions for any required steps you need to complete to ensure that your cluster is ready for an upgrade. The default location is the CVU work directory.

  The -fixup -method flag defines the method by which root scripts are run. The -method flag requires one of the following options:

  • sudo: Run as a user on the sudoers list.

  • root: Run as the root user.

  If you select sudo, then enter the -location flag to provide the path to Sudo on the server, and enter the -user flag to provide the user account with Sudo privileges.

• -verbose

  Use the -verbose flag to produce detailed output of individual checks.

B.6.2 Example of Verifying System Upgrade Readiness for Grid Infrastructure

You can verify that the permissions required for installing Oracle Clusterware have been configured by running the following command:

$ ./runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome 
/u01/app/11.2.0/grid -dest_crshome /u01/app/12.1.0/grid -dest_version
12.1.0.1 -fixup -verbose

B.8.1 Performing a Rolling Upgrade from an Earlier Release

Use the following procedure to upgrade the cluster from an earlier release:

1. Start the installer, and select the option to upgrade an existing Oracle Clusterware and Oracle ASM installation.

2. On the node selection page, select all nodes.

3. Select installation options as prompted. Oracle recommends that you configure root script automation, so that the rootupgrade.sh script can be run automatically during the upgrade.

   If you have an AIX HACMP vendor clusterware installation, then proceed to step 5.

4. If you do not have an HACMP cluster, then run root scripts, using either automatically or manually:

   • Running root scripts automatically

     If you have configured root script automation, then use the pause between batches to relocate services from the nodes running the previous release to the new release.

   • Running root scripts manually

     If you have not configured root script automation, then when prompted, run the rootupgrade.sh script on each node in the cluster that you want to upgrade.

     If you run root scripts manually, then run the script on the local node first. The script shuts down the earlier release installation, replaces it with the new Oracle Clusterware release, and starts the new Oracle Clusterware installation.

     After the script completes successfully, you can run the script in parallel on all nodes except for one, which you select as the last node. When the script is run successfully on all the nodes except the last node, run the script on the last node.

   If you do not have an HACMP cluster, then proceed to step 10.

5. If you have AIX HACMP vendor clusterware on an Oracle Clusterware cluster, then start the installer on one node in the cluster, and select the option to upgrade an existing Oracle Clusterware and Oracle ASM installation.

6. For the HACMP cluster, shut down the previous release Oracle Clusterware software stack on the cluster node of the cluster.

7. Run the rootpre.sh script on the cluster node that you have shut down. The script replaces some libraries from the earlier release installation with the new Oracle Clusterware release.

8. After the rootpre.sh script is run on the node, run rootupgrade.sh script. The upgraded Oracle Clusterware stack and AUTOSTART resources are started on the node.

9. Repeat steps 6 through 8 on each HACMP cluster member node.

10. After running the rootupgrade.sh script on the last node in the cluster, if you are upgrading from a release earlier than Oracle Grid Infrastructure 11g Release 2 (11.2.0.1), and left the check box labeled ASMCA checked, as is the default, then Oracle Automatic Storage Management Configuration Assistant ASMCA runs automatically, and the Oracle Grid Infrastructure upgrade is complete. If you unchecked the box during the interview stage of the upgrade, then ASMCA is not run automatically.

    If an earlier release of Oracle Automatic Storage Management (Oracle ASM) is installed, then the installer starts ASMCA to upgrade Oracle ASM to 12c Release 1 (12.1). You can choose to upgrade Oracle ASM at this time, or upgrade it later.

    Oracle recommends that you upgrade Oracle ASM at the same time that you upgrade the Oracle Clusterware binaries. Until Oracle ASM is upgraded, Oracle Database installations that use Oracle ASM cannot be installed. Until Oracle ASM is upgraded, the 12c Release 1 (12.1) Oracle ASM management tools in the Grid home (for example, srvctl) do not work.

11. Because the Oracle Grid Infrastructure home is in a different location than the former Oracle Clusterware and Oracle ASM homes, update any scripts or applications that use utilities, libraries, or other files that reside in the Oracle Clusterware and Oracle ASM homes.

B.8.2 Completing an Oracle Clusterware Upgrade when Nodes Become Unreachable

If some nodes become unreachable in the middle of an upgrade, then you cannot complete the upgrade, because the upgrade script (rootupgrade.sh) did not run on the unreachable nodes. Because the upgrade is incomplete, Oracle Clusterware remains in the previous release. You can confirm that the upgrade is incomplete by entering the command crsctl query crs activeversion.

To resolve this problem, run the rootupgrade.sh command with the -force flag on any of the nodes where the rootupgrade.sh script has already completed as follows:

Grid_home/rootupgrade -force

For example:

# /u01/app/12.1.0/grid/rootupgrade -force

This command forces the upgrade to complete. Verify that the upgrade has completed by using the command crsctl query crs activeversion. The active release should be the upgrade release.

The force cluster upgrade command has the following limitations:

• All active nodes must be upgraded to the newer release

• All inactive nodes (accessible or inaccessible) may be either upgraded or not upgraded

• Inaccessible nodes have the following restrictions after you run the force command:

  • After patch set upgrades, you can delete the node from the cluster. If the node becomes accessible later, and the patch release upgrade path is supported, then you can upgrade it to the new patch release.

B.8.3 Upgrading Inaccessible Nodes After Forcing an Upgrade

Starting with Oracle Grid Infrastructure 12c, after you compete a force cluster upgrade command, you can join inaccessible nodes to the cluster as an alternative to deleting the nodes, which was required in earlier releases. To use this option, Oracle Grid Infrastructure 12c Release 1 (12.1) software must already be installed on the nodes.

To complete the upgrade of inaccessible or unreachable nodes:

1. Log in as the Grid user on the node that is to be joined to the cluster.

2. Change directory to the /crs/install directory in the Oracle Grid Infrastructure 12c Release 1 (12.1) Grid home. For example:

   $ cd /u01/12.1.0/grid/crs/install
   

3. Run the following PERL command, where existingnode is the name of the option and upgraded_node is the upgraded node:

   $ rootupgrade.sh -join -existingnode upgraded_node
   

   Note:

   The -join operation is not supported for Oracle Clusterware releases earlier than 11.2.0.1.0. In such cases, delete the node and add it to the clusterware using the addNode command.

B.8.4 Changing the First Node for Install and Upgrade

If the first node becomes inaccessible, you can force another node to be the first node for installation or upgrade. During installation, if root.sh fails to complete on the first node, run the following command on another node using the -force option:

root.sh -force -first

For upgrade, run the following command:

rootupgrade.sh -force -first

B.9.1 About Upgrading Oracle ASM Separately

If you must complete upgrade of Oracle ASM separately, then note the following requirements

• The active release of Oracle Clusterware must be 12c Release 1 (12.1). To determine the active release, enter the following command:

  $ crsctl query crs activeversion
  

• You can upgrade a single instance Oracle ASM installation to a clustered Oracle ASM installation. However, you can only upgrade an existing single instance Oracle ASM installation if you run the installation from the node on which the Oracle ASM installation is installed. You cannot upgrade a single instance Oracle ASM installation on a remote node.

• You must ensure that any rebalance operations on your existing Oracle ASM installation are completed before starting the upgrade process.

• During the upgrade process, you alter the Oracle ASM instances to an upgrade state. You do not need to shut down database clients unless they are on Oracle ACFS. However, because this upgrade state limits Oracle ASM operations, you should complete the upgrade process soon after you begin. The following are the operations allowed when an Oracle ASM instance is in the upgrade state:

  • Diskgroup mounts and dismounts

  • Opening, closing, resizing, or deleting database files

  • Recovering instances

  • Queries of fixed views and packages: Users are allowed to query fixed views and run anonymous PL/SQL blocks using fixed packages, such as dbms_diskgroup)

B.9.2 Using ASMCA to Upgrade Oracle ASM

Complete the following tasks if you must upgrade from an Oracle ASM release where Oracle ASM was installed in a separate Oracle home, or if the Oracle ASM portion of Oracle Grid Infrastructure failed to complete:

1. On the node you plan to start the upgrade, set the environment variable ASMCA_ROLLING_UPGRADE as true. For example:

   $ export ASMCA_ROLLING_UPGRADE=true
   

2. From the Oracle Grid Infrastructure 12c Release 1 (12.1) home, start ASMCA. For example:

   $ cd /u01/12.1/grid/bin
   $ ./asmca
   

3. Select Upgrade.

   ASM Configuration Assistant upgrades Oracle ASM in succession for all nodes in the cluster.

4. After you complete the upgrade, run the command to unset the ASMCA_ROLLING_UPGRADE environment variable.

B.10.1 About Individual (One-Off) Oracle ASM Patches

Individual patches are called one-off patches. An Oracle ASM one-off patches is available for a specific released release of Oracle ASM. If a patch you want is available, then you can download the patch and apply it to Oracle ASM using the OPatch Utility. The OPatch inventory keeps track of the patches you have installed for your release of Oracle ASM. if there is a conflict between the patches you have installed and patches you want to apply, then the OPatch Utility advises you of these conflicts.

B.10.2 About Oracle ASM Software Patch Levels

The software patch level for Oracle Grid Infrastructure represents the set of all one-off patches applied to the Oracle Grid Infrastructure software release, including Oracle ASM. The release is the release number, in the format of major, minor, and patch set release format. For example, with the release number 12.1.0.1, the major release is 12, the minor release is 1, and 0.0 is the patch set number. With one-off patches, the major and minor release remains the same, though the patch levels change each time you apply or roll back an interim patch.

As with standard upgrades to Oracle Grid Infrastructure, at any given point in time for normal operation of the cluster, all the nodes in the cluster must have the same software release and patch level. Because one-off patches can be applied as rolling upgrades, all possible patch levels on a particular software release are compatible with each other.

B.10.3 Patching Oracle ASM to a Software Patch Level

Starting with Oracle Grid Infrastructure 12c Release 1 (12.1), a new cluster state called "Rolling Patch" is available. This mode is similar to the existing "Rolling Upgrade" mode in terms of the Oracle ASM operations allowed in this quiesce state.

1. Download patches you want to apply from My Oracle Support:

   https://support.oracle.com

   Select the Patches and Updates tab to locate the patch.

   Oracle recommends that you select Recommended Patch Advisor, and enter the product group, release, and platform for your software. My Oracle Support provides you with a list of the most recent patch set updates (PSUs) and critical patch updates (CPUs).

   Place the patches in an accessible directory, such as /tmp.

2. Change directory to the /opatch directory in the Grid home. For example:

   $ cd /u01/app/12.1.0/grid/opatch
   

3. Review the patch documentation for the patch you want to apply, and complete all required steps before starting the patch upgrade.

B.11.1 Updating the Enterprise Manager Cloud Control Target After Upgrades

1. Log in to Enterprise Manager Cloud Control.

2. Navigate to the Targets menu, and then to the Cluster page.

3. Click a cluster target that was upgraded.

4. Click Cluster, then Target Setup, and then Monitoring Configuration from the menu.

5. Update the value for Oracle Home with the new Grid home path.

6. Save the updates.

B.11.2 Updating the Enterprise Manager Agent Base Directory After Upgrades

1. Navigate to the bin directory in the Management Agent home.

   The Agent Base directory is a directory where the Management Agent home is created. The Management Agent home is in the path Agent_Base_Directory/core/EMAgent_Version. For example, if the Agent Base directory is /u01/app/emagent, then the Management Agent home is created as /u01/app/emagent/core/12.1.0.1.0.

2. In the /u01/app/emagent/core/12.1.0.1.0/bin directory, open the file emctl with a text editor.

3. Locate the parameter CRS_HOME, and update the parameter to the new Grid home path.

4. Repeat steps 1-3 on each node of the cluster with an Enterprise Manager agent.

B.14.1 About Downgrading Oracle Clusterware After an Upgrade

Downgrading Oracle Clusterware restores the Oracle Clusterware configuration to the state it was in before the Oracle Clusterware 12c Release 1 (12.1) upgrade. Any configuration changes you performed during or after the Oracle Grid Infrastructure 12c Release 1 (12.1) upgrade are removed and cannot be recovered.

In the downgrade procedures, the following variables are used:

• local node is the first node on which you ran the rootupgrade script.

• remote nodes are all other nodes where you ran the rootupgrade script.

To restore Oracle Clusterware to the previous release, use the downgrade procedure for the release to which you want to downgrade.

B.14.2 Downgrading to Releases Before 11g Release 2 (11.2.0.2)

1. On all remote nodes, use the command syntax Grid_home/crs/install/rootcrs.sh -downgrade to stop the 12c Release 1 (12.1) resources, and shut down the Oracle Grid Infrastructure 12c Release 1 (12.1) stack.

2. After the rootcrs.sh -downgrade script has completed on all remote nodes, on the local node use the command syntax Grid_home/crs/install/rootcrs.sh -downgrade -lastnode.

   For example:

   # /u01/app/12.1.0/grid/crs/install/rootcrs.sh -downgrade  -lastnode
   

   Note:

   With Oracle Grid Infrastructure 12c, you no longer need to provide the location of the previous release Grid home or release number.

3. On any of the cluster member nodes where the rootupgrade.sh script has run successfully:

   1. Log in as the Oracle Grid Infrastructure installation owner

   2. Use the following command to start the installer, where /u01/app/12.1.0/grid is the location of the new (upgraded) Grid home:

   $ cd /u01/app/12.1.0/grid/oui/bin
   ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList
   -silent CRS=false ORACLE_HOME=/u01/app/12.1.0/grid
   

   Add the flag -cfs if the Grid home is a shared home.

4. On any of the cluster member nodes where the rootupgrade.sh script has run successfully:

   1. Log in as the Oracle Grid Infrastructure installation owner (grid).

   2. Use the following command to start the installer, where the path you provide for the flag ORACLE_HOME is the location of the home directory from the earlier Oracle Clusterware installation

      For example:

      $ cd /u01/app/12.1.0/grid/oui/bin
      $ ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=true ORACLE_HOME=/u01/app/crs
      

   3. For downgrades to 11.1 and earlier releases

      If you are downgrading to Oracle Clusterware 11g Release 1 (11.1) or an earlier release, then you must run root.sh manually from the earlier release Oracle Clusterware home to complete the downgrade after you complete step b.

      OUI prompts you to run root.sh manually from the earlier release Oracle Clusterware installation home in sequence on each member node of the cluster to complete the downgrade. After you complete this task, downgrade is completed.

      Running root.sh from the earlier release Oracle Clusterware installation home restarts the Oracle Clusterware stack, starts up all the resources previously registered with Oracle Clusterware in the earlier release, and configures the old initialization scripts to run the earlier release Oracle Clusterware stack.

      After completing the downgrade, update the entry for Oracle ASM instance in the oratab file (/etc/oratab or /var/opt/oracle/oratab) on every node in the cluster as follows:

      +ASM<instance#>:<RAC-ASM home>:N

B.14.3 Downgrading to 11g Release 2 (11.2.0.2) or Later Release

1. On all remote nodes, use the command syntax Grid_home/crs/install/rootcrs.sh -downgrade to stop the 12c Release 1 (12.1) resources, and shut down the Oracle Grid Infrastructure 12c Release 1 (12.1) stack.

2. Use the command syntax rootcrs.sh -downgrade. For example:

   # /u01/app/12.1.0/grid/crs/install/rootcrs.sh -downgrade
   

3. After the rootcrs.sh -downgrade script has completed on all remote nodes, on the local node use the command syntax Grid_home/crs/install/rootcrs.sh -downgrade -lastnode.

   For example:

   # /u01/app/12.1.0/grid/crs/install/rootcrs.sh -downgrade  -lastnode
   

   Note:

   Starting with Oracle Grid Infrastructure 12c Release 1 (12.1), you no longer need to provide the location of the earlier release Grid home or earlier release number.

4. On each cluster member node where the rootupgrade.sh script has run successfully:

   1. Log in as the Oracle Grid Infrastructure installation owner.

   2. Use the following command to start the installer, where /u01/app/12.1.0/grid is the location of the new (upgraded) Grid home:

   ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=false ORACLE_HOME=/u01/app/12.1.0/grid
   

   Add the flag -cfs if the Grid home is a shared home.

5. On each cluster member node where the rootupgrade script has run successfully:

   1. Log in as the Oracle Grid Infrastructure installation owner.

   2. Use the following command to start the installer, where the path you provide for the flag ORACLE_HOME is the location of the home directory from the earlier Oracle Clusterware installation

      For example:

      $ cd /u01/app/12.1.0/grid/oui/bin
      $ ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=true ORACLE_HOME=/u01/app/crs
      

   3. For downgrades to 11.2.0.2

      If you are downgrading to Oracle Clusterware 11g Release 1 (11.2.0.1),. then you must start the Oracle Clusterware stack manually after you complete step b.

      On each node, start Oracle Clusterware from the earlier release Oracle Clusterware home using the command crsctl start crs. For example, where the earlier release home is /u01/app/11.2.0/grid, use the following command on each node:

      /u01/app/11.2.0/grid/bin/crsctl start crs

   4. For downgrades to 12.1.0.1

      If you are downgrading to Oracle Grid Infrastructure 12c Release 1 (12.1.0.1), then run the following commands to configure the Grid Management Database:

      1. Start the 12.1.0.1 Oracle Clusterware stack on all nodes.

      2. On any node, remove the MGMTDB resource as follows:

         12101_Grid_home/bin/srvctl remove mgmtdb
         

      3. Run DBCA in the silent mode from the 12.1.0.1 Oracle home and create the Management Database as follows:

         12101_Grid_home/bin/dbca -silent -createDatabase -templateName
         MGMTSeed_Database.dbc -sid -MGMTDB -gdbName _mgmtdb -storageType ASM
         -diskGroupName ASM_DG_NAME
         -datafileJarLocation 12101_grid_home/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords
         

      4. Configure the Management Database by running the Configuration Assistant from the location 12101_Grid_home/bin/mgmtca.