B How to Upgrade to Oracle Grid Infrastructure 12c Release 1

B.6.1 Checks to Complete Before Upgrade an Existing Oracle Clusterware Installation

Complete the following tasks before starting an upgrade:

1. For each node, use Cluster Verification Utility to ensure that you have completed preinstallation steps. It can generate Fixup scripts to help you to prepare servers. In addition, the installer will help you to ensure all required prerequisites are met.

   Ensure that you have information you will need during installation, including the following:

   • An Oracle base location for Oracle Clusterware.

   • An Oracle Grid Infrastructure home location that is different from your existing Oracle Clusterware location.

   • SCAN name and addresses, and other network addresses, as described in Chapter 4.

   • Privileged user operating system groups, as described in Chapter 5.

   • root user access, to run scripts as root during installation, using one of the options described in Section 7.1.1.

2. For the installation owner running the installation, if you have environment variables set for the existing installation, then unset the environment variables $ORACLE_HOME and $ORACLE_SID, as these environment variables are used during upgrade. For example:

   $ unset ORACLE_BASE
   $ unset ORACLE_HOME
   $ unset ORACLE_SID
   

   See Also:

   Section B.2, "Unset Oracle Environment Variables"

B.6.2 Running the ORAchk Upgrade Readiness Assessment

The ORAchk (Oracle RAC Configuration Audit Tool) Upgrade Readiness Assessment can be used to obtain an automated upgrade-specific health check for upgrades to Oracle Grid Infrastructure 11.2.0.3, 11.2.0.4, and 12.1.0.1. You can run the ORAchk Upgrade Readiness Assessment tool and automate many of the manual pre-upgrade and post upgrade checks.

Oracle recommends that you download and run the latest version of ORAchk from My Oracle Support. For information about downloading, configuring, and running ORAchk configuration audit tool, refer to My Oracle Support note 1457357.1, which is available at the following URL:

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1457357.1

B.7.1 About the CVU Grid Upgrade Validation Command Options

Navigate to the staging area for the upgrade, where the runcluvfy.sh command is located, and run the command runcluvfy.sh stage -pre crsinst -upgrade to check the readiness of your Oracle Clusterware installation for upgrades. Running runcluvfy.sh with the -pre crsinst -upgrade flags performs system checks to confirm if the cluster is in a correct state for upgrading from an existing clusterware installation.

The command uses the following syntax, where variable content is indicated by italics:

runcluvfy.sh stage -pre crsinst -upgrade [-rolling] -src_crshome src_Gridhome -dest_crshome dest_Gridhome -dest_version dest_release
[-fixup][-method {sudo|root} [-location dir_path] [-user user_name]] [-verbose]

The options are:

• -rolling

  Use this flag to verify readiness for rolling upgrades.

• -src_crshome src_Gridhome

  Use this flag to indicate the location of the source Oracle Clusterware or Grid home that you are upgrading, where src_Gridhome is the path to the home to upgrade.

• -dest_crshome dest_Gridhome

  Use this flag to indicate the location of the upgrade Grid home, where dest_ Gridhome is the path to the Grid home.

• -dest_version dest_version

  Use the dest_version flag to indicate the release number of the upgrade, including any patchset. The release number must include the five digits designating the release to the level of the platform-specific patch. For example: 11.2.0.2.0.

• -fixup [-fixupdir fixupdirpath]

  Use the -fixup flag to indicate that you want to generate instructions for any required steps you must complete to ensure that your cluster is ready for an upgrade. The default location is the CVU work directory. If you want to place the fixup instructions in a different directory, then add the flag -fixupdir, and provide the path to the directory where you want to put the instructions for required fixes.

• -verbose

  Use the -verbose flag to produce detailed output of individual checks

B.7.2 Example of Verifying System Upgrade Readiness for Grid Infrastructure

You can verify that the permissions required for installing Oracle Clusterware have been configured by running the following command:

$ ./runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome 
/u01/app/grid/11.2.0.1 -dest_crshome /u01/app/grid/11.2.0.2 -dest_version 11.2.0.3.0 -fixup -fixupdir /home/grid/fixup -verbose

B.7.3 Verifying System Readiness for Oracle Database Upgrades

Use Cluster Verification Utility to assist you with system checks in preparation for starting a database upgrade. The installer runs the appropriate CVU checks automatically, and either prompts you to fix problems, or provides a fixup script to be run on all nodes in the cluster before proceeding with the upgrade.

B.9.1 Performing a Standard Upgrade from an Earlier Release

Use the following procedure to upgrade the cluster from an earlier release:

1. Start the installer, and select the option to upgrade an existing Oracle Clusterware and Oracle ASM installation.

2. On the node selection page, select all nodes.

3. Select installation options as prompted. Oracle recommends that you configure root script automation, so that the rootupgrade.sh script can be run automatically during the upgrade.

4. Run root scripts, using either automatically or manually:

   • Running root scripts automatically

     If you have configured root script automation, then use the pause between batches to relocate services from the nodes running the previous release to the new release.

   • Running root scripts manually

     If you have not configured root script automation, then when prompted, run the rootupgrade.sh script on each node in the cluster that you want to upgrade.

     If you run root scripts manually, then run the script on the local node first. The script shuts down the earlier release installation, replaces it with the new Oracle Clusterware release, and starts the new Oracle Clusterware installation.

     After the script completes successfully, you can run the script in parallel on all nodes except for one, which you select as the last node. When the script is run successfully on all the nodes except the last node, run the script on the last node.

5. After running the rootupgrade.sh script on the last node in the cluster, if you are upgrading from a release earlier than Oracle Grid Infrastructure 11g Release 2 (11.2.0.1), and left the check box labeled ASMCA checked, as is the default, then Oracle Automatic Storage Management Configuration Assistant ASMCA runs automatically, and the Oracle Grid Infrastructure upgrade is complete. If you unchecked the box during the interview stage of the upgrade, then ASMCA is not run automatically.

   If an earlier release of Oracle Automatic Storage Management (Oracle ASM) is installed, then the installer starts ASMCA to upgrade Oracle ASM to 12c Release 1 (12.1). You can choose to upgrade Oracle ASM at this time, or upgrade it later.

   Oracle recommends that you upgrade Oracle ASM at the same time that you upgrade the Oracle Clusterware binaries. Until Oracle ASM is upgraded, Oracle Database installations that use Oracle ASM cannot be installed. Until Oracle ASM is upgraded, the 12c Release 1 (12.1) Oracle ASM management tools in the Grid home (for example, srvctl) do not work.

6. Because the Oracle Grid Infrastructure home is in a different location than the former Oracle Clusterware and Oracle ASM homes, update any scripts or applications that use utilities, libraries, or other files that reside in the Oracle Clusterware and Oracle ASM homes.

B.9.2 Completing an Oracle Clusterware Upgrade when Nodes Become Unreachable

If some nodes become unreachable in the middle of an upgrade, then you cannot complete the upgrade, because the upgrade script (rootupgrade.sh) did not run on the unreachable nodes. Because the upgrade is incomplete, Oracle Clusterware remains in the previous release. You can confirm that the upgrade is incomplete by entering the command crsctl query crs activeversion.

To resolve this problem, run the rootupgrade.sh command with the -force flag on any of the nodes where the rootupgrade.sh script has already completed as follows:

Grid_home/rootupgrade -force

For example:

# /u01/app/12.1.0/grid/rootupgrade -force

This command forces the upgrade to complete. Verify that the upgrade has completed by using the command crsctl query crs activeversion. The active release should be the upgrade release.

The force cluster upgrade command has the following limitations:

• All active nodes must be upgraded to the newer release

• All inactive nodes (accessible or inaccessible) may be either upgraded or not upgraded

• Inaccessible nodes have the following restrictions after you run the force command:

  • After patch set upgrades, you can delete the node from the cluster. If the node becomes accessible later, and the patch release upgrade path is supported, then you can upgrade it to the new patch release.

B.9.3 Upgrading Inaccessible Nodes After Forcing an Upgrade

Starting with Oracle Grid Infrastructure 12c, after you compete a force cluster upgrade command, you can join inaccessible nodes to the cluster as an alternative to deleting the nodes, which was required in earlier releases. To use this option, Oracle Grid Infrastructure 12c Release 1 (12.1) software must already be installed on the nodes.

To complete the upgrade of inaccessible or unreachable nodes:

1. Log in as the Grid user on the node that is to be joined to the cluster.

2. Change directory to the /crs/install directory in the Oracle Grid Infrastructure 12c Release 1 (12.1) Grid home. For example:

   $ cd /u01/12.1.0/grid/crs/install
   

3. Run the following PERL command, where existingnode is the node in the cluster on which you are running the command, and upgraded_node is the inaccessible or unreachable node that you want to join to the cluster:

   $ rootcrs.sh -join -existingnode upgraded_node
   

B.9.4 Changing the First Node for Install and Upgrade

If the first node becomes inaccessible, you can force another node to be the first node for installation or upgrade. During installation, if root.sh fails to complete on the first node, run the following command on another node using the -force option:

root.sh -force -first

For upgrade, run the following command:

rootupgrade.sh -force -first

B.9.5 Performing Rolling Upgrades of Oracle Automatic Storage Management

Review the following sections for information about Oracle ASM rolling upgrade options:

• About Full Release Rolling Upgrades of Oracle ASM

• About Individual (One-Off) Oracle ASM Patches

• About Oracle Grid Infrastructure Software Patch Levels

B.9.6 Preparing to Complete a Separate Upgrade of Oracle ASM

If you must complete upgrade of Oracle ASM separately, then note the following requirements

• The active release of Oracle Clusterware must be 12c Release 1 (12.1). To determine the active release, enter the following command:

  $ crsctl query crs activeversion
  

• You can upgrade a single instance Oracle ASM installation to a clustered Oracle ASM installation. However, you can only upgrade an existing single instance Oracle ASM installation if you run the installation from the node on which the Oracle ASM installation is installed. You cannot upgrade a single instance Oracle ASM installation on a remote node.

• You must ensure that any rebalance operations on your existing Oracle ASM installation are completed before starting the upgrade process.

• During the upgrade process, you alter the Oracle ASM instances to an upgrade state. You do not need to shut down database clients unless they are on Oracle ACFS. However, because this upgrade state limits Oracle ASM operations, you should complete the upgrade process soon after you begin. The following are the operations allowed when an Oracle ASM instance is in the upgrade state:

  • Diskgroup mounts and dismounts

  • Opening, closing, resizing, or deleting database files

  • Recovering instances

  • Queries of fixed views and packages: Users are allowed to query fixed views and run anonymous PL/SQL blocks using fixed packages, such as dbms_diskgroup)

B.9.7 Patching Oracle ASM Separately from an Oracle Grid Infrastructure Upgrade

Complete the following tasks if you must upgrade from an Oracle ASM release where Oracle ASM was installed in a separate Oracle home, or if the Oracle ASM portion of Oracle Grid Infrastructure failed to complete:

1. On the node you plan to start the upgrade, set the environment variable ASMCA_ROLLING_UPGRADE as true. For example:

   $ export ASMCA_ROLLING_UPGRADE=true
   

2. From the Oracle Grid Infrastructure 12c Release 1 (12.1) home, start ASMCA. For example:

   $ cd /u01/12.1/grid/bin
   $ ./asmca
   

3. Select Upgrade.

   ASM Configuration Assistant upgrades Oracle ASM in succession for all nodes in the cluster.

4. After you complete the upgrade, run the command to unset the ASMCA_ROLLING_UPGRADE environment variable.

B.9.8 Upgrading Oracle ASM to a Software Patch Level

Starting with Oracle Grid Infrastructure 12c Release 1 (12.1), a new cluster state called "Rolling Patch" is available. This mode is similar to the existing "Rolling Upgrade" mode in terms of the Oracle ASM operations allowed in this quiesce state.

1. Download patches you want to apply from My Oracle Support:

   https://support.oracle.com

   Select the Patches and Updates tab to locate the patch.

   Oracle recommends that you select Recommended Patch Advisor, and enter the product group, release, and platform for your software. My Oracle Support provides you with a list of the most recent patch set updates (PSUs) and critical patch updates (CPUs).

   Place the patches in an accessible directory, such as /tmp.

2. Change directory to the /opatch directory in the Grid home. For example:

   $ cd /u01/app/12.1.0/grid/opatch
   

3. Review the patch documentation for the patch you want to apply, and complete all required steps before starting the patch upgrade.

B.10.1 Updating the Enterprise Manager Cloud Control Target After Upgrades

1. Log in to Enterprise Manager Cloud Control.

2. Navigate to the Targets menu, and then to the Cluster page.

3. Click a cluster target that was upgraded.

4. Click Cluster, then Target Setup, and then Monitoring Configuration from the menu.

5. Update the value for Oracle Home with the new Grid home path.

6. Save the updates.

B.10.2 Updating the Enterprise Manager Agent Base Directory After Upgrades

1. Navigate to the bin directory in the Agent Base Directory.

   The Agent Base Directory is a directory where the Management Agent home is created. It is in the path EMAgent_home/core/EMAgent_Version/bin. For example, if the Agent Base Directory is /u01/app/grid, then the Management Agent home is created as /u01/app/emagent/core/12.1.0.1.0/bin

2. In the bin directory, open the file emctl with a text editor.

3. Locate the parameter CRS_HOME, and update the parameter to the new Grid home path.

4. Repeat steps 1-3 on each node of the cluster with an Enterprise Manager agent.

B.12.1 About Downgrading Oracle Clusterware After an Upgrade

Downgrading Oracle Clusterware restores the Oracle Clusterware configuration to the state it was in before the Oracle Clusterware 12c Release 1 (12.1) upgrade. Any configuration changes you performed during or after the Oracle Grid Infrastructure 12c Release 1 (12.1) upgrade are removed and cannot be recovered.

In the downgrade procedures, the following variables are used:

• local node is the first node on which you ran the rootupgrade script.

• remote nodes are all other nodes where you ran the rootupgrade script.

To restore Oracle Clusterware to the previous release, use the downgrade procedure for the release to which you want to downgrade.

B.12.2 Downgrading to Releases Before 11g Release 2 (11.2.0.2)

1. On all remote nodes, use the command syntax Grid_home/crs/install/rootcrs.sh -downgrade [-force] to stop the 12c Release 1 (12.1) resources, and shut down the Oracle Grid Infrastructure 12c Release 1 (12.1) stack.

2. After the rootcrs.sh -downgrade script has completed on all remote nodes, on the local node use the command syntax Grid_home/crs/install/rootcrs.sh -downgrade -lastnode.

   For example:

   # /u01/app/12.1.0/grid/crs/install/rootcrs.sh -downgrade  -lastnode
   

   Note:

   With Oracle Grid Infrastructure 12c, you no longer need to provide the location of the previous release Grid home or release number.

3. On any of the cluster member nodes where the rootupgrade.sh script has run successfully:

   1. Log in as the Oracle Grid Infrastructure installation owner

   2. Use the following command to start the installer, where /u01/app/12.1.0/grid is the location of the new (upgraded) Grid home:

   ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=false ORACLE_HOME=/u01/app/12.1.0/grid
   

   Add the flag -cfs if the Grid home is a shared home.

4. On each cluster member node where the rootupgrade.sh script has run successfully:

   1. Log in as the Oracle Grid Infrastructure installation owner (grid)

   2. Use the following command to start the installer, where the path you provide for the flag ORACLE_HOME is the location of the home directory from the earlier Oracle Clusterware installation

      For example:

      $ cd /u01/app/12.1.0/grid/oui/bin
      $ ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=true ORACLE_HOME=/u01/app/crs
      

   3. For downgrades to 11.1 and earlier releases

      If you are downgrading to Oracle Clusterware 11g Release 1 (11.1) or an earlier release, then you must run root.sh manually from the earlier release Oracle Clusterware home to complete the downgrade after you complete step b.

      OUI prompts you to run root.sh manually from the earlier release Oracle Clusterware installation home in sequence on each member node of the cluster to complete the downgrade. After you complete this task, downgrade is completed.

      Running root.sh from the earlier release Oracle Clusterware installation home restarts the Oracle Clusterware stack, starts up all the resources previously registered with Oracle Clusterware in the earlier release, and configures the old initialization scripts to run the earlier release Oracle Clusterware stack.

      After completing the downgrade, update the entry for Oracle ASM instance in the oratab file (/etc/oratab or /var/opt/oracle/oratab) on every node in the cluster as follows:

      +ASM<instance#>:<RAC-ASM home>:N

B.12.3 Downgrading to 11g Release 2 (11.2.0.2) or Later Release

1. On all remote nodes, use the command syntax Grid_home/crs/install/rootcrs.sh -downgrade to stop the 12c Release 1 (12.1) resources, and shut down the Oracle Grid Infrastructure 12c Release 1 (12.1) stack.

2. Use the command syntax rootcrs.sh -downgrade. For example:

   # /u01/app/12.1.0/grid/crs/install/rootcrs.sh -downgrade
   

3. After the rootcrs.sh -downgrade script has completed on all remote nodes, on the local node use the command syntax Grid_home/crs/install/rootcrs.sh -downgrade -lastnode.

   For example:

   # /u01/app/12.1.0/grid/crs/install/rootcrs.sh -downgrade  -lastnode
   

   Note:

   Starting with Oracle Grid Infrastructure 12c Release 1 (12.1), you no longer need to provide the location of the earlier release Grid home or earlier release number.

4. On each cluster member node where the rootupgrade.sh script has run successfully:

   1. Log in as the Oracle Grid Infrastructure installation owner

   2. Use the following command to start the installer, where /u01/app/12.1.0/grid is the location of the new (upgraded) Grid home:

   $ cd /u01/app/12.1.0/grid/oui/bin
   ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=false ORACLE_HOME=/u01/app/12.1.0/grid
   

   Add the flag -cfs if the Grid home is a shared home.

5. On any of the cluster member nodes where the rootupgrade script has run successfully:

   1. Log in as the Oracle Grid Infrastructure installation owner

   2. Use the following command to start the installer, where the path you provide for the flag ORACLE_HOME is the location of the home directory from the earlier Oracle Clusterware installation

      For example:

      $ cd /u01/app/12.1.0/grid/oui/bin
      $ ./runInstaller -nowait -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=true ORACLE_HOME=/u01/app/crs
      

   3. For downgrades to 11.2.0.2

      If you are downgrading to Oracle Clusterware 11g Release 1 (11.2.0.1), then you must start the Oracle Clusterware stack manually after you complete step b.

      On each node, start Oracle Clusterware from the earlier release Oracle Clusterware home using the command crsctl start crs. For example, where the earlier release home is /u01/app/11.2.0/grid, use the following command on each node:

      /u01/app/11.2.0/grid/bin/crsctl start crs

   4. For downgrades to 12.1.0.1

      If you are downgrading to Oracle Grid Infrastructure 12c Release 1 (12.1.0.1), then run the following commands to configure the Grid Management Database:

      1. Start the 12.1.0.1 Oracle Clusterware stack on all nodes.

      2. On any node, remove the MGMTDB resource as follows:

         12101_Grid_home/bin/srvctl remove mgmtdb
         

      3. Run DBCA in the silent mode from the 12.1.0.1 Oracle home and create the Management Database as follows:

         12101_Grid_home/bin/dbca -silent -createDatabase -templateName
         MGMTSeed_Database.dbc -sid -MGMTDB -gdbName _mgmtdb -storageType ASM
         -diskGroupName ASM_DG_NAME
         -datafileJarLocation 12101_grid_home/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords
         

      4. Configure the Management Database by running the Configuration Assistant from the location 12101_Grid_home/bin/mgmtca.