6 Managing Oracle Cluster Registry and Voting Files

Migrating Oracle Cluster Registry to Oracle Automatic Storage Management

To improve Oracle Clusterware storage manageability, OCR is configured, by default, to use Oracle ASM in Oracle Database 12c. With the Oracle Clusterware storage residing in an Oracle ASM disk group, you can manage both database and clusterware storage using Oracle Enterprise Manager.

However, if you upgrade from a previous version of Oracle Clusterware, you can migrate OCR to reside on Oracle ASM, and take advantage of the improvements in managing Oracle Clusterware storage.

To migrate OCR to Oracle ASM using OCRCONFIG:

1. Ensure the upgrade to Oracle Clusterware 12c is complete. Run the following command to verify the current running version:

   $ crsctl query crs activeversion
   

2. Use the Oracle ASM Configuration Assistant (ASMCA) to configure and start Oracle ASM on all nodes in the cluster.

   See Also:

   Oracle Automatic Storage Management Administrator's Guide for more information about using ASMCA

3. Use ASMCA to create an Oracle ASM disk group that is at least the same size of the existing OCR and has at least normal redundancy.

   Notes:

   • If OCR is stored in an Oracle ASM disk group with external redundancy, then Oracle recommends that you add another OCR location to another disk group to avoid the loss of OCR, if a disk fails in the disk group.

     Oracle does not support storing OCR on different storage types simultaneously, such as storing OCR on both Oracle ASM and a shared file system, except during a migration.

   • If an Oracle ASM instance fails on any node, then OCR becomes unavailable on that particular node.

     If the CRSD process running on the node affected by the Oracle ASM instance failure is the OCR writer, the majority of the OCR locations are stored in Oracle ASM, and you attempt I/O on OCR during the time the Oracle ASM instance is down on this node, then CRSD stops and becomes inoperable. Cluster management is now affected on this particular node.

     Under no circumstances will the failure of one Oracle ASM instance on one node affect the whole cluster.

   • Ensure that Oracle ASM disk groups that you create are mounted on all of the nodes in the cluster.

   See Also:

   Oracle Grid Infrastructure Installation Guide for more detailed sizing information

4. To add OCR to an Oracle ASM disk group, ensure that the Oracle Clusterware stack is running and run the following command as root:

   # ocrconfig -add +new_disk_group
   

   You can run this command more than once if you add multiple OCR locations. You can have up to five OCR locations. However, each successive run must point to a different disk group.

5. To remove storage configurations no longer in use, run the following command as root:

   # ocrconfig -delete old_storage_location
   

   Run this command for every configured OCR.

The following example shows how to migrate two OCRs to Oracle ASM using OCRCONFIG.

# ocrconfig -add +new_disk_group
# ocrconfig -delete /ocrdata/ocr_1
# ocrconfig -delete /ocrdata/ocr_2

# ocrconfig -add /nas/ocr

Adding, Replacing, Repairing, and Removing Oracle Cluster Registry Locations

The Oracle installation process for Oracle Clusterware gives you the option of automatically mirroring OCR. You can manually put the mirrored OCRs on a shared Network File System (NFS), or on any cluster file system that is certified by Oracle. Alternatively, you can place OCR on Oracle ASM and allow it to create mirrors automatically, depending on the redundancy option you select.

This section includes the following topics:

• Adding an Oracle Cluster Registry Location

• Removing an Oracle Cluster Registry Location

• Replacing an Oracle Cluster Registry Location

• Repairing an Oracle Cluster Registry Configuration on a Local Node

• Overriding the Oracle Cluster Registry Data Loss Protection Mechanism

You can manually mirror OCR, as described in the "Adding an Oracle Cluster Registry Location" section, if you:

• Upgraded to Oracle Clusterware 12c but did not choose to mirror OCR during the upgrade

• Created only one OCR location during the Oracle Clusterware installation

Oracle recommends that you configure:

• At least three OCR locations, if OCR is configured on non-mirrored or non-redundant storage. Oracle strongly recommends that you mirror OCR if the underlying storage is not RAID. Mirroring can help prevent OCR from becoming a single point of failure.

• At least two OCR locations if OCR is configured on an Oracle ASM disk group. You should configure OCR in two independent disk groups. Typically this is the work area and the recovery area.

• At least two OCR locations if OCR is configured on mirrored hardware or third-party mirrored volumes.

In addition to mirroring OCR locations, you can also:

• Replace an OCR location if there is a misconfiguration or other type of OCR error, as described in the "Replacing an Oracle Cluster Registry Location" section.

• Repair an OCR location if Oracle Database displays an OCR failure alert in Oracle Enterprise Manager or in the Oracle Clusterware alert log file, as described in the "Repairing an Oracle Cluster Registry Configuration on a Local Node" section.

• Remove an OCR location if, for example, your system experiences a performance degradation due to OCR processing or if you transfer your OCR to RAID storage devices and choose to no longer use multiple OCR locations, as described in the "Removing an Oracle Cluster Registry Location" section.

# ocrconfig -add +asm_disk_group | file_name

ocrconfig -repair -replace current_OCR_location -replacement target_OCR_location

Backing Up Oracle Cluster Registry

This section describes how to back up OCR content and use it for recovery. The first method uses automatically generated OCR copies and the second method enables you to issue a backup command manually:

• Automatic backups: Oracle Clusterware automatically creates OCR backups every four hours. At any one time, Oracle Database always retains the last three backup copies of OCR. The CRSD process that creates the backups also creates and retains an OCR backup for each full day and after each week. You cannot customize the backup frequencies or the number of files that Oracle Database retains.

• Manual backups: Run the ocrconfig -manualbackup command on a node where the Oracle Clusterware stack is up and running to force Oracle Clusterware to perform a backup of OCR at any time, rather than wait for the automatic backup. You must run the command as a user with administrative privileges. The -manualbackup option is especially useful when you want to obtain a binary backup on demand, such as before you make changes to OCR. The OLR only supports manual backups.

When the clusterware stack is down on all nodes in the cluster, the backups that are listed by the ocrconfig -showbackup command may differ from node to node.

This section includes the following topics:

• Listing Backup Files

• Changing Backup Location

Listing Backup Files

Run the following command to list the backup files:

ocrconfig -showbackup

The ocrconfig -showbackup command displays the backup location, timestamp, and the originating node name of the backup files that Oracle Clusterware creates. By default, the -showbackup option displays information for both automatic and manual backups but you can include the auto or manual flag to display only the automatic backup information or only the manual backup information, respectively.

Run the following command to inspect the contents and verify the integrity of the backup file:

ocrdump -backupfile backup_file_name

You can use any backup software to copy the automatically generated backup files at least once daily to a different device from where the primary OCR resides.

The default location for generating backups on Linux or UNIX systems is Grid_home/cdata/cluster_name, where cluster_name is the name of your cluster. The Windows default location for generating backups uses the same path structure. Because the default backup is on a local file system, Oracle recommends that you include the backup file created with the OCRCONFIG utility as part of your operating system backup using standard operating system or third-party tools.

Changing Backup Location

Run the following command to change the location where OCR creates backups:

ocrconfig -backuploc file_name

The file_name variable in the preceding command can be a full directory path name that is accessible by all nodes, or it can be an Oracle ASM disk group that is mounted on all nodes. You must migrate OCR to Oracle ASM before changing the OCR backup location to an Oracle ASM disk group. You can change the OCR backup location to an Oracle ASM disk group only if there is at least one Oracle ASM OCR location in a separate disk group.

For example, to specify an OCR backup location in a directory:

# ocrconfig -backuploc Grid_home/cdata/cluster3

To specify an OCR backup location in an Oracle ASM disk group:

# ocrconfig –backuploc +bkupdg

Restoring Oracle Cluster Registry

If a resource fails, then before attempting to restore OCR, restart the resource. As a definitive verification that OCR failed, run ocrcheck and if the command returns a failure message, then both the primary OCR and the OCR mirror have failed. Attempt to correct the problem using the OCR restoration procedure for your platform.

• Restoring the Oracle Cluster Registry on Linux or UNIX Systems

• Restoring the Oracle Cluster Registry on Windows Systems

• Restoring the Oracle Cluster Registry in an Oracle Restart Environment

Restoring the Oracle Cluster Registry on Linux or UNIX Systems

If you are storing OCR on an Oracle ASM disk group, and that disk group is corrupt, then you must restore the Oracle ASM disk group using Oracle ASM utilities, and then mount the disk group again before recovering OCR. Recover OCR by running the ocrconfig -restore command, as instructed in the following procedure.

Use the following procedure to restore OCR on Linux or UNIX systems:

1. List the nodes in your cluster by running the following command on one node:

   $ olsnodes
   

2. Depending on whether OCR is located in an Oracle ASM disk group or on network attached storage (NAS), stop Oracle Clusterware by running one of the following commands as root on all of the nodes:

   If OCR is located in an Oracle ASM disk group, then stop the Oracle Clusterware daemon:

   # crsctl stop crs
   

   If the preceding command returns any error due to OCR corruption, then force stop Oracle Clusterware by running the following command as root on all of the nodes:

   # crsctl stop crs -f
   

3. If you are restoring OCR to a cluster file system or network file system, then run the following command as root to restore OCR with an OCR backup that you can identify in "Listing Backup Files":

   # ocrconfig -restore file_name
   

   Note:

   If the OCR backup file is located in an Oracle ASM disk group, then ensure that the disk group exists and is mounted.

   After you complete this step, proceed to step 11.

   If you are not restoring OCR to a cluster file system or network file system, then proceed to the next step.

4. Start the Oracle Clusterware stack on one node in exclusive mode by running the following command as root:

   # crsctl start crs -excl -nocrs
   

   The -nocrs option ensures that the CRSD process and OCR do not start with the rest of the Oracle Clusterware stack.

   Ignore any errors that display.

5. Check whether CRSD is running by running the following command:

   $ crsctl status resource ora.crsd -init
   

   If CRSD is running, then stop it by running the following command as root:

   # crsctl stop resource ora.crsd -init
   

   Caution:

   Do not use the -init flag with any other command unless directed to do so by My Oracle Support.

6. To restore OCR to an Oracle ASM disk group, you must first create a disk group using SQL*Plus that has the same name as the disk group you want to restore and mount it on the local node.

   If you cannot mount the disk group locally, then run the following SQL*Plus command:

   SQL> drop diskgroup disk_group_name force including contents;
   

7. Restore OCR with an OCR backup that you can identify in "Listing Backup Files" by running the following command as root:

   # ocrconfig -restore file_name
   

   Notes:

   • If the original OCR location does not exist, then you must create an empty (0 byte) OCR location before you run the ocrconfig -restore command.

   • Ensure that the OCR devices that you specify in the OCR configuration exist and that these OCR devices are valid.

   • If you configured OCR in an Oracle ASM disk group, then ensure that the Oracle ASM disk group exists and is mounted.

   • If the OCR backup file is located in an Oracle ASM disk group, then ensure that the disk group exists and is mounted.

   See Also:

   • Oracle Grid Infrastructure Installation Guide for information about creating OCRs

   • Oracle Automatic Storage Management Administrator's Guide for more information about Oracle ASM disk group management

8. Verify the integrity of OCR:

   # ocrcheck
   

9. Stop Oracle Clusterware on the node where it is running in exclusive mode:

   # crsctl stop crs -f
   

10. Run the ocrconfig -repair -replace command as root on all the nodes in the cluster where you did not the ocrconfig -restore command. For example, if you ran the ocrconfig -restore command on node 1 of a four-node cluster, then you must run the ocrconfig -repair -replace command on nodes 2, 3, and 4.

11. Begin to start Oracle Clusterware by running the following command as root on all of the nodes:

    # crsctl start crs
    

12. Verify OCR integrity of all of the cluster nodes that are configured as part of your cluster by running the following CVU command:

    $ cluvfy comp ocr -n all -verbose
    

Restoring the Oracle Cluster Registry on Windows Systems

If you are storing OCR on an Oracle ASM disk group, and that disk group is corrupt, then you must restore the Oracle ASM disk group using Oracle ASM utilities, and then mount the disk group again before recovering OCR. Recover OCR by running the ocrconfig -restore command.

Use the following procedure to restore OCR on Windows systems:

1. List the nodes in your cluster by running the following command on one node:

   C:\>olsnodes
   

2. Stop Oracle Clusterware by running the following command as a member of the Administrators group on all of the nodes:

   C:\>crsctl stop crs
   

   If the preceding command returns any error due to OCR corruption, stop Oracle Clusterware by running the following command as a member of the Administrators group on all of the nodes:

   C:\>crsctl stop crs -f
   

3. Start the Oracle Clusterware stack on one node in exclusive mode by running the following command as a member of the Administrators group:

   C:\>crsctl start crs -excl -nocrs
   

   The -nocrs option ensures that the CRSD process and OCR do not start with the rest of the Oracle Clusterware stack.

   Ignore any errors that display.

4. Restore OCR with the OCR backup file that you identified in "Listing Backup Files" by running the following command as a member of the Administrators group:

   C:\>ocrconfig -restore file_name
   

   Make sure that the OCR devices that you specify in the OCR configuration exist and that these OCR devices are valid.

   Notes:

   • Ensure that the OCR devices that you specify in the OCR configuration exist and that these OCR devices are valid.

   • Ensure that the Oracle ASM disk group you specify exists and is mounted.

   See Also:

   • Oracle Grid Infrastructure Installation Guide for information about creating OCRs

   • Oracle Automatic Storage Management Administrator's Guide for more information about Oracle ASM disk group management

5. Verify the integrity of OCR:

   C:\>ocrcheck
   

6. Stop Oracle Clusterware on the node where it is running in exclusive mode:

   C:\>crsctl stop crs -f
   

7. Begin to start Oracle Clusterware by running the following command as a member of the Administrators group on all of the nodes:

   C:\>crsctl start crs
   

8. Run the following Cluster Verification Utility (CVU) command to verify OCR integrity of all of the nodes in your cluster database:

   C:\>cluvfy comp ocr -n all -verbose
   

   See Also:

   Appendix A, "Cluster Verification Utility Reference" for more information about enabling and using CVU

Restoring the Oracle Cluster Registry in an Oracle Restart Environment

Use the following procedure to restore OCR in an Oracle Restart environment:

1. Stop Oracle High Availability Services by running the following command as root on all of the nodes:

   # crsctl stop has [-f]
   

2. Run the ocrcheck -config command to determine the OCR location and then create an empty (0 byte) OCR location with appropriate permissions in that location.

3. Restore OCR by running the following command as root:

   # crsctl pin css -n host_name
   

   Notes:

   Ensure that the OCR devices that you specify in the OCR configuration exist and that these OCR devices are valid.

   See Also:

   Oracle Grid Infrastructure Installation Guide for information about creating OCRs

4. Run the ocrcheck command to verify the integrity of OCR:

5. Start Oracle High Availability Services by running the following command on all of the nodes:

   $ crsctl start has
   

Diagnosing Oracle Cluster Registry Problems

You can use the OCRDUMP and OCRCHECK utilities to diagnose OCR problems.

Administering Oracle Cluster Registry with Export and Import Commands

In addition to using the automatically created OCR backup files, you should also export OCR contents before and after making significant configuration changes, such as adding or deleting nodes from your environment, modifying Oracle Clusterware resources, and upgrading, downgrading or creating a database. Do this by using the ocrconfig -export command, which exports OCR content to a file format.

Using the ocrconfig -export command also enables you to restore OCR using the -import option if your configuration changes cause errors. For example, if you have configuration problems that you cannot resolve, or if you are unable to restart Oracle Clusterware after such changes, then restore your configuration using the procedure for your platform.

Oracle recommends that you use either automatic or manual backups, and the ocrconfig -restore command instead of the ocrconfig -export and ocrconfig -import commands to restore OCR for the following reasons:

• A backup is a consistent snapshot of OCR, whereas an export is not.

• Backups are created when the system is online. You must shut down Oracle Clusterware on all nodes in the cluster to get a consistent snapshot using the ocrconfig -export command.

• You can inspect a backup using the OCRDUMP utility. You cannot inspect the contents of an export.

• You can list backups with the ocrconfig -showbackup command, whereas you must keep track of all generated exports.

This section includes the following topics:

• Importing Oracle Cluster Registry Content on Linux or UNIX Systems

• Importing Oracle Cluster Registry Content on Windows Systems

Oracle Local Registry

In Oracle Clusterware 12c, each node in a cluster has a local registry for node-specific resources, called an Oracle Local Registry (OLR), that is installed and configured when Oracle Clusterware installs OCR. Multiple processes on each node have simultaneous read and write access to the OLR particular to the node on which they reside, regardless of whether Oracle Clusterware is running or fully functional.

By default, OLR is located at Grid_home/cdata/host_name.olr on each node.

Manage OLR using the OCRCHECK, OCRDUMP, and OCRCONFIG utilities as root with the -local option.

• You can check the status of OLR on the local node using the OCRCHECK utility, as follows:

  # ocrcheck -local
  
  Status of Oracle Cluster Registry is as follows :
          Version                  :          3
          Total space (kbytes)     :     262132
          Used space (kbytes)      :       9200
          Available space (kbytes) :     252932
          ID                       :  604793089
          Device/File Name         : /private2/crs/cdata/localhost/dglnx6.olr
                                     Device/File integrity check succeeded
  
          Local OCR integrity check succeeded
  

• You can display the content of OLR on the local node to the text terminal that initiated the program using the OCRDUMP utility, as follows:

  # ocrdump -local -stdout
  

• You can perform administrative tasks on OLR on the local node using the OCRCONFIG utility.

  • To export OLR to a file:

    # ocrconfig –local –export file_name
    

    Notes:

    • Oracle recommends that you use the -manualbackup and -restore commands and not the -import and -export commands.

    • When exporting OLR, Oracle recommends including "olr", the host name, and the timestamp in the name string. For example:

      olr_myhost1_20090603_0130_export
      

  • To import a specified file to OLR:

    # ocrconfig –local –import file_name
    

  • To manually back up OLR:

    # ocrconfig –local –manualbackup
    

    Note:

    The OLR is backed up after an installation or an upgrade. After that time, you can only manually back up the OLR. Automatic backups are not supported for the OLR. You should create a new backup when you migrate OCR from Oracle ASM to other storage, or when you migrate OCR from other storage to Oracle ASM.

    The default backup location for the OLR is in the path Grid_home/cdata/host_name.

  • To view the contents of the OLR backup file:

    ocrdump -local -backupfile olr_backup_file_name
    

  • To change the OLR backup location:

    ocrconfig -local -backuploc new_olr_backup_path
    

  • To restore OLR:

    # crsctl stop crs
    # ocrconfig -local -restore file_name
    # ocrcheck -local
    # crsctl start crs
    $ cluvfy comp olr
    

Upgrading and Downgrading the Oracle Cluster Registry Configuration

When you upgrade Oracle Clusterware, it automatically runs the ocrconfig -upgrade command. To downgrade, follow the downgrade instructions for each component and also downgrade OCR using the ocrconfig -downgrade command. If you are upgrading OCR, then you can use the OCRCHECK utility to verify the integrity of OCR.

Storing Voting Files on Oracle ASM

Oracle ASM manages voting files differently from other files that it stores. If you choose to store your voting files in Oracle ASM, then Oracle ASM stores all the voting files for the cluster in the disk group you choose. You cannot use voting files stored in Oracle ASM and voting files not stored in Oracle ASM in the same cluster.

Once you configure voting files on Oracle ASM, you can only make changes to the voting files' configuration using the crsctl replace votedisk command. This is true even in cases where there are no working voting files. Despite the fact that crsctl query css votedisk reports zero vote disks in use, Oracle Clusterware remembers the fact that Oracle ASM was in use and the replace verb is required. Only after you use the replace verb to move voting files back to non-Oracle ASM storage are the verbs add css votedisk and delete css votedisk again usable.

The number of voting files you can store in a particular Oracle ASM disk group depends upon the redundancy of the disk group.

By default, Oracle ASM puts each voting file in its own failure group within the disk group. A failure group is a subset of the disks in a disk group. Failure groups define disks that share components, such that if one fails then other disks sharing the component might also fail. An example of what you might define as a failure group would be a set of SCSI disks sharing the same SCSI controller. Failure groups are used to determine which Oracle ASM disks to use for storing redundant data. For example, if two-way mirroring is specified for a file, then redundant copies of file extents must be stored in separate failure groups.

The redundancy level that you choose for the Oracle ASM disk group determines how Oracle ASM mirrors files in the disk group, and determines the number of disks and amount of disk space that you require. If the voting files are in a disk group, then the disk groups that contain Oracle Clusterware files (OCR and voting files) have a higher minimum number of failure groups than other disk groups because the voting files are stored in quorum failure groups.

A quorum failure group is a special type of failure group that is used to store the Oracle Clusterware voting files. The quorum failure group is used to ensure that a quorum of the specified failure groups are available. When Oracle ASM mounts a disk group that contains Oracle Clusterware files, the quorum failure group is used to determine if the disk group can be mounted if there is a loss of one or more failure groups. Disks in the quorum failure group do not contain user data, therefore a quorum failure group is not considered when determining redundancy requirements in respect to storing user data.

Redundancy levels include:

• External redundancy: An external redundancy disk group requires a minimum of one disk device. The effective disk space in an external redundancy disk group is the sum of the disk space in all of its devices.

  Because Oracle ASM does not mirror data in an external redundancy disk group, Oracle recommends that you use external redundancy with storage devices such as RAID, or other similar devices that provide their own data protection mechanisms.

• Normal redundancy: A normal redundancy disk group requires a minimum of two disk devices (or two failure groups). The effective disk space in a normal redundancy disk group is half the sum of the disk space in all of its devices.

  For Oracle Clusterware files, a normal redundancy disk group requires a minimum of three disk devices (two of the three disks are used by failure groups and all three disks are used by the quorum failure group) and provides three voting files and one OCR and mirror of the OCR. When using a normal redundancy disk group, the cluster can survive the loss of one failure group.

• High redundancy: In a high redundancy disk group, Oracle ASM uses three-way mirroring to increase performance and provide the highest level of reliability. A high redundancy disk group requires a minimum of three disk devices (or three failure groups). The effective disk space in a high redundancy disk group is one-third the sum of the disk space in all of its devices.

  For Oracle Clusterware files, a high redundancy disk group requires a minimum of five disk devices (three of the five disks are used by failure groups and all five disks are used by the quorum failure group) and provides five voting files and one OCR and two mirrors of the OCR. With high redundancy, the cluster can survive the loss of two failure groups.

Using the crsctl replace votedisk command, you can move a given set of voting files from one Oracle ASM disk group into another, or onto a certified file system. If you move voting files from one Oracle ASM disk group to another, then you can change the number of voting files by placing them in a disk group of a different redundancy level as the former disk group.

Backing Up Voting Files

Oracle Clusterware automatically backs up voting file data in OCR as part of any configuration change and automatically restores the data to any voting file added. If all voting files are corrupted, however, you can restore them as described in "Restoring Voting Files".

Restoring Voting Files

If all of the voting files are corrupted, then you can restore them, as follows:

1. Restore OCR as described in "Restoring Oracle Cluster Registry", if necessary.

   This step is necessary only if OCR is also corrupted or otherwise unavailable, such as if OCR is on Oracle ASM and the disk group is no longer available.

   See Also:

   Oracle Automatic Storage Management Administrator's Guide for more information about managing Oracle ASM disk groups

2. Run the following command as root from only one node to start the Oracle Clusterware stack in exclusive mode, which does not require voting files to be present or usable:

   # crsctl start crs -excl
   

3. Run the crsctl query css votedisk command to retrieve the list of voting files currently defined, similar to the following:

   $ crsctl query css votedisk
   --  -----    -----------------                --------- ---------
   ##  STATE    File Universal Id                File Name Disk group
    1. ONLINE   6f57843d89464c46ea747362e8a3fa43 (/dev/sdb1) [DATA]
    2. ONLINE   7c54856e98474f61bf349401e7c9fb95 (/dev/sdc1) [DATA]
    3. ONLINE   9c46232b76234f61fc934673d5c8ec59 (/dev/sdd1) [DATA]
   

   This list may be empty if all voting files were corrupted, or may have entries that are marked as status 3 or OFF.

4. Depending on where you store your voting files, do one of the following:

   • If the voting files are stored in Oracle ASM, then run the following command to migrate the voting files to the Oracle ASM disk group you specify:

     crsctl replace votedisk +asm_disk_group
     

     The Oracle ASM disk group to which you migrate the voting files must exist in Oracle ASM. You can use this command whether the voting files were stored in Oracle ASM or some other storage device.

   • If you did not store voting files in Oracle ASM, then run the following command using the File Universal Identifier (FUID) obtained in the previous step:

     $ crsctl delete css votedisk FUID
     

     Add a voting file, as follows:

     $ crsctl add css votedisk path_to_voting_disk
     

5. Stop the Oracle Clusterware stack as root:

   # crsctl stop crs
   

   Note:

   If the Oracle Clusterware stack is running in exclusive mode, then use the -f option to force the shutdown of the stack.

6. Restart the Oracle Clusterware stack in normal mode as root:

   # crsctl start crs
   

Adding, Deleting, or Migrating Voting Files

You can add, remove, and migrate voting files after you install Oracle Clusterware. Note that the commands you use to do this are different, depending on whether your voting files are located in Oracle ASM, or are located in another storage option.

Modifying voting files that are stored in Oracle ASM

• To display the voting file FUID and file path of each current voting file, run the crsctl query css votedisk command to display output similar to the following:

  $ crsctl query css votedisk
  --  -----    -----------------                --------- ---------
  ##  STATE    File Universal Id                File Name Disk group
   1. ONLINE   6f57843d89464c46ea747362e8a3fa43 (/dev/sdb1) [DATA]
   2. ONLINE   7c54856e98474f61bf349401e7c9fb95 (/dev/sdc1) [DATA]
   3. ONLINE   9c46232b76234f61fc934673d5c8ec59 (/dev/sdd1) [DATA]
  

  This command returns a disk sequence number, the status of the disk, the FUID, the path of the disk, and the name of the Oracle ASM disk group on which the disk is stored.

• To migrate voting files from Oracle ASM to an alternative storage device, specify the path to the non-Oracle ASM storage device with which you want to replace the Oracle ASM disk group using the following command:

  $ crsctl replace votedisk path_to_voting_disk
  

  You can run this command on any node in the cluster.

• To replace all voting files not stored in Oracle ASM with voting files managed by Oracle ASM in an Oracle ASM disk group, run the following command:

  $ crsctl replace votedisk +asm_disk_group
  

Modifying voting files that are not stored on Oracle ASM:

• To display the voting file FUID and file path of each current voting file, run the following command:

  $ crsctl query css votedisk
  ##  STATE    File Universal Id                File Name Disk group
  --  -----    -----------------                --------- ---------
   1. ONLINE   7c54856e98474f61bf349401e7c9fb95 (/cfs/host09_vd3) []
  

  This command returns a disk sequence number, the status of the disk, the FUID, and the path of the disk and no name of an Oracle ASM disk group.

• To add one or more voting files, run the following command, replacing the path_to_voting_disk variable with one or more space-delimited, complete paths to the voting files you want to add:

  $ crsctl add css votedisk path_to_voting_disk [...]
  

• To replace voting file A with voting file B, you must add voting file B, and then delete voting file A. To add a new disk and remove the existing disk, run the following command, replacing the path_to_voting_diskB variable with the fully qualified path name of voting file B:

  $ crsctl add css votedisk path_to_voting_diskB -purge
  

  The -purge option deletes existing voting files.

• To remove a voting file, run the following command, specifying one or more space-delimited, voting file FUIDs or comma-delimited directory paths to the voting files you want to remove:

  $ crsctl delete css votedisk {FUID | path_to_voting_disk[...]}
  

# crsctl start crs -excl

# crsctl replace votedisk path_to_voting_disk

Migrating voting files to Oracle ASM

To migrate voting files to Oracle ASM, specify the Oracle ASM disk group name in the following command:

$ crsctl replace votedisk +asm_disk_group

You can run this command on any node in the cluster.

Verifying the voting file location

After modifying the voting file, verify the voting file location, as follows:

$ crsctl query css votedisk