6 Configuring Storage for Oracle Grid Infrastructure and Oracle RAC

6.1.2.1 About Oracle ACFS and Oracle ADVM

Oracle ACFS extends Oracle ASM technology to support of all of your application data in both single instance and cluster configurations. Oracle ADVM provides volume management services and a standard disk device driver interface to clients. Oracle Automatic Storage Management Cluster File System communicates with Oracle ASM through the Oracle Automatic Storage Management Dynamic Volume Manager interface.

6.1.2.2 Restrictions and Guidelines for Oracle ACFS

Note the following about Oracle ACFS:

• Oracle ACFS and Oracle ADVM are not supported on IBM AIX Workload Partitions (WPARs).

• Oracle Automatic Storage Management Cluster File System (Oracle ACFS) provides a general purpose file system. You can place Oracle Database binaries and Oracle Database files on this system, but you cannot place Oracle Clusterware files on Oracle ACFS.

  For policy-managed Oracle Flex Cluster databases, be aware that Oracle ACFS can run on Hub Nodes, but cannot run on Leaf Nodes. For this reason, Oracle RAC binaries cannot be placed on Oracle ACFS on Leaf Nodes.

• For Oracle Flex Clusters, Leaf Nodes cannot mount an Oracle home on ACFS from the Hub Nodes; it is not supported for some nodes to access the same Oracle home using NFS while other nodes use ACFS for the same Oracle home path.

• Oracle Restart does not support root-based Oracle Clusterware resources. For this reason, the following restrictions apply if you run Oracle ACFS on an Oracle Restart Configuration

  • You must manually load and unload Oracle ACFS drivers.

  • You must manually mount and unmount Oracle ACFS file systems, after the Oracle ASM instance is running

  • With Oracle Restart no Oracle Grid Infrastructure resources can be defined for file systems. Therefore ACFS file systems cannot be used for database homes or data files.

  • You cannot place Oracle ACFS database home file systems into the Oracle ACFS mount registry. The mount registry is entirely removed with Oracle Grid Infrastructure 12c release 1.

• You cannot place Oracle Clusterware binaries and files on Oracle ACFS.

• With Oracle Grid Infrastructure for a cluster, creating Oracle data files on an Oracle ACFS file system is supported from Oracle Database 12c Release 1.

• You can place Oracle Database binaries and administrative files (for example, trace files) on Oracle ACFS.

• Oracle ACFS does not support replication or encryption with Oracle Database data files, tablespace files, control files, and redo logs.

6.1.3.1 General Storage Considerations for Oracle Clusterware

Oracle Clusterware voting disks are used to monitor cluster node status, and Oracle Cluster Registry (OCR) files contain configuration information about the cluster. You can place voting disks and OCR files either in an ASM diskgroup, or on a cluster file system or shared network file system. Storage must be shared; any node that does not have access to an absolute majority of voting disks (more than half) will be restarted.

6.1.3.2 General Storage Considerations for Oracle RAC

For Standard Edition and Standard Edition 2 (SE2) Oracle RAC installations, Oracle ASM is the only supported storage option for database and recovery files. For all installations, Oracle recommends that you create at least two separate Oracle ASM disk groups: One for Oracle Database data files, and one for recovery files. Oracle recommends that you place the Oracle Database disk group and the recovery files disk group in separate failure groups.

If you do not use Oracle ASM, then Oracle recommends that you place the data files and the Fast Recovery Area in shared storage located outside of the Oracle home, in separate locations, so that a hardware failure does not affect availability.

Note the following additional guidelines for supported storage options:

• You can choose any combination of the supported storage options for each file type provided that you satisfy all requirements listed for the chosen storage options.

• If you intend to use Oracle ASM with Oracle RAC, and you are configuring a new Oracle ASM instance, then your system must meet the following conditions:

  • All nodes on the cluster have Oracle Clusterware and Oracle ASM 12c Release 1 (12.1) installed as part of an Oracle Grid Infrastructure for a cluster installation.

  • Any existing Oracle ASM instance on any node in the cluster is shut down.

• If you do not have a storage option that provides external file redundancy, then you must configure at least three voting disk areas to provide voting disk redundancy.

6.2.4.1 About Direct NFS Client Storage

With Oracle Database, instead of using the operating system kernel NFS client, you can configure Oracle Database to access NFS servers directly using an Oracle internal client called Direct NFS Client. Direct NFS Client supports NFSv3, NFSv4 and NFSv4.1 protocols (excluding the Parallel NFS extension) to access the NFS server.

To enable Oracle Database to use Direct NFS Client, the NFS file systems must be mounted and available over regular NFS mounts before you start installation. Direct NFS Client manages settings after installation. If Oracle Database cannot open an NFS server using Direct NFS Client, then Oracle Database uses the platform operating system kernel NFS client. You should still set the kernel mount options as a backup, but for normal operation, Direct NFS Client uses its own NFS client.

Direct NFS Client supports up to four network paths to the NFS server. Direct NFS Client performs load balancing across all specified paths. If a specified path fails, then Direct NFS Client reissues I/O commands over any remaining paths.

Some NFS file servers require NFS clients to connect using reserved ports. If your filer is running with reserved port checking, then you must disable reserved port checking for Direct NFS Client to operate. To disable reserved port checking, consult your NFS file server documentation.

For NFS servers that restrict port range, you can use the insecure option to enable clients other than root to connect to the NFS server. Alternatively, you can disable Direct NFS Client as described in Section 6.3.10, "Disabling Direct NFS Client Oracle Disk Management Control of NFS".

6.2.4.2 About Direct NFS Client Configuration

Direct NFS Client uses either the configuration file $ORACLE_HOME/dbs/oranfstab or the operating system mount tab file /etc/mtab to find out what mount points are available. If oranfstab is not present, then by default Direct NFS Client servers mount entries found in /etc/mtab. No other configuration is required. You can use oranfstab to specify additional specific Oracle Database operations to use Direct NFS Client. For example, you can use oranfstab to specify additional paths for a mount point.

Direct NFS Client supports up to four network paths to the NFS server. Direct NFS Client performs load balancing across all specified paths. If a specified path fails, then Direct NFS Client reissues I/O commands over any remaining paths.

6.2.4.3 Using the oranfstab File with Direct NFS Client

If you use Direct NFS Client, then you can choose to use a new file specific for Oracle data file management, oranfstab, to specify additional options specific for Oracle Database to Direct NFS Client. For example, you can use oranfstab to specify additional paths for a mount point. You can add the oranfstab file either to /etc or to $ORACLE_HOME/dbs.

With shared Oracle homes, when the oranfstab file is placed in $ORACLE_HOME/dbs, the entries in the file are specific to a single database. In this case, all nodes running an Oracle RAC database use the same $ORACLE_HOME/dbs/oranfstab file. In non-shared Oracle RAC installs, oranfstab must be replicated on all nodes.

When the oranfstab file is placed in /etc, it is globally available to all Oracle databases, and can contain mount points used by all Oracle databases running on nodes in the cluster, including standalone databases. However, on Oracle RAC systems, if the oranfstab file is placed in /etc, then you must replicate the file /etc/oranfstab file on all nodes, and keep each /etc/oranfstab file synchronized on all nodes, just as you must with the /etc/fstab file.

In all cases, mount points must be mounted by the kernel NFS system, even when they are being served using Direct NFS Client. Refer to your vendor documentation to complete operating system NFS configuration and mounting.

6.2.4.4 About Mounting NFS Storage Devices with Direct NFS Client

Direct NFS Client determines mount point settings to NFS storage devices based on the configurations in /etc/mtab, which are changed with configuring the /etc/fstab file.

Direct NFS Client searches for mount entries in the following order:

1. $ORACLE_HOME/dbs/oranfstab

2. /etc/oranfstab

3. /etc/mtab

Direct NFS Client uses the first matching entry as the mount point.

Oracle Database requires that mount points be mounted by the kernel NFS system even when served through Direct NFS Client.

If Oracle Database uses Direct NFS Client mount points configured using oranfstab, then it first verifies kernel NFS mounts by cross-checking entries in oranfstab with operating system NFS mount points. If a mismatch exists, then Direct NFS Client logs an informational message, and does not operate.

If Oracle Database cannot open an NFS server using Direct NFS Client, then Oracle Database uses the platform operating system kernel NFS client. In this case, the kernel NFS mount options must be set up as defined in Section 6.3.3, "Checking NFS Mount and Buffer Size Parameters for Oracle RAC". Additionally, an informational message is logged into the Oracle alert and trace files indicating that Direct NFS Client could not connect to an NFS server.

Section 6.1.1, "Supported Storage Options" lists the file types that are supported by Direct NFS Client.

The Oracle files resident on the NFS server that are served by Direct NFS Client are also accessible through the operating system kernel NFS client.

6.2.6.1 Overview of Requirements for Using HACMP with Oracle Clusterware

You must define one Multi-node Disk Heartbeat (MNDHB) network for each Oracle Clusterware voting disk. Each MNDHB and voting disk pair must be located on a single hard disk, separate from the other pairs. You must also configure MNDHB so that the node is halted if access is lost to a quorum of the MNDHB networks in the enhanced concurrent volume group.

To reduce the likelihood of a cluster partition, IBM recommends that HACMP is deployed with multiple IP networks and at least one non-IP network. The non-IP networks can be implemented using RS232 or disk heart-beating. For systems using Oracle RAC and HACMP enhanced concurrent resources (enhanced concurrent logical volumes) for database storage, you must configure MNDHB networks.

Install, configure and have HACMP running before installing Oracle Clusterware. For an Oracle RAC configuration, do not use HACMP for IP failovers on the Oracle RAC network interfaces (public, VIP or private). These network interfaces should not be configured to use HACMP IP failover, as Oracle Clusterware manages VIP failovers for Oracle RAC. The Oracle RAC network interfaces are bound to individual nodes and Oracle RAC instances. Problems can occur with Oracle Clusterware if HACMP reconfigures IP addresses over different interfaces, or fails over addresses across nodes. You only can use HACMP for failover of IP address on Oracle RAC nodes if Oracle RAC does not use these addresses.

6.2.6.2 Deploying HACMP and MDNDHB for Oracle Clusterware

Complete the following tasks, replacing each term in italics with the appropriate response for your system, or carrying out the action described and entering the appropriate response for your image:

1. Start HACMP.

2. Enter the following command to ensure that the HACMP clcomdES daemon is running:

   # lssrc -s clcomdES
   

   If the daemon is not running, then start it using the following command:

   # startsrc –s clcomdES
   

3. Ensure that your versions of HACMP and AIX meet the system requirements listed in Section 3.6, "Operating System Requirements for IBM AIX on POWER Systems (64-Bit)".

4. Create HACMP cluster and add the Oracle Clusterware nodes. For example:

   # smitty cm_add_change_show_an_hacmp_cluster.dialog
   * Cluster Name [mycluster] 
   

5. Create an HACMP cluster node for each Oracle Clusterware node. For example:

   # smitty cm_add_a_node_to_the_hacmp_cluster_dialog 
   * Node Name [mycluster_node1]
   Communication Path to Node [] 
   

6. Create HACMP Ethernet heartbeat networks. The HACMP configuration requires network definitions. Select NO for the IP address takeover for these networks, since they are used by Oracle Clusterware.

   Create at least two network definitions: one for the Oracle public interface and a second one for the Oracle private (cluster interconnect) network. Additional Ethernet heartbeat networks can be added if desired.

   For example:

   # smitty cm_add_a_network_to_the_hacmp_cluster_select 
   - select ether network 
   * Network Name [my_network_name] 
   * Network Type ether 
   * Netmask [my.network.netmask.here] 
   * Enable IP Address Takeover via IP Aliases [No] 
   IP Address Offset for Heart beating over IP Aliases [] 
   

7. For each of the networks added in the previous step, define all of the IP names for each Oracle Clusterware node associated with that network, including the public, private and VIP names for each Oracle Clusterware node. For example:

   # smitty cm_add_communication_interfaces_devices.select 
   - select: Add Pre-defined Communication Interfaces and Devices / Communication Interfaces / desired network 
   * IP Label/Address [node_ip_address] 
   * Network Type ether 
   * Network Name some_network_name 
   * Node Name [my_node_name] 
   Network Interface [] 
   

8. Create an HACMP resource group for the enhanced concurrent volume group resource with the following options:

   # smitty config_resource_group.dialog.custom 
   * Resource Group Name [my_resource_group_name] 
   * Participating Nodes (Default Node Priority) [mynode1,mynode2,mynode3] 
   Startup Policy Online On All Available Nodes 
   Fallover Policy Bring Offline (On Error Node Only) 
   Fallback Policy Never Fallback 
   

9. Create an AIX enhanced concurrent volume group (Big VG, or Scalable VG) using either the command smitty mkvg, or using command lines. The VG must contain at least one hard disk for each voting disk. You must configure at least three voting disks.

   In the following example, where you see default, accept the default response:

   # smitty _mksvg 
   VOLUME GROUP name [my_vg_name] PP SIZE in MB 
   * PHYSICAL VOLUME names [mydisk1,mydisk2,mydisk3] 
   Force the creation of a volume group? no 
   Activate volume group AUTOMATICALLY no at system restart? 
   Volume Group MAJOR NUMBER [] 
   Create VG Concurrent Capable? enhanced concurrent 
   Max PPs per VG in kilobytes default
   Max Logical Volumes default
   

10. Under "Change/Show Resources for a Resource Group (standard)", add the concurrent volume group to the resource group added in the preceding steps.

    For example:

    # smitty cm_change_show_resources_std_resource_group_menu_dmn.select 
    - select_resource_group_from_step_6
    Resource Group Name shared_storage 
    Participating Nodes (Default Node Priority) mynode1,mynode2,mynode3
    Startup Policy Online On All Available Nodes 
    Fallover Policy Bring Offline (On Error Node Only) 
    Fallback Policy Never Fallback 
    Concurrent Volume Groups [enter_VG_from_step_7]
    Use forced varyon of volume groups, if necessary false 
    Application Servers [] 
    

11. Using the following command, ensure that one MNDHB network is defined for each Oracle Clusterware voting disk. Each MNDHB and voting disk pair must be collocated on a single hard disk, separate from the other pairs. The MNDHB network and Voting Disks exist on shared logical volumes in an enhanced concurrent logical volume managed by HACMP as an enhanced concurrent resource. For each of the hard disks in the VG created in step 6 on which you want to place a voting disk logical volume (LV), create a MNDHB LV.

    # smitty cl_add_mndhb_lv 
    - select_resource_group_defined_in_step_6
    * Physical Volume name enter F4, then select a hard disk
    Logical Volume Name [] 
    Logical Volume Label [] 
    Volume Group name ccvg 
    Resource Group Name shared_storage 
    Network Name [n]
    

    Note:

    When you define the LVs for the Oracle Clusterware voting disks, they should be defined on the same disks: one for each disk, as used in this step for the MNDHB LVs.

12. Configure MNDHB so that the node is halted if access is lost to a quorum of the MNDHB networks in the enhanced concurrent volume group. For example:

    # smitty cl_set_mndhb_response 
    - select_the_VG_created_in_step_7 
    On loss of access Halt the node 
    Optional notification method [] 
    Volume Group ccvg 
    

13. Verify and Synchronize HACMP configuration. For example:

    # smitty cm_initialization_and_standard_config_menu_dmn 
    - select "Verify and Synchronize HACMP Configuration" 
    

    Enter Yes if prompted: "Would you like to import shared VG: ccvg, in resource group my_resource_group onto node: mynode to node: racha702 [Yes / No]:"

14. Add the Add the HACMP cluster node IP names to the file /usr/es/sbin/cluster/etc/rhosts.

6.2.6.3 Upgrading an Existing Oracle Clusterware and HACMP Installation

Complete the following procedure:

1. Back up all databases, and back up the Oracle Cluster Registry (OCR)

2. Shut down on all nodes all Oracle RAC databases, all node applications, and Oracle Clusterware.

3. Enter the following command to disable Oracle Clusterware from starting when nodes are restarted:

   # crsctl disable crs
   

4. Shut down HACMP on all nodes.

5. Install HACMP APAR IZ01809, following the directions in the README included with that APAR.

6. Determine if the existing voting disk LVs are already on separate hard disks, and if each of these disks have sufficient space (at least 256 MB for the MNDHB LVs. If this is true, then create a MNDHB LV on each of the hard disks. If this is not true, then create new MNDHB LVs and new voting disk LVs, located on separate hard disks using the following command, responding to the sections in italics with the appropriate information for your system:

   # smitty cl_add_mndhb_lv 
   - Select_resource_group
   * Physical Volume name Enter F4, then select disk for the MNDHB and Voting Disk pair
   Logical Volume Name [] 
   Logical Volume Label [] 
   Volume Group name ccvg 
   Resource Group Name shared_storage 
   Network Name [net_diskhbmulti_01] 
   

7. Verify and Synchronize HACMP configuration.

8. Start HACMP on all nodes.

9. If you added new LVs for voting disks in step 5, then replace each of the existing voting disks with the new ones.

10. Enter the following command to re-enable Oracle Clusterware:

    # crsctl enable crs
    

11. Start Oracle Clusterware on all nodes, and verify that all resources start correctly.

6.4.1.1 Identifying Storage Requirements for Oracle Automatic Storage Management

To identify the storage requirements for using Oracle ASM, you must determine how many devices and the amount of free disk space that you require. To complete this task, follow these steps:

1. Determine whether you want to use Oracle ASM for Oracle Clusterware files (OCR and voting files), Oracle Database files, recovery files, or all files except for Oracle Clusterware or Oracle Database binaries. Oracle Database files include data files, control files, redo log files, the server parameter file, and the password file.

   Note:

   • You do not have to use the same storage mechanism for Oracle Clusterware, Oracle Database files and recovery files. You can use a shared file system for one file type and Oracle ASM for the other.

   • There are two types of Oracle Clusterware files: OCR files and voting files. Each type of file can be stored on either Oracle ASM or a cluster file system. All the OCR files or all the voting files must use the same type of storage. You cannot have some OCR files stored in Oracle ASM and other OCR files in a cluster file system. However, you can use one type of storage for the OCR files and a different type of storage for the voting files if all files of each type use the same type of storage.

2. Choose the Oracle ASM redundancy level to use for the Oracle ASM disk group.

   The redundancy level that you choose for the Oracle ASM disk group determines how Oracle ASM mirrors files in the disk group and determines the number of disks and amount of free disk space that you require. If the voting files are in a disk group, then the disk groups that contain Oracle Clusterware files (OCR and voting files) have a higher minimum number of failure groups than other disk groups because the voting files are stored in quorum failure groups.

   A quorum failure group is a special type of failure group that is used to store the Oracle Clusterware voting files. The quorum failure group is used to ensure that a quorum of the specified failure groups are available. When Oracle ASM mounts a disk group that contains Oracle Clusterware files, the quorum failure group is used to determine if the disk group can be mounted in the event of the loss of one or more failure groups. Disks in the quorum failure group do not contain user data, therefore a quorum failure group is not considered when determining redundancy requirements in respect to storing user data.

   The redundancy levels are as follows:

   • External redundancy

     An external redundancy disk group requires a minimum of one disk device. The effective disk space in an external redundancy disk group is the sum of the disk space in all of its devices.

     Because Oracle ASM does not mirror data in an external redundancy disk group, Oracle recommends that you use external redundancy with storage devices such as RAID, or other similar devices that provide their own data protection mechanisms.

   • Normal redundancy

     In a normal redundancy disk group, to increase performance and reliability, Oracle ASM by default uses two-way mirroring. A normal redundancy disk group requires a minimum of two disk devices (or two failure groups). The effective disk space in a normal redundancy disk group is half the sum of the disk space in all of its devices.

     For Oracle Clusterware files, a normal redundancy disk group requires a minimum of three disk devices (two of the three disks are used by failure groups and all three disks are used by the quorum failure group) and provides three voting files and one OCR (one primary and one secondary copy). With normal redundancy, the cluster can survive the loss of one failure group.

     For most installations, Oracle recommends that you select normal redundancy.

   • High redundancy

     In a high redundancy disk group, Oracle ASM uses three-way mirroring to increase performance and provide the highest level of reliability. A high redundancy disk group requires a minimum of three disk devices (or three failure groups). The effective disk space in a high redundancy disk group is one-third the sum of the disk space in all of its devices.

     For Oracle Clusterware files, a high redundancy disk group requires a minimum of five disk devices (three of the five disks are used by failure groups and all five disks are used by the quorum failure group) and provides five voting files and one OCR (one primary and two secondary copies). With high redundancy, the cluster can survive the loss of two failure groups.

     While high redundancy disk groups do provide a high level of data protection, you should consider the greater cost of additional storage devices before deciding to select high redundancy disk groups.

   Note:

   After a disk group is created, you cannot alter the redundancy level of the disk group.

3. Determine the total amount of disk space that you require for Oracle Clusterware files, and for the database files and recovery files.

   Use Table 6-4 and Table 6-5 to determine the minimum number of disks and the minimum disk space requirements for installing Oracle Clusterware files, and installing the starter database, where you have voting files in a separate disk group:

   Table 6-4 Total Oracle Clusterware Storage Space Required by Redundancy Type

   Redundancy Level	Minimum Number of Disks	Oracle Cluster Registry (OCR) Files	Voting Files	Both File Types	Total Storage with Grid Infrastructure Management Repository
   External	1	400 MB	300 MB	700 MB	At least 5.9 GB for a cluster with 4 nodes or less (5.2 GB + 400 MB + 300 MB). Additional space required for clusters with 5 or more nodes. For example, a six-node cluster allocation should be at least 6.9 GB: (5.2 GB +2*(500 MB) +400 MB + 300 MB).
   Normal	3	At least 400 MB for each failure group, or 800 MB	900 MB	1.7 GBFoot 1	At least 12.1 GB for a cluster with 4 nodes or less (2*5.2 GB + 2*400 MB + 3*300 MB). Additional space required for clusters with 5 or more nodes. For example, for a six-node cluster allocation should be at least 14.1 GB: (2 * (5.2 GB +2*(500 MB)) +(2 * 400 MB) +(3 * 300 MB)).
   High	5	At least 400 MB for each failure group, or 1.2 GB	1.5 GB	2.7 GB	At least 18.3 GB for a cluster with 4 nodes or less (3* 5.2 GB + 3*400 MB + 5*300 MB). Additional space required for clusters with 5 or more nodes. For example, for a six-node cluster allocation should be at least 21.3 GB: (3* (5.2 GB +2*(500 MB))+(3 * 400 MB) +(5 * 300 MB)).

   
   

   ^Footnote 1 If you create a disk group during installation, then it must be at least 2 GB.

   Note:

   If the voting files are in a disk group, be aware that disk groups with Oracle Clusterware files (OCR and voting files) have a higher minimum number of failure groups than other disk groups.

   If you create a disk group as part of the installation in order to install the OCR and voting files, then the installer requires that you create these files on a disk group with at least 2 GB of available space.

   Table 6-5 Total Oracle Database Storage Space Required by Redundancy Type

   Redundancy Level	Minimum Number of Disks	Database Files	Recovery Files	Both File Types
   External	1	1.5 GB	3 GB	4.5 GB
   Normal	2	3 GB	6 GB	9 GB
   High	3	4.5 GB	9 GB	13.5 GB

   
   

4. Determine an allocation unit size. Every Oracle ASM disk is divided into allocation units (AU). An allocation unit is the fundamental unit of allocation within a disk group. You can select the AU Size value from 1, 2, 4, 8, 16, 32 or 64 MB, depending on the specific disk group compatibility level. The default value is set to 1 MB.

5. For Oracle Clusterware installations, you must also add additional disk space for the Oracle ASM metadata. You can use the following formula to calculate the disk space requirements (in MB) for OCR and voting files, and the Oracle ASM metadata:

   total = [2 * ausize * disks] + [redundancy * (ausize * (nodes * (clients + 1) + 30) + (64 * nodes) + 533)]

   Where:

   • redundancy = Number of mirrors: external = 1, normal = 2, high = 3.

   • ausize = Metadata AU size in megabytes.

   • nodes = Number of nodes in cluster.

   • clients - Number of database instances for each node.

   • disks - Number of disks in disk group.

   For example, for a four-node Oracle RAC installation, using three disks in a normal redundancy disk group, you require an additional X MB of space:

   [2 * 1 * 3] + [2 * (1 * (4 * (4 + 1)+ 30)+ (64 * 4)+ 533)] = 1684 MB

   To ensure high availability of Oracle Clusterware files on Oracle ASM, you need to have at least 2 GB of disk space for Oracle Clusterware files in three separate failure groups, with at least three physical disks. Each disk must have at least 1 GB of capacity to ensure that there is sufficient space to create Oracle Clusterware files.

6. Optionally, identify failure groups for the Oracle ASM disk group devices.

   If you intend to use a normal or high redundancy disk group, then you can further protect your database against hardware failure by associating a set of disk devices in a custom failure group. By default, each device comprises its own failure group. However, if two disk devices in a normal redundancy disk group are attached to the same SCSI controller, then the disk group becomes unavailable if the controller fails. The controller in this example is a single point of failure.

   To protect against failures of this type, you could use two SCSI controllers, each with two disks, and define a failure group for the disks attached to each controller. This configuration would enable the disk group to tolerate the failure of one SCSI controller.

   Note:

   Define custom failure groups after installation, using the GUI tool ASMCA, the command line tool asmcmd, or SQL commands.

   If you define custom failure groups, then for failure groups containing database files only, you must specify a minimum of two failure groups for normal redundancy disk groups and three failure groups for high redundancy disk groups.

   For failure groups containing database files and clusterware files, including voting files, you must specify a minimum of three failure groups for normal redundancy disk groups, and five failure groups for high redundancy disk groups.

   Disk groups containing voting files must have at least 3 failure groups for normal redundancy or at least 5 failure groups for high redundancy. Otherwise, the minimum is 2 and 3 respectively. The minimum number of failure groups applies whether or not they are custom failure groups.

7. If you are sure that a suitable disk group does not exist on the system, then install or identify appropriate disk devices to add to a new disk group. Use the following guidelines when identifying appropriate disk devices:

   • All of the devices in an Oracle ASM disk group should be the same size and have the same performance characteristics.

   • Do not specify multiple partitions on a single physical disk as a disk group device. Each disk group device should be on a separate physical disk.

   • Although you can specify a logical volume as a device in an Oracle ASM disk group, Oracle does not recommend their use because it adds a layer of complexity that is unnecessary with Oracle ASM. In addition, Oracle RAC requires a cluster logical volume manager in case you decide to use a logical volume with Oracle ASM and Oracle RAC.

     Oracle recommends that if you choose to use a logical volume manager, then use the logical volume manager to represent a single LUN without striping or mirroring, so that you can minimize the impact of the additional storage layer.

6.4.1.2 Creating Files on a NAS Device for Use with Oracle ASM

If you have a certified NAS storage device, then you can create zero-padded files in an NFS mounted directory and use those files as disk devices in an Oracle ASM disk group.

To create these files, follow these steps:

1. If necessary, create an exported directory for the disk group files on the NAS device.

   Refer to the NAS device documentation for more information about completing this step.

2. Switch user to root.

3. Create a mount point directory on the local system. For example:

   # mkdir -p /mnt/asm
   

4. To ensure that the NFS file system is mounted when the system restarts, add an entry for the file system in the mount file /etc/vfstab.

   See Also:

   My Oracle Support note 359515.1 for updated NAS mount option information, available at the following URL:

   https://support.oracle.com
   

   For more information about editing the mount file for the operating system, refer to the man pages. For more information about recommended mount options, refer to Configuring Operating System NFS Mount and Buffer Size Parameters.

5. Enter a command similar to the following to mount the NFS file system on the local system:

   # mount /mnt/asm
   

6. Choose a name for the disk group to create. For example: sales1.

7. Create a directory for the files on the NFS file system, using the disk group name as the directory name. For example:

   # mkdir /mnt/asm/nfsdg
   

8. Use commands similar to the following to create the required number of zero-padded files in this directory:

   # dd if=/dev/zero of=/mnt/asm/nfsdg/disk1 bs=1024k count=1000
   

   This example creates 1 GB files on the NFS file system. You must create one, two, or three files respectively to create an external, normal, or high redundancy disk group.

9. Enter commands similar to the following to change the owner, group, and permissions on the directory and files that you created, where the installation owner is grid, and the OSASM group is asmadmin:

   # chown -R grid:asmadmin /mnt/asm
   # chmod -R 660 /mnt/asm
   

10. If you plan to install Oracle RAC or a standalone Oracle Database, then during installation, edit the Oracle ASM disk discovery string to specify a regular expression that matches the file names you created. For example:

    /mnt/asm/sales1/
    

    Note:

    During installation, disk paths mounted on Oracle ASM are listed as default database storage candidate disks.

6.4.1.3 Using an Existing Oracle ASM Disk Group

Select from the following choices to store either database or recovery files in an existing Oracle ASM disk group, depending on installation method:

• If you select an installation method that runs Database Configuration Assistant in interactive mode, then you can decide whether you want to create a disk group, or to use an existing one.

  The same choice is available to you if you use Database Configuration Assistant after the installation to create a database.

• If you select an installation method that runs Database Configuration Assistant in noninteractive mode, then you must choose an existing disk group for the new database; you cannot create a disk group. However, you can add disk devices to an existing disk group if it has insufficient free space for your requirements.

To determine if an existing Oracle ASM disk group exists, or to determine if there is sufficient disk space in a disk group, you can use Oracle Enterprise Manager Cloud Control or the Oracle ASM command line tool (asmcmd) as follows:

1. Connect to the Oracle ASM instance and start the instance if necessary:

   $ $ORACLE_HOME/bin/asmcmd
   ASMCMD> startup
   

2. Enter one of the following commands to view the existing disk groups, their redundancy level, and the amount of free disk space in each one:

   ASMCMD> lsdb
   

   or:

   $ORACLE_HOME/bin/asmcmd -p lsdg
   

3. From the output, identify a disk group with the appropriate redundancy level and note the free space that it contains.

4. If necessary, install or identify the additional disk devices required to meet the storage requirements listed in the previous section.

   Note:

   If you are adding devices to an existing disk group, then Oracle recommends that you use devices that have the same size and performance characteristics as the existing devices in that disk group.

6.4.1.4 Configuring Disk Devices for Oracle ASM

You can configure raw disks for use as Oracle Automatic Storage Management (Oracle ASM) disk groups. To use Oracle ASM with raw disks, you must create sufficient partitions for your data files, and then bind the partitions to raw disks. Make a list of the raw disk names you create for the data files, and have the list available during database installation.

In the following procedure, you are directed to set physical volume IDs (PVIDs) for raw disks. Oracle recommends that you complete the entire procedure, even if you are certain that you do not have PVIDs configured on your system, to prevent the possibility of configuration issues.

/dev/rdlmfdrv8
/dev/rdlmfdrv9

Use the following procedure to configure disks:

1. If necessary, install the disks that you intend to use for the disk group and restart the system.

2. Identify or create the disks that you want to include in the Oracle ASM disk group. As the root user, enter the following command on any node to identify the device names for the disk devices that you want to use:

   # /usr/sbin/lspv | grep -i none 
   

   This command displays information similar to the following for each disk device that is not configured in a volume group:

   hdisk17         0009005fb9c23648                    None  
   

   In this example, hdisk17 is the device name of the disk and 0009005fb9c23648 is the physical volume ID (PVID).

3. If a disk device that you want to use does not have a PVID, then enter a command similar to the following to assign one to it, where n is the number of the hdisk:

   # chdev -l hdiskn -a pv=yes
   

   Note:

   If you have an existing PVID, then chdev overwrites the existing PVID. Be aware that if you have applications depending on the previous PVID, then they will fail.

4. On each of the other nodes, enter a command similar to the following to identify the device name associated with each PVID on that node:

   # /usr/sbin/lspv | grep -i "0009005fb9c23648"
   

   The output from this command should be similar to the following:

   hdisk18         0009005fb9c23648                    None
   

   In this example, the device name associated with the disk device (hdisk18) is different on this node.

5. If the device names are the same on all nodes, then enter commands similar to the following on all nodes to change the owner, group, and permissions on the character raw device files for the disk devices where grid is the Oracle Grid Infrastructure installation owner, and asmadmin is the OSASM group:

   # chown grid:asmadmin /dev/rhdiskn
   # chmod 660 /dev/rhdiskn
   

6. To enable simultaneous access to a disk device from multiple nodes, you must set the appropriate Object Data Manager (ODM) attribute, depending on the type of reserve attribute used by your disks. The following section describes how to perform this task using hdisk logical names. Refer to your operating system documentation to find logical device names.

   To determine the reserve setting your disks use, enter the following command, where n is the hdisk device number:

   # lsattr -E -l hdiskn | grep reserve_
   

   The response is either a reserve_lock setting, or a reserve_policy setting. If the attribute is reserve_lock, then ensure that the setting is reserve_lock = no. If the attribute is reserve_policy, then ensure that the setting is reserve_policy = no_reserve.

   If necessary, change the setting with the chdev command using the following syntax, where n is the hdisk device number:

   chdev -l hdiskn -a [ reserve_lock=no | reserve_policy=no_reserve ]
   

   For example, to change a setting for the device hdisk4 from reserve_lock=yes to reserve_lock=no, enter the following command:

   # chdev -l hdisk4  -a  reserve_lock=no
   

   To verify that the setting is correct on all disk devices, enter the following command:

   # lsattr -El hdiskn | grep reserve
   

7. Enter commands similar to the following on any node to clear the PVID from each disk device that you want to use:

   # /usr/sbin/chdev -l hdiskn -a pv=clear
   

   When you are installing Oracle Clusterware, you must enter the paths to the appropriate device files when prompted for the path of the OCR and Oracle Clusterware voting disk. For example:

   /dev/rhdisk10
   

6.4.2.1 Identifying and Using Existing Oracle Database Disk Groups on Oracle ASM

The following section describes how to identify existing disk groups and determine the free disk space that they contain.

• Optionally, identify failure groups for the Oracle Automatic Storage Management disk group devices.

  If you intend to use a normal or high redundancy disk group, then you can further protect your database against hardware failure by associating a set of disk devices in a custom failure group. By default, each device comprises its own failure group. However, if two disk devices in a normal redundancy disk group are attached to the same SCSI controller, then the disk group becomes unavailable if the controller fails. The controller in this example is a single point of failure.

  To protect against failures of this type, you could use two SCSI controllers, each with two disks, and define a failure group for the disks attached to each controller. This configuration would enable the disk group to tolerate the failure of one SCSI controller.

  Note:

  If you define custom failure groups, then you must specify a minimum of two failure groups for normal redundancy and three failure groups for high redundancy.

6.4.2.2 Creating Disk Groups for Oracle Database Data Files

If you are sure that a suitable disk group does not exist on the system, then install or identify appropriate disk devices to add to a new disk group. Use the following guidelines when identifying appropriate disk devices:

• All of the devices in an Oracle Automatic Storage Management disk group should be the same size and have the same performance characteristics.

• Do not specify multiple partitions on a single physical disk as a disk group device. Oracle Automatic Storage Management expects each disk group device to be on a separate physical disk.

• Although you can specify logical volumes as devices in an Oracle ASM disk group, Oracle does not recommend their use. Non-shared logical volumes are not supported with Oracle RAC. If you want to use logical volumes for your Oracle RAC database, then you must use shared and concurrent raw logical volumes created by a cluster-aware logical volume manager.

6.4.2.3 Creating and Using Oracle ASM Credentials File

An Oracle ASM Storage Client does not have Oracle ASM running on the nodes and uses Oracle ASM storage services in a different client cluster.

To create Oracle ASM credentials file, from the Grid_home/bin directory on the Storage Server, run the following command on one of the member nodes, where credential_file is the name and path location of the Oracle ASM credentials file you create:

Grid_home/bin/asmcmd mkcc client_cluster_name credential_file

For example:

Grid_home/bin/asmcmd mkcc clientcluster1 /home/grid/clientcluster1_credentials.xml

Copy the Oracle ASM credentials file to a secure path on the client cluster node where you run the client cluster installation. The Oracle Installation user must have permissions to access that file. Oracle recommends that no other user is granted permissions to access the Oracle ASM credentials file. During installation, you are prompted to provide a path to the file.