3 Configuring Storage for Oracle Grid Infrastructure and Oracle RAC

3.1.1 General Storage Considerations for Oracle Grid Infrastructure

Oracle Clusterware voting disks are used to monitor cluster node status, and the Oracle Cluster Registry (OCR) is a file that contains the configuration information and status of the cluster. The installer automatically initializes the OCR during the Oracle Clusterware installation. Oracle Database Configuration Assistant (DBCA) uses the OCR for storing the configurations for the cluster databases that it creates.

You can place voting disks and OCR files either in an Oracle ASM disk group, or on a cluster file system. Storage must be shared; any node that does not have access to an absolute majority of voting disks (more than half) will be restarted.

For a storage option to meet high availability requirements, the files stored on the disk must be protected by data redundancy, so that if one or more disks fail, then the data stored on the failed disks can be recovered. This redundancy can be provided externally using Redundant Array of Independent Disks (RAID) devices, or logical volumes on multiple physical devices and implement the stripe-and-mirror- everything methodology, also known as SAME. If you do not have a RAID devices or logical volumes, then you can create additional copies, or mirrors, of the files on different file systems. If you choose to mirror the files, then you must provide disk space for additional OCR files and at least two additional voting disk files.

Each OCR location should be placed on a different disk. For voting disk file placement, ensure that each file is configured so that it does not share any hardware device or disk, or other single point of failure with the other voting disks. Any node that does not have available to it an absolute majority of voting disks configured (more than half) will be restarted.

Use the following guidelines when choosing storage options for the Oracle Clusterware files:

• You can choose any combination of the supported storage options for each file type if you satisfy all requirements listed for the chosen storage options.

• You can use Oracle ASM 11g release 2 (11.2) for shared storage. You cannot use earlier Oracle ASM releases to do this.

• If you do not have a storage option that provides external file redundancy, then you must configure at least three voting disk locations to provide redundancy.

• The Oracle Grid Infrastructure home (Grid home) cannot be stored on a shared file system; it must be installed on a local disk.

• If you choose to store Oracle Clusterware files on Oracle ASM and use redundancy for the disk group, then Oracle ASM automatically maintains the ideal number of voting files based on the redundancy of the diskgroup. The voting files are created within a single disk group and you cannot add extra voting files to this disk group manually.

3.1.2 General Storage Considerations for Oracle RAC

For all Oracle RAC installations, you must choose the shared storage options to use for Oracle Database files. Oracle Database files include data files, control files, redo log files, the server parameter file, and the password file. To enable automated backups during the installation, you must also choose a shared storage option to use for recovery files (the fast recovery area).

• Guidelines for Choosing a Storage Option

• Guidelines for Using Oracle ASM Disk Groups for Storage

• Guidelines for Using Direct Network File System (NFS) with Oracle RAC

• Guidelines for Using Oracle ACFS with Oracle RAC

• Guidelines for Placing Data Files on a File System

• Guidelines for Placing Recovery Files on a File System

3.2.1 Supported Storage Options for Oracle Grid Infrastructure

You cannot install the Oracle Grid Infrastructure software on a cluster file system. The Oracle Grid Infrastructure home (Grid home) must be on a local, NTFS formatted disk.

There are two ways of storing the shared Oracle Clusterware files:

• Oracle ASM: You can install Oracle Clusterware files (OCR and voting disks) in Oracle ASM disk groups.

  Oracle ASM is the required database storage option for Typical installations, and for Standard Edition Oracle RAC installations. It is an integrated, high-performance database file system and disk manager for Oracle Clusterware and Oracle Database files. It performs striping and mirroring of database files automatically.

  Note:

  • You can no longer use OUI to install Oracle Clusterware or Oracle Database files directly on raw devices.

  • Only one Oracle ASM instance is permitted for each node regardless of the number of database instances on the node.

  • To store the Oracle Clusterware files in an Oracle ASM disk group, the disk group compatibility must be at least 11.2. See Oracle Automatic Storage Management Administrator's Guide for more information about disk group compatibility.

• OCFS for Windows: OCFS for Windows is the only other supported cluster file system that you can use to storage of Oracle Clusterware and Oracle RAC files on Microsoft Windows platforms. OCFS for Windows is not the same as OCFS2, which is available on Linux platforms.

  OCFS for Windows is included in the installation media for Oracle Grid Infrastructure and Oracle RAC on Microsoft Windows platforms and is installed automatically with Oracle Clusterware. However, for new installations, Oracle recommends that you use Oracle ASM to store the OCR and voting disk files. Oracle does not recommend using OCFS for Windows for Oracle Clusterware files.

  Note:

  You cannot put Oracle Clusterware files on Oracle Automatic Storage Management Cluster File System (Oracle ACFS). You cannot install Oracle Grid Infrastructure on a cluster file system.

  See Also:

  The Certify page on My Oracle Support for supported cluster file systems. See Section 2.4, "Checking Hardware and Software Certification on My Oracle Support".

Table 3-1, "Supported Storage Options for Oracle Clusterware and Oracle RAC Files and Home Directories" shows the storage options supported for storing Oracle Clusterware and Oracle RAC files.

3.2.2 Supported Storage Options for Oracle RAC

There are several ways of storing Oracle Database (Oracle RAC) files that must be shared across all the nodes:

• Oracle ASM: You can create Oracle RAC data files and recovery files in Oracle ASM disk groups.

  Oracle ASM is the required database storage option for Typical installations, and for Standard Edition Oracle RAC installations.

  Note:

  You can no longer use OUI to install Oracle Clusterware or Oracle Database files or binaries directly on raw devices.

  Only one Oracle ASM instance is permitted for each node regardless of the number of database instances on the node.

• A supported shared file system: Supported file systems include the following:

  • OCFS for Windows: OCFS for Windows is a cluster file system used to store Oracle Database binary and data files. If you intend to use OCFS for Windows for your database storage, then you should create partitions large enough for all the database and recovery files when you create the unformatted disk partitions that are used by OCFS for Windows.

  • Oracle ACFS: Oracle ACFS provides a general purpose file system that can store administrative files as well as external general purpose data files. You can install the Oracle Database software on Oracle ACFS.

    Note:

    You cannot put Oracle Clusterware or Oracle Database data files on Oracle ACFS.

  See Also:

  The Certify page on My Oracle Support for supported cluster file systems. See Section 2.4, "Checking Hardware and Software Certification on My Oracle Support".

• Network File System (NFS) with Oracle Direct NFS client: You can configure Oracle RAC to access NFS V3 servers directly using an Oracle internal Direct NFS client.

  Note:

  You cannot use Direct NFS to store Oracle Clusterware files. You can only use Direct NFS to store Oracle Database files. To install Oracle RAC on Windows using Direct NFS, you must have access to a shared storage method other than NFS for the Oracle Clusterware files.

  See Also:

  Section 3.8.1, "About Direct NFS Storage" for more information on using Direct NFS

Table 3-1 shows the storage options supported for storing Oracle Clusterware and Oracle RAC files.

3.3.1 Requirements for Using a Cluster File System for Oracle Clusterware Files

To use OCFS for Windows for Oracle Clusterware files, you must comply with the following requirements:

• If you choose to place your OCR files on a shared file system, then Oracle recommends that one of the following is true:

  • The disks used for the file system are on a highly available storage device, (for example, a RAID device that implements file redundancy)

  • At least two file systems are mounted, and you use the features of Oracle Clusterware 11g release 2 (11.2) to provide redundancy for the OCR and voting disks

• If you use a RAID device to store the Oracle Clusterware files, then you must have a partition that has at least 280 megabyte (MB) of available space for the OCR and 280 MB for each voting disk.

• If you use the redundancy features of Oracle Clusterware to provide high availability for the OCR and voting disk files, then you need a minimum of three file systems, and each one must have 560 MB of available space for the OCR and voting disk.

  Note:

  The smallest partition size that OCFS for Windows can use is 500 MB

For example, to store all OCR and voting disk files on a cluster file system that does not provide redundancy at the hardware level (external redundancy), you should have approximately 2 GB of storage available over a minimum of three volumes (three separate volume locations for the OCR and voting disk files, one on each volume). If you use a file system that provides data redundancy, then you need only one physical disk with 280 MB of available space to store the OCR and 560 MB of available space for each voting disk file.

3.3.2 Identifying Storage Requirements for Using Oracle ASM for Shared Storage

To identify the storage requirements for using Oracle ASM, you must determine how many devices and the amount of free disk space that you require. To complete this task, follow these steps:

1. Determine whether you want to use Oracle ASM for Oracle Clusterware files (OCR and voting disks), Oracle Database data files, recovery files, or all file types.

   Note:

   • You do not have to use the same storage mechanism for data files and recovery files. You can store one type of file in a cluster file system while storing the other file type within Oracle ASM. If you plan to use Oracle ASM for both data files and recovery files, then you should create separate Oracle ASM disk groups for the data files and the recovery files.

   • All the OCR files or voting disk files must be located on either Oracle ASM or a cluster file system. You cannot have some files of one type in Oracle ASM and other Oracle Clusterware files in a cluster file system. You can have OCR stored on Oracle ASM and voting disk files stored on a cluster file system, but all files of each type must use the same storage type.

   If you plan to enable automated backups for your Oracle RAC database, then you must place the fast recovery area on shared storage. You can choose Oracle ASM as the shared storage mechanism for recovery files by specifying an Oracle ASM disk group for the fast recovery area. Depending how you choose to create a database during the installation, you have the following options:

   • If you created the Oracle ASM disk groups prior to performing the installation, and then select an installation method that runs DBCA in interactive mode (for example, by choosing the Advanced database configuration option), then you can decide whether you want to use the same Oracle ASM disk group for data files and recovery files. You can also choose to use different disk groups for each file type. Ideally, you should create separate Oracle ASM disk groups for data files and recovery files.

     The same choice is available to you if you use DBCA after the installation to create a database.

   • If you select an installation type that runs DBCA in non-interactive mode, then you must use the same Oracle ASM disk group for data files and recovery files. The Oracle ASM disk group you select must have been created prior to starting the installation or DBCA.

2. Choose the Oracle ASM redundancy level to use for the Oracle ASM disk group.

   The redundancy level that you choose for the Oracle ASM disk group determines how Oracle ASM mirrors files in the disk group, and determines the number of disks and amount of disk space that you require. The redundancy levels are as follows:

   • External redundancy

     An external redundancy disk group requires a minimum of one disk device. The effective disk space in an external redundancy disk group is the sum of the disk space in all of its devices.

     Because Oracle ASM does not mirror data in an external redundancy disk group, Oracle recommends that you use external redundancy with storage devices such as RAID, or other similar devices that provide their own data protection mechanisms.

     Even if you select external redundancy, you must have at least three voting disks configured, as each voting disk is an independent entity, and cannot be mirrored.

   • Normal redundancy

     A normal redundancy disk group requires a minimum of two disk devices (or two failure groups). The effective disk space in a normal redundancy disk group is half the sum of the disk space in all of its devices.

     For Oracle Clusterware files, a normal redundancy disk group requires a minimum of three disk devices and provides three voting disk files, one OCR, and two OCR copies (one primary and one secondary mirror). When using a normal redundancy disk group, the cluster can survive the loss of one failure group.

     For most installations, Oracle recommends that you select normal redundancy disk groups.

   • High redundancy

     In a high redundancy disk group, Oracle ASM uses three-way mirroring to increase performance and provide the highest level of reliability. A high redundancy disk group requires a minimum of three disk devices (or three failure groups). The effective disk space in a high redundancy disk group is one-third the sum of the disk space in all of its devices.

     For Oracle Clusterware files, a high redundancy disk group requires a minimum of five disk devices and provides five voting disk files, one OCR, and three copies of the OCR (one primary and two secondary mirrors). With high redundancy, the cluster can survive the loss of two failure groups.

     While high redundancy disk groups provide a high level of data protection, you should consider the greater cost of additional storage devices before deciding to use this redundancy level.

   Note:

   After a disk group is created, you cannot alter the redundancy level of the disk group.

3. Determine the total amount of disk space that you require for the Oracle Clusterware files.

   Use Table 3-2 to determine the minimum number of disks and the minimum disk space requirements for installing Oracle Clusterware using Oracle ASM for shared storage:

   Table 3-2 Oracle Clusterware Disk Space for Oracle ASM by Redundancy Type

   File Types Stored	Minimum Number of Disks	Disk or Disk Partition Sizes
   OCR and voting disks in an external redundancy disk group	1	At least 300 MB for each voting disk file and 300 MB for each OCR
   OCR and voting disks in a normal redundancy disk group	3	At least 600 MB for the OCR and its copies, at least 900 MB for the voting disk files, or at least 1.5 GB for both files types in one disk group. Note: If you create a disk group during installation, then it must be at least 2 GB in size.
   OCR and voting disks in a high redundancy disk group	5	At least 900 MB for the OCR and its copies, at least 1.5 GB for the voting disk files, or at least 2.4 GB for both files types in one disk group.

   
   

   Note:

   If the voting disk files are in a disk group, then note that disk groups that contain Oracle Clusterware files (OCR and voting disk files) have a higher minimum number of failure groups than other disk groups.

   If you create a disk group for the OCR and voting disk files as part of the installation, then the installer requires that you create these files on a disk group with at least 2 GB of available space.

   A quorum failure group is a special type of failure group. Disks in quorum failure groups do not contain user data. A quorum failure group is not considered when determining redundancy requirements in respect to storing user data. However, a quorum failure group counts when mounting a disk group.

   To ensure high availability of Oracle Clusterware files on Oracle ASM, you must have at least 2 GB of disk space for Oracle Clusterware files in three separate failure groups, with at least three physical disks. Each disk must have at least 1 GB of capacity to ensure that there is sufficient space to create Oracle Clusterware files.

4. Determine the total amount of disk space that you require for the Oracle database files and recovery files.

   Use the following table to determine the minimum number of disks and the minimum disk space requirements for installing the starter database:

   Table 3-3 Total Oracle Database Storage Space Required by Redundancy Type

   Redundancy Level	Minimum Number of Disks	Database Files	Recovery Files	Both File Types
   External	1	1.5 GB	3 GB	4.5 GB
   Normal	2	3 GB	6 GB	9 GB
   High	3	4.5 GB	9 GB	13.5 GB

   
   

   Note:

   The file sizes listed in the previous table are estimates of minimum requirements for a new installation (or a database without any user data). The file sizes for your database will be larger.

5. Determine if you can use an existing disk group.

   If an Oracle ASM instance currently exists on the system, then you can use an existing disk group to meet these storage requirements. If necessary, you can add disks to an existing disk group during the installation.

   See Section 3.3.2.1, "Using an Existing Oracle ASM Disk Group" for more information about using an existing disk group.

6. Optionally, identify failure groups for the Oracle ASM disk group devices.

   Note:

   You only have to complete this step if you plan to use an installation method that includes configuring Oracle ASM disk groups before installing Oracle RAC, or creating an Oracle RAC database.

   If you intend to use a normal or high redundancy disk group, then you can further protect your database against hardware failure by associating a set of disk devices in a custom failure group. Failure groups define Oracle ASM disks that share a common potential failure mechanism. By default, each device comprises its own failure group. If you choose to define custom failure groups, then note the following:

   • You must specify a minimum of two failure groups for normal redundancy disk groups and three failure groups for high redundancy disk groups.

   • If the disk group contains data files and Oracle Clusterware files, including the voting disk files, then you must specify a minimum of three failure groups for normal redundancy disk groups and five failure groups for high redundancy disk groups.

   • Disk groups containing voting disk files must have at least three failure groups for normal redundancy or at least five failure groups for high redundancy. If the disk group does not contain the voting disk files, then the minimum number of required failure groups is two for normal redundancy and three for high redundancy. The minimum number of failure groups applies whether or not they are custom failure groups.

   If two disk devices in a normal redundancy disk group are attached to the same small computer system interface (SCSI) controller, then the disk group becomes unavailable if the controller fails. The SCSI controller in this example is a single point of failure. To protect against failures of this type, you could use two SCSI controllers, each with two disks, and define a failure group for the disks attached to each controller. This configuration enables the disk group to tolerate the failure of one SCSI controller.

   Note:

   You can define custom failure groups after installation of Oracle Grid Infrastructure using the GUI tool ASMCA, the command-line tool asmcmd, or SQL*Plus commands.

   For more information about Oracle ASM failure groups, refer to Oracle Automatic Storage Management Administrator's Guide.

7. If you are sure that a suitable disk group does not exist on the system, then install or identify appropriate disk devices to add to a new disk group. Use the following guidelines when identifying appropriate disk devices:

   • All of the devices in an Oracle ASM disk group should be the same size and have the same performance characteristics.

   • Do not specify multiple partitions on a single physical disk as a disk group device. Oracle ASM expects each disk group device to be on a separate physical disk.

   • Nonshared logical partitions are not supported with Oracle RAC. If you want to use logical partitions for your Oracle RAC database, then you must use shared logical volumes created by a logical volume manager such as diskpart.msc.

3.3.3 Restrictions for Disk Partitions Used By Oracle ASM

Be aware of the following restrictions when configuring disk partitions for use with Oracle ASM:

• You cannot use primary partitions for storing Oracle Clusterware files while running OUI to install Oracle Clusterware as described in Chapter 4, "Installing Oracle Grid Infrastructure for a Cluster". You must create logical drives inside extended partitions for the disks to be used by Oracle Clusterware files and Oracle ASM.

• With x64 Windows, you can create up to 128 primary partitions for each disk.

• You can create shared directories only on primary partitions and logical drives.

• Oracle recommends that you limit the number of partitions you create on a single disk to prevent disk contention. Therefore, you may prefer to use extended partitions rather than primary partitions.

For these reasons, you might prefer to use extended partitions for storing Oracle software files and not primary partitions.

3.3.4 Requirements for Using a Shared File System

To use a shared file system for Oracle Clusterware, Oracle ASM, and Oracle RAC, the file system must comply with the following requirements:

• To use a cluster file system, it must be a supported cluster file system, as listed in the section Section 3.2, "Supported Storage Options for Oracle Grid Infrastructure and Oracle RAC".

• To use NFS, it must be on a certified network attached storage (NAS) device. Access the My Oracle Support Web site as described in Section 2.4, "Checking Hardware and Software Certification on My Oracle Support" to find a list of certified NAS devices.

• If you choose to place your OCR files on a shared file system, then Oracle recommends that one of the following is true:

  • The disks used for the file system are on a highly available storage device, for example, a RAID device.

  • At least two file systems are mounted, and use the features of Oracle Clusterware 11g release 2 (11.2) to provide redundancy for the OCR.

• If you choose to place the Oracle RAC data files on a shared file system, then one of the following should be true:

  • The disks used for the file system are on a highly available storage device, (for example, a RAID device).

  • The file systems consist of at least two independent file systems, with the data files on one file system, and the recovery files on a different file system.

• The user account with which you perform the installation (oracle or grid) must have write permissions to create the files in the path that you specify for the shared storage.

Use Table 3-4 and Table 3-5 to determine the minimum size for shared file systems:

In Table 3-4 and Table 3-5, the total required volume size is cumulative. For example, to store all Oracle Clusterware files on the shared file system with normal redundancy, you should have at least 2 GB of storage available over a minimum of three volumes (three separate volume locations for the OCR and two OCR mirrors, and one voting disk on each volume). You should have a minimum of three physical disks, each at least 500 MB, to ensure that voting disks and OCR files are on separate physical disks. If you also use this shared storage for Oracle RAC, using one volume for data files and one volume for recovery files, then you should have at least 3.5 GB available storage over two volumes, and at least 5.5 GB available total for all volumes.

3.3.5 Requirements for Files Managed by Oracle

If you use OCFS for Windows or Oracle ASM for your database files, then your database is created by default with files managed by Oracle Database. When using the Oracle Managed files feature, you need specify only the database object name instead of file names when creating or deleting database files.

Configuration procedures are required to enable Oracle Managed Files.

3.5.1 Disabling Write Caching

You must disable write caching on all disks that will be used to share data between the nodes in your cluster. Perform these steps to disable write caching:

1. Click Start, then select Control Panel, then Administrative Tools, then Computer Management, then Device Manager, and then Disk drives

2. Expand the Disk drives and double-click the first drive listed.

3. Under the Policies tab for the selected drive, uncheck the option that enables write caching.

4. Double-click each of the other drives that will be used by Oracle Clusterware and Oracle RAC and disable write caching as described in the previous step.

3.5.2 Enabling Automounting for Windows

If you are using Windows Server 2003 R2 Enterprise Edition or Datacenter Edition, then you must enable disk automounting, as it is disabled by default. For other Windows releases, even though the automount feature is enabled by default, you should verify that automount is enabled.

You must enable automounting when using:

• Raw partitions for Oracle RAC

• OCFS for Windows

• Oracle Clusterware

• Raw partitions for single-node database installations

• Logical drives for Oracle ASM

If you upgrade the operating system from one version of Windows to another (for example, Windows Server 2003 to Windows Advanced Server 2003), then you must repeat this procedure after the upgrade is finished.

To determine if automatic mounting of new volumes is enabled, use the following commands:

c:\> diskpart
DISKPART> automount
Automatic mounting of new volumes disabled.

To enable automounting:

1. Enter the following commands at a command prompt:

   c:\> diskpart
   DISKPART> automount enable
   Automatic mounting of new volumes enabled.
   

2. Type exit to end the diskpart session

3. Repeat steps 1 and 2 for each node in the cluster.

4. When you have prepared all of the cluster nodes in your Windows Server 2003 R2 system as described in the previous steps, restart all of the nodes.

3.6.1 Creating Disk Partitions for Oracle ASM

To use direct-attached storage (DAS) or storage area network (SAN) disks for Oracle ASM, each disk must have a partition table. Oracle recommends creating exactly one partition for each disk that encompasses the entire disk.

Use Microsoft Computer Management utility or the command line tool diskpart to create the partitions. Ensure that you create the partitions without drive letters. After you have created the partitions, the disks can be configured.

3.6.2 Marking Disk Partitions for Oracle ASM Before Installation

The only partitions that OUI displays for Windows systems are logical drives that are on disks that do not contain a primary partition, and have been marked (or stamped) with asmtool. Configure the disks before installation either by using asmtoolg (graphical user interface (GUI) version) or using asmtool (command line version). You also have the option of using the asmtoolg utility during Oracle Grid Infrastructure for a cluster installation.

The asmtoolg and asmtool utilities only work on partitioned disks; you cannot use Oracle ASM on unpartitioned disks. You can also use these tools to reconfigure the disks after installation. These utilities are installed automatically as part of Oracle Grid Infrastructure.

The following section describes the asmtoolg and asmtool functions and commands.

asmtool -add [-force]
\Device\Harddisk1\Partition1 ORCLDISKASM0
\Device\Harddisk2\Partition1 ORCLDISKASM2...

asmtool -addprefix ORCLDISKASM [-force]
\Device\Harddisk1\Partition1
\Device\Harddisk2\Partition1...

asmtool -create \\server\share\file 1000

asmtool -create D:\asm\asmfile02.asm 240

asmtool -list

asmtool -delete ORCLDISKASM0 ORCLDISKASM1...

3.7.1 Formatting Drives to Use OCFS for Windows after Installation

If you installed Oracle Grid Infrastructure, and you want to use OCFS for Windows for storage for Oracle RAC, then run the ocfsformat.exe command from the Grid_home\cfs directory using the following syntax:

Grid_home\cfs\OcfsFormat /m link_name /c ClusterSize_in_KB /v volume_label /f /a

Where:

• /m link_name is the mountpoint for this file system which you want to format with OCFS for Windows. On Windows, provide a drive letter corresponding to the logical drive.

• ClusterSize_in_KB is the Cluster size or allocation size for the OCFS for Windows volume (this option must be used with the /a option or else the default size of 4 kilobytes (KB) is used)

  Note:

  The Cluster size is essentially the block size. Recommended values are 1024 (1 MB) if the OCFS for Windows disk partition is to be used for Oracle datafiles and 4 (4 KB) if the OCFS for Windows disk partition is to be used for the Oracle home.

• volume_label is an optional volume label

• The /f option forces the format of the specified volume

• The /a option, if specified, forces OcfsFormat to use the clustersize specified with the /c option

For example, to create an OCFS for Windows formatted shared disk partition named DATA, mounted as U:, using a shared disk with a nondefault cluster size of 1 MB, you would use the following command:

ocfsformat /m U: /c 1024 /v DATA /f /a

3.8.1 About Direct NFS Storage

Oracle Disk Manager (ODM) can manage NFS on its own. This is referred to as Direct NFS. Direct NFS implements NFS version 3 protocol within the Oracle Database kernel. This change enables monitoring of NFS status using the ODM interface. The Oracle Database kernel driver tunes itself to obtain optimal use of available resources.

Starting with Oracle Database 11g release 1 (11.1), you can configure Oracle Database to access NFS version 3 servers directly using Direct NFS. This enables the storage of data files on a supported NFS system.

If Oracle Database cannot open an NFS server using Direct NFS, then an informational message is logged into the Oracle alert and trace files indicating that Direct NFS could not be established.

The Oracle files resident on the NFS server that are served by the Direct NFS Client can also be accessed through a third party NFS client. Management of Oracle data files created with Direct NFS should be done according to the guidelines specified in the "Managing Datafiles and Tempfiles" chapter of Oracle Database Administrator's Guide.

3.8.2 About the Oranfstab File for Direct NFS

If you use Direct NFS, then you must create a configuration file, oranfstab, to specify the options, attributes, and parameters that enable Oracle Database to use Direct NFS. Direct NFS looks for the mount point entries in Oracle_home\database\oranfstab. It uses the first matched entry as the mount point. You must create the oranfstab file in the Oracle_home\database directory.

When the oranfstab file is placed in Oracle_home\database, the entries in the file are specific to a single database. For Oracle RAC installations in a shared Oracle home, the oranfstab file is globally available to all database instances. All instances that use the shared Oracle home use the same Oracle_home\database\oranfstab file. For a nonshared Oracle home, because all the Oracle RAC instances use the same oranfstab file, you must replicate the oranfstab file on all of the nodes. Also, you must keep the oranfstab file synchronized on all the nodes.

3.8.3 Mounting NFS Storage Devices with Direct NFS

Direct NFS determines mount point settings for NFS storage devices based on the configuration information in oranfstab. If Oracle Database cannot open an NFS server using Direct NFS, then an error message is written into the Oracle alert and trace files indicating that Direct NFS could not be established.

3.8.4 Specifying Network Paths for a NFS Server

Direct NFS can use up to four network paths defined in the oranfstab file for an NFS server. The Direct NFS client performs load balancing across all specified paths. If a specified path fails, then Direct NFS re-issues all outstanding requests over any remaining paths.

Use the following global dynamic performance views for managing Direct NFS usage with your Oracle RAC database:

• GV$DNFS_SERVERS: Lists the servers that are accessed using Direct NFS.

• GV$DNFS_FILES: Lists the files that are currently open using Direct NFS.

• GV$DNFS_CHANNELS: Shows the open network paths, or channels, to servers for which Direct NFS is providing files.

• GV$DNFS_STATS: Lists performance statistics for Direct NFS.

3.8.5 Enabling the Direct NFS Client

To enable the Direct NFS Client, you must add an oranfstab file to Oracle_home\database. When oranfstab is placed in this directory, the entries in this file are specific to one particular database. The Direct NFS Client searches for the mount point entries as they appear in oranfstab. The Direct NFS Client uses the first matched entry as the mount point.

Complete the following procedure to enable the Direct NFS Client:

1. Create an oranfstab file with the following attributes for each NFS server accessed by Direct NFS:

   • server: The NFS server name.

   • path: Up to four network paths to the NFS server, specified either by internet protocol (IP) address, or by name, as displayed using the ifconfig command on the NFS server.

   • local: Up to 4 network interfaces on the database host, specified by IP address, or by name, as displayed using the ipconfig command on the database host.

   • export: The exported path from the NFS server. Use a UNIX-style path.

   • mount: The corresponding local mount point for the exported volume. Use Windows-style path.

   • mnt_timeout: (Optional) Specifies the time (in seconds) for which Direct NFS client should wait for a successful mount before timing out. The default timeout is 10 minutes (600).

   • uid: (Optional) The UNIX user ID to be used by Direct NFS to access all NFS servers listed in oranfstab. The default value is uid:65534, which corresponds to user:nobody on the NFS server.

   • gid: (Optional) The UNIX group ID to be used by Direct NFS to access all NFS servers listed in oranfstab. The default value is gid:65534, which corresponds to group:nogroup on the NFS server.

   The mount point specified in the oranfstab file represents the local path where the database files would reside normally, as if Direct NFS was not used. For example, if the location for the data files if the database did not use Direct NFS would be C:\app\oracle\oradata\orcl directory, then you specify C:\app\oracle\oradata\orcl for the NFS virtual mount point in the corresponding oranfstab file.

   Example 3-1 and Example 3-2 provide examples of how the Direct NFS attributes can be used in an oranfstab file.

   Note:

   • Direct NFS ignores a uid or gid value of 0.

   • The exported path from the NFS server must be accessible for read/write/execute by the user with the uid, gid specified in oranfstab. If neither uid nor gid is listed, then the exported path must be accessible by the user with uid:65534 and gid:65534.

2. Replace the standard ODM library, oraodm11.dll, with the ODM NFS library.

   Oracle Database uses the ODM library, oranfsodm11.dll, to enable Direct NFS. To replace the ODM library, complete the following steps:

   1. Change directory to Oracle_home\bin.

   2. Shut down the Oracle Database instance on a node using the Server Control Utility (SRVCTL).

   3. Enter the following commands:

      copy oraodm11.dll oraodm11.dll.orig
      copy /Y oranfsodm11.dll oraodm11.dll 
      

   4. Restart the Oracle Database instance using SRVCTL.

   5. Repeat Step a to Step d for each node in the cluster.

server: MyDataServer1
local: 132.34.35.10
path: 132.34.35.12
local: 132.34.55.10
path: 132.34.55.12
export: /vol/oradata1 mount: C:\APP\ORACLE\ORADATA\ORCL
 

server: MyDataServer2
local: LocalInterface1
path: NfsPath1
local: LocalInterface2
path: NfsPath2
local: LocalInterface3
path: NfsPath3
local: LocalInterface4
path: NfsPath4
export: /vol/oradata2 mount: C:\APP\ORACLE\ORADATA\ORCL2
export: /vol/oradata3 mount: C:\APP\ORACLE\ORADATA\ORCL3

3.8.6 Performing Basic File Operations Using the ORADNFS Utility

ORADNFS is a utility which enables the database administrators to perform basic file operations over Direct NFS Client on Microsoft Windows platforms.

ORADNFS is a multi-call binary, which is a single binary that acts like many utilities. You must be a member of the local ORA_DBA group to use ORADNFS. To execute commands using ORADNFS you issue the command as an argument on the command line.

The following command prints a list of commands available with ORADNFS:

C:\> oradnfs help

To display the list of files in the NFS directory mounted as C:\ORACLE\ORADATA, use the following command:

C:\> oradnfs ls C:\ORACLE\ORADATA\ORCL

3.8.7 Disabling Direct NFS Client

Use one of the following methods to disable the Direct NFS client:

• Remove the oranfstab file.

• Restore the original oraodm11.dll file by reversing the process you completed in Section 3.8.5, "Enabling the Direct NFS Client".

• Remove the specific NFS server or export paths in the oranfstab file.