D How to Complete Preinstallation Tasks Manually

This appendix provides instructions for how to complete configuration tasks manually that Cluster Verification Utility (CVU) and Oracle Universal Installer normally complete during installation. Use this appendix as a guide if you cannot use the fixup script.

This appendix contains the following information:

D.1 Configuring Kernel Parameters for Linux

This section contains the following topics:

Note:

Unless otherwise specified, the kernel parameter and shell limit values shown in the following table are minimum values only. For production database systems, Oracle recommends that you tune these values to optimize the performance of the system. See the operating system documentation for more information about tuning kernel parameters.

D.1.1 Minimum Parameter Settings for Installation

During the Oracle Database installation, you can generate and run the fixup script to check and set the kernel parameter values required for successful installation of the database. This script updates required kernel parameters, if necessary, to minimum values.

If you cannot use the fixup script, then review the following table to set the values manually:

Parameter Value File
semmsl

semmns

semopm

semmni

250

32000

100

128

/proc/sys/kernel/sem
shmall 40 percent of the size of physical memory in pages

Note: If the server supports multiple databases, or uses a large SGA, then set this parameter to a value that is equal to the total amount of shared memory, in 4K pages, that the system can use at one time.

/proc/sys/kernel/shmall
shmmax Half the size of physical memory in bytes

See My Oracle Support Note 567506.1 for additional information about configuring shmmax.

/proc/sys/kernel/shmmax
shmmni 4096 /proc/sys/kernel/shmmni
panic_on_oops 1 /proc/sys/kernel/panic_on_oops
file-max 6815744 /proc/sys/fs/file-max
aio-max-nr 1048576

Note: This value limits concurrent outstanding requests and should be set to avoid I/O subsystem failures.

/proc/sys/fs/aio-max-nr
ip_local_port_range Minimum: 9000

Maximum: 65500

See Setting UDP and TCP Kernel Parameters Manually

/proc/sys/net/ipv4/ip_local_port_range
rmem_default 262144 /proc/sys/net/core/rmem_default
rmem_max 4194304 /proc/sys/net/core/rmem_max
wmem_default 262144 /proc/sys/net/core/wmem_default
wmem_max 1048576 /proc/sys/net/core/wmem_max

Note:

If the current value for any parameter is greater than the value listed in this table, then the Fixup scripts do not change the value of that parameter.

D.1.2 Displaying and Changing Kernel Parameter Values

Enter the commands shown in the following table to display the current values of the kernel parameters. Note these values and identify any values that you must change:

Parameter Command
semmsl, semmns, semopm, and semmni # /sbin/sysctl -a | grep sem

This command displays the value of the semaphore parameters in the order listed.

shmall, shmmax, and shmmni # /sbin/sysctl -a | grep shm

This command displays the details of the shared memory segment sizes.

file-max # /sbin/sysctl -a | grep file-max

This command displays the maximum number of file handles.

ip_local_port_range # /sbin/sysctl -a | grep ip_local_port_range

This command displays a range of port numbers.

rmem_default # /sbin/sysctl -a | grep rmem_default
rmem_max # /sbin/sysctl -a | grep rmem_max
wmem_default # /sbin/sysctl -a | grep wmem_default
wmem_max # /sbin/sysctl -a | grep wmem_max
aio-max-nr # /sbin/sysctl -a | grep aio-max-nr

If the value of any kernel parameter is different from the minimum value, then perform the following:

  1. Using any text editor, create or edit the /etc/sysctl.conf file, and add or edit lines similar to the following. For example:

    Note:

    Include lines only for the kernel parameter values to change. For the semaphore parameters (kernel.sem), you must specify all four values. If any of the current values are larger than the minimum value, then specify the larger value.
    fs.aio-max-nr = 1048576
    fs.file-max = 6815744
    kernel.shmall = 2097152
    kernel.shmmax = 4294967295
    kernel.shmmni = 4096
    kernel.sem = 250 32000 100 128
    net.ipv4.ip_local_port_range = 9000 65500
    net.core.rmem_default = 262144
    net.core.rmem_max = 4194304
    net.core.wmem_default = 262144
    net.core.wmem_max = 1048576
    

    By specifying the values in the /etc/sysctl.conf file, they persist when you restart the system. On SUSE Linux Enterprise Server systems, enter the following command to ensure that the system reads the /etc/sysctl.conf file when it restarts:

    # /sbin/chkconfig boot.sysctl on
    
  2. Enter the following command to change the current values of the kernel parameters:

    # /sbin/sysctl -p
    

    Review the output from this command to verify that the values are correct. If the values are incorrect, edit the /etc/sysctl.conf file, then enter this command again.

  3. Enter the command /sbin/sysctl -a to confirm that the values are set correctly.

  4. After updating the values of the kernel parameters in the /etc/sysctl.conf file, either restart the computer, or run the command sysctl -p to make the changes in the /etc/sysctl.conf file available in the active kernel memory.

D.1.3 Additional Parameter and Kernel Settings for SUSE Linux

On SUSE Linux Enterprise Server systems only, complete the following steps as needed:

  1. Enter the following command to cause the system to read the /etc/sysctl.conf file when it restarts:

    # /sbin/chkconfig boot.sysctl on
    
  2. Enter the GID of the oinstall group as the value for the parameter /proc/sys/vm/hugetlb_shm_group. Doing this grants members of oinstall a group permission to create shared memory segments.

    For example, where the oinstall group GID is 501:

    # echo 501 > /proc/sys/vm/hugetlb_shm_group
    

    After running this command, use vi to add the following text to /etc/sysctl.conf, and enable the boot.sysctl script to run on system restart:

    vm.hugetlb_shm_group=501
    

    Note:

    Only one group can be defined as the vm.hugetlb_shm_group.

D.2 Setting UDP and TCP Kernel Parameters Manually

If you do not use a Fixup script or CVU to set ephemeral ports, then set TCP/IP ephemeral port range parameters to provide enough ephemeral ports for the anticipated server workload. Ensure that the lower range is set to at least 9000 or higher, to avoid Well Known ports, and to avoid ports in the Registered Ports range commonly used by Oracle and other server ports. Set the port range high enough to avoid reserved ports for any applications you may intend to use. If the lower value of the range you have is greater than 9000, and the range is large enough for your anticipated workload, then you can ignore Oracle Universal Installer warnings regarding the ephemeral port range.

For example, with IPv4, use the following command to check your current range for ephemeral ports:

$ cat /proc/sys/net/ipv4/ip_local_port_range
32768 61000

In the preceding example, the lowest port (32768) and the highest port (61000) are set to the default range.

If necessary, update the UDP and TCP ephemeral port range to a range high enough for anticipated system workloads, and to ensure that the ephemeral port range starts at 9000 and above. For example:

# echo 9000 65500 > /proc/sys/net/ipv4/ip_local_port_range

Oracle recommends that you make these settings permanent. For example, as root, use a text editor to open /etc/sysctl.conf, and add or change to the following: net.ipv4.ip_local_port_range = 9000 65500, and then restart the network (# /etc/rc.d/init.d/network restart). Refer to your Linux distribution system administration documentation for detailed information about how to automate this ephemeral port range alteration on system restarts.

See Also:

"Setting TCP Network Protocol Buffer for Direct NFS Client" if you use Direct NFS Client

D.3 Configuring Storage Paths and Disk Devices

Oracle recommends that you use Oracle ASM Filter Driver (Oracle ASMFD) to maintain device persistence. However, you can choose to use Oracle Automatic Storage Management library driver (Oracle ASMLIB) or set udev rules for device persistence.

This section contains the following topics:

D.3.1 Configuring Storage Device Path Persistence Using Oracle ASMLIB

Review the following section to configure Oracle ASMLIB:

Note:

Oracle ASMLIB is not supported on IBM:Linux on System z.

D.3.1.1 About Oracle ASM with Oracle ASMLIB

The Oracle Automatic Storage Management library driver simplifies the configuration and management of block disk devices by eliminating the need to rebind block disk devices used with Oracle Automatic Storage Management (Oracle ASM) each time the system is restarted.

With Oracle ASMLIB, you define the range of disks you want to have made available as Oracle ASM disks. Oracle ASMLIB maintains permissions and disk labels that are persistent on the storage device, so that the label is available even after an operating system upgrade.

Note:

If you configure disks using Oracle ASMLIB, then you must change the disk discovery string to ORCL:*. If the diskstring is set to ORCL:*, or is left empty (""), then the installer discovers these disks.

See Also:

"Using Oracle ASM Library Driver" in Oracle Automatic Storage Management Administrator's Guide

D.3.1.2 Configuring Oracle ASMLIB to Maintain Block Devices

To use Oracle ASMLIB to configure Oracle ASM devices, complete the following tasks:

Note:

To create a database during the installation using the Oracle ASM library driver, you must choose an installation method that runs ASMCA in interactive mode. You must also change the disk discovery string to ORCL:*.
D.3.1.2.1 Installing and Configuring Oracle ASMLIB Software

Oracle ASMLIB is included with the Oracle Linux packages, and with SUSE Linux Enterprise Server 11. If you are a member of the Unbreakable Linux Network, then you can install the Oracle ASMLIB RPMs by subscribing to the Oracle Linux channel, and using yum to retrieve the most current package for your system and kernel. For additional information, see the following URL:

http://www.oracle.com/technetwork/topics/linux/asmlib/index-101839.html

To install and configure the Oracle Automatic Storage Management library driver software manually, perform the following steps:

  1. Enter the following command to determine the kernel version and architecture of the system:

    # uname -rm
    
  2. If necessary, download the required Oracle Automatic Storage Management library driver packages from the Oracle Technology Network website:

    http://www.oracle.com/technetwork/server-storage/linux/downloads/index-088143.html

    Note:

    You must install oracleasm-support package version 2.0.1 or later to use Oracle ASMLIB on Red Hat Enterprise Linux 5 Advanced Server. Oracle ASMLIB is already included with SUSE Linux Enterprise Server distributions.

    See Also:

    My Oracle Support note 1089399.1 for information about Oracle ASMLIB support with Red Hat distributions:

    https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=1089399.1

  3. Switch to the root user:

    $ su -
    
  4. Install the following packages in sequence, where version is the version of the Oracle Automatic Storage Management library driver, arch is the system architecture, and kernel is the version of the kernel that you are using:

    oracleasm-support-version.arch.rpm
    oracleasm-kernel-version.arch.rpm
    oracleasmlib-version.arch.rpm
    

    Enter a command similar to the following to install the packages:

    # rpm -ivh oracleasm-support-version.arch.rpm \
               oracleasm-kernel-version.arch.rpm \
               oracleasmlib-version.arch.rpm
    

    For example, if you are using the Red Hat Enterprise Linux 5 AS kernel on an AMD64 system, then enter a command similar to the following:

    # rpm -ivh oracleasm-support-2.1.3-1.el5.x86_64.rpm \
               oracleasm-2.6.18-194.26.1.el5xen-2.0.5-1.el5.x86_64.rpm \
               oracleasmlib-2.0.4-1.el5.x86_64.rpm
    
  5. Enter the following command to run the oracleasm initialization script with the configure option:

    # /usr/sbin/oracleasm configure -i
    

    Note:

    The oracleasm command in /usr/sbin is the command you should use. The /etc/init.d path is not deprecated, but the oracleasm binary in that path is now used typically for internal commands.
  6. Enter the following information in response to the prompts that the script displays:

    Prompt Suggested Response
    Default user to own the driver interface: Standard groups and users configuration: Specify the Oracle software owner user (for example, oracle)

    Job role separation groups and users configuration: Specify the Grid Infrastructure software owner (for example, grid)

    Default group to own the driver interface: Standard groups and users configuration: Specify the OSDBA group for the database (for example, dba).

    Job role separation groups and users configuration: Specify the OSASM group for storage administration (for example, asmadmin).

    Start Oracle ASM Library driver on boot (y/n): Enter y to start the Oracle Automatic Storage Management library driver when the system starts.
    Scan for Oracle ASM disks on boot (y/n) Enter y to scan for Oracle ASM disks when the system starts.

    The script completes the following tasks:

    • Creates the /etc/sysconfig/oracleasm configuration file

    • Creates the /dev/oracleasm mount point

    • Mounts the ASMLIB driver file system

      Note:

      The Oracle ASMLIB file system is not a regular file system. It is used only by the Oracle ASM library to communicate with the Oracle ASMLIB.
  7. Enter the following command to load the oracleasm kernel module::

    # /usr/sbin/oracleasm init
    
D.3.1.2.2 Configuring Disk Devices to Use Oracle ASMLIB

To configure the disk devices to use in an Oracle Automatic Storage Management disk group, perform the following steps:

  1. If you intend to use IDE, SCSI, or RAID devices in the Oracle Automatic Storage Management disk group, then perform the following steps:

    1. Install or configure the disk devices that you intend to use for the disk group and restart the system.

    2. Enter the following command to identify the device name for the disks to use:

      # /sbin/fdisk -l
      

      Depending on the type of disk, the device name can vary:

      Disk Type Device Name Format Description
      IDE disk
      /dev/hdxn
      
      In this example, x is a letter that identifies the IDE disk, and n is the partition number. For example, /dev/hda is the first disk on the first IDE bus.
      SCSI disk
      /dev/sdxn
      
      In this example, x is a letter that identifies the SCSI disk, and n is the partition number. For example, /dev/sda is the first disk on the first SCSI bus.
      RAID disk
      /dev/rd/cxdypz
      /dev/ida/cxdypz
      
      Depending on the RAID controller, RAID devices can have different device names. In the examples shown, x is a number that identifies the controller, y is a number that identifies the disk, and z is a number that identifies the partition. For example, /dev/ida/c0d1 is the second logical drive on the first controller.

      To include devices in a disk group, you can specify either whole-drive device names or partition device names.

      Note:

      Oracle recommends that you create a single whole-disk partition on each disk to use.
    3. Use either fdisk or parted to create a single whole-disk partition on the disk devices.

  2. Enter a command similar to the following to mark a disk as an Oracle Automatic Storage Management disk:

    # /usr/sbin/oracleasm createdisk DISK1 /dev/sdb1
    

    In this example, DISK1 is a name assigned to the disk.

    Note:

    • The disk names you specify can contain uppercase letters, numbers, and the underscore character. They must start with an uppercase letter.

    • To create a database during the installation using the Oracle Automatic Storage Management library driver, you must change the disk discovery string to ORCL:*.

    • If you are using a multi-pathing disk driver with Oracle ASM, then make sure that you specify the correct logical device name for the disk.

D.3.1.2.3 Administering Oracle ASMLIB and Disks

To administer the Oracle Automatic Storage Management library driver and disks, use the /usr/sbin/oracleasm initialization script with different options, as described in the following table:

Table D-1 Disk Management Tasks Using ORACLEASM

Task Command Example Description

Configure or reconfigure ASMLIB

oracleasm configure -i

Use the configure option to reconfigure the Oracle Automatic Storage Management library driver, if necessary.

To see command options, enter oracleasm configure without the -i flag.

Change system restart load options for ASMLIB

oracleasm enable

Options are disable and enable.

Use the disable and enable options to change the actions of the Oracle Automatic Storage Management library driver when the system starts. The enable option causes the Oracle Automatic Storage Management library driver to load when the system starts.

Load or unload ASMLIB without restarting the system

oracleasm restart

Options are start, stop and restart.

Use the start, stop, and restart options to load or unload the Oracle Automatic Storage Management library driver without restarting the system.

Mark a disk for use with ASMLIB

oracleasm createdisk VOL1 /dev/sda1

Use the createdisk option to mark a disk device for use with the Oracle Automatic Storage Management library driver and give it a name, where labelname is the name you want to use to mark the device, and devicepath is the path to the device:

oracleasm createdisk labelname devicepath

Unmark a named disk device

oracleasm deletedisk VOL1

Use the deletedisk option to unmark a named disk device, where diskname is the name of the disk:

oracleasm deletedisk diskname

Caution: Do not use this command to unmark disks that are being used by an Oracle Automatic Storage Management disk group. You must delete the disk from the Oracle Automatic Storage Management disk group before you unmark it.

Determine if ASMLIB is using a disk device

oracleasm querydisk

Use the querydisk option to determine if a disk device or disk name is being used by the Oracle Automatic Storage Management library driver, where diskname_devicename is the name of the disk or device that you want to query:

oracleasm querydisk diskname_devicename

List Oracle ASMLIB disks

oracleasm listdisks

Use the listdisks option to list the disk names of marked Oracle ASM library driver disks.

Identify disks marked as ASMLIB disks

oracleasm scandisks

Use the scandisks option to enable cluster nodes to identify which shared disks have been marked as ASMLIB disks on another node.

Rename ASMLIB disks

oracleasm renamedisk VOL1 VOL2

Use the renamedisk option to change the label of an Oracle ASM library driver disk or device by using the following syntax, where manager specifies the manager device, label_device specifies the disk you intend to rename, as specified either by OracleASM label name or by the device path, and new_label specifies the new label you want to use for the disk:

oracleasm renamedisk [-l manager] [-v] label_device new_label

Use the -v flag to provide a verbose output for debugging.

Caution: You must ensure that all Oracle Database and Oracle ASM instances have ceased using the disk before you relabel the disk. If you do not do this, then you may lose data.


D.3.1.3 Deinstalling Oracle ASMLIB

If Oracle ASMLIB is installed but you do not use it for device path persistence, then deinstall Oracle ASMLIB:

  1. Stop Oracle ASM and any running database instance:

    $ srvctl stop asm
    $ srvctl stop instance -d db_unique_name
    
  2. Log in as root.

  3. Stop the Oracle Restart stack:

    # cd Grid_home/bin
    # crsctl stop has
    
  4. Stop Oracle ASMLIB:

    # /etc/init.d/oracleasm disable
    
  5. Remove the oracleasm library and tools RPMs:

    # rpm -e oracleasm-support
    # rpm -e oracleasmlib
    
  6. Check if any oracleasm RPMs are remaining:

    # rpm -qa| grep oracleasm
    
  7. If any oracleasm configuration files are remaining, remove them:

    # rpm -qa| grep oracleasm | xargs rpm -e
    

    Oracle ASMLIB and associated RPMs are now removed.

  8. Start the Oracle Restart stack. Optionally, you can install and configure Oracle ASM Filter Driver (Oracle ASMFD) before starting the Oracle Restart stack.

D.3.2 Configuring Disk Devices Manually for Oracle Automatic Storage Management

This section contains the following information about preparing disk devices manually for use by Oracle ASM:

Note:

The operation of udev depends on the Linux version, vendor, and storage configuration.

D.3.2.1 About Device File Names and Ownership for Linux

By default, the device file naming scheme udev dynamically creates device file names when the server is started, and assigns ownership of them to root. If udev applies default settings, then it changes Oracle device file names and owners for the disks, making the disks inaccessible when the server is restarted. For example, a voting disk on a device named /dev/sdd owned by the user grid may be on a device named /dev/sdf owned by root after restarting the server.

If you use Oracle ASMFD, then you do not have to ensure permissions and device path persistency in udev.

If you do not use Oracle ASMFD, then you must create a custom rules file. Linux vendors customize their udev configurations and use different orders for reading rules files. For example, on some Linux distributions when udev is started, it sequentially carries out rules (configuration directives) defined in rules files. These files are in the path /etc/udev/rules.d/. Rules files are read in lexical order. For example, rules in the file 10-wacom.rules are parsed and carried out before rules in the rules file 90-ib.rules.

When specifying the device information in the UDEV rules file, ensure that the OWNER, GROUP and MODE are specified before all other characteristics in the order shown. For example, to include the characteristic ACTION on the UDEV line, specify ACTION after OWNER, GROUP, and MODE.

Where rules files describe the same devices, on the supported Linux kernel versions, the last file read is the one that is applied.

D.3.2.2 Configuring a Permissions File for Disk Devices for Oracle ASM

To configure a permissions file for disk devices for use by Oracle ASM, complete the following tasks:

  1. Configure SCSI devices as trusted devices, while listed, by editing the /etc/scsi_id.config file and adding "options=-g" to the file. For example:

    # cat > /etc/scsi_id.config
    vendor="ATA",options=-p 0x80
    options=-g
    
  2. Using a text editor, create a udev rules file for the Oracle ASM devices, setting permissions to 0660 for the installation owner and the group whose members are administrators of the grid infrastructure software.

    For example, on Oracle Linux, to create a role-based configuration rules.d file, where the installation owner is grid, and the OSASM group is asmadmin, enter commands similar to the following:

    # vi /etc/udev/rules.d/99-oracle-asmdevices.rules
    
    KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",
    RESULT=="14f70656e66696c00000000", OWNER="grid", GROUP="asmadmin", MODE="0660"
    KERNEL=="sd?2", BUS=="scsi", PROGRAM=="/sbin/scsi_id",
    RESULT=="14f70656e66696c00000000", OWNER="grid", GROUP="asmadmin", MODE="0660"
    KERNEL=="sd?3", BUS=="scsi", PROGRAM=="/sbin/scsi_id",
    RESULT=="14f70656e66696c00000000", OWNER="grid", GROUP="asmadmin", MODE="0660"
    
  3. Load updated block device partition tables on the server using: /sbin/partprobe devicename. You must do this as the root user.

  4. Enter the command to restart the udev service.

    On Oracle Linux, and Red Hat Enterprise Linux, the commands are:

    # /sbin/udevcontrol reload_rules
    # /sbin/start_udev
    

    On SUSE Linux Enterprise Server, the command is:

    # /etc/init.d boot.udev restart
    

    Verify that the device permissions and ownerships are set correctly.