2 Administering Oracle Clusterware

How Server Pools Work

Server pools divide the cluster into groups of servers hosting the same or similar resources. They distribute a uniform workload (a set of Oracle Clusterware resources) over several servers in the cluster. For example, you can restrict Oracle databases to run only in certain server pools. When you enable role-separated management, you can explicitly grant permission to operating system users to change attributes of certain server pools.

Top-level server pools:

• Logically divide the cluster

• Are always exclusive, meaning that one server can only reside in one particular server pool at a certain point in time

Server pools each have three attributes that they are assigned when they are created:

• MIN_SIZE: The minimum number of servers the server pool should contain. If the number of servers in a server pool is below the value of this attribute, then Oracle Clusterware automatically moves servers from elsewhere into the server pool until the number of servers reaches the attribute value.

• MAX_SIZE: The maximum number of servers the server pool should contain.

• IMPORTANCE: A number from 0 to 1000 (0 being least important) that ranks a server pool among all other server pools in a cluster.

  Table 2-2 lists and describes all server pool attributes.

  Table 2-2 Server Pool Attributes

  Attribute	Values and Format	Description
  ACL	String in the following format: owner:user:rwx,pgrp:group:rwx,other::r—	Defines the owner of the server pool and which privileges are granted to various operating system users and groups. The server pool owner defines the operating system user of the owner, and which privileges that user is granted. The value of this optional attribute is populated at the time a server pool is created based on the identity of the process creating the server pool, unless explicitly overridden. The value can subsequently be changed, if such a change is allowed based on the existing privileges of the server pool. In the string: owner: The operating system user of the server pool owner, followed by the privileges of the owner pgrp: The operating system group that is the primary group of the owner of the server pool, followed by the privileges of members of the primary group other: Followed by privileges of others r: Read only w: Modify attributes of the pool or delete it x: Assign resources to this pool By default, the identity of the client that creates the server pool is the owner. Also by default, root, and the user specified in owner have full privileges. You can grant required operating system users and operating system groups their privileges by adding the following lines to the ACL attribute: user:username:rwx group:group_name:rwx
  ACTIVE_SERVERS	A string of server names in the following format: server_name1 server_name2 ...	Oracle Clusterware automatically manages this attribute, which contains the space-delimited list of servers that are currently assigned to a server pool.
  EXCLUSIVE_POOLS	String	This optional attribute indicates if servers assigned to this server pool are shared with other server pools. A server pool can explicitly state that it is exclusive of any other server pool that has the same value for this attribute. Two or more server pools are mutually exclusive when the sets of servers assigned to them do not have a single server in common. For example, server pools A and B must be exclusive if they both set the value of this attribute to foo_A_B. Top-level server pools are mutually exclusive, by default.
  IMPORTANCE	Any integer from 0 to 1000	Relative importance of the server pool, with 0 denoting the lowest level of importance and 1000, the highest. This optional attribute is used to determine how to reconfigure the server pools when a node joins or leaves the cluster. The default value is 0.
  MAX_SIZE	Any nonnegative integer or -1 (no limit)	The maximum number of servers a server pool can contain. This attribute is optional and is set to -1 (no limit), by default. Note: A value of -1 for this attribute spans the entire cluster.
  MIN_SIZE	Any nonnegative integer	The minimum size of a server pool. If the number of servers contained in a server pool is below the number you specify in this attribute, then Oracle Clusterware automatically moves servers from other pools into this one until that number is met. Note: The value of this optional attribute does not set a hard limit. It governs the priority for server assignment whenever the cluster is reconfigured. The default value is 0.
  NAME	String	The name of the server pool, which you must specify when you create the server pool. Server pool names must be unique within the domain of names of user-created entities, such as resources, types, and servers. A server pool name can contain any platform-supported characters except the exclamation point (!) and the tilde (~). A server pool name cannot begin with a period nor with ora.
  PARENT_POOLS	A string of space-delimited server pool names in the following format: sp1 sp2 ...	Use of this attribute makes it possible to create nested server pools. Server pools listed in this attribute are referred to as parent server pools. A server pool included in a parent server pool is referred to as a child server pool.
  SERVER_NAMES	A string of space-delimited server names in the following format: server1 server2 ...	A list of candidate node names that may be associated with a server pool. If this optional attribute is empty, Oracle Clusterware assumes that any server may be assigned to any server pool, to the extent allowed by values of other attributes, such as PARENT_POOLS. The server names identified as candidate node names are not validated to confirm that they are currently active cluster members. Cluster administrators can use this attribute to define servers as candidates that have not yet been added to the cluster.

  
  

You manage server pools that are managing Oracle RAC databases with the Server Control (SRVCTL) utility. Use the Oracle Clusterware Control (CRSCTL) utility to manage all other server pools. Only cluster administrators have permission to create top-level server pools.

When Oracle Clusterware is installed, two server pools are created automatically: Generic and Free. All servers in a new installation are assigned to the Free server pool, initially. Servers move from Free to newly defined server pools automatically. When you upgrade Oracle Clusterware from a previous release, all nodes are assigned to the Generic server pool, to ensure compatibility with database releases before Oracle Database 11g release 2 (11.2).

The Free Server Pool

The Free server pool contains servers that are not assigned to any other server pools. The attributes of the Free server pool are restricted, as follows:

• SERVER_NAMES, MIN_SIZE, and MAX_SIZE cannot be edited by the user

• IMPORTANCE and ACL can be edited by the user

The Generic Server Pool

The Generic server pool stores pre-11g release 2 (11.2) Oracle Databases and administrator-managed databases that have fixed configurations. Additionally, the Generic server pool contains servers that match either of the following:

• Servers that you specified in the HOSTING_MEMBERS resource attribute of all resources of the application resource type

  See Also:

  • "HOSTING_MEMBERS" for more information about this attribute

  • "Resource Types" for more information about resource types

• Servers with names you specified in the SERVER_NAMES attribute of the server pools that list the Generic server pool as a parent server pool

The Generic server pool's attributes are restricted, as follows:

• No one can modify configuration attributes of the Generic server pool (all attributes are read-only)

• When you specify a server name in the HOSTING_MEMBERS resource attribute, Oracle Clusterware only allows it if the server is:

  • Online and exists in the Generic server pool

  • Online and exists in the Free server pool, in which case Oracle Clusterware moves the server into the Generic server pool

  • Online and exists in any other server pool and the client is either a cluster administrator or is allowed to use the server pool's servers, in which case, the server is moved into the Generic server pool

  • Offline and the client is a cluster administrator

• When you register a child server pool with the Generic server pool, Oracle Clusterware only allows it if the server names pass the same requirements as previously specified for the resources.

  Servers are initially considered for assignment into the Generic server pool at cluster startup time or when a server is added to the cluster, and only after that to other server pools.

Servers Moving from Server Pool to Server Pool

If the number of servers in a server pool falls below the value of the MIN_SIZE attribute for the server pool (such as when a server fails), based on values you set for the MIN_SIZE and IMPORTANCE attributes for all server pools, Oracle Clusterware can move servers from other server pools into the server pool whose number of servers has fallen below the value for MIN_SIZE. Oracle Clusterware selects servers from other server pools to move into the deficient server pool that meet the following criteria:

• For server pools that have a lower IMPORTANCE value than the deficient server pool, Oracle Clusterware can take servers from those server pools even if it means that the number of servers falls below the value for the MIN_SIZE attribute.

• For server pools with equal or greater IMPORTANCE, Oracle Clusterware only takes servers from those server pools if the number of servers in a server pool is greater than the value of its MIN_SIZE attribute.

IPMI Postinstallation Configuration with Oracle Clusterware

When you install IPMI during Oracle Clusterware installation, you configure failure isolation in two phases. Before you start the installation, you install and enable the IPMI driver in the server operating system, and configure the IPMI hardware on each node (IP address mode, admin credentials, and so on), as described in Oracle Grid Infrastructure Installation Guide. When you install Oracle Clusterware, the installer collects the IPMI administrator user ID and password, and stores them in an Oracle Wallet in node-local storage, in OLR.

After you complete the server configuration, complete the following procedure on each cluster node to register IPMI administrators and passwords on the nodes.

1. Start Oracle Clusterware, which allows it to obtain the current IP address from IPMI. This confirms the ability of the clusterware to communicate with IPMI, which is necessary at startup.

   If Oracle Clusterware was running before IPMI was configured, you can shut Oracle Clusterware down and restart it. Alternatively, you can use the IPMI management utility to obtain the IPMI IP address and then use CRSCTL to store the IP address in OLR by running a command similar to the following:

   crsctl set css ipmiaddr 192.168.10.45
   

2. Use CRSCTL to store the previously established user ID and password for the resident IPMI in OLR by running the crsctl set css ipmiadmin command, and supplying password at the prompt. For example:

   crsctl set css ipmiadmin administrator_name
   IPMI BMC password: password
   

   This command validates the supplied credentials and fails if another cluster node cannot access the local IPMI using them.

   After you complete hardware and operating system configuration, and register the IPMI administrator on Oracle Clusterware, IPMI-based failure isolation should be fully functional.

Modifying IPMI Configuration Using CRSCTL

To modify an existing IPMI-based failure isolation configuration (for example to change IPMI passwords, or to configure IPMI for failure isolation in an existing installation), use CRSCTL with the IPMI configuration tool appropriate to your platform. For example, to change the administrator password for IPMI, you must first modify the IMPI configuration as described in Oracle Grid Infrastructure Installation Guide, and then use CRSCTL to change the password in OLR.

The configuration data needed by Oracle Clusterware for IPMI is kept in an Oracle Wallet in OCR. Because the configuration information is kept in a secure store, it must be written by the Oracle Clusterware installation owner account (the Grid user), so you must log in as that installation user.

Use the following procedure to modify an existing IPMI configuration:

1. Enter the crsctl set css ipmiadmin administrator_name command. For example, with the user IPMIadm:

   crsctl set css ipmiadmin IPMIadm
   

   Provide the administrator password. Oracle Clusterware stores the administrator name and password for the local IPMI in OLR.

   After storing the new credentials, Oracle Clusterware can retrieve the new credentials and distribute them as required.

2. Enter the crsctl set css ipmiaddr bmc_ip_address command. For example:

   crsctl set css ipmiaddr 192.0.2.244
   

   This command stores the new IPMI IP address of the local IPMI in OLR, After storing the IP address, Oracle Clusterware can retrieve the new configuration and distribute it as required.

3. Enter the crsctl get css ipmiaddr command. For example:

   crsctl get css ipmiaddr
   

   This command retrieves the IP address for the local IPMI from OLR and displays it on the console.

4. Remove the IPMI configuration information for the local IPMI from OLR and delete the registry entry, as follows:

   crsctl unset css ipmiconfig
   

Removing IPMI Configuration Using CRSCTL

You can remove an IPMI configuration from a cluster using CRSCTL if you want to stop using IPMI completely or if IPMI was initially configured by someone other than the user that installed Oracle Clusterware. If the latter is true, then Oracle Clusterware cannot access the IPMI configuration data and IPMI is not usable by the Oracle Clusterware software, and you must reconfigure IPMI as the user that installed Oracle Clusterware.

To completely remove IPMI, perform the following steps. To reconfigure IPMI as the user that installed Oracle Clusterware, perform steps 3 and 4, then repeat steps 2 and 3 in "Modifying IPMI Configuration Using CRSCTL".

1. Disable the IPMI driver and eliminate the boot-time installation, as follows:

   /sbin/modprobe –r
   

   See Also:

   Oracle Grid Infrastructure Installation Guide for your platform for more information about the IPMI driver

2. Disable IPMI-over-LAN for the local IPMI using either ipmitool or ipmiutil, to prevent access over the LAN or change the IPMI administrator user ID and password.

3. Ensure that Oracle Clusterware is running and then use CRSCTL to remove the IPMI configuration data from OLR by running the following command:

   crsctl unset css ipmiconfig
   

4. Restart Oracle Clusterware so that it runs without the IPMI configuration by running the following commands as root:

   # crsctl stop crs
   # crsctl start crs
   

About Private Networks and Network Interfaces

Oracle Clusterware requires that each node is connected through a private network (in addition to the public network). The private network connection is referred to as the cluster interconnect. Table 2-5 describes how the network interface card (NIC) and the private IP address are stored.

Oracle only supports clusters in which all of the nodes use the same network interface connected to the same subnet (defined as a global interface with the oifcfg command). You cannot use different network interfaces for each node (node-specific interfaces). Refer to Appendix D, "Oracle Interface Configuration Tool (OIFCFG) Command Reference" for more information about global and node-specific interfaces.

Redundant Interconnect Usage

You can define multiple interfaces for Redundant Interconnect Usage by classifying the interfaces as private either during installation or after installation using the oifcfg setif command. When you do, Oracle Clusterware creates from one to four (depending on the number of interfaces you define) highly available IP (HAIP) addresses, which Oracle Database and Oracle ASM instances use to ensure highly available and load balanced communications.

The Oracle software (including Oracle RAC, Oracle ASM, and Oracle ACFS, all 11g release 2 (11.2.0.2), or later), by default, uses these HAIP addresses for all of its traffic, allowing for load balancing across the provided set of cluster interconnect interfaces. If one of the defined cluster interconnect interfaces fails or becomes non-communicative, then Oracle Clusterware transparently moves the corresponding HAIP address to one of the remaining functional interfaces.

Consequences of Changing Interface Names Using OIFCFG

The consequences of changing interface names depend on which name you are changing, and whether you are also changing the IP address. In cases where you are only changing the interface names, the consequences are minor. If you change the name for the public interface that is stored in OCR, then you also must modify the node applications for the cluster. Therefore, you must stop the node applications for this change to take effect.

https://metalink.oracle.com

Changing a Network Interface

You can change a network interface and its associated subnet address using the following procedure.

1. Ensure that Oracle Clusterware is running on all of the cluster nodes by running the following command:

   olsnodes -s
   

   The command returns output similar to the following, showing that Oracle Clusterware is running on all of the nodes in the cluster:

   ./olsnodes -s
   myclustera Active
   myclusterc Active
   myclusterb Active
   

2. Ensure that the replacement interface is configured and operational in the operating system on all of the nodes. Use the ifconfig command (or ipconfig on Windows) for your platform. For example, on Linux, use:

   /sbin/ifconfig..
   

3. Add the new interface to the cluster as follows, providing the name of the new interface and the subnet address (in this example, eth2 and 10.220.52.0, respectively), using the following command:

   $ oifcfg setif -global eth2 10.220.52.0:cluster_interconnect
   

   Note:

   You can use wildcards in the preceding command example.

4. Ensure that the previous step completes. Then you can remove the former subnet, as follows, by providing the name and subnet address of the former interface (in this example, eth1 and 10.220.65.0, respectively):

   $ oifcfg delif -global eth1 10.220.65.0
   

5. Verify the current configuration using the following command:

   oifcfg getif
   

   For example:

   $ oifcfg getif
   eth2 10.220.52.0 global cluster_interconnect
   eth0 10.220.16.0 global public
   

6. On all of the nodes, stop Oracle Clusterware by running the following command as the root user:

   # crsctl stop crs
   

7. When the cluster stops, you can deconfigure the deleted network interface in the operating system using the following command:

   $ ifconfig down
   

   At this point, the IP address from network interfaces for the old subnet is deconfigured from Oracle Clusterware. This command does not affect the configuration of the IP address on the operating system.

8. Restart Oracle Clusterware by running the following command on each cluster member node as the root user:

   # crsctl start crs
   

   The changes take effect when Oracle Clusterware restarts.

   If you use the CLUSTER_INTERCONNECTS initialization parameter, then you must update it to reflect the changes.

Changing the Network Interface for the Interconnect

To change the network interface for the private interconnect (for example, eth1), you must perform the change on all nodes (globally).

This procedure changes the network interface and IP address on each node in the cluster that Oracle Clusterware and Oracle Database were previously using.

To change the network interface, perform the following steps:

1. Ensure that the Oracle Clusterware stack is up and running on all cluster nodes.

2. Use operating system commands (ifconfig, or the command for your system) to ensure that the new or replacement interface is configured and up on all cluster member nodes.

3. On a single node in the cluster add the new global interface specification:

   $ oifcfg setif -global if_name/subnet:cluster_interconnect
   

   You can use wildcards with the interface name. For example, oifcfg setif -global "eth*/192.168.0.0:cluster_interconnect is valid syntax. However, be careful to avoid ambiguity with other addresses or masks used with other cluster interfaces. If you use wildcards, then you see a warning similar to the following:

   eth* 192.168.0.0 global cluster_interconnect
   PRIF-29: Warning: wildcard in network parameters can cause mismatch
   among GPnP profile, OCR, and system
   

   Note:

   Legacy network configuration does not support wildcards; thus wildcards are resolved using current node configuration at the time of the update.

   See Also:

   Appendix D, "Oracle Interface Configuration Tool (OIFCFG) Command Reference" for more information about using OIFCFG commands

4. Verify the configuration with the oifcfg getif command.

5. Remove the former subnet, as follows, providing the name and subnet address of the former interface:

   oifcfg delif -global if_name/subnet
   

   For example:

   $ oifcfg delif -global eth1/10.10.0.0
   

   Caution:

   This step should be performed only after a replacement interface is committed into the Grid Plug and Play configuration. Simple deletion of cluster interfaces without providing a valid replacement can result in invalid cluster configuration.

6. Stop Oracle Clusterware on all nodes by running the following command as root on each node:

   # crsctl stop crs
   

   Note:

   With cluster network configuration changes, the cluster must be fully stopped; do not use rolling stops and restarts.

7. Update the network configuration settings on the operating system.

   You must update the operating system configuration changes, because changes made using ifconfig are not persistent.

   See Also:

   Your operating system documentation for more information about how to make ifconfig commands persistent

8. Restart Oracle Clusterware by issuing the following command as root user on all nodes:

   # crsctl start crs
   

   You must restart Oracle Clusterware after running the oifcfg delif command, because Oracle Clusterware, Oracle ASM, and Oracle RAC continue to use the former subnet until they are restarted.