J Troubleshooting Oracle Clusterware

CHM Services

CHM consists of the following services:

• System Monitor Service

• Cluster Logger Service

• Oracle Grid Infrastructure Management Repository

System Monitor Service

There is one system monitor service on every node. The system monitor service (osysmond) is a real-time, monitoring and operating system metric collection service that sends the data to the cluster logger service. The cluster logger service receives the information from all the nodes and persists in an Oracle Grid Infrastructure Management Repository database.

Cluster Logger Service

There is one cluster logger service (OLOGGERD) for every 32 nodes in a cluster. Another OLOGGERD is spawned for every additional 32 nodes (which can be a sum of Hub and Leaf Nodes). If the cluster logger service fails (because the service is not able come up after a fixed number of retries or the node where it was running is down), then Oracle Clusterware starts OLOGGERD on a different node. The cluster logger service manages the operating system metric database in the Oracle Grid Infrastructure Management Repository.

Oracle Grid Infrastructure Management Repository

The Oracle Grid Infrastructure Management Repository:

• Is an Oracle database that stores real-time operating system metrics collected by CHM. You configure the Oracle Grid Infrastructure Management Repository during an installation of or upgrade to Oracle Clusterware 12c on a cluster.

  Note:

  If you are upgrading Oracle Clusterware to Oracle Clusterware 12c and Oracle Cluster Registry (OCR) and the voting file are stored on raw or block devices, then you must move them to Oracle ASM or a shared file system before you upgrade your software.

• Runs on one node in the cluster (this must be a Hub Node in an Oracle Flex Cluster configuration), and must support failover to another node in case of node or storage failure.

  You can locate the Oracle Grid Infrastructure Management Repository on the same node as the OLOGGERD to improve performance and decrease private network traffic.

• Communicates with any cluster clients (such as OLOGGERD and OCLUMON) through the private network. Oracle Grid Infrastructure Management Repository communicates with external clients over the public network, only.

• Data files are located in the same disk group as the OCR and voting file.

  If OCR is stored in an Oracle ASM disk group called +MYDG, then configuration scripts will use the same disk group to store the Oracle Grid Infrastructure Management Repository.

  Oracle increased the Oracle Clusterware shared storage requirement to accommodate the Oracle Grid Infrastructure Management Repository, which can be a network file system (NFS), cluster file system, or an Oracle ASM disk group.

• Size and retention is managed with OCLUMON.

Collecting CHM Data

You can collect CHM data from any node in the cluster by running the Grid_home/bin/diagcollection.pl script on the node.

To run the data collection script on only the node where the cluster logger service is running:

1. Run the following command to identify the node running the cluster logger service:

   $ Grid_home/bin/oclumon manage -get master
   

2. Run the following command from a writable directory outside the Grid home as a privileged user on the cluster logger service node to collect all the available data in the Oracle Grid Infrastructure Management Repository:

   # Grid_home/bin/diagcollection.pl --collect
   

   On Windows, run the following commands:

   C:\Grid_home\perl\bin\perl.exe
   C:\Grid_home\bin\diagcollection.pl --collect
   

   The diagcollection.pl script creates a file called chmosData_host_name_time_stamp.tar.gz, similar to the following:

   chmosData_stact29_20121006_2321.tar.gz
   

To limit the amount of data you want collected, enter the following command on a single line:

# Grid_home/bin/diagcollection.pl --collect --chmos
   --incidenttime time --incidentduration duration

In the preceding command, the format for the --incidenttime argument is MM/DD/YYYY24HH:MM:SS and the format for the --incidentduration argument is HH:MM. For example:

# Grid_home/bin/diagcollection.pl --collect --crshome Grid_home
   --chmos --incidenttime 07/21/2013 01:00:00 --incidentduration 00:30

oclumon debug

Use the oclumon debug command to set the log level for the CHM services.

Syntax

oclumon debug [log daemon module:log_level] [version]

Parameters

log daemon module:log_level

version

Example

The following example sets the log level of the system monitor service (osysmond):

$ oclumon debug log osysmond CRFMOND:3

oclumon dumpnodeview

Use the oclumon dumpnodeview command to view log information from the system monitor service in the form of a node view.

A node view is a collection of all metrics collected by CHM for a node at a point in time. CHM attempts to collect metrics every five seconds on every node. Some metrics are static while other metrics are dynamic.

A node view consists of eight views when you display verbose output:

• SYSTEM: Lists system metrics such as CPU COUNT, CPU USAGE, and MEM USAGE

• TOP CONSUMERS: Lists the top consuming processes in the following format:

  metric_name: 'process_name(process_identifier) utilization'
  

• PROCESSES: Lists process metrics such as PID, name, number of threads, memory usage, and number of file descriptors

• DEVICES: Lists device metrics such as disk read and write rates, queue length, and wait time per I/O

• NICS: Lists network interface card metrics such as network receive and send rates, effective bandwidth, and error rates

• FILESYSTEMS: Lists file system metrics, such as total, used, and available space

• PROTOCOL ERRORS: Lists any protocol errors

• CPUS: Lists statistics for each CPU

You can generate a summary report that only contains the SYSTEM and TOP CONSUMERS views.

"Metric Descriptions" lists descriptions for all the metrics associated with each of the views in the preceding list.

Example J-1 shows an example of a node view.

Syntax

oclumon dumpnodeview [[-allnodes] | [-n node1 node2 noden] [-last "duration"] | 
[-s "time_stamp" -e "time_stamp"] [-i interval] [-v]] [-h]

Parameters

-allnodes

-n node1 node2

-last "duration"

"23:05:00"

-s "time_stamp" -e "time_stamp"

"2011-05-10 23:05:00"

-i interval

-v

-h

Usage Notes

• In certain circumstances, data can be delayed for some time before it is replayed by this command. For example, the crsctl stop cluster -all command can cause data delay. After running crsctl start cluster -all, it may take several minutes before oclumon dumpnodeview shows any data collected during the interval.

• The default is to continuously dump node views. To stop continuous display, use Ctrl+C on Linux and Windows.

• Both the local system monitor service (osysmond) and the cluster logger service (ologgerd) must be running to obtain node view dumps.

Examples

The following example dumps node views from node1, node2, and node3 collected over the last twelve hours:

$ oclumon dumpnodeview -n node1 node2 node3 -last "12:00:00"

The following example displays node views from all nodes collected over the last fifteen minutes at a 30 second interval:

$ oclumon dumpnodeview -allnodes -last "00:15:00" -i 30

Metric Descriptions

This section includes descriptions of the metrics in each of the seven views that comprise a node view listed in the following tables.

#pcpus

#vcpus

cpuht

chipname

cpu

cpuq

physmemfree

physmemtotal

mcache

swapfree

swaptotal

hugepagetotal

hugepagefree

hugepagesize

ior

iow

ios

swpin

swpout

pgin

pgout

netr

netw

procs

procsoncpu

rtprocs

rtprocsoncpu

#fds

#sysfdlimit

#disks

#nics

nicErrors

name

pid

#procfdlimit

cpuusage

privmem

shm

workingset

#fd

#threads

priority

nice

state

ior

iow

ios

qlen

wait

type

netrr

netwr

neteff

nicerrors

pktsin

pktsout

errsin

errsout

indiscarded

outdiscarded

inunicast

type

innonunicast

latency

total

mount

type

used

available

used%

ifree%

IPHdrErr

IPAddrErr

IPUnkProto

IPReasFail

IPFragFail

TCPFailedConn

TCPEstRst

TCPRetraSeg

UDPUnkPort

UDPRcvErr

^Footnote 1 All protocol errors are cumulative values since system startup.

cpuid

sys-usage

user-usage

nice

usage

iowait

----------------------------------------
Node: rwsak10 Clock: '14-04-16 18.47.25 PST8PDT' SerialNo:155631
----------------------------------------

SYSTEM:
#pcpus: 2 #vcpus: 24 cpuht: Y chipname: Intel(R) cpu: 1.23 cpuq: 0
physmemfree: 8889492 physmemtotal: 74369536 mcache: 55081824 swapfree: 18480404
swaptotal: 18480408 hugepagetotal: 0 hugepagefree: 0 hugepagesize: 2048 ior: 132
iow: 236 ios: 23 swpin: 0 swpout: 0 pgin: 131 pgout: 235 netr: 72.404
netw: 97.511 procs: 969 procsoncpu: 6 rtprocs: 62 rtprocsoncpu N/A #fds: 32640
#sysfdlimit: 6815744 #disks: 9 #nics: 5 nicErrors: 0

TOP CONSUMERS:
topcpu: 'osysmond.bin(30981) 2.40' topprivmem: 'oraagent.bin(14599) 682496'
topshm: 'ora_dbw2_oss_3(7049) 2156136' topfd: 'ocssd.bin(29986) 274'
topthread: 'java(32255) 53'

CPUS:

cpu18: sys-2.93 user-2.15 nice-0.0 usage-5.8 iowait-0.0 steal-0.0
.
.
.

PROCESSES:

name: 'osysmond.bin' pid: 30891 #procfdlimit: 65536 cpuusage: 2.40 privmem: 35808
shm: 81964 #fd: 119 #threads: 13 priority: -100 nice: 0 state: S
.
.
.

DEVICES:

sdi ior: 0.000 iow: 0.000 ios: 0 qlen: 0 wait: 0 type: SYS
sda1 ior: 0.000 iow: 61.495 ios: 629 qlen: 0 wait: 0 type: SYS
.
.
.

NICS:

lo netrr: 39.935  netwr: 39.935  neteff: 79.869  nicerrors: 0 pktsin: 25
pktsout: 25  errsin: 0  errsout: 0  indiscarded: 0  outdiscarded: 0
inunicast: 25 innonunicast: 0  type: PUBLIC
eth0 netrr: 1.412  netwr: 0.527  neteff: 1.939  nicerrors: 0 pktsin: 15
pktsout: 4  errsin: 0  errsout: 0  indiscarded: 0  outdiscarded: 0
inunicast: 15  innonunicast: 0  type: PUBLIC  latency: <1

FILESYSTEMS:

mount: / type: rootfs total: 563657948 used: 78592012 available: 455971824
used%: 14 ifree%: 99 GRID_HOME
.
.
.

PROTOCOL ERRORS:

IPHdrErr: 0 IPAddrErr: 0 IPUnkProto: 0 IPReasFail: 0 IPFragFail: 0
TCPFailedConn: 5197 TCPEstRst: 717163 TCPRetraSeg: 592 UDPUnkPort: 103306
UDPRcvErr: 70

oclumon manage

Use the oclumon manage command to view and change configuration information from the system monitor service.

Syntax

oclumon manage -repos {{changeretentiontime time} | {changerepossize 
memory_size}} | -get {key1 [key2 ...] | alllogger [-details] | mylogger [-details]}

Parameters

-repos 
{{changeretentiontime time} |
{changerepossize memory_size}}

-get key1 [key2 ...]

-h

Usage Notes

• The local system monitor service must be running to change the retention time of the CHM repository.

• The Cluster Logger Service must be running to change the retention time of the CHM repository.

Examples

The following examples show commands and sample output:

$ oclumon manage -get MASTER
Master = node1

$ oclumon manage -get alllogger -details
Logger = node1
Nodes = node1,node2

$ oclumon manage -repos changeretentiontime 86400

$ oclumon manage -repos changerepossize 6000

oclumon version

Use the oclumon version command to obtain the version of CHM that you are using.

Syntax

oclumon version

Example

This command produces output similar to the following:

Cluster Health Monitor (OS), Version 12.1.0.0.2 - Production Copyright 2013
Oracle. All rights reserved.

tfactl print

Use the tfactl print command to print information from the Berkeley database.

Syntax

tfactl print [status | config | directories | hosts | actions | repository | cookie

Parameters

status

config

directories

hosts

actions

repository

cookie

Example

The tfactl print config command returns output similar to the following:

Configuration parameter                                Value
------------------------------------------------------------
TFA Version                                          2.5.1.5
Automatic diagnostic collection                          OFF
Trimming of files during diagcollection                   ON
Repository current size (MB) in node1                    526
Repository maximum size (MB) in node1                  10240
Trace Level                                                1

In the preceding sample output:

• Automatic diagnostic collection: When ON (default is OFF), if scanning an alert log, then finding specific events in those logs triggers diagnostic collection.

• Trimming of files during diagcollection: Determines if TFA will trim large files to contain only data that is within specified time ranges. When this is OFF, no trimming of trace files occurs for automatic diagnostic collection.

• Repository current size in MB: How much space in the repository is currently used.

• Repository maximum size in MB: The maximum size of storage space in the repository. Initially, the maximum size is set to the smaller of either 10GB or 50% of free space in the file system.

• Trace Level: 1 is the default, and the values 2, 3, and 4 have increasing verbosity. While you can set the trace level dynamically for the running TFA daemon, increasing the trace level significantly impacts the performance of TFA, and should only be done at the request of My Oracle Support.

tfactl purge

Use the tfactl purge command to delete diagnostic collections from the TFA repository that are older than a specific time.

Syntax

tfactl purge -older number[h | d]

Specify a number followed by h or d (to specify a number of hours or days, respectively) to remove files older than the specified time constraint. For example:

# tfactl purge -older 30d

The preceding command removes files older than 30 days.

tfactl directory

Use the tfactl directory command to add a directory to, or remove a directory from, the list of directories that will have their trace or log files analyzed. You can also use this command to change the directory permissions. When a directory is added by auto discovery, it is added as public, which means that any file in that directory can be collected by any user that has permission to run the tfactl diagcollect command. This is only important when sudo is used to allow users other than root to run TFACTL commands. If a directory is marked as private, then TFA determines which user is running TFACTL commands and verifies that the user has permissions to see the files in the directory before allowing any files to be collected.

Syntax

tfactl directory [add directory | remove directory | modify directory -private | -public]

Parameters

add directory

remove directory

modify directory
-private | -public

Usage Notes

You must add all trace directory names to the Berkeley database, so that TFA will collect file metadata in that directory. The discovery process finds most directories but if new or undiscovered directories are required, then you can add these manually using the tfactl directory command. When you add a directory using TFACTL, TFA attempts to determine whether the directory is for the database, Oracle Clusterware, operating system logs, or some other component, and for which database or instance. If TFA cannot determine this information, then it returns an error and requests that you enter the information, similar to the following:

# tfactl directory add /tmp

Failed to add directory to TFA. Unable to determine parameters for directory: /tmp
Please enter component for this Directory [RDBMS|CRS|ASM|INSTALL|OS|CFGTOOLS] : RDBMS
Please enter database name for this Directory :MYDB
Please enter instance name for this Directory :MYDB1

tfactl host

Use the tfactl host command to add hosts to, or remove hosts from, the TFA cluster.

Syntax

tfactl host [add host_name | remove host_name]

Specify a host name to add or remove, as in the following example:

# tfactl host add myhost.domain.com

Usage Notes

Using the tfactl host add command notifies the local TFA about other nodes on your network. When you add a host, TFA contacts that host and, if TFA is running on that host, then both hosts synchronize their host list. TFA authenticates that a host can be added using a cookie. If the host to be added does not have the correct cookie, then you must retrieve that cookie from an existing host in the cluster and set it on the host being added, similar to the following:

#tfactl host add node2

Failed to add host: node2 as the TFA cookies do not match.
To add the host successfully, try the following steps:
1. Get the cookie in node1 using:
  ./tfa_home/bin/tfactl print cookie
2. Set the cookie from Step 1 in node2 using:
  ./tfa_home/bin/tfactl set cookie=<COOKIE>
3. After Step 2, add host again:
  ./tfa_home/bin/tfactl host add node2

After you successfully add the host, all clusterwide commands will activate on all nodes registered in the Berkeley database.

tfactl set

Use the tfactl set command to adjust the manner in which TFA runs.

Syntax

tfactl set [autodiagcollect=ON | OFF | cookie=UID | trimfiles=ON | OFF |
tracelevel=1 | 2 | 3 | 4 | reposizeMB=number | repositorydir=directory] [-c]

Parameters

autodiagcollect=ON | OFF

cookie=UID

trimfiles=ON | OFF

tracelevel=1 | 2 | 3 | 4

-c

Automatic Diagnostic Collection

After TFA initially gathers trace file metadata, the daemon monitors all files that are determined to be alert logs using tail, so that TFA can take action when certain strings occur.

By default, these logs are database alert logs, Oracle ASM alert logs, and Oracle Clusterware alert logs. When specific patterns occur in the logs saved to the Berkeley database, automatic diagnostic collection may take place.

Exactly what is collected depends on the pattern that is matched. TFA may just store information on the pattern matched or may initiate local diagnostic collection. Although TFA always monitors the logs and collects information into its database, automatic diagnostic collection only happens if it is enabled first using the tfactl set command.

tfactl diagcollect

Use the tfactl diagcollect command to perform on-demand diagnostic collection. You can configure a number of different parameters to determine how large or detailed a collection you want. You can specify a specific time of an incident or a time range for data to be collected, and determine if whole files that have relevant data should be collected or just a time interval of data from those files.

Syntax

tfactl diagcollect [-all | -database all | database_1,database_2,... | -asm | -crs | -os | -install]
[-node all | local | node_1,node_2,...][-tag description] [-z file_name]
[-since numberh | d | -from "mmm/dd/yyyy hh:mm:ss" -to "mmm/dd/yyyy hh:mm:ss"
| -for "mmm/dd/yyyy hh:mm:ss" [-nocopy] [-nomonitor]]

Parameters

-all | -database all
 | database_1,database_2,...
 | -asm | -crs | -os |
 -install

-node all | local |
 node_1,node_2,...

-tag description

-z file_name

-since numberh | d |
 -from "mmm/dd/yyyy hh:mm:ss" 
-to "mmm/dd/yyyy hh:mm:ss" |
 -for "mmm/dd/yyyy hh:mm:ss"

-nocopy

-nomonitor

Examples

• The following command trims and zips all files updated in the last four hours, including chmos and osw data, from across the cluster and collects it on the initiating node:

  # tfactl diagcollect
  

• The following command trims and zips all files updated in the last eight hours, including chmos and osw data, from across the cluster and collects it on the initiating node:

  # tfactl diagcollect –all –since 8h
  

• The following command trims and zips all files from databases hrdb and fdb updated in the last one day and collects it on the initiating node:

  # tfactl diagcollect -database hrdb,fdb -since 1d -z foo
  

• The following command trims and zips all Oracle Clusterware files, operating system logs, and chmos and osw data from node1 and node2 updated in the last six hours, and collects it on the initiating node:

  # tfactl diagcollect –crs -os -node node1,node2 -since 6h
  

• The following command trims and zips all Oracle ASM logs from node1 updated between July 4, 2014 and July 5, 2015 at 21:00, and collects it on the initiating node:

  # tfactl diagcollect -asm -node node1 -from Jul/4/2014 -to "Jul/5/2014 21:00:00"
  

• The following command trims and zips all log files updated on July 2, 2014 and collects it on the initiating node:

  # tfactl diagcollect -for Jul/2/2014
  

• The following command trims and zips all log files updated from 09:00 on July 2, 2014, to 09:00 on July 3, 2014, which is 12 hours before and after the time specified in the command, and collects it on the initiating node:

  # tfactl diagcollect -for "Jul/2/2014 21:00:00"