12 Monitoring WebLogic Domains

When using Enterprise Manger version 11.1 and a Secure Socket Layer (SSL) protocol to discover and monitor WebLogic servers, the Intelligent Agent must be able to "trust" the server before it can establish a secure communication link. The Agent maintains a Java Keystore (JKS) truststore containing certificates of servers with which it can establish a secure connection.

The Agent truststore is located at the following location:

$ORACLE_HOME/sysman/config/montrust/AgentTrust.jks

The Agent comes with nine well-known CA certificates.

Important:

It is recommended that customers using WebLogic t3s in a production environment use certificates signed by a well-known Certification Authority (CA), such as VeriSign or Thawte, on their WebLogic servers. A few popular Root CA certificates are available out-of-box in the Agent's JKS-based truststore and does not require any action by the customer. However, if self-signed certificates or the default (out-of-box) demo certificate are being used on the Weblogic servers, then the following step is needed to explicitly import the Root CA certificate for these server certificates to the Agent's truststore.

Updating the Agent truststore is required on ALL Enterprise Manger Agent's involved in the discovery and monitoring of the WebLogic domain using t3s/iiops.

Updating the Agent Truststore

To update the Agent truststore (AgentTrust.jks), you use emctl. If the default demo certificate, or a self-signed certificate is being used on the WebLogic servers for t3s/iiops, then the Root CA certificate for this must be added to the AgentTrust.jks in order for the Agent to be able to discover and monitor these WebLogic servers and J2EE applications using t3s. An emctl command is provided for this purpose.

emctl secure add_trust_cert_to_jks [-password <password> -trust_certs_loc <loc> -alias <alias>]

where

password - password to the AgentTrust.jks (if not specified will be prompted for)
trust_certs_loc - location of the cert file to import
alias - alias for the cert to import

Importing a Demo WebLogic Server Root CA Certificate.

To import the Root CA certificate for a Demo WebLogic server into the Agent's truststore, the emctl secure command needs to be executed from the host on which the Agent is located.

<ORACLE_HOME>/bin/emctl secure add_trust_cert_to_jks -password "welcome"

The following example demonstrates a typical session using the secure command with the add_trust_cert_to_jks option.

Example 12-1 Sample Session

./emctl secure add_trust_cert_to_jks -password welcome 
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0 
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.

Message   :   Certificate was added to keystore
ExitStatus: SUCCESS

The default out-of-box password for the AgentTrust.jks is "welcome" and it is recommended that this be changed using the JDK keytool utility. If no password is specified along with the emctl command, the system will prompt you for the password.

Importing a Custom Root CA Certificate

If the WebLogic servers are secured with another certificate, such as a self-signed certificate, then that Root CA certificate must be imported into the Agent's truststore as follows:

<ORACLE_HOME>/bin/emctl secure add_trust_cert_to_jks -password "welcome" trust_certs_loc <location of certificate> -alias <certificate-alias>

Changing the Default AgentTrust.jks Password Using Keytool

The following JVM keytool utility command will let you change the default out-of-box password to the AgentTrust.jks.

<OH>/jdk/bin/keytool -storepasswd -keystore AgentTrust.jks -storepass welcome -new myNewPass

Discovering and Monitoring weblogic domains where Admin Channel is enabled

When the Administration channel is enabled on a WebLogic 9.x or higher domain, additional steps are required on all Agents to enable discovery and monitoring functionality for these domains. You must generate and install WebLogic fullclient jar files into the $ORACLE/sysman/jlib directory of each Agent(s) monitoring the WebLogic domain.

Update the Agent's JKS-based truststore using the emctl command specified in the preceding sections. For example: ./emctl secure add_trust_cert_to_jks -password welcome to populate the Agent's truststore with the Root CA certificate of the WebLogic demo certificate or other custom certificate.
From the BEA_HOME/wlserver10.3/server/lib directory generate a wlfullclient.jar as per instructions in the following link
```
http://download.oracle.com/docs/cd/E12840_01/wls/docs103/client/jarbuilder.html
```
Basically, you invoke java -jar wljarbuilder.jar from above directory which will result in a wlfullclient.jar file being created there.
Copy over wlfullclient.jar and wlcipher.jar from above location to the $ORACLE_HOME/sysman/jlib directory for each of the Agents monitoring/discovering this WebLogic domain secured via AdminChannel and restart the Agent

Collecting JVM Performance Metrics for WebLogic Servers

In order to collect JVM performance metrics from platform MBeans, the Mbeans must be made accessible via the runtime MBeanServer. To do this, from the WebLogic console, set PlatformMBeanServerEnabled=true. (Domain->Advanced)

Note:

Only applies to WebLogic server installations where Java Required Files (JRF) are not installed.

Setting the PlatformMBeanServerUsed Attribute

If you are using WebLogic server versions 9.2.0.40, 10.0.2.0, 10.3.1 and 10.3.2 and certain patch releases of 9.x, you must explicitly set the PlatformMBeanServerUsed attribute to TRUE in addition to setting the PlatformMBeanServerEnabled (shown in the previous section). You set the PlatformMBeanServerUsed attribute using the WebLogic Scripting Tool (WLST), as shown in the next section.

Note:

From 10.3.3 onwards, the default out-of-box behavior enables platform MBeans to be accessible via runtime MBeanServers. Hence, this section can be skipped.

Activating Platform MBeans on WebLogicServer 9.x to 10.3.2 versions

The following WebLogic Scripting Tool session shown in Example 12-2 demonstrates how to use check and set the PlatformMBeanServerUsed attribute.

User actions are shown in bold.

Example 12-2 Setting PlatformMBeanServerUsed

cd common/bin/

ade:[ sparmesw_easvr ] [sparmesw@stacc20 bin]$ ./wlst.sh 

CLASSPATH=/net/stacc20/scratch/shiphomes/wl/wl10/patch_wls1002/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/net/stacc20/scratch/shiphomes/wl/wl10/patch_cie640/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/net/stacc20/scratch/shiphomes/wl/wl10/jrockit_150_15/lib/tools.jar:/net/stacc20/scratch/shiphomes/wl/wl10/wlserver_10.0/server/lib/weblogic_sp.jar:/net/stacc20/scratch/shiphomes/wl/wl10/wlserver_10.0/server/lib/weblogic.jar:/net/stacc20/scratch/shiphomes/wl/wl10/modules/features/weblogic.server.modules_10.0.2.0.jar:/net/stacc20/scratch/shiphomes/wl/wl10/modules/features/com.bea.cie.common-plugin.launch_2.1.2.0.jar:/net/stacc20/scratch/shiphomes/wl/wl10/wlserver_10.0/server/lib/webservices.jar:/net/stacc20/scratch/shiphomes/wl/wl10/modules/org.apache.ant_1.6.5/lib/ant-all.jar:/net/stacc20/scratch/shiphomes/wl/wl10/modules/net.sf.antcontrib_1.0b2.0/lib/ant-contrib.jar:PATH=/net/stacc20/scratch/shiphomes/wl/wl10/wlserver_10.0/server/bin:/net/stacc20/scratch/shiphomes/wl/wl10/modules/org.apache.ant_1.6.5/bin:/net/stacc20/scratch/shiphomes/wl/wl10/jrockit_150_15/jre/bin:/net/stacc20/scratch/shiphomes/wl/wl10/jrockit_150_15/bin:/home/sparmesw/products/valgrind/bin:/ade/sparmesw_easvr/oracle/jdk/bin:/ade/sparmesw_easvr/oracle/work/middleware/oms/perl/bin:/bin:/usr/local/bin:/usr/local/remote/packages/firefox-1.5.0.3:/ade/sparmesw_easvr/oratst/bin:/ade/sparmesw_easvr/oracle/buildtools/bin:/ade/sparmesw_easvr/oracle/emdev/merge:/ade/sparmesw_easvr/oracle/emdev/utl:/ade/sparmesw_easvr/oracle/utl:/pdp/pds/utl:/ade/sparmesw_easvr/oracle/work/middleware/oms/bin:/ade/sparmesw_easvr/oracle/nlsrtl3/bin:/opt/SUNWspro/bin:/usr/ccs/bin:/usr/bin:/usr/sbin:/ade/sparmesw_easvr/oracle/opmn/bin:/usr/X11R6/bin:/home/sparmesw/products/valgrind/bin:/home/sparmesw/products/valgrind/bin:/usr/kerberos/bin:/home/sparmesw/products/valgrind/bin:/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin:/usr/local/ade/bin:/bin:/usr/local/bin

Your environment has been set.

CLASSPATH=/net/stacc20/scratch/shiphomes/wl/wl10/patch_wls1002/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/net/stacc20/scratch/shiphomes/wl/wl10/patch_cie640/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/net/stacc20/scratch/shiphomes/wl/wl10/jrockit_150_15/lib/tools.jar:/net/stacc20/scratch/shiphomes/wl/wl10/wlserver_10.0/server/lib/weblogic_sp.jar:/net/stacc20/scratch/shiphomes/wl/wl10/wlserver_10.0/server/lib/weblogic.jar:/net/stacc20/scratch/shiphomes/wl/wl10/modules/features/weblogic.server.modules_10.0.2.0.jar:/net/stacc20/scratch/shiphomes/wl/wl10/modules/features/com.bea.cie.common-plugin.launch_2.1.2.0.jar:/net/stacc20/scratch/shiphomes/wl/wl10/wlserver_10.0/server/lib/webservices.jar:/net/stacc20/scratch/shiphomes/wl/wl10/modules/org.apache.ant_1.6.5/lib/ant-all.jar:/net/stacc20/scratch/shiphomes/wl/wl10/modules/net.sf.antcontrib_1.0b2.0/lib/ant-contrib.jar::/net/stacc20/scratch/shiphomes/wl/wl10/wlserver_10.0/common/eval/pointbase/lib/pbembedded51.jar:/net/stacc20/scratch/shiphomes/wl/wl10/wlserver_10.0/common/eval/pointbase/lib/pbtools51.jar:/net/stacc20/scratch/shiphomes/wl/wl10/wlserver_10.0/common/eval/pointbase/lib/pbclient51.jar

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

wls:/offline>

wls:/offline> connect('weblogic','welcome1','stacc20:7501')
Connecting to t3://stacc20:7501 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'base_domain'.

Warning: An insecure protocol was used to connect to the server. To ensure on-the-wire security, the SSL port or Admin port should be used instead.

wls:/base_domain/serverConfig> edit()
Location changed to edit tree. This is a writable tree with DomainMBean as the root. To make changes you will need to start an edit session via startEdit(). 

For more help, use help(edit)

wls:/base_domain/edit> startEdit()
Starting an edit session ...
Started edit session, please be sure to save and activate your changes once you are done.

wls:/base_domain/edit !> cd('JMX')

wls:/base_domain/edit/JMX !> ls()
drw-   base_domain

wls:/base_domain/edit/JMX !> cd ('base_domain')

wls:/base_domain/edit/JMX/base_domain !> ls()
-rw-   CompatibilityMBeanServerEnabled              true
-rw-   DomainMBeanServerEnabled                     true
-rw-   EditMBeanServerEnabled                       true
-rw-   InvocationTimeoutSeconds                     0
-rw-   ManagementEJBEnabled                         true
-rw-   Name                                         base_domain
-rw-   Notes                                        null
-rw-   PlatformMBeanServerEnabled                   true
-rw-   PlatformMBeanServerUsed                      false **
-rw-   RuntimeMBeanServerEnabled                    true
-r--   Type                                         JMX

-r-x   freezeCurrentValue                           Void : String(attributeName)
-r-x   isSet                                        Boolean : String(propertyName
)
-r-x   restoreDefaultValue                          Void : String(attributeName)
-r-x   unSet                                        Void : String(propertyName)

wls:/base_domain/edit/JMX/base_domain !> set('PlatformMBeanServerUsed','true')
wls:/base_domain/edit/JMX/base_domain !> ls()

-rw-   CompatibilityMBeanServerEnabled              true
-rw-   DomainMBeanServerEnabled                     true
-rw-   EditMBeanServerEnabled                       true
-rw-   InvocationTimeoutSeconds                     0
-rw-   ManagementEJBEnabled                         true
-rw-   Name                                         base_domain
-rw-   Notes                                        null
-rw-   PlatformMBeanServerEnabled                   true
-rw-   PlatformMBeanServerUsed                      true  **
-rw-   RuntimeMBeanServerEnabled                    true
-r--   Type                                         JMX
-r-x   freezeCurrentValue                           Void : String(attributeName)
-r-x   isSet                                        Boolean : String(propertyName
)
-r-x   restoreDefaultValue                          Void : String(attributeName)
-r-x   unSet                                        Void : String(propertyName)

wls:/base_domain/edit/JMX/base_domain !> activate()
Activating all your changes, this may take a while ... 
The edit lock associated with this edit session is released once the activation is completed.

The following non-dynamic attribute(s) have been changed on MBeans 
that require server re-start: **
MBean Changed : com.bea:Name=base_domain,Type=JMX
Attributes changed : PlatformMBeanServerUsed

Activation completed
wls:/base_domain/edit/JMX/base_domain> ade:[ sparmesw_easvr ] [sparmesw@stacc20 bin]$ 
ade:[ sparmesw_easvr ] [sparmesw@stacc20 bin]$

** NOTE: PlatformMBeanServerUsed attribute is present in WebLogic releases 10.3.1.0 and 10.3.2.0 and also for certain patch releases of prior versions. If above PlatformMBeanServerUsed attribute is NOT present, or if it is present and already set to true, then running the commands are not necessary.