14 Application Server Management

Out-of-Box Management Using Oracle Application Server Control

Every Oracle Application Server 10g instance is installed with an Application Server Control to help you manage that particular application server instance. Application Server Control provides Web-based management features designed to monitor and administer application server instances, farms, and clusters. You can also deploy applications, monitor real-time performance, manage security, and configure the application server components.

Application Server Control relies heavily on various underlying technologies to discover, monitor, and administer the application servers in an environment. It consists of the Application Server Control console and its underlying technologies:

• Oracle Dynamic Monitoring Service (DMS)

• Oracle Process Management Notification (OPMN)

• Distributed Configuration Management (DCM)

• A local version of the Oracle Management Agent specifically designed to gather monitoring data

However, you can manage only one application server at a time using Application Server Control. Typically, you have multiple application servers in your enterprise configuration, and managing all these instances using individual Application Server Controls becomes very difficult.

For centralized management and additional management functionality (for example, application service level management, deployments, historical data collections for performance trending alerts, and so on), you can use Grid Control.

Centralized Management Using Enterprise Manager Grid Control

While Application Server Control provides standalone management for a single application server instance and its components, Grid Control provides centralized management of multiple application servers in your environment.

For example, if you have ten application servers installed on ten different hosts, then you can manage all these ten application servers and hosts through a single window using Grid Control. With the help of Management Agents deployed on each host, Grid Control automatically discovers the application servers on these hosts and begins monitoring them using default monitoring levels, notification rules, and other default settings.

Both Application Server Control and Grid Control have their own application server home pages that provide easy access to key information required by the administrators. The Application Server Home page on Grid Control (Figure 14-1) provides:

• Application server status, responsiveness, and performance data

• Resource usage for the application server and its components

• List of core components that were installed and configured for the application server, and links to their home pages

• Functionality to start, stop, and restart any of those core components

• Alerts and diagnostic drill-downs so you can identify and resolve problems quickly

• Links to Application Server Control for administration operations such as starting and stopping components, modifying configurations, and deploying applications

• Links to other pages in Grid Control that might be helpful in accomplishing your given task

Figure 14-1 Application Server Home Page

What Gets Monitored?

Grid Control helps you monitor the following application server types and components:

Viewing Critical Information

This section describes how you can use Grid Control to view critical information pertaining to application server targets.

General Information

Grid Control provides general information about the targets including their status and availability. This helps you understand how the target is performing, where it is deployed, what is its version, where is its home directory, and so on. If the target is a logical target that groups its inherent members, then details about the membership are also provided.

Figure 14-2 General Section

Alerts and Host Alerts

Grid Control shows the number of critical, warning, and error alerts generated for the past 24 hours. These alerts indicate that a particular metric condition has been encountered. For example, an alert is triggered when a metric threshold is reached. Using these details, you can drill down to investigate the target and the problem that triggered the alert. For more information about alerts, see Automated Monitoring and Alerts in this chapter.

Figure 14-3 Alerts and Host Alerts Sections

Policy Violations

Grid Control also shows the number of informational, warning, and critical policy rules violated for the application server target. You can see the roll up of individual policy compliance scores for that target. Compliance scores provide a quick way to determine the health of the your target. You can also see the policy trend overview for the last 24 hours, for the last week, for the last month, or for a user-define time period, and determine your course of action in solving the policy violations.

Grid Control also shows the time when security policy rules were last evaluated, and shows the roll up of individual policy compliance scores for that target.

Figure 14-4 Policy Violations and Security Sections

Viewing Information About Applications Deployed

Grid Control provides details about the J2EE applications deployed on the application server instances. You can quickly assess which applications are most active and which are using the most system resources.

Figure 14-5 Viewing Details about the Applications Deployed

Description of "Figure 14-5 Viewing Details about the Applications Deployed"

Grid Control gives you the flexibility to update the information about applications based on current, real-time application performance, or based on historical data from the Management Repository. For example, you can see which applications were the most active over the past 24 hours, over the past week, or over the past month.

Viewing Members

For logical, composite targets like OracleAS Farms and Clusters, Grid Control provides rolled-up information on the member targets that make up the group. It provides a "Members" tab that helps you view the members associated with that particular composite target; check their status; and start, stop, or restart each of them.

Figure 14-6 Viewing Details about Member Targets

Grid Control also helps you study the High Availability (HA) grouping done for the members of an OracleAS DCM Managed Cluster. A High Availability Group is a group composed of similar individual components of application server instances clustered together in a DCM managed cluster. For example, an OC4J High Availability Group has a group of OC4J instances in an OracleAS Cluster.

Monitoring Performance

This section describes the features offered by Grid Control that help you monitor the performance of application server targets.

Automated Monitoring and Alerts

Grid Control provides a comprehensive set of features that facilitates automated monitoring and generation of alerts. Oracle Management Agent on a host automatically discovers the Oracle Application Server targets on that host, and helps Grid Control perform unattended monitoring of their status, health, and performance.

Grid Control gathers and evaluates diagnostic information from these targets distributed across the enterprise, and an extensive array of application server performance metrics are automatically monitored against predefined thresholds.

For example, Grid Control can automatically monitor:

• The CPU or memory consumption of the application server, including detailed monitoring of individual Java Virtual Machines (JVMs) being run by the server's Oracle Application Server Containers for J2EE (OC4J) instances

• J2EE application responsiveness from the application down through individual servlets and Enterprise JavaBeans (EJBs)

• HTTP Server session volumes, connection duration, and error rates

• Oracle Application Server Web Cache hit rates and volumes

• Top servlets based on number of requests, maximum processing time, and highest average processing time

If an Oracle Application Server or any of its core components go down, or if a performance metric crosses a warning or critical threshold, then an alert is generated by Grid Control and a notification is sent to you. Grid Control supports notifications via e-mail (including e-mail-to-page systems), SNMP traps, and/or by running custom scripts.

When you receive an alert notification, Grid Control makes it easy for you to investigate the problem and take corrective actions wherever required. For example, notification of excessive CPU consumption by OC4J may lead to investigation of the applications running in that container. By using the Top J2EE Applications tab of the Application Server Home page (Figure 14-7) in Grid Control, you can quickly identify the highest volume or least responsive application. You can then drill down and diagnose application's servlets, Java Server Pages (JSPs), or EJBs to identify the bottleneck.

Figure 14-7 Monitoring the Performance of Your Deployed J2EE Applications

You can set up corrective actions to automatically resolve an alert condition. These corrective actions ensure that routine responses to alerts are automatically executed, thereby saving you time and ensuring that problems are dealt with before they noticeably impact the users.

You can also use monitoring templates to simplify the task of standardizing monitoring settings across your enterprise. You can specify the monitoring settings once and apply them to all Oracle Application Server targets. A Monitoring template defines all Grid Control parameters you would normally set to monitor an Oracle Application Server target, such as:

• Target type to which the template applies

• Metrics (including user-defined metrics), thresholds, metric collection schedules, and corrective actions

When a change is made to a template, you can reapply the template across affected Oracle Application Server targets in order to propagate the new changes. You can reapply monitoring templates as often as needed.

Viewing Topology

Besides the monitoring features provided for Oracle Application Servers, OracleAS Farms, and OracleAS DCM Managed Clusters, Grid Control also provides a topology view for these targets. You can view their topology to understand what application servers and components are running on which hosts, how these components are related to each other, and how requests are routed through different layers of the deployment. This kind of visualization of the data center enterprise topology helps administrators effectively monitor, manage, and validate the enterprise architecture.

Grid Control provides three different views of topology. Each view provides the overlay of some key metrics of components including current status, number of alerts and policy violations, and CPU/memory utilization performance metrics.

• Host View shows the physical view of deployment and visually shows the relationship between hosts and various components and instances hosted by them.

• Routing Overview shows the routing view of topology and provides an end-to-end view of wired component and application flow through them. The routing includes routing from OracleAS Web Cache to Oracle HTTP Server to OC4J to DB instances.

• Routing Details also shows the protocol and port used by various components to route requests to other components in the topology.

Figure 14-8 Topology View of an OracleAS Farm

These routing views enable you to fix various topology configuration problems immediately. For example, if an OC4J instance is not accepting any user requests, the routing details view confirms if the front-ending Oracle HTTP Server instances are configured correctly to route user requests to that OC4J instance. OracleAS Web Caches, Oracle HTTP Server, or OC4J components are displayed in the same box, if they provide redundancy for each other.

For example, a set of OracleAS Web Caches are combined in the same box if all of them are routing to the same set of Oracle HTTP Server instances. Similarly, a set of OC4J instances are combined in the same box if they are routed by the same set of Oracle HTTP Server instances, and are also hosting the same set of J2EE applications. As the like-components are grouped together, the topology visual representation in routing overview or routing details view provides instant Root Cause Analysis for the service availability problems.

Monitoring Dashboard

Grid Control also provides access to Oracle System Monitoring Dashboard that helps you monitor the health of a target. However, this feature is provided only for OracleAS Farms, OracleAS DCM Managed Clusters, Oracle Application Servers, OC4J High Availability Groups, and HTTP Server High Availability Groups.

Oracle System Monitoring Dashboard presents information using intuitive icons and graphics that let you spot recent changes and quickly identify and respond to problems. You can customize the display attributes to match information requirements of managed targets, monitor status indicators for recent problems, and see new alerts that have been triggered since the dashboard was last viewed.

Figure 14-9 Oracle System Monitoring Dashboard

Managing and Creating Blackouts

Grid Control comes with a bundle of performance and health metrics that enable automated monitoring of application targets in your environment. When a metric reaches the predefined warning or critical threshold, Grid Control generates an alert and notifies the administrators.

However, there are occasions when you want to perform maintenance work on your application server targets, and not want any alerts to be generated while you are bringing them down. In this case, you can schedule a blackout and suspend monitoring of the application server targets.

Blackouts allow you to suspend any data collection activity on one or more monitored targets, thus allowing you to perform scheduled maintenance on targets. If you continue monitoring during these periods, the collected data will show trends and other monitoring information that are not the result of normal day-to-day operations. To get a more accurate, long-term picture of a target's performance, you can use blackouts to exclude these special-case situations from data analysis.

Grid Control allows you to define new blackouts; view the status of existing blackouts; and edit, stop, and delete blackouts that are not required. You will find the Black Out option in the General section on the home page of any application server target. For composite targets like OracleAS Farms, you will find this option in the Administration tab.

Managing and Creating Jobs

For OracleAS Farms, OracleAS DCM Managed Clusters, and OracleAS Clusters, Grid Control provides a job system that allows you to create jobs. You can also view details about the jobs that are scheduled, running, suspended, or the ones that have a problem. You can schedule a job directly from these composite targets to run an OS command on the members or start, stop, or restart any OPMN component.

Creating Services from J2EE or Web Applications

Grid Control allows you to create services directly from a J2EE or Web application to help you track their availability and performance. Understandably, the business applications running on an application server simply reflect the quality of service rendered by an organization. Considering their criticality and complexity, it becomes imperative for any organization to ensure that they are always available and capable of servicing the requests within an acceptable turnaround time.

By creating services from a J2EE or Web application, you can group your applications as an entity that provides a useful function to its users. This helps you monitor the overall performance of the service that represents a function offered to the users, and also monitor the infrastructure components upon which the service depends. You can also receive alerts when there is a problem, identify common issues within the system, diagnose causes of failures, and resolve them.

Figure 14-10 Creating Service from J2EE or Web Applications

Every time you create a service for J2EE or Web application, Grid Control creates a service dependency to identify the components on which the selected application depends. Here, the components refer to infrastructure components like hosts, databases, application servers, and so on that work together to host your J2EE or Web applications. After identifying the components required for that application, Grid Control creates a system with those components, and then creates a service and associates it with that application.

So, every time you create a service, Grid Control checks to see if a dependency mapping is already available for the selected application. If it is available, then a service is created using that existing system rather than creating another system with exactly the same members.

According to the availability of this dependency mapping for an application, you can either create a new service with a new system, create a new service with an existing system, or refresh a service.

Creating Infrastructure and Aggregate Services

Grid Control allows you to create infrastructure services for Oracle BPEL Process Managers.

An infrastructure service is a dependency service that is created to identify the infrastructure components on which the selected Oracle BPEL Process Manager depends. Here, the infrastructure components refer to hosts, databases, application servers, and so on that work together to host the Oracle BPEL Process Manager.

You can either create an infrastructure service with a new system or an existing system, or simply refresh an existing infrastructure service, if there is already one existing. By creating infrastructure services and systems, you can better manage your BPEL targets and also the components on which the BPEL targets depend.

Figure 14-11 Creating Infrastructure Service for Oracle BPEL Process Manager

Once you create an infrastructure service for an Oracle BPEL Process Manager target, Grid Control allows you to create an aggregate service for every process within that Oracle BPEL target. An aggregate service is a logical grouping of services, in this case, infrastructure services and availability services. An availability service is a service that is created when a SOAP (Simple Object Access Protocol) test is added for the first time to the partner links associated with the selected Oracle BPEL processes. Aggregate Services give you a bird's-eye view of the services that have been created for the Oracle BPEL target and helps you monitor their availability, performance, and usage.

Figure 14-12 Creating Aggregate Service for Oracle BPEL Process

Adding SOAP Tests for Partner Links

Another feature that is exclusive to Oracle BPEL Process Manager is the SOAP Test feature. Grid Control allows you to add SOAP tests for partner links associated with a selected Oracle BPEL process in order to test their availability.

SOAP is an XML-based message protocol used by Web Services of these partner links. Each SOAP message consists of a single SOAP envelope. An envelope defines how to process the message, who should process the message, and whether processing is optional or mandatory.

Grid Control allows you to select a particular method specific to a port type exposed by the Web service, and provide values for the input parameters that it should use. The SOAP test invokes the selected method and checks if the Web service is up. This way, you can track the performance of the partner links and know when they are down.

Provisioning BPEL Processes

Grid Control provides deployment procedures for provisioning BPEL processes on selected BPEL Process Managers. The BPEL Process Provisioning deployment procedure comes with the following predefined phases:

• Stage Setup: This phase sets up the BPEL processes and other dependencies. Essentially, it sets up the BPEL suitcase files, stages them to a temporary location for deployment, and stages the deploy.jar on the target hosts.

• Deploy: This phase deploys the BPEL processes on the selected target BPEL Process Managers.

• Clean Setup: This phase cleans up the files and folders staged on the individual hosts.

The BPEL process suitcase files have to be placed as generic components in Oracle Software Library, and once it is available for deployment, the deployment procedure can be run to deploy the processes on BPEL Process Managers.

Creating Log Rotation Jobs

Oracle Application Server components generate log files containing messages that record all types of events, including startup and shutdown information, errors, warning messages, access information on HTTP requests, and additional information.

However, the information recorded in log files is voluminous, thus making it difficult to track what update was made at what time. Also because of the huge quantity of information updated periodically, the log files grow in size and occupy more space on the system over a period of time. The only way to manage these log files is to manually archive the contents to another file and store them in a different location.

Considering these impediments, Grid Control has been enhanced with a log rotation feature that helps you manage the logs of Oracle Application Server components more effectively. In particular, you can use Grid Control to:

• Schedule a job that automatically rotates a log at the scheduled date and time

• Manage space on your system by storing the rotated log files in a different directory

Grid Control allows you to view the logs of a particular Oracle Application Server component type and select the ones that need to be rotated. Note that a log rotation job can also be part of a multi-task job.

When a log rotation job is executed, Grid Control automatically stops the component whose logs have to be rotated. After it is stopped, the content from its existing log file is moved to another file that is distinguished with the timestamp when it was actually rotated. The original log file is kept empty for new log details to be populated. Once this is done, Grid Control restarts the component.

Managing Configurations

Grid Control provides a suite of configuration management functions that can be performed on Oracle Application Server and its components (OracleAS Web Cache, Oracle HTTP Server, OC4J, and BPEL Process Manager).

Oracle Management Agent collects configuration information about Oracle Application Server targets from their respective configuration files, and communicates this information over HTTP/HTTPS to Oracle Management Service, which stores it in the Management Repository. This information is periodically collected and updated while maintaining the audit of changes. Enterprise Manager's configuration management capabilities efficiently guide the users to desired configuration data in a particular component.

In addition to collecting and tracking hardware and software installations (binaries version number/patch level) of Oracle Application Server targets, Grid Control also tracks configuration details of core components (OracleAS Web Cache, Oracle HTTP Server, OC4J, and BPEL Process Manager) of all Oracle Application Server instances. You can compare these configuration details and view the differences and similarities between the core components. You have the flexibility to compare two configurations in the Management Repository or two saved configuration files. You can also compare one configuration with multiple configurations or one configuration in the Management Repository with a saved configuration file.

Using Grid Control, you can search configurations across application servers and find configuration anomalies - whether they are a mismatch of an install/patch version of Oracle Application Server software, or they are a mismatch of software configuration data for the core components of Oracle Application Server. You can perform more intelligent searches to identify all the components hosting a particular application or other resources.

You can also perform some out-of-box searches. The Administration page in Grid Control provided for Oracle Application Server targets allows you to search one or more:

• Origin servers

• Application servers with particular installation settings

• Data sources used by the applications deployed across your enterprise configuration

• J2EE applications deployed in a particular OC4J instance, application server instance, or host

• Modules of J2EE applications deployed across your enterprise configuration

• Application server ports across your enterprise topology

In addition, for BPEL Process Manager targets, you can view the BPEL Processes, its different versions, and the suitcase files associated with each version. You can also compare the BPEL Process suitcase files of different versions and track the changes that were made to a version. shows you how the versions can be selected and compared. This allows you to identify the cause for improved or deteriorated performance due to a change in the BPEL Process suitcase file.

Figure 14-13 BPEL Process Suitcase File Comparison

Extensible Monitoring

Many administrators often require custom logic to be written to check for conditions specific to their application environments. Grid Control allows integration of application instrumentation in Grid Control's event monitoring infrastructure. If application developers expose application instrumentation using standards like JMX or Web Services operations, then you can build management plug-ins for the instrumentation using easy-to-use command line tools, and leverage Grid Control's event monitoring system to monitor it. You do not have to edit any XM files or write any integration code to integrate such instrumentation in Grid Control. Follow these procedures to integrate application-defined instrumentation in Grid Control:

• Use Command Line Interfaces that analyze MBean interfaces for JMX and WSDL for Web Services and create management plug-ins

• Import Management Plug-in Archive in Grid Control

• Deploy Management Plug-in to Management Agents

• Create Target-type instances for the target types defined in Management Plug-in Archive

• Leverage Grid Control's event monitoring system including monitoring templates, corrective actions, historical and real time metric views, alerts, customization of notification rules, and methods on events generated from application instrumentation metrics

Diagnosing Performance Issues with Top Reports

When you are troubleshooting performance problems, it can be helpful to know which servlets or JSPs are the most active. By using the Top Servlets or Top JSPs performance links of the Application Server Performance page (Figure 14-14 and Figure 14-15) in Grid Control, you can identify the top Java servlets or JSPs running on the application server instance. You can then sort them to identify the servlets and JSPs by number of requests, maximum processing time, or highest average processing time.

Figure 14-14 Monitoring the Top Servlets for a J2EE Application

Figure 14-15 Monitoring the Top JSPs for a J2EE Application

Analyzing Historical Performance

As with all Grid Control diagnostics, the application server diagnostic reports can be based on current or historical data. Application server metrics are collected and stored in the Management Repository, so you can analyze the data well after the situation has changed. For example, you can use historical data and diagnostic reports to research an application performance problem that occurred days or even weeks ago.

You can even provide a customized time period for which the data should be retrieved from the Management Repository. You can customize the time period for:

• Pre-defined range of the last 24 hours, last 7 days, or last 31 days

• Customized range of any number of days, weeks, months, or years

• Any start date and end date (such that the duration is not greater than 99 years)

Managing Backup and Recovery of the Application Server Environment

Backup and recovery refers to the various strategies and procedures involved in guarding against hardware failures and data loss, and reconstructing data should there be a loss. A comprehensive backup strategy should involve a coordinated approach to backing up your entire application server environment, including the middle tiers and the Application Server Infrastructure Oracle homes.

Grid Control helps you manage the backup and recovery of a single application server or a group of application servers.

Using Grid Control, you can:

• Schedule backups

• Restore application server backups for recovery

• Display the status of backup jobs

• Display the status of recovery jobs

• Configure the required settings for backup

For more information on the types of backups and recommended strategy for performing backup and recovery, see the Oracle Application Server Administrator's Guide.

Cloning and Patching the Application Server Environment

Using Grid Control's automated provisioning tools, you can ensure standardization in your data center and also significantly reduce the time spent on these tasks. To consistently maintain standardization in the topology, it is recommended that the new instances be added through ÒcloningÓ rather than Òinstall and configureÓ.

Cloning ensures that the new instance is installed and configured exactly like other instances in the enterprise topology. Cloning is the process of copying an existing installation to a different location while preserving its configuration and deployments. Grid Control's cloning wizard automates the duplication of application server installations; specifically, the directories where the Oracle homes reside. Its "multicasting" capability also helps you create multiple clones on multiple target hosts in a single operation.

Using a direct link to Oracle MetaLink, Grid Control proactively and regularly retrieves the list of critical patches that have to be applied on Oracle Application Server installations. Grid Control also analyzes the data center environment and notifies you of patches that are applicable to their application server instances. All other patches can also be manually found in the context of a specific target. You can also automate the application of patches using robust job system infrastructure. You can apply the patches instantly or schedule the application in the maintenance window while backing out Oracle Application Server instances in the maintenance window.