Chapter 26. Healthcheck Monitoring and Application Performance

Table of Contents

Introduction
Healthcheck Monitors Web Components
Healthcheck Error Reporting
Healthcheck Tests Performed
Optimizing Performance
Performance Hogs
Cache Tuning
Database Optimization
Indexing
Conclusion
Contributors

Note

If you find this EDG documentation helpful please consider DONATING! to keep the doc alive and current.

 Maintainer:David Lloyd

Now that you have your application developed and deployed, what tools are available for you to monitor and maintain your application? You want to ensure that the individual components of Expresso and your custom components are operating correctly, to ensure that your applications stays operational and available to users, and to ensure that it's performance is acceptable and does not degrade over time. Tools exist within Expresso to verify it's own functioning and the function of any other web application. Expresso Health Check utility provides periodic system and performance tests as means to monitor the various components of a web and/or intranet and to ensure that they remain up and operational at all times. It is designed to be run by a system scheduler.

Introduction

Expresso Health Check utility provides a means to monitor the various components of a web and/or intranet and to ensure that they remain up and operational at all times. It is designed to be run by a system scheduler (cron, ROBOT, NT Scheduler) at short intervals (e.g. once every 30 minutes).

This chapter also covers the area of application performance.

Healthcheck Monitors Web Components

Designed to be run on a timer on the server. It checks that:

  1. Web services are running

  2. The database is up

  3. The ORB is operational

  4. Expresso Framework is running correctly

  5. The ServletRunner is operational

  6. Pilot/PIP is operation (if applicable at site)

  7. Email services are operational.

  8. Performance is within acceptable range for specified urls

Tests may be defined which execute a specific URL and measure how long it takes to respond, as well as checking for a specific response. This helps ensure the sytem is operating correctly.

Healthcheck Error Reporting

All except the last item on the above list, give a warning message in email, the last one just gets a big warning on the console & a message in the log. Health Check is an essential utility for large sites when each of these services may be on a different server.

If any of these checks fails, the HealthCheck utility stops what it is doing and attempts to notify the system administrator by one of several means:

  • It logs a message into the server log (red)

  • It attempts to send an email to the SYSERROR event list of users

  • It displays a prominent message on the console (in case all other notifications fail)

Health Check can also be monitored real-time by a special System Status/Health Check Servlet web page, which automatically updates itself at a user-specified interval (e.g. every 30 seconds). It performs all of the selected health check functions, and shows a check mark for every successful check. It also shows a distinctive "warning" icon for any function that fails, along with a detailed description of the problem & an audio warning (so the system administrator is alerted even if not actually watching the page). This page can be left up on a console for a continuous monitoring of the overall system health at all times.

Healthcheck Tests Performed

HealthCheck performs the following tests:

  • Verifies that the control database is operational by connecting to it & reading setup values

  • Verifies that the Object Request Broker is operational by connecting to it

  • Verifies that the Application Framework is operational by connecting to it

  • Verifies that the SMTP email server is operational by sending a test e-mail to a fictional user

  • Verifies that the servlet runtime system is operational by running a special status servlet Using our extensibility philosophy, Health Check can be configured to check other system functions, i.e. could verify that PIP/Pilot is operational by requesting status.

To Use HealthCheck, select the "Operation" menu choice in Expresso's Admin menu. Scroll down to the heading, "Periodic System and Performance Tests." There you will see a list of functions like the below:

Table 26.1. Healthcheck Functions

HealthCheck FunctionDescription
Run Performance Tests 
Run HealthCheck 
Performance Test Definitions 
Performance Test Stats 
Performance Test Sets 
Performance Test Set Details 

Optimizing Performance

It is important to concentrate your efforts on the areas where the largest gain can be experienced - in order to do this, you must understand where the "hot spots" are in your application. Expresso can help you do this, and then help you perform more extensive analysis of those "hot spots" to make sure they are running as quickly as possible.

Performance Hogs

There are a number of things that can slow down an Expresso application. By checking this list carefully before beginning any more intensive performance tuning, you can verify that you are not "driving with the parking brake on" and can often solve performance problems immediately.

Some of the items to watch for are:

  • Logging:

    Log4j, which is the integrated logging mechanism built into Expresso, is a very capable package, and it's performance impact is very small compared to many logging methods. Still, the fastest logging is no logging at all, and careful adjustment of the expressoLogging.xml file, and the corresponding file for your custom application can result in a large performance benefit.

    The first logging issue to look for is of course to select the appropriate logging level for your running application. If you are still testing and debugging, then log priorities of "debug" for at least some objects might be appropriate, but in a production environment you should need no more than "info", and preferably only "warn". One of the easiest ways to see if you have something set to too detailed a level is to read the log file, looking for "debug" or "info" messages that you do not really need to see in production - then find out why these messages are still enabled and edit the xml configuration file accordingly.

    Another important logging performance issue is the format of the logging message: the "ConversionPattern" parameter that can be supplied when setting up a logging channel can have a significant impact on performance. During testing and debugging, it is often very useful to be able to see the exact class, method, and line number that a log message originated from. Unfortunately, during production the processing required to produce this information is very expensive, and can be the cause of a performance issue. Review the log4j documentation included with Expresso for details on how to specify less "expensive" patterns for logging entries for production systems.

  • dbTest on:

    In the property file, the dbTest property supplies an SQL statement that is used to verify if a database connection is "alive" (e.g. connected to the database and operational) before passing the connection back from the connection pool when a new connection is requested. This is helpful in situations where the connection could time out or otherwise become unavailable while the connection pool is still maintaining the connection - for example, an external data warehouse connection that is only used infrequently. In normal production, however, this option can be expensive, particularly if the query being specified is not very fast on that particular database. For example, if you specified a dbTest of "select * from USERLOGIN" you would certainly notice a sever performance decrease, as the entire USERLOGIN table would be retrieved every time a new connection was requested (which is a great many times).

    Ideally, leave out the dbTest property where it's not needed, or at least be sure that it is a fast operation for the specific database.

  • dbCache off:

    In the properties file, the dbCache property is by default "y", e.g. database object caching is enabled. It is possible to specify dbCache=n, which disables caching for database objects entirely, and as a result does not take advantage of caching as a performance boost for database accesses. Be sure you leave out "dbCache=n" for best production performance.

  • Bad Setup Values:

    For best performance, it is important to review your Setup values for correctness. For example, if you should have specified an email server that is incorrect our outside your local network, your application could be spending a lot of fruitless CPU cycles trying to attach to this server to send email notifications.

    The best way to catch such problems is to review the log, looking for any unusual errors or warnings and tracking them back to their source. Often the culprit will be a bad Setup value.

Cache Tuning

Unless you have specifically turned off the option to do so, (see the properties file documentation) Expresso as it runs will collect information about the effectiveness of the DB Object caching system. This information includes how many read operations (retrieve(), searchAndRetrieve(), find() and so forth) have been made for a particular DB object (in a given db/context), and how many of these reads were able to use the cache to find the record they were looking for. The idea of course, is that generally speaking the more often the record can be read from cache, instead of being retrieved from the database, the faster your application will run.

The "Status" servlet is used to retrieve this information, which also computes the percentage of "hits" to the database object that were supplied by the cache - what you're looking for in this listing is a large number of total hits with a low percentage (or zero percentage) of hits to the cache. This indicates a DBObject that is heavily used by read operations in your application, but which is not caching very much. The listing from status also shows you the size of the currently specified cache for this database object - if it is zero, then you may not have an entry for the object in the "Database Object Page Limits" table (accessed from the "Setup" page in Expresso).

There are a number of other factors you should consider when tuning caching for database objects:

  • Available Memory:

    Caching can use substantial amounts of memory, so you should check the amount of available memory to your application before increasing the cache sizes for objects too much. You can see a "snapshot" of current memory usage from the status servlet, but you may wish to read the log to see what memory usage has been over time. You can adjust the logging detail level for the "core.cache.CacheManager object" to see periodic memory reports. If you see a number of instances where the cache manager is having to clear caches in order to get more memory (e.g. to bring it over the specified minimum - again see the property file documentation for details), then you probably cannot increase caching much without actually slowing down the application.

    Be sure that you are launching your java virtual machine (or your applications server) with the appropriate options to make enough memory available to your application - just because you physically have enough memory in your system doesn't mean that your JVM has use of this memory. See the -X options for your Java runtime for details on how to increase this limit.

  • Cached Items:

    Some database objects may show high activity, and low cache hits, but still not be a problem. This is particularly true of items that are always cached by Expresso, such as Setup values, security entries for db objects and controllers, and a few others.

  • Write Patterns:

    The effectiveness of caching is also influenced by the patterns in which your application writes to the database - if a particular db object is changed, it's cache entry is removed (so that stale data is not read) and it must be read from the database again the next time it is accessed. If you have an object that has a lot of cache set up for it, but is still not caching very successfully, it may be that this object is being written to very frequently.

    This is more of an application design issue than something that can be directly addressed by cache tuning.

By careful manipulation of the records in the "Database Object Page Limit" table, you can make best use of your available memory via db object caching. This is often the single most important optimization step you can take.

Database Optimization

If all of the above optimization and performance tuning techniques indicate that there is a problem with database access slowing down the system (despite tuning the db cache), then it is sometimes required to analyze the database queries themselves in order to determine which queries are causing the problem.

The DBConnection object, which is responsible for executing the actual SQL queries built by DB objects, can help perform this analysis: turn on the logging priority for this object to "debug" using the expressoLogging.xml file, and monitor the results. Each SQL query will have it's elapsed time recorded, and you can use this information to pick out the worst offenders and either modify your application to perform better queries, or use database indices to make them faster.

Indexing

See the section on DBobjects for information about setting up Indexes for faster access on non-primary-key fields.

Conclusion

Contributors

The following persons have contributed their time to this chapter:

Note

Was this EDG documentation helpful? Do you wish to express your appreciation for the time expended over years developing the EDG doc? We now accept and appreciate monetary donations. Your support will keep the EDG doc alive and current. Please click the Donate button and enter ANY amount you think the EDG doc is worth. In appreciation of a $35+ donation, we'll give you a subscription service by emailing you notifications of doc updates; and donations $75+ will also receive an Expresso T-shirt. All online donation forms are SSL secured and payment can be made via Credit Card or your Paypal account. Thank you in advance.

Copyright © 2001-2004 Jcorporate Ltd. All rights reserved.