GC log configuration and tools for analyse

Garbage Collection Monitoring 

The Garbage Collection Monitoring refers to the process of figuring out how JVM is running Garbage Collection (GC). The GC log is a highly important tool for revealing potential improvements to the heap and GC configuration or the object allocation pattern of the application. For each GC happening, the GC log provides exact data about its results and duration.

How to enable gc logs in wildfly application server

After this quick overview of GC, let's move on to the practical part about the configuration for wildfly.

Edit the file in $WILDFLY_HOME/bin/standalone.conf and append the following line:

JAVA_OPTS="$JAVA_OPTS -XX:+PrintGC -Xloggc:<your_custom_path>/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps"

Where:

• -XX:+PrintGC activates the “simple” GC logging mode.
• -Xloggc sets the path and the name of gc log file.
• -XX:+PrintGCTimeStamps and -XX:+PrintGCDateStamps add time and date.

Similarly, in JBoss AS edit the file $JBOSS_HOME/bin/standalone.conf.

Rolling GC Logs

As of Oracle Java 1.6_34 (or 1.7_2 in the latest minor version or 1.8_x), the GC logs can be automatically rolled at a certain size and retain only a certain number of logs.
Add this second line immediately after the previous one (into standalone.conf file):

JAVA_OPTS="$JAVA_OPTS -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=<numberOfFiles> -XX:GCLogFileSize=<size>" 

Where 

• <numberOfFiles> is just an integer. 
• <size> is the size of the file (e.g 16K is 16 kilobytes, 128M is 128 megabytes, etc.). Rolled files are appened with .<number>, where earlier numbered files are the older files. 

Example

Set gc log file size to 16 kilobytes and number of file to 3:

JAVA_OPTS="$JAVA_OPTS -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=16K -XX:GCLogFileSize=3" 

[Trabolshouting] GC log rotation data lose on application restart

Issue (JDK-6950794):

After a restart, the first log file gc.log.0 is overwritten and the data of that file is not rolled to gc.log.1 and hence lost. 

Possible solutions:

1) Add the timestamp or date in filename of gc log with following java option:

DATE=`date +%Y-%m-%d-%H-%M`

JAVA_OPTS="$JAVA_OPTS -Xloggc:<your_custom_path>/gc-$DATE.log

 or

JAVA_OPTS="$JAVA_OPTS -Xloggc:<your_custom_path>/gc-%t.log

2) Add pid of wildfly in filename

JAVA_OPTS="$JAVA_OPTS -Xloggc:<your_custom_path>/gc-%p.log

Tools for analyse the gc log files

After the gc log configuration and after and after having collected enough data into gc log during the execution of the application, you are ready to analyse these log files. There are lot a tools for this analysis. Some of these are the following:

GCeasy is an online tool for analyse the gc log, indeed, no need to register, dow
nload & install. Just upload GC log file & review the report instantly. It detects memory leaks, GC pauses, Consecutive Full GCs and several such problems. It provides a tool report very sophisticaed and precise metrics on the G1 GC logs. In addition it parses all formats of Android GC logs including dalvikvm GC Log, Android RunTime (ART) GC Log. For more details about GCeasy see the official web page: http://gceasy.io/

Scalyr is cloud based and it aggregates all of your operational data and provides a ric
h suite of analysis tools to bring that data to life. It analyzies terabytes of data per day and provide instant, real-time parsing to bring structure (and beauty) to otherwise chaotic logs and metrics.
Some her featureas are:

• Collect everything - Web access logs, system logs, application logs; System, process, and custom server metrics; External probes, server pings, and more - all in one place.
• Lightning-fast search - Search hundreds of GBs/sec across all your servers. Most searches take less than a second. Learn more about how we search so fast.
• Manage log volume - Choose which logs are collected; subsample or filter noisy or sensitive logs before they leave your servers. Archive logs to Amazon S3 for long-term storage.
• Live Tail - Watch new events arrive in near real-time. Apply filters and see only the messages that you care about.

For more details about Scalyr see official web page: https://www.scalyr.com/.

GCViewer is a free open source tool to visualize data produced by the Java VM options -verbose:gc and -Xloggc:<file>. It also calculates garbage collection related performance metrics (throughput, accumulated pauses, longest pause, etc.).
This can be very useful when tuning the garbage collection of a particular application by changing generation sizes or setting the initial heap size.
You can also export the data in CSV (comma separated values) format, which may easily be imported into spreadsheet applications for further processing.
The sources and binaries are available on github in https://github.com/chewiebug/GCViewer. The origial version (http://www.tagtraum.com/gcviewer.html) was used for JDK 1.5, the download link is http://www.tagtraum.com/gcviewer-download.html.

Internal Links

May be of interest to you:

1. Wildfly getting started

External Links

1. Wildfly official page
2. GCEasy
3. SCALYR
4. GCViewer (github)