GC log configuration and tools for analyse

Garbage Collection Monitoring 

The Garbage Collection Monitoring refers to the process of figuring out how JVM is running Garbage Collection (GC). The GC log is a highly important tool for revealing potential improvements to the heap and GC configuration or the object allocation pattern of the application. For each GC happening, the GC log provides exact data about its results and duration.

How to enable gc logs in wildfly application server

After this quick overview of GC, let's move on to the practical part about the configuration for wildfly.

Edit the file in $WILDFLY_HOME/bin/standalone.conf and append the following line:

JAVA_OPTS="$JAVA_OPTS -XX:+PrintGC -Xloggc:<your_custom_path>/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps"

Where:

• -XX:+PrintGC activates the “simple” GC logging mode.
• -Xloggc sets the path and the name of gc log file.
• -XX:+PrintGCTimeStamps and -XX:+PrintGCDateStamps add time and date.

Similarly, in JBoss AS edit the file $JBOSS_HOME/bin/standalone.conf.

Rolling GC Logs

As of Oracle Java 1.6_34 (or 1.7_2 in the latest minor version or 1.8_x), the GC logs can be automatically rolled at a certain size and retain only a certain number of logs.
Add this second line immediately after the previous one (into standalone.conf file):

JAVA_OPTS="$JAVA_OPTS -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=<numberOfFiles> -XX:GCLogFileSize=<size>" 

Where 

• <numberOfFiles> is just an integer. 
• <size> is the size of the file (e.g 16K is 16 kilobytes, 128M is 128 megabytes, etc.). Rolled files are appened with .<number>, where earlier numbered files are the older files. 

Example

Set gc log file size to 16 kilobytes and number of file to 3:

JAVA_OPTS="$JAVA_OPTS -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=16K -XX:GCLogFileSize=3" 

[Trabolshouting] GC log rotation data lose on application restart

Issue (JDK-6950794):

After a restart, the first log file gc.log.0 is overwritten and the data of that file is not rolled to gc.log.1 and hence lost. 

Possible solutions:

1) Add the timestamp or date in filename of gc log with following java option:

DATE=`date +%Y-%m-%d-%H-%M`

JAVA_OPTS="$JAVA_OPTS -Xloggc:<your_custom_path>/gc-$DATE.log

 or

JAVA_OPTS="$JAVA_OPTS -Xloggc:<your_custom_path>/gc-%t.log

2) Add pid of wildfly in filename

JAVA_OPTS="$JAVA_OPTS -Xloggc:<your_custom_path>/gc-%p.log

Tools for analyse the gc log files

After the gc log configuration and after and after having collected enough data into gc log during the execution of the application, you are ready to analyse these log files. There are lot a tools for this analysis. Some of these are the following:

GCeasy is an online tool for analyse the gc log, indeed, no need to register, dow
nload & install. Just upload GC log file & review the report instantly. It detects memory leaks, GC pauses, Consecutive Full GCs and several such problems. It provides a tool report very sophisticaed and precise metrics on the G1 GC logs. In addition it parses all formats of Android GC logs including dalvikvm GC Log, Android RunTime (ART) GC Log. For more details about GCeasy see the official web page: http://gceasy.io/

Scalyr is cloud based and it aggregates all of your operational data and provides a ric
h suite of analysis tools to bring that data to life. It analyzies terabytes of data per day and provide instant, real-time parsing to bring structure (and beauty) to otherwise chaotic logs and metrics.
Some her featureas are:

• Collect everything - Web access logs, system logs, application logs; System, process, and custom server metrics; External probes, server pings, and more - all in one place.
• Lightning-fast search - Search hundreds of GBs/sec across all your servers. Most searches take less than a second. Learn more about how we search so fast.
• Manage log volume - Choose which logs are collected; subsample or filter noisy or sensitive logs before they leave your servers. Archive logs to Amazon S3 for long-term storage.
• Live Tail - Watch new events arrive in near real-time. Apply filters and see only the messages that you care about.

For more details about Scalyr see official web page: https://www.scalyr.com/.

GCViewer is a free open source tool to visualize data produced by the Java VM options -verbose:gc and -Xloggc:<file>. It also calculates garbage collection related performance metrics (throughput, accumulated pauses, longest pause, etc.).
This can be very useful when tuning the garbage collection of a particular application by changing generation sizes or setting the initial heap size.
You can also export the data in CSV (comma separated values) format, which may easily be imported into spreadsheet applications for further processing.
The sources and binaries are available on github in https://github.com/chewiebug/GCViewer. The origial version (http://www.tagtraum.com/gcviewer.html) was used for JDK 1.5, the download link is http://www.tagtraum.com/gcviewer-download.html.

Internal Links

May be of interest to you:

1. Wildfly getting started

External Links

1. Wildfly official page
2. GCEasy
3. SCALYR
4. GCViewer (github)

Wildfly getting started

Introduction

Wildfly is an application server authored by Red Hat. It is written in Java, and implements the Java Platform, Enterprise Edition (Java EE) specification. It runs on multiple platforms.

Wildfly versions earlier than 8.0.0 are available under the name of JBoss Application Server. Indeed in November 2014, JBoss Application Server was renamed WildFly.

WildFly is free and open-source software, subject to the requirements of the GNU Lesser General Public License (LGPL), version 2.1.

Installing Wildfly

In this page you can download all wildfly versions. Unzip (or untar) in a folder (that show you how to WILDFLY_HOME). Wildfly requires the Java SE 8 or later.  Now add in $WILDFLY_HOME/bin/standalone.conf your java settings, for example:

 export JAVA_HOME=/opt/jdk/1.8.0_91
 export PATH=$PATH:$JAVA_HOME/bin
 export CLASSPATH=$JAVA_HOME/lib 

So now you can starting your wildfly

 cd $WILDFLY_FOLDER/bin
 ./standalone.sh

if you want the wildfly process in background and active after your logut from the server so use the following command:

 nohup $WILDFLY_HOME/bin/standalone.sh > $WILDFLY_HOME/standalone/log/nohup.out &

Finally for stop you can kill the java process.

Wildfly Configuration

Memory Configuration

If the wildfly shows the following error during starting:

 Unrecognized VM option 'MetaspaceSize=96M'
 Could not create the Java virtual machine.

so it's needs an update of memory size, in $WILDFLY_HOME/bin/standalone.conf the default memory size is:

JAVA_OPTS="-Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true"

you increase the memory according your needs, for example:

 JAVA_OPTS="-Xms2048m -Xmx2048m -XX:MetaspaceSize=256M -XX:MaxMetaspaceSize=512m -Djava.net.preferIPv4Stack=true" 

Enable the remote Access

The Wildfly default configuration in $WILDFLY_HOME/standalone/configuration/standalone.xml file about the interfaces is the following:

<interfaces>
        <interface name="management">
            <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
        </interface>
        <interface name="public">
            <inet-address value="${jboss.bind.address:127.0.0.1}"/>
        </interface>
</interfaces>

If you want provides the remote access of your Wildfly you need the following replace the configuration in standalone.xml as below:

<interfaces> 
        <interface name="management">
            <inet-address value="${jboss.bind.address.management:0.0.0.0}"/>
        </interface>
        <interface name="public">
            <inet-address value="${jboss.bind.address:0.0.0.0}"/>
        </interface>
        <interface name="unsecure">
            <inet-address value="${jboss.bind.address.unsecure:0.0.0.0}"/>
        </interface>
</interfaces>

After the Wildfly restart, it will be accessible from remote hosts.

Oracle Data Source 

Follow the steps below to install the oracle data source:

1. Download the ojdbc driver from Oracle. 
2. Create a subfolder $WILDFLY_HOME/modules/system/layers/base/com/oracle/db/main.
3. Copy the driver (downloaded in step 1) in folder created in step 2.
4. Always in this folder, create a file with name module.xml. 
5. Edit module.xml as the example below. Note: replace <version> according the driver downloaded in step 1.

<?xml version="1.0" encoding="UTF-8"?>
<module xmlns="urn:jboss:module:1.1" name="com.oracle.db">
    <resources>
        <resource-root path="ojdbc<version>.jar"/>
        <!-- Insert resources here -->
    </resources>
    <dependencies>
        <module name="javax.api"/>
        <module  name="javax.transaction.api"/>
    </dependencies>
</module>

6. Add the oracle driver editing $WILDFLY_HOME/standalone/configuration/standalone.xml, xml tag with name <drivers>. 

<driver name="oracle" module="com.oracle.db">
        <driver-class>oracle.jdbc.driver.OracleDriver</driver-class>
</driver>

7. Always in this file, add the datasource in <datasources>. Replace the highlighted params: 

8. <datasource jndi-name="java:/[name]" pool-name="[pool-name]" enabled="true" use-java-context="true">
        <connection-url>jdbc:oracle:thin:@[hostname]:1521:[sid]</connection-url>
        <driver>oracle</driver>
        <security>
             <user-name>[your_user]</user-name>
             <password>[password]</password>
        </security>
   </datasource>

   Always in this file, add in <subsystem xmlns="urn:jboss:domain:ee:4.0">
   

   <global-modules>
        <module name="com.oracle.db" slot="main"/>
   </global-modules>

Enable the CXF logger

Add in $WILDFLY_HOME/standalone/configuration/standalone.xml file (in system-properties xml tag, if the tag is not exist, you can add it as child node of server xml tag) the following system property:

 <system-properties>  
  ....  
  <property name="org.apache.cxf.logging.enabled" value="true"/>  
  ....  
 </system-properties>  

Alternatively you can use the following setting value="pretty" for nicely formatted xml output.

 <system-properties>  
  ....  
  <property name="org.apache.cxf.logging.enabled" value="pretty"/>  
  ....  
 </system-properties>  

Also in this case it is necessary to restart.

GZip

gzip (GNU zip) is a compression utility designed to be a replacement for compress. To improve the performance of a web application you can take advantage of gzip to compress web contents as js, css, etc ...
Add in $WILDFLY_HOME/standalone/configuration/standalone.xml file (in system-properties xml tag, if the tag is not exist, you can add it as child node of server xml tag)  the following system properties:

<system-properties>
        .......
        <property name="org.apache.coyote.http11.Http11Protocol.COMPRESSION" value="on"/>
        <property name="org.apache.coyote.http11.Http11Protocol.COMPRESSION_MIME_TYPES" value="text/javascript,text/css,text/html"/>
        .......
</system-properties>

Deployment Scanner configuration

The deployment scanner is only used in standalone mode. To enable the hot deploy the deployment-scanner subsystem must be to set the auto-deploy-exploded to "true" in $WILDFLY_HOME/standalone/configuration/standalone.xml.

<subsystem xmlns="urn:jboss:domain:deployment-scanner:1.0">
    <deployment-scanner 
        scan-interval="5000"
        relative-to="jboss.server.base.dir" 
        path="deployments"
        auto-deploy-exploded="true" />
</subsystem>

Now copy the package (war, ear, rar, etc..) in $WILDFLY_HOME/standalone/deployments folder and create an empty file with <package_name>.dodeploy:

 cp my-app.war $WILDFLY_HOME/standalone/deployments
 touch my-app.war.dodeploy

When the deployment is done with success,  Wildfly will automatically create the following file:

 my-app.war.deployed

Instead for undeploy, remove the file with suffix ".deployed" and create the file with suffix .skipdeploy, if we want undeploy the package of previus example:

 rm my-app.war.deployed
 touch my-app.war.skipdeploy

When the undeployment is done with success, Wildfly will automatically create the following file:

 my-app.war.undeployed

For previous versions with name JBoss, the undeploy is different. Indeed, you mast create a file with suffix .undeploy. Example:

 rm my-app.war.deployed
touch my-app.war.undeploy

When the undeployment is done with success,  JBoss AS will automatically create the following file:

 my-app.war.undeployed

External Links

1. Wildfly official page
2. Wildfly download page
3. Oracle Drive - Download Page