logstash-logback-encoder

[!IMPORTANT] This document applies to the next version under development.

    See here for documentation on the latest released version.

Logstash Logback Encoder

Provides logback encoders, layouts, and appenders to log in JSON and other formats supported by Jackson.

Supports both regular LoggingEvents (logged through a Logger) and AccessEvents (logged via logback-access).

Originally written to support output in logstash's JSON format, but has evolved into a highly-configurable, general-purpose, structured logging mechanism for JSON and other Jackson dataformats. The structure of the output, and the data it contains, is fully configurable.

Contents:

• Including it in your project
• Java Version Requirements
• Usage
  • UDP Appenders
  • TCP Appenders
    • Keep-alive
    • Multiple Destinations
    • Reconnection Delay
    • Connection Timeout
    • Write Buffer Size
    • Write Timeout
    • Initial Send Delay
    • SSL
  • Async Appenders
    • RingBuffer Size
    • RingBuffer Full
    • Graceful Shutdown
    • Wait Strategy
  • Appender Listeners
  • Encoders / Layouts
  • Threads and ThreadLocals
• LoggingEvent Fields
  • Standard Fields
  • MDC fields
  • Key Value Pair fields
  • Context fields
  • Caller Info Fields
  • Custom Fields
    • Global Custom Fields
    • Event-specific Custom Fields
• AccessEvent Fields
  • Standard Fields
  • Header Fields
• Customizing Jackson
  • Data Format
  • Customizing TokenStreamFactory, ObjectMapper, and JsonGenerator
  • Registering Jackson Modules
  • Customizing Character Escapes
• Masking
• Customizing Standard Field Names
• Customizing Version
• Customizing Timestamp
• Customizing LoggingEvent Message
• Customizing AccessEvent Message
• Customizing Logger Name Length
• Customizing Stack Traces
  • Omit Common Frames
  • Truncate after Regex
  • Exclude Frames per Regex
  • Omit Throwable Messages
  • Maximum Depth per Throwable
  • Maximum Trace Size (bytes)
  • Classname Shortening
  • Custom Line Separator
  • Root Cause First
  • Conditional Output
  • Stack Hashes
  • Using with PatternLayout
• Registering Additional Providers
• Prefix/Suffix/Separator
• Composite Encoder/Layout
  • Providers common to LoggingEvents and AccessEvents
  • Providers for LoggingEvents
  • Providers for AccessEvents
  • Nested JSON Provider
  • Pattern JSON Provider
    • LoggingEvent patterns
    • AccessEvent patterns
  • Custom JSON Provider
• Status Listeners
• Joran/XML Configuration
  • Duration Property
  • Comma separated list of values

Including it in your project

Maven style:

<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>9.0</version>
    
</dependency>

<dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-classic</artifactId>
    <version>1.5.20</version>
    
</dependency>

If you get ClassNotFoundException/NoClassDefFoundError/NoSuchMethodError at runtime, then ensure the required dependencies (and appropriate versions) as specified in the pom file from the maven repository exist on the runtime classpath. Specifically, the following need to be available on the runtime classpath:

• jackson-databind / jackson-core / jackson-annotations >= 3.0.0
• logback-core >= 1.5.0
• logback-classic >= 1.5.0 (required for logging LoggingEvents)
• logback-access >= 2.0.0 (required for logging AccessEvents)
• slf4j-api (usually comes as a transitive dependency of logback-classic)
• java-uuid-generator (required if the uuid provider is used)

Older versions than the ones specified in the pom file might work, but the versions in the pom file are what testing has been performed against. Support for logback versions prior to 1.3.0 was removed in logstash-logback-encoder 7.4. Support for jackson versions prior to 3.0.0 was removed in logstash-logback-encoder 9.0.

If you are using logstash-logback-encoder in a project (such as spring-boot) that also declares dependencies on any of the above libraries, you might need to tell maven explicitly which versions to use to avoid conflicts. You can do so using maven's dependencyManagement feature. For example, to ensure that maven doesn't pick different versions of logback-core, logback-classic, and logback-access, add this to your project's pom.xml

…

Java Version Requirements

Usage

To log using JSON format, you must configure logback to use either:

• an appender provided by the logstash-logback-encoder library, OR
• an appender provided by logback (or another library) with an encoder or layout provided by the logstash-logback-encoder library

The appenders, encoders, and layouts provided by the logstash-logback-encoder library are as follows:

These encoders/layouts can generally be used by any logback appender (such as RollingFileAppender).

The general composite JSON encoders/layouts can be used to output any JSON format/data by configuring them with various JSON providers. The Logstash encoders/layouts are really just extensions of the general composite JSON encoders/layouts with a pre-defined set of providers.

The logstash encoders/layouts are easier to configure if you want to use the standard logstash version 1 output format. Use the composite encoders/layouts if you want to heavily customize the output, or if you need to use logstash version 0 output.

The *AsyncDisruptorAppender appenders are similar to logback's AsyncAppender, except that a LMAX Disruptor RingBuffer is used as the queuing mechanism, as opposed to a BlockingQueue. These async appenders can delegate to any other underlying logback appender.

UDP Appenders

To output JSON for LoggingEvents to a syslog/UDP channel, use the LogstashUdpSocketAppender with a LogstashLayout or LoggingEventCompositeJsonLayout in your logback.xml, like this:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appender name="stash" class="net.logstash.logback.appender.LogstashUdpSocketAppender">
        <host>MyAwesomeSyslogServer</host>
        
        <port>514</port>
        
        <layout class="net.logstash.logback.layout.LogstashLayout"/>
    </appender>
    
    <root level="all">
        <appender-ref ref="stash" />
    </root>
</configuration>

You can further customize the JSON output by customizing the layout as described in later sections.

For example, to configure global custom fields, you can specify

<appender name="stash" class="net.logstash.logback.appender.LogstashUdpSocketAppender">
    <host>MyAweso