fluent-plugin-elasticsearch

Fluent::Plugin::Elasticsearch, a plugin for Fluentd

Send your logs to Elasticsearch (and search them with Kibana maybe?)

Note: For Amazon Elasticsearch Service please consider using fluent-plugin-aws-elasticsearch-service

Current maintainers: Hiroshi Hatake | @cosmo0920, Kentaro Hayashi | @kenhys

• Installation
• Usage
  • Index templates
• Configuration
  • host
  • port
  • cloud_id
  • cloud_auth
  • emit_error_for_missing_id
  • hosts
  • user, password, path, scheme, ssl_verify
  • logstash_format
  • logstash_prefix
  • logstash_prefix_separator
  • logstash_dateformat
  • pipeline
  • time_key_format
  • time_precision
  • time_key
  • time_key_exclude_timestamp
  • include_timestamp
  • utc_index
  • suppress_type_name
  • target_index_key
  • target_type_key
  • target_index_affinity
  • template_name
  • template_file
  • template_overwrite
  • customize_template
  • rollover_index
  • index_date_pattern
  • deflector_alias
  • application_name
  • index_prefix
  • templates
  • max_retry_putting_template
  • fail_on_putting_template_retry_exceed
  • fail_on_detecting_es_version_retry_exceed
  • max_retry_get_es_version
  • request_timeout
  • reload_connections
  • reload_on_failure
  • resurrect_after
  • include_tag_key, tag_key
  • id_key
  • parent_key
  • routing_key
  • remove_keys
  • remove_keys_on_update
  • remove_keys_on_update_key
  • retry_tag
  • write_operation
  • time_parse_error_tag
  • reconnect_on_error
  • with_transporter_log
  • content_type
  • include_index_in_url
  • http_backend
  • http_backend_excon_nonblock
  • prefer_oj_serializer
  • compression_level
  • Client/host certificate options
  • Proxy Support
  • Buffer options
  • Hash flattening
  • Generate Hash ID
  • sniffer_class_name
  • selector_class_name
  • reload_after
  • validate_client_version
  • unrecoverable_error_types
  • verify_es version at startup
  • default_elasticsearch_version
  • custom_headers
  • api_key
  • Not seeing a config you need?
  • Dynamic configuration
  • Placeholders
  • Multi workers
  • log_es_400_reason
  • suppress_doc_wrap
  • ignore_exceptions
  • exception_backup
  • bulk_message_request_threshold
  • enable_ilm
  • ilm_policy_id
  • ilm_policy
  • ilm_policies
  • ilm_policy_overwrite
  • truncate_caches_interval
  • use_legacy_template
  • metadata section
    • include_chunk_id
    • chunk_id_key
• Configuration - Elasticsearch Input
• Configuration - Elasticsearch Filter GenID
• Configuration - Elasticsearch Output Data Stream
• Elasticsearch permissions
• Troubleshooting
• Contact
• Contributing
• Running tests

Requirements

[!IMPORTANT] You need to change the version of elasticsearch gem, according to the version of Elasticsearch you use. Please refer to the following example.

• https://docs.fluentd.org/container-deployment/docker-compose#step-1-create-fluentd-image-with-your-config--plugin

NOTE: Since fluent-plugin-elasticsearch 5.3.0, it requires faraday 2.0 or later.

NOTE: For v0.12 version, you should use 1.x.y version. Please send patch into v0.12 branch if you encountered 1.x version's bug.

NOTE: This documentation is for fluent-plugin-elasticsearch 2.x or later. For 1.x documentation, please see v0.12 branch.

NOTE: Using Index Lifecycle management(ILM) feature needs to install elasticsearch-xpack gem v7.4.0 or later.

Installation

$ gem install fluent-plugin-elasticsearch

Usage

In your Fluentd configuration, use @type elasticsearch. Additional configuration is optional, default values would look like this:

  @type elasticsearch
  host localhost
  port 9200
  index_name fluentd
  type_name fluentd

NOTE: type_name parameter will be used fixed _doc value for Elasticsearch 7.

NOTE: type_name parameter will make no effect for Elasticsearch 8.

Index templates

This plugin creates Elasticsearch indices by merely writing to them. Consider using Index Templates to gain control of what get indexed and how. See this example for a good starting point.

Configuration

host

host user-custom-host.domain # default localhost

You can specify Elasticsearch host by this parameter.

Note: Since v3.3.2, host parameter supports builtin placeholders. If you want to send events dynamically into different hosts at runtime with elasticsearch_dynamic output plugin, please consider to switch to use plain elasticsearch output plugin. In more detail for builtin placeholders, please refer to Placeholders section.

To use IPv6 address on host parameter, you can use the following styles:

string style

To use string style, you must quote IPv6 address due to prevent to be interpreted as JSON:

host "[2404:7a80:d440:3000:192a:a292:bd7f:ca10]"

raw style

You can also specify raw IPv6 address. This will be handled as [specified IPv6 address]:

host 2404:7a80:d440:3000:192a:a292:bd7f:ca10

port

port 9201 # defaults to 9200

You can specify Elasticsearch port by this parameter.

cloud_id

cloud_id test-dep:ZXVyb3BlLXdlc3QxLmdjcC5jbG91ZC5lcy5pbyRiYZTA1Ng==

You can specify Elasticsearch cloud_id by this parameter.

If you specify cloud_id option then cloud_auth option is required. If you specify cloud_id option, host, port, user and password options are ignored.

cloud_auth

cloud_auth 'elastic:slkjdaooewkd87iqQ2O8EQYV'

You can specify Elasticsearch cloud_auth by this parameter.

emit_error_for_missing_id

emit_error_for_missing_id true

When write_operation is configured to anything other then index, setting this value to true will cause the plugin to emit_error_event of any records which do not include an _id field. The default (false) behavior is to silently drop the records.

hosts

hosts host1:port1,host2:port2,host3:port3

You can specify multiple Elasticsearch hosts with separator ",".

If you specify multiple hosts, this plugin will load balance updates to Elasticsearch. This is an elasticsearch-ruby feature, the default strategy is round-robin.

If you specify hosts option, host and port options are ignored.

host user-custom-host.domain # ignored
port 9200                    # ignored
hosts host1:port1,host2:port2,host3:port3

If you specify hosts option without port, port option is used.

port 9200
hosts host1:port1,host2:port2,host3 # port3 is 9200

Note: If you will use scheme https, do not include "https://" in your hosts ie. host "https://domain", this will cause ES cluster to be unreachable and you will receive an error "Can not reach Elasticsearch cluster"

Note: Up until v2.8.5, it was allowed to embed the username/password in the URL. However, this syntax is deprecated as of v2.8.6 because it was found to cause serious connection problems (See #394). Please migrate your settings to use the user and password field (described below) instead.

IPv6 addresses

When you want to specify IPv6 addresses, you must specify schema together:

hosts http://[2404:7a80:d440:3000:de:7311:6329:2e6c]:port1,http://[2404:7a80:d440:3000:de:7311:6329:1e6c]:port2,http://[2404:7a80:d440:3000:de:6311:6329:2e6c]:port3

If you don't specify hosts with schema together, Elasticsearch plugin complains Invalid URI for them.

user, password, path, scheme, ssl_verify

user demo
password secret
path /elastic_search/
scheme https

You can specify user and password for HTTP Basic authentication.

And this plugin will escape required URL encoded characters within %{} placeholders.

user %{demo+}
password %{@secret}

Specify ssl_verify false to skip ssl verification (defaults to true)

logstash_format

logstash_format true # defaults to false

This is meant to make writing data into Elasticsearch indices compatible to what Logstash calls them. By doing this, one could take advantage of Kibana. See logstash_prefix and logstash_dateformat to customize this index name pattern. The index name will be #{logstash_prefix}-#{formatted_date}

:warning: Setting this option to true will ignore the index_name setting. The default index name prefix is logstash-.

include_timestamp

include_timestamp true # defaults to false

Adds a @timestamp field to the log, following all settings logstash_format does, except without the restrictions on index_name. This allows one to log to an alias in Elasticsearch and utilize the rollover API.

logstash_prefix

logstash_prefix mylogs # defaults to "logstash"

logstash_prefix_separator

logstash_prefix_separator _ # defaults to "-"

logstash_dateformat

The strftime format to generate index target index name when logstash_format is set to true. By default, the records are inserted into index logstash-YYYY.MM.DD. This option, alongw