适用于 Fluentd 的 Amazon Kinesis 输出插件
A fluentd output plugin to send events to [Amazon Kinesis Data Streams][streams] and [Amazon Data Firehose][firehose]. The plugin also supports [KPL Aggregated Record Format][kpl].
This gem includes following three output plugins:
kinesis_streamskinesis_firehosekinesis_streams_aggregatedThe plugin is also described in [official Fluentd document][fluentd-doc-kinesis].
Note: This README is for the latest v3. Plugin v3 is almost compatible with v2. If you use v1, see [v1 README][v1-readme].
Simply use RubyGems:
$ gem install fluent-plugin-kinesis --no-documentIf you would like to build by yourself and install, you can build and install as follows:
$ git clone https://github.com/awslabs/aws-fluent-plugin-kinesis.git
$ cd aws-fluent-plugin-kinesis
$ bundle install
$ bundle exec rake build
$ bundle exec rake installIf using td-agent v4 or lower (EOL), use td-agent-gem:
$ td-agent-gem install fluent-plugin-kinesisIf using fluent-package v5 or higher, use fluent-gem:
$ fluent-gem install fluent-plugin-kinesis| fluent-plugin-kinesis | fluentd | ruby |
|---|---|---|
| >= 3.7.0 | >= 0.14.22 | >= 2.7.1 |
| >= 3.6.0 && = 0.14.22 | >= 2.7.1 | |
| >= 3.5.0 && = 0.14.22 | >= 2.4.2 | |
| >= 3.2.0 && = 0.14.22 | >= 2.3 | |
| >= 3.0.0 && = 0.14.10 | >= 2.1 | |
| >= 2.0.0 && = 0.12.35 | >= 2.1 | |
| = 0.10.58 | >= 2.0 |
When you run this plugin on Amazon EC2 instances or container services, use instance profiles to assume role. If you want to use specific credentials, see Credentials.
In your Fluentd configuration, use @type kinesis_streams. The configuration would look like this:
@type kinesis_streams
region us-east-1
stream_name YOUR_STREAM
partition_key key # Otherwise, use random partition keyFor more details, see Configuration: kinesis_streams.
In your Fluentd configuration, use @type kinesis_firehose. The configuration would look like this:
@type kinesis_firehose
region us-east-1
delivery_stream_name YOUR_STREAMFor more details, see Configuration: kinesis_firehose.
In your Fluentd configuration, use @type kinesis_streams_aggregated. The configuration would look like this:
@type kinesis_streams_aggregated
region us-east-1
stream_name YOUR_STREAM
# Unlike kinesis_streams, there is no way to use dynamic partition key.
# fixed_partition_key or random.For more details, see Configuration: kinesis_streams_aggregated.
The following parameters are kinesis_streams specific configurations.
Name of the stream to put data.
As of Fluentd v1, you can use placeholders for this stream_name parameter. Note that chunk keys are required in your buffer section attributes for placeholders to work.
The following configuration shows kinesis_streams output plugin that applies extract_placeholders on stream_name:
# chunk_key: tag
# ${tag} will be replaced with actual tag string
@type kinesis_streams
stream_name ${tag}-stream
# ...The value of timekey in buffer chunk keys can be extracted using strptime placeholders like this:
# chunk_key: tag and time
@type kinesis_streams
stream_name ${tag}-stream-%Y%m%d
# ...You can also use custom placeholder like this:
# chunk_key: $.kubernetes.annotations.kinesis_stream
@type kinesis_streams
stream_name "${$.kubernetes.annotations.kinesis_stream}"
# ...For more details, see [Placeholders in Config: Buffer Section][fluentd-buffer-section-placeholders].
A key to extract partition key from JSON object. Default nil, which means partition key will be generated randomly.
The following parameters are kinesis_firehose specific configurations.
Name of the delivery stream to put data.
As of Fluentd v1, placeholders are supported. For more details, see stream_name for kinesis_streams plugin and [Placeholders in Config: Buffer Section][fluentd-buffer-section-placeholders].
Boolean. Default true. If it is enabled, the plugin adds new line character (\n) to each serialized record.
Before appending \n, plugin calls chomp and removes separator from the end of each record as chomp_record is true. Therefore, you don't need to enable chomp_record option when you use kinesis_firehose output with default configuration (append_new_line is true). If you want to set append_new_line false, you can choose chomp_record false (default) or true (compatible format with plugin v2).
Integer. Default 0 (no aggregation). When set to a value greater than 0, the plugin concatenates multiple small records into a single larger record up to the specified size (in bytes) before sending to Firehose via PutRecordBatch. This can reduce costs when using Direct PUT ingestion, which applies a 5 KB per-record billing roundup. The value cannot exceed max_record_size (default 1 MB). This is similar to the aggregatedRecordSizeBytes configuration in Kinesis Agent.
The following parameters are kinesis_streams_aggregated specific configurations.
Name of the stream to put data.
As of Fluentd v1, placeholders are supported. For more details, see stream_name for kinesis_streams plugin and [Placeholders in Config: Buffer Section][fluentd-buffer-section-placeholders].
A value of fixed partition key. Default nil, which means partition key will be generated randomly. Note that all records will go to single shard if you specify this option.
To put records into Amazon Kinesis Data Streams or Amazon Data Firehose, you need to provide AWS security credentials. Without specifying credentials in config file, this plugin automatically fetches credentials just following AWS SDK for Ruby does (environment variable, shared profile, or instance profile).
This plugin uses almost same configurations as [fluent-plugin-s3][fluent-plugin-s3-credentials], but also supports several additional configurations like aws_ses_token for temporary credentials.
These parameters are required when your agent is not running on EC2 instance with an IAM Role. When using an IAM role, make sure to configure instance_profile_credentials. Usage can be found below.
AWS access key id.
AWS secret key.
AWS session token. This parameter is optional, but can be provided if using MFA or temporary credentials when your agent is not running on EC2 instance with an IAM Role.
The number of attempts to make (with exponential backoff) when loading instance profile credentials from the EC2 metadata service using an IAM role. Defaults to 5 retries.
Typically, you can use AssumeRole for cross-account access or federation.
@type kinesis_streams
role_arn ROLE_ARN
role_session_name ROLE_SESSION_NAMESee also:
The Amazon Resource Name (ARN) of the role to assume.
An identifier for the assumed role session.
An IAM policy in JSON format.
The duration, in seconds, of the role session. The value can range from 900 seconds (15 minutes) to 3600 seconds (1 hour). By default, the value is set to 3600 seconds.
A unique identifier that is used by third parties when assuming roles in their customers' accounts.
Proxy url for proxying requests to amazon sts service api. This needs to be set up independently from global http_proxy parameter for the use case in which requests to kinesis api are going via kinesis vpc endpoint but requests to sts api have to go via http proxy. It should be added to assume_role_credentials section in the next format:
sts_http_proxy http://[username:password]@hostname:portSTS API endpoint url. This can be used to override the default global STS API endpoint of sts.amazonaws.com. Using regional endpoints may be preferred to reduce latency, and are required if utilizing a PrivateLink VPC Endpoint for STS API calls.
Similar to the assume_role_credentials, but for usage in EKS.
@type kinesis_streams
role_arn ROLE_ARN
role_session_name ROLE_SESSION_NAME
web_identity_token_file AWS_WEB_IDENTITY_TOKEN_FILESee also:
The Amazon Resource Name (ARN) of the role to assume.
An identifier for the assumed role session.
The absolute path to the file on disk containing the OIDC token.
An IAM policy in JSON format.
The duration, in seconds, of the role session. The value can range from 900 seconds (15 minutes) to 43200 seconds (12 hours). By default, the value is set to 3600 seconds (1 hour).
Retrieve t
暂无开放 Issues,或尚未同步最近议题。