百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
A

aws-fluent-plugin-kinesis

> 编程语言
开源

适用于 Fluentd 的 Amazon Kinesis 输出插件

289 stars0 点赞3 次浏览
访问官网GitHub

工具介绍

适用于 Fluentd 的 Amazon Kinesis 输出插件

Fluent plugin for Amazon Kinesis

A fluentd output plugin to send events to [Amazon Kinesis Data Streams][streams] and [Amazon Data Firehose][firehose]. The plugin also supports [KPL Aggregated Record Format][kpl].

This gem includes following three output plugins:

  • kinesis_streams
  • kinesis_firehose
  • kinesis_streams_aggregated

The plugin is also described in [official Fluentd document][fluentd-doc-kinesis].

Note: This README is for the latest v3. Plugin v3 is almost compatible with v2. If you use v1, see [v1 README][v1-readme].

Installation

Simply use RubyGems:

$ gem install fluent-plugin-kinesis --no-document

If you would like to build by yourself and install, you can build and install as follows:

$ git clone https://github.com/awslabs/aws-fluent-plugin-kinesis.git
$ cd aws-fluent-plugin-kinesis
$ bundle install
$ bundle exec rake build
$ bundle exec rake install

If using td-agent v4 or lower (EOL), use td-agent-gem:

$ td-agent-gem install fluent-plugin-kinesis

If using fluent-package v5 or higher, use fluent-gem:

$ fluent-gem install fluent-plugin-kinesis

Requirements

fluent-plugin-kinesis fluentd ruby
>= 3.7.0 >= 0.14.22 >= 2.7.1
>= 3.6.0 && = 0.14.22 >= 2.7.1
>= 3.5.0 && = 0.14.22 >= 2.4.2
>= 3.2.0 && = 0.14.22 >= 2.3
>= 3.0.0 && = 0.14.10 >= 2.1
>= 2.0.0 && = 0.12.35 >= 2.1
= 0.10.58 >= 2.0

Getting Started

When you run this plugin on Amazon EC2 instances or container services, use instance profiles to assume role. If you want to use specific credentials, see Credentials.

kinesis_streams

In your Fluentd configuration, use @type kinesis_streams. The configuration would look like this:

  @type kinesis_streams
  region us-east-1
  stream_name YOUR_STREAM
  partition_key key  # Otherwise, use random partition key

For more details, see Configuration: kinesis_streams.

kinesis_firehose

In your Fluentd configuration, use @type kinesis_firehose. The configuration would look like this:

  @type kinesis_firehose
  region us-east-1
  delivery_stream_name YOUR_STREAM

For more details, see Configuration: kinesis_firehose.

kinesis_streams_aggregated

In your Fluentd configuration, use @type kinesis_streams_aggregated. The configuration would look like this:

  @type kinesis_streams_aggregated
  region us-east-1
  stream_name YOUR_STREAM
  # Unlike kinesis_streams, there is no way to use dynamic partition key.
  # fixed_partition_key or random.

For more details, see Configuration: kinesis_streams_aggregated.

Configuration

  • Configuration: Plugin
    • Configuration: kinesis_streams
    • Configuration: kinesis_firehose
    • Configuration: kinesis_streams_aggregated
  • Configuration: Credentials
    • AWS key and secret authentication
    • \ section
    • \ section
    • \ section
    • \ section
    • \ section
  • Configuration: Performance
    • \ section
    • Configuration: Batch Request
  • Configuration: Format
    • \ section
    • \ section
  • Configuration: AWS SDK

Configuration: Plugin

Configuration: kinesis_streams

The following parameters are kinesis_streams specific configurations.

stream_name

Name of the stream to put data.

As of Fluentd v1, you can use placeholders for this stream_name parameter. Note that chunk keys are required in your buffer section attributes for placeholders to work.

The following configuration shows kinesis_streams output plugin that applies extract_placeholders on stream_name:

# chunk_key: tag
# ${tag} will be replaced with actual tag string

  @type kinesis_streams
  stream_name ${tag}-stream

  
    # ...

The value of timekey in buffer chunk keys can be extracted using strptime placeholders like this:

# chunk_key: tag and time

  @type kinesis_streams
  stream_name ${tag}-stream-%Y%m%d

  
    # ...

You can also use custom placeholder like this:

# chunk_key: $.kubernetes.annotations.kinesis_stream

  @type kinesis_streams
  stream_name "${$.kubernetes.annotations.kinesis_stream}"

  
    # ...

For more details, see [Placeholders in Config: Buffer Section][fluentd-buffer-section-placeholders].

partition_key

A key to extract partition key from JSON object. Default nil, which means partition key will be generated randomly.

Configuration: kinesis_firehose

The following parameters are kinesis_firehose specific configurations.

delivery_stream_name

Name of the delivery stream to put data.

As of Fluentd v1, placeholders are supported. For more details, see stream_name for kinesis_streams plugin and [Placeholders in Config: Buffer Section][fluentd-buffer-section-placeholders].

append_new_line

Boolean. Default true. If it is enabled, the plugin adds new line character (\n) to each serialized record.
Before appending \n, plugin calls chomp and removes separator from the end of each record as chomp_record is true. Therefore, you don't need to enable chomp_record option when you use kinesis_firehose output with default configuration (append_new_line is true). If you want to set append_new_line false, you can choose chomp_record false (default) or true (compatible format with plugin v2).

aggregated_record_size

Integer. Default 0 (no aggregation). When set to a value greater than 0, the plugin concatenates multiple small records into a single larger record up to the specified size (in bytes) before sending to Firehose via PutRecordBatch. This can reduce costs when using Direct PUT ingestion, which applies a 5 KB per-record billing roundup. The value cannot exceed max_record_size (default 1 MB). This is similar to the aggregatedRecordSizeBytes configuration in Kinesis Agent.

Configuration: kinesis_streams_aggregated

The following parameters are kinesis_streams_aggregated specific configurations.

stream_name (kinesis_streams_aggregated)

Name of the stream to put data.

As of Fluentd v1, placeholders are supported. For more details, see stream_name for kinesis_streams plugin and [Placeholders in Config: Buffer Section][fluentd-buffer-section-placeholders].

fixed_partition_key

A value of fixed partition key. Default nil, which means partition key will be generated randomly. Note that all records will go to single shard if you specify this option.

Configuration: Credentials

To put records into Amazon Kinesis Data Streams or Amazon Data Firehose, you need to provide AWS security credentials. Without specifying credentials in config file, this plugin automatically fetches credentials just following AWS SDK for Ruby does (environment variable, shared profile, or instance profile).

This plugin uses almost same configurations as [fluent-plugin-s3][fluent-plugin-s3-credentials], but also supports several additional configurations like aws_ses_token for temporary credentials.

AWS key and secret authentication

These parameters are required when your agent is not running on EC2 instance with an IAM Role. When using an IAM role, make sure to configure instance_profile_credentials. Usage can be found below.

aws_key_id (required)

AWS access key id.

aws_sec_key (required)

AWS secret key.

aws_ses_token

AWS session token. This parameter is optional, but can be provided if using MFA or temporary credentials when your agent is not running on EC2 instance with an IAM Role.

aws_iam_retries

The number of attempts to make (with exponential backoff) when loading instance profile credentials from the EC2 metadata service using an IAM role. Defaults to 5 retries.

\ section

Typically, you can use AssumeRole for cross-account access or federation.

  @type kinesis_streams

  
    role_arn          ROLE_ARN
    role_session_name ROLE_SESSION_NAME

See also:

  • Using IAM Roles - AWS Identity and Access Management
  • Aws::STS::Client
  • Aws::AssumeRoleCredentials

role_arn (required)

The Amazon Resource Name (ARN) of the role to assume.

role_session_name (required)

An identifier for the assumed role session.

policy

An IAM policy in JSON format.

duration_seconds

The duration, in seconds, of the role session. The value can range from 900 seconds (15 minutes) to 3600 seconds (1 hour). By default, the value is set to 3600 seconds.

external_id

A unique identifier that is used by third parties when assuming roles in their customers' accounts.

sts_http_proxy

Proxy url for proxying requests to amazon sts service api. This needs to be set up independently from global http_proxy parameter for the use case in which requests to kinesis api are going via kinesis vpc endpoint but requests to sts api have to go via http proxy. It should be added to assume_role_credentials section in the next format:

sts_http_proxy http://[username:password]@hostname:port

sts_endpoint_url

STS API endpoint url. This can be used to override the default global STS API endpoint of sts.amazonaws.com. Using regional endpoints may be preferred to reduce latency, and are required if utilizing a PrivateLink VPC Endpoint for STS API calls.

\ section

Similar to the assume_role_credentials, but for usage in EKS.

  @type kinesis_streams

  
    role_arn          ROLE_ARN
    role_session_name ROLE_SESSION_NAME
    web_identity_token_file AWS_WEB_IDENTITY_TOKEN_FILE

See also:

  • Using IAM Roles - AWS Identity and Access Management
  • IAM Roles For Service Accounts
  • Aws::STS::Client
  • Aws::AssumeRoleWebIdentityCredentials

role_arn (required)

The Amazon Resource Name (ARN) of the role to assume.

role_session_name (required)

An identifier for the assumed role session.

web_identity_token_file (required)

The absolute path to the file on disk containing the OIDC token.

policy

An IAM policy in JSON format.

duration_seconds

The duration, in seconds, of the role session. The value can range from 900 seconds (15 minutes) to 43200 seconds (12 hours). By default, the value is set to 3600 seconds (1 hour).

\ section

Retrieve t

Issues· 2 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Rubyamazon-kinesisfluent-pluginfluentdkinesis-firehose

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月18日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言