Too many open files on Amazon Linux 2

Describe the bug

We are facing issue of too many open files on a newly built Amazon Linux 2 AMI.

LSOF command returns a count of ~5k when there are no containers running and around 40-50k when containers spin up. With td-agent in running state, it rises to 400-500k open files.

A lower td agent version works well on Amazon Linux 1 though, with 2-5k open files

To Reproduce

Create a Amazon Linux 2 instance and download latest version of td-agent - 4.4.2 Keep some conf files in /etc/td-agent/ folder and run the following

lsof | wc -l

lsof | awk '{ print $1 " " $2; }' | sort -rn | uniq -c | sort -rn | head -15

Expected behavior

Too many open files message would be shown

lsof | awk '{ print $1 " " $2; }' | sort -rn | uniq -c | sort -rn | head -15

67562 event_loo 15244 52503 flush_thr 15244 52503 enqueue_t 15244 984 dockerd 4312 552 auditbeat 4191 492 agent 4876 407 ruby 15244 324 container 4763 319 container 3915 235 socket_ma 15239 235 signal_th 15239 235 fluentd 15239 234 gssproxy 3417 209 ssm-agent 4568 209 amazon-ss 4132

lsof |wc -l 179205

Your Environment

- Fluentd version: 1.15.3
- TD Agent version: 4.4.2
- Operating system: Amazon Linux 2
- Kernel version: 4.14.301-224.520.amzn2.x86_64

Your Configuration

<source>
  @type tail
  format json
  pos_file /tmp/hcs.log
  path /var/log/app/hcs/hcs_api.log
  tag hcs.log
</source>

<match hcs.log>
 @type kafka_buffered
 output_include_time true
 output_include_tag true

 brokers  <brokers_here1,2,.....,n>
 flush_interval 3s

 default_topic kafka-HelpCenterService
 output_data_type json
        buffer_type file
        buffer_path /var/log/td-agent/buffer/hcs
        buffer_chunk_limit 100m
        buffer_queue_limit 256
</match>

Your Error Log

Too many open files message would be shown

lsof | awk '{ print $1 " " $2; }' | sort -rn | uniq -c | sort -rn | head -15

  67562 event_loo 15244
  52503 flush_thr 15244
  52503 enqueue_t 15244
    984 dockerd 4312
    552 auditbeat 4191
    492 agent 4876
    407 ruby 15244
    324 container 4763
    319 container 3915
    235 socket_ma 15239
    235 signal_th 15239
    235 fluentd 15239
    234 gssproxy 3417
    209 ssm-agent 4568
    209 amazon-ss 4132

lsof |wc -l
179205

Additional context

No response