Skip to content

Update config values#269

Open
sparrc wants to merge 1 commit into
aws-observability:mainfrom
sparrc:sparrc-config-upd
Open

Update config values#269
sparrc wants to merge 1 commit into
aws-observability:mainfrom
sparrc:sparrc-config-upd

Conversation

@sparrc

@sparrc sparrc commented Jan 23, 2026

Copy link
Copy Markdown

Description of changes:

I have made a selection of changes to the default config for the following reasons.

Overall the changes are meant to limit runaway memory usage on the fluent-bit container, by setting mem_buf_limit on all input plugins.

One other thing to consider would be to add two new config parameters: workers for the output plugins and threaded for the input plugins, which enables multi-threading on the fluent-bit process to increase efficiency and free up fluent-bit's main thread. These options, however, require that the user is using aws-for-fluent-bit version 3+. Is this an assumption we can safely make?

Change Reason
Flush 5 → 2 Logs are flushed every second instead of every 5 seconds, reducing size of output plugin buffers
storage.max_chunks_up 16 Limits memory chunks to 16 if filesystem storage is enabled
Removed storage.type filesystem from tail inputs The tail plugin already tracks file offsets via the DB option, so filesystem buffering is redundant and creates excessive disk IO. Filesystem buffering also
Mem_Buf_Limit 10MB on tail inputs These were previously using filesystem storage, so setting a mem_buf_limit to keep memory usage in check
Added Mem_Buf_Limit 10MB to systemd inputs Ensures all inputs have memory limits for consistent behavior
retry_limit 5 on all outputs Retries failed sends up to 5 times before dropping logs

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@sparrc sparrc force-pushed the sparrc-config-upd branch from c4ffaa2 to 0229998 Compare January 23, 2026 23:11
@J-Swift

J-Swift commented Apr 2, 2026

Copy link
Copy Markdown

We are seeing regular OOM kills on our fluent-bit pods and I found this PR while researching after AI basically recommended a similar set of changes. It feels off putting to need to modify this on my end if the underlying isnt configured appropriately

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants