Skip to content

in_s3: stop retrying SQS messages that cannot be decoded - #484

Draft
Watson1978 wants to merge 1 commit into
fluent:masterfrom
Watson1978:in_s3-validate-sqs-message-structure
Draft

in_s3: stop retrying SQS messages that cannot be decoded#484
Watson1978 wants to merge 1 commit into
fluent:masterfrom
Watson1978:in_s3-validate-sqs-message-structure

Conversation

@Watson1978

@Watson1978 Watson1978 commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

The problem

Posting hello to the queue is enough to stall an in_s3 source indefinitely.

is_valid_queue only checks that the top-level Records (or detail) key exists, so a body without the nested s3.object.key gets through it and raises in get_raw_key. The rescue in run turns every per-message error into :skip_delete, which tells QueuePoller to leave the message in the queue. SQS redelivers it after the visibility timeout, it fails identically, and this repeats until the retention period expires — up to 14 days of CPU, log volume and SQS requests, with a backtrace logged on every pass and nothing ever ingested. skip_delete is false by default, so no configuration is needed to reach this, and anything that can put a message on the queue can trigger it.

The change

  • Validate a message down to its object key before touching S3, and skip the ones that cannot name an object — a body that is not JSON, is not an object, is not an S3 notification, has no usable key, or has a key that decodes to invalid UTF-8. Those can never succeed on redelivery, so they are skipped and deleted instead of retried forever
  • Leave every failure after the key is known — a failed get, a corrupt archive, a parse error — on :skip_delete. The object exists there, a misconfigured s3_bucket would otherwise drain the whole backlog, and a redrive policy with a dead letter queue is the right tool for it
  • Keep the decoding inside the per-message rescue, so an unforeseen error still falls back to :skip_delete rather than escaping QueuePoller#poll and halting ingestion for retry_error_interval seconds
  • Log one warn per skipped message so a mistake stays visible: an SNS envelope, or an event_bridge_mode that does not match the queue, used to be dropped in silence and looked exactly like an idle queue
  • Route everything logged through one helper that truncates to 256 bytes and scrubs, because an object key can be megabytes long and can hold invalid UTF-8 that JSON.generate rejects when the log event record is built

Notes

process now takes the object key rather than the parsed body. Tests go from 108 to 143, and they redeliver the same message to prove a skipped one is handled exactly once while a failed fetch is still retried. Two pre-existing limitations are unchanged and left for separate changes: only Records[0] is examined, and add_object_metadata reports the URL-escaped key.

@Watson1978
Watson1978 marked this pull request as draft August 17, 2026 06:03
@Watson1978 Watson1978 changed the title in_s3: validate SQS message structure before processing in_s3: stop retrying SQS messages that can never succeed Aug 17, 2026
@Watson1978
Watson1978 force-pushed the in_s3-validate-sqs-message-structure branch 3 times, most recently from fdc9195 to 22c6239 Compare August 17, 2026 08:26
@Watson1978 Watson1978 changed the title in_s3: stop retrying SQS messages that can never succeed in_s3: stop retrying SQS messages that cannot be decoded Aug 17, 2026
is_valid_queue only checked that the top-level Records or detail key
existed, so a message missing the nested s3.object.key reached
get_raw_key and raised. The rescue in run turned that into
:skip_delete, the message was never removed, and SQS redelivered it
after the visibility timeout to fail the same way until the retention
period expired. skip_delete is false by default and anyone who can put
a message on the queue can trigger it, a body of "hello" included.

- Make get_raw_key total and move decoding, validation and match_regexp
  into object_key_from, where a permanent failure skips the message
- Keep object_key_from inside the per-message rescue, so anything
  unforeseen still falls back to :skip_delete rather than escaping
  QueuePoller#poll and halting ingestion for retry_error_interval
- Report every skipped message through one helper that truncates and
  scrubs what it logs, since an object key can be megabytes long and can
  hold invalid UTF-8 that JSON.generate rejects for the log event record
- Leave failures after the key is known, such as a failed get or a
  corrupt archive, on :skip_delete, because the object exists and a
  redrive policy with a dead letter queue is the right tool there
- Redeliver the same message in tests to prove a skipped message is
  handled once, and assert the warn each malformed shape produces

Signed-off-by: Shizuo Fujita <fujita@clear-code.com>
@Watson1978
Watson1978 force-pushed the in_s3-validate-sqs-message-structure branch from 195d9d1 to 8ea2375 Compare August 17, 2026 12:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant