docs: s3: SSE configuration docs#2365
Conversation
Signed-off-by: Gabriel Yamin <gabrielyamin98@gmail.com>
📝 WalkthroughWalkthroughThe S3 output plugin docs add two public configuration parameters, Changes
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Suggested labels
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@GabrielYamin I think this is functionality waiting for the PR to be merged right? |
|
@patrick-stephens, The BL PR is still in draft, I'll be opening it for review very soon. |
|
PR is ready for review @patrick-stephens |
Signed-off-by: Gabriel Yamin <43830000+GabrielYamin@users.noreply.github.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
pipeline/outputs/s3.md (1)
117-132: Document required KMS permissions for SSE-KMS encryption.The permissions section only lists
s3:PutObject, but when usingsse: aws:kmsorsse: aws:kms:dsse, additional KMS permissions are required. Without these, uploads will fail with permission errors.Required KMS permissions:
kms:GenerateDataKey- Required for encrypting objectskms:Decrypt- Required if using customer-managed keys📋 Suggested documentation update
Add a new subsection after line 132:
### Additional permissions for SSE-KMS When using server-side encryption with AWS KMS (`sse: aws:kms` or `sse: aws:kms:dsse`), the following KMS permissions are also required: ```json { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": [ "kms:GenerateDataKey", "kms:Decrypt" ], "Resource": "arn:aws:kms:region:account-id:key/key-id" }] }Replace
region,account-id, andkey-idwith your specific KMS key details. If using the AWS-managed S3 key (whensse_kms_key_idis not specified), permissions are managed automatically.</details> </blockquote></details> </blockquote></details>🧹 Nitpick comments (1)
pipeline/outputs/s3.md (1)
579-617: Consider adding examples for SSE-S3 and DSSE-KMS.The example demonstrates SSE-KMS effectively, but users may also need guidance on:
- SSE-S3 (simpler, no KMS key required): Just set
sse: AES256- DSSE-KMS (dual-layer encryption): Use
sse: aws:kms:dssewith a KMS key ARNAdding brief examples of these variants would provide more complete documentation coverage.
📝 Suggested additional examples
Add after line 617:
An example using SSE-S3 encryption (S3-managed keys): {% tabs %} {% tab title="fluent-bit.yaml" %} ```yaml pipeline: outputs: - name: s3 match: '*' bucket: your-bucket region: us-east-1 sse: AES256{% endtab %}
{% tab title="fluent-bit.conf" %}[OUTPUT] Name s3 Match * bucket your-bucket region us-east-1 sse AES256{% endtab %}
{% endtabs %}</details> </blockquote></details> </blockquote></details> <!-- This is an auto-generated comment by CodeRabbit for review status -->
|
@GabrielYamin I do agree with Vale in some of the comments that it is a bit acronym/initialism heavy so useful to introduce the acronym/initialism. Can you check and resolve any comments raised by CI? |
|
@patrick-stephens Do you think that if I introduce the acronyms in the SSE config definition it would suffice for the key_id config? or should I introduce them in both? |
|
Yeah I'd just introduce it somewhere at the top or use the full name the first time it is present in the table/text - whilst I agree most folks should be aware I would still err on the side of caution. You don't need to define the term but at least clarify what the acronym/initialism stands for. It's also pretty easy to get different definitions sometimes for the same 3 letters so being explicit never hurts. I'm not sure if there are conflicting definitions in this case but you can never know all possible usage now and in the future - it doesn't take much effort to be clear and resolves any potential issue (even if there are none!). If anything is already covered as well but CI is flagging it then just make a comment to that effect and resolve it. |
Signed-off-by: Gabriel Yamin <43830000+GabrielYamin@users.noreply.github.com>
|
I have updated the descriptions, let me know if it's good now or you want me to adjust it a bit more. |
eschabell
added
waiting-on-code-merge
waiting-on-review
|
@GabrielYamin while we wait on the code review, can you address the failing check (linting)? |
|
@eschabell All of these are present in the I think that spelling them out in every instance of the acronym is redundant. |
eschabell
left a comment
eschabell
left a comment
There was a problem hiding this comment.
@GabrielYamin to resolve the lint vale issues, please address the below change requests?
| | `s3_key_format` | Format string for keys in S3. This option supports a UUID, strftime time formatters, a syntax for selecting parts of the Fluent log tag using a syntax inspired by the `rewrite_tag` filter. Add `$UUID` in the format string to insert a random string. Add `$INDEX` in the format string to insert an integer that increments each upload. The `$INDEX` value saves in the `store_dir`. Add `$TAG` in the format string to insert the full log tag. Add `$TAG[0]` to insert the first part of the tag in the S3 key. The tag is split into parts using the characters specified with the `s3_key_format_tag_delimiters` option. Add the extension directly after the last piece of the format string to insert a key suffix. To specify a key suffix in `use_put_object` mode, you must specify `$UUID`. See [S3 Key Format](#s3-key-format-and-tag-delimiters). Time in `s3_key` is the timestamp of the first record in the S3 file. | `/fluent-bit-logs/$TAG/%Y/%m/%d/%H/%M/%S` | | ||
| | `s3_key_format_tag_delimiters` | A series of characters which will be used to split the tag into `parts` for use with the s3_key_format option. | `.` | | ||
| | `send_content_md5` | Send the Content-MD5 header with `PutObject` and UploadPart requests, as is required when Object Lock is enabled. | `false` | | ||
| | `sse` | Server-side encryption (SSE) for S3 objects. Set to `AES256` for S3-managed keys (SSE-S3), `aws:kms` for AWS Key Management Service (SSE-KMS), or `aws:kms:dsse` for dual-layer server-side encryption with KMS (DSSE-KMS). | _none_ | |
There was a problem hiding this comment.
@GabrielYamin if you enclose the offending word in back-ticks, this will resolve.
| | `s3_key_format_tag_delimiters` | A series of characters which will be used to split the tag into `parts` for use with the s3_key_format option. | `.` | | ||
| | `send_content_md5` | Send the Content-MD5 header with `PutObject` and UploadPart requests, as is required when Object Lock is enabled. | `false` | | ||
| | `sse` | Server-side encryption (SSE) for S3 objects. Set to `AES256` for S3-managed keys (SSE-S3), `aws:kms` for AWS Key Management Service (SSE-KMS), or `aws:kms:dsse` for dual-layer server-side encryption with KMS (DSSE-KMS). | _none_ | | ||
| | `sse_kms_key_id` | AWS KMS key Resource Name (ARN) for server-side encryption. Only applicable when `sse` is set to `aws:kms` or `aws:kms:dsse`. If not specified, the default AWS-managed KMS key for S3 will be used. | _none_ | |
| {% endtab %} | ||
| {% endtabs %} | ||
|
|
||
| An example using SSE-KMS encryption: |
3 participants
Summary by CodeRabbit
✏️ Tip: You can customize this high-level summary in your review settings.