Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions packages/aws/_dev/build/docs/cloudtrail.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,21 @@ The `number_of_workers` setting defines the number of workers assigned to readin

**Recommendation:** Set `number_of_workers` to **5 or less** and `scan_frequency` to **5m or more**, regardless of how many log groups match `log_group_name_prefix`. Workers will iterate through the matching log groups within each scan interval. The default value is `1`.

#### S3 polling mode considerations

When using the "Collect logs via S3 Bucket" option in polling mode, the integration lists and processes all objects in the bucket. For buckets containing large volumes of historical logs, this can cause high memory usage and potential out-of-memory (OOM) errors.

**Important:** If you provide both a bucket ARN and an SQS Queue URL, the integration ignores the SQS URL and operates in polling mode, attempting to process the entire bucket. To use SQS mode, disable "Collect logs via S3 Bucket" and provide only the SQS Queue URL.

**Recommendation:** Use SQS mode when possible to avoid scanning the entire bucket.

If you must use polling mode, configure these advanced options to limit which S3 objects are processed:

- **Ignore Older Timespan** (`ignore_older`): Skip S3 objects older than the specified duration (for example, `48h`, `30d`).
- **Start Timestamp** (`start_timestamp`): Only process objects newer than the specified time (`YYYY-MM-DDTHH:MM:SSZ`).

If you experience timeouts (`ListObjectsV2, context canceled`), also consider increasing `bucket_list_interval` to reduce listing frequency.

## Logs reference

The `cloudtrail` data stream collects AWS CloudTrail logs. CloudTrail monitors events like
Expand Down
16 changes: 6 additions & 10 deletions packages/aws/changelog.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# newer versions go on top
- version: "6.14.2"
- version: "6.15.0"
changes:
- description: Fix NextToken invalidation in Security Hub, GuardDuty, and Inspector by removing the upper time bound from paginated queries.
type: bugfix
link: https://github.com/elastic/integrations/pull/18830
- description: Add S3 polling mode performance recommendations to CloudTrail docs
type: enhancement
link: https://github.com/elastic/integrations/pull/18887
Comment on lines +2 to +6
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- version: "6.15.0"
changes:
- description: Fix NextToken invalidation in Security Hub, GuardDuty, and Inspector by removing the upper time bound from paginated queries.
type: bugfix
link: https://github.com/elastic/integrations/pull/18830
- description: Add S3 polling mode performance recommendations to CloudTrail docs
type: enhancement
link: https://github.com/elastic/integrations/pull/18887
- version: "6.15.0"
changes:
- description: Add S3 polling mode performance recommendations to CloudTrail docs
type: enhancement
link: https://github.com/elastic/integrations/pull/18887
- version: "6.14.2"
changes:
- description: Fix NextToken invalidation in Security Hub, GuardDuty, and Inspector by removing the upper time bound from paginated queries.
type: bugfix
link: https://github.com/elastic/integrations/pull/18830

- version: "6.14.1"
changes:
- description: Handle null EvaluationResults in AWS Config GetComplianceDetailsByConfigRule response.
Expand All @@ -26,7 +26,7 @@
link: https://github.com/elastic/integrations/pull/18365
- version: "6.11.0"
changes:
- description: Enable agentless deployment for AWS Transitgateway metrics.
- description: Enable agentless deployment for AWS Transitgateway metrics.
type: enhancement
link: https://github.com/elastic/integrations/pull/18389
- version: "6.10.0"
Expand Down Expand Up @@ -103,11 +103,7 @@
link: https://github.com/elastic/integrations/pull/17552
- version: "6.2.0"
changes:
- description: |
Add support for the `aws.waf.captcha_response.*`, `aws.waf.challenge_response.*`, `aws.waf.format_version`, `aws.waf.ja4_fingerprint`,
`aws.waf.labels.name`, `aws.waf.oversize_fields`, `aws.waf.request_body_size`, `aws.waf.request_body_size_inspected_by_waf`, `aws.waf.response_code_sent`,
`aws.waf.request_headers_inserted.*`, `host.domain`, `tls.client.ja3`, `url.fragment` and `url.scheme` in waf data stream.
`aws.waf.request_headers_inserted.*`, `tls.client.ja3`, `url.registered_domain`, `url.fragment` and `url.scheme` in waf data stream.
- description: "Add support for the `aws.waf.captcha_response.*`, `aws.waf.challenge_response.*`, `aws.waf.format_version`, `aws.waf.ja4_fingerprint`, \n`aws.waf.labels.name`, `aws.waf.oversize_fields`, `aws.waf.request_body_size`, `aws.waf.request_body_size_inspected_by_waf`, `aws.waf.response_code_sent`, \n`aws.waf.request_headers_inserted.*`, `host.domain`, `tls.client.ja3`, `url.fragment` and `url.scheme` in waf data stream.\n`aws.waf.request_headers_inserted.*`, `tls.client.ja3`, `url.registered_domain`, `url.fragment` and `url.scheme` in waf data stream.\n"
type: enhancement
link: https://github.com/elastic/integrations/pull/17205
- version: "6.1.1"
Expand Down
15 changes: 15 additions & 0 deletions packages/aws/docs/cloudtrail.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,21 @@ The `number_of_workers` setting defines the number of workers assigned to readin

**Recommendation:** Set `number_of_workers` to **5 or less** and `scan_frequency` to **5m or more**, regardless of how many log groups match `log_group_name_prefix`. Workers will iterate through the matching log groups within each scan interval. The default value is `1`.

#### S3 polling mode considerations

When using the "Collect logs via S3 Bucket" option in polling mode, the integration lists and processes all objects in the bucket. For buckets containing large volumes of historical logs, this can cause high memory usage and potential out-of-memory (OOM) errors.

**Important:** If you provide both a bucket ARN and an SQS Queue URL, the integration ignores the SQS URL and operates in polling mode, attempting to process the entire bucket. To use SQS mode, disable "Collect logs via S3 Bucket" and provide only the SQS Queue URL.

**Recommendation:** Use SQS mode when possible to avoid scanning the entire bucket.

If you must use polling mode, configure these advanced options to limit which S3 objects are processed:

- **Ignore Older Timespan** (`ignore_older`): Skip S3 objects older than the specified duration (for example, `48h`, `30d`).
- **Start Timestamp** (`start_timestamp`): Only process objects newer than the specified time (`YYYY-MM-DDTHH:MM:SSZ`).

If you experience timeouts (`ListObjectsV2, context canceled`), also consider increasing `bucket_list_interval` to reduce listing frequency.

## Logs reference

The `cloudtrail` data stream collects AWS CloudTrail logs. CloudTrail monitors events like
Expand Down
2 changes: 1 addition & 1 deletion packages/aws/manifest.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 3.4.0
name: aws
title: AWS
version: 6.14.2
version: 6.15.0
description: Collect logs and metrics from Amazon Web Services (AWS) with Elastic Agent.
type: integration
categories:
Expand Down
Loading