diff --git a/packages/aws/_dev/build/docs/cloudtrail.md b/packages/aws/_dev/build/docs/cloudtrail.md index 09adb17301a..62ff38ddcf7 100644 --- a/packages/aws/_dev/build/docs/cloudtrail.md +++ b/packages/aws/_dev/build/docs/cloudtrail.md @@ -61,6 +61,21 @@ The `number_of_workers` setting defines the number of workers assigned to readin **Recommendation:** Set `number_of_workers` to **5 or less** and `scan_frequency` to **5m or more**, regardless of how many log groups match `log_group_name_prefix`. Workers will iterate through the matching log groups within each scan interval. The default value is `1`. +#### S3 polling mode considerations + +When using the "Collect logs via S3 Bucket" option in polling mode, the integration lists and processes all objects in the bucket. For buckets containing large volumes of historical logs, this can cause high memory usage and potential out-of-memory (OOM) errors. + +**Important:** If you provide both a bucket ARN and an SQS Queue URL, the integration ignores the SQS URL and operates in polling mode, attempting to process the entire bucket. To use SQS mode, disable "Collect logs via S3 Bucket" and provide only the SQS Queue URL. + +**Recommendation:** Use SQS mode when possible to avoid scanning the entire bucket. + +If you must use polling mode, configure these advanced options to limit which S3 objects are processed: + +- **Ignore Older Timespan** (`ignore_older`): Skip S3 objects older than the specified duration (for example, `48h`, `30d`). +- **Start Timestamp** (`start_timestamp`): Only process objects newer than the specified time (`YYYY-MM-DDTHH:MM:SSZ`). + +If you experience timeouts (`ListObjectsV2, context canceled`), also consider increasing `bucket_list_interval` to reduce listing frequency. + ## Logs reference The `cloudtrail` data stream collects AWS CloudTrail logs. CloudTrail monitors events like diff --git a/packages/aws/changelog.yml b/packages/aws/changelog.yml index 2955601ae8c..959762fe0b5 100644 --- a/packages/aws/changelog.yml +++ b/packages/aws/changelog.yml @@ -1,9 +1,9 @@ # newer versions go on top -- version: "6.14.2" +- version: "6.15.0" changes: - - description: Fix NextToken invalidation in Security Hub, GuardDuty, and Inspector by removing the upper time bound from paginated queries. - type: bugfix - link: https://github.com/elastic/integrations/pull/18830 + - description: Add S3 polling mode performance recommendations to CloudTrail docs + type: enhancement + link: https://github.com/elastic/integrations/pull/18887 - version: "6.14.1" changes: - description: Handle null EvaluationResults in AWS Config GetComplianceDetailsByConfigRule response. @@ -26,7 +26,7 @@ link: https://github.com/elastic/integrations/pull/18365 - version: "6.11.0" changes: - - description: Enable agentless deployment for AWS Transitgateway metrics. + - description: Enable agentless deployment for AWS Transitgateway metrics. type: enhancement link: https://github.com/elastic/integrations/pull/18389 - version: "6.10.0" @@ -103,11 +103,7 @@ link: https://github.com/elastic/integrations/pull/17552 - version: "6.2.0" changes: - - description: | - Add support for the `aws.waf.captcha_response.*`, `aws.waf.challenge_response.*`, `aws.waf.format_version`, `aws.waf.ja4_fingerprint`, - `aws.waf.labels.name`, `aws.waf.oversize_fields`, `aws.waf.request_body_size`, `aws.waf.request_body_size_inspected_by_waf`, `aws.waf.response_code_sent`, - `aws.waf.request_headers_inserted.*`, `host.domain`, `tls.client.ja3`, `url.fragment` and `url.scheme` in waf data stream. - `aws.waf.request_headers_inserted.*`, `tls.client.ja3`, `url.registered_domain`, `url.fragment` and `url.scheme` in waf data stream. + - description: "Add support for the `aws.waf.captcha_response.*`, `aws.waf.challenge_response.*`, `aws.waf.format_version`, `aws.waf.ja4_fingerprint`, \n`aws.waf.labels.name`, `aws.waf.oversize_fields`, `aws.waf.request_body_size`, `aws.waf.request_body_size_inspected_by_waf`, `aws.waf.response_code_sent`, \n`aws.waf.request_headers_inserted.*`, `host.domain`, `tls.client.ja3`, `url.fragment` and `url.scheme` in waf data stream.\n`aws.waf.request_headers_inserted.*`, `tls.client.ja3`, `url.registered_domain`, `url.fragment` and `url.scheme` in waf data stream.\n" type: enhancement link: https://github.com/elastic/integrations/pull/17205 - version: "6.1.1" diff --git a/packages/aws/docs/cloudtrail.md b/packages/aws/docs/cloudtrail.md index 1e201d9b246..aaa9f5ae260 100644 --- a/packages/aws/docs/cloudtrail.md +++ b/packages/aws/docs/cloudtrail.md @@ -61,6 +61,21 @@ The `number_of_workers` setting defines the number of workers assigned to readin **Recommendation:** Set `number_of_workers` to **5 or less** and `scan_frequency` to **5m or more**, regardless of how many log groups match `log_group_name_prefix`. Workers will iterate through the matching log groups within each scan interval. The default value is `1`. +#### S3 polling mode considerations + +When using the "Collect logs via S3 Bucket" option in polling mode, the integration lists and processes all objects in the bucket. For buckets containing large volumes of historical logs, this can cause high memory usage and potential out-of-memory (OOM) errors. + +**Important:** If you provide both a bucket ARN and an SQS Queue URL, the integration ignores the SQS URL and operates in polling mode, attempting to process the entire bucket. To use SQS mode, disable "Collect logs via S3 Bucket" and provide only the SQS Queue URL. + +**Recommendation:** Use SQS mode when possible to avoid scanning the entire bucket. + +If you must use polling mode, configure these advanced options to limit which S3 objects are processed: + +- **Ignore Older Timespan** (`ignore_older`): Skip S3 objects older than the specified duration (for example, `48h`, `30d`). +- **Start Timestamp** (`start_timestamp`): Only process objects newer than the specified time (`YYYY-MM-DDTHH:MM:SSZ`). + +If you experience timeouts (`ListObjectsV2, context canceled`), also consider increasing `bucket_list_interval` to reduce listing frequency. + ## Logs reference The `cloudtrail` data stream collects AWS CloudTrail logs. CloudTrail monitors events like diff --git a/packages/aws/manifest.yml b/packages/aws/manifest.yml index bf789bf24ec..c0665493806 100644 --- a/packages/aws/manifest.yml +++ b/packages/aws/manifest.yml @@ -1,7 +1,7 @@ format_version: 3.4.0 name: aws title: AWS -version: 6.14.2 +version: 6.15.0 description: Collect logs and metrics from Amazon Web Services (AWS) with Elastic Agent. type: integration categories: