Skip to content

[cloudflare_logpush] fix breaking-change bugs#18852

Draft
brijesh-elastic wants to merge 4 commits intoelastic:mainfrom
brijesh-elastic:cloudflare_logpush-2.0.0
Draft

[cloudflare_logpush] fix breaking-change bugs#18852
brijesh-elastic wants to merge 4 commits intoelastic:mainfrom
brijesh-elastic:cloudflare_logpush-2.0.0

Conversation

@brijesh-elastic
Copy link
Copy Markdown
Collaborator

@brijesh-elastic brijesh-elastic commented May 6, 2026

Proposed commit message

cloudflare_logpush: fix breaking-change bugs

Fix several field mapping and ingest pipeline correctness bugs across data streams:
- Fix `cloudflare_logpush.gateway_http.quarantined` mapping by changing its type from
`keyword` to `boolean` to match the actual values returned by Cloudflare.
- Fix `http_request` HTTP header field naming by renaming `header` to `headers` to align
with the field definitions and the underlying Cloudflare payload.
- Fix `gateway_dns` resolved IP modeling by replacing the flat
`cloudflare_logpush.gateway_dns.resolved_ip` field with the nested
`cloudflare_logpush.gateway_dns.resolved_ip_details.ips`, so per-IP category
metadata can be captured alongside the IP values.
- Fix `dlp_forensic_copies` `preserve_duplicate_custom_fields` toggle so that the correct
custom fields (`triggered_rule_id`, `datetime`) — which have ECS counterparts — are
removed when the toggle is OFF. Previously the remove processor referenced fields
the pipeline never produced, so the toggle had no effect for these fields.
- Fix integer-to-keyword type casting for `*IDs` fields in `gateway_dns`, `gateway_http`,
and `gateway_network` by replacing `rename` processors with `convert` processors of
`type: string`, so values are consistently stored as strings.

The fixes above are causing breaking changes:
- `cloudflare_logpush.gateway_http.quarantined` mapping changes from `keyword` to `boolean`.
Existing indices will reject the new value type, so users must roll over the
data stream (or reindex) for the new mapping to take effect.
- The `http_request` HTTP header fields are renamed from `header` to `headers`. Saved searches,
dashboards, queries, and detection rules referencing `cloudflare_logpush.http_request.*.header`
will no longer return data and must be updated to use `headers`.
- The `cloudflare_logpush.gateway_dns.resolved_ip` field is removed; resolved IPs now live under
`cloudflare_logpush.gateway_dns.resolved_ip_details.ips`. Saved searches, dashboards, queries,
and detection rules referencing the old field must be updated.
- In the `dlp_forensic_copies` data stream, when `preserve_duplicate_custom_fields` is
OFF (the default), `cloudflare_logpush.dlp_forensic_copies.triggered_rule_id` and
`cloudflare_logpush.dlp_forensic_copies.datetime` are no longer present in documents.
Users that depend on these custom fields must turn `preserve_duplicate_custom_fields`
ON, or migrate to the corresponding ECS fields.
- The `*IDs` fields in `gateway_dns`, `gateway_http`, and `gateway_network` are now indexed as
strings rather than the raw JSON integers Cloudflare emits. Queries, dashboards, and detection rules
comparing these fields against numeric values must be updated to use string values.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

  • Clone integrations repo.
  • Install elastic package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/cloudflare_logpush directory.
  • Run the following command to run tests.

elastic-package test -v

@brijesh-elastic brijesh-elastic self-assigned this May 6, 2026
@brijesh-elastic brijesh-elastic added documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. breaking change Integration:cloudflare_logpush Cloudflare Logpush Category: Integration quality Category: Quality used for SI planning Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] labels May 6, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 6, 2026

✅ Vale Linting Results

No issues found on modified lines!


The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod Bot commented May 6, 2026

🚀 Benchmarks report

Package cloudflare_logpush 👍(21) 💚(11) 💔(10)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
gateway_dns 5617.98 4444.44 -1173.54 (-20.89%) 💔
magic_ids 14285.71 11627.91 -2657.8 (-18.6%) 💔
nel_report 35714.29 30303.03 -5411.26 (-15.15%) 💔
network_session 5076.14 4255.32 -820.82 (-16.17%) 💔
access_request 5524.86 3194.89 -2329.97 (-42.17%) 💔
workers_trace 14705.88 7751.94 -6953.94 (-47.29%) 💔
gateway_http 5263.16 4366.81 -896.35 (-17.03%) 💔
gateway_network 7518.8 5649.72 -1869.08 (-24.86%) 💔
dlp_forensic_copies 19607.84 13888.89 -5718.95 (-29.17%) 💔
dns 13333.33 9259.26 -4074.07 (-30.56%) 💔

To see the full report comment with /test benchmark fullreport

@elasticmachine
Copy link
Copy Markdown

💚 Build Succeeded

History

cc @brijesh-elastic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking change Category: Integration quality Category: Quality used for SI planning documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:cloudflare_logpush Cloudflare Logpush Team:SDE-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants