-
Notifications
You must be signed in to change notification settings - Fork 583
[Azure Logs] Add dedicated aadgraphactivitylogs data stream #18880
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
terrancedejesus
wants to merge
25
commits into
main
Choose a base branch
from
enhancement/azure-ad-graph-activitylogs
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
25 commits
Select commit
Hold shift + click to select a range
ee437f5
[Azure Logs] Add aadgraphactivitylogs data stream
terrancedejesus 2c88052
Update changelog for azure.aadgraphactivitylogs dataset
terrancedejesus 7f60d1d
Apply suggestion from @terrancedejesus
terrancedejesus c1a75c0
Apply suggestion from @terrancedejesus
terrancedejesus 82dd31b
azure/aadgraphactivitylogs: update codeowners
terrancedejesus 9c4fb1e
Merge branch 'main' into enhancement/azure-ad-graph-activitylogs
terrancedejesus 357c09f
azure/aadgraphactivitylogs: update codeowners
terrancedejesus ecae992
Merge branch 'main' into enhancement/azure-ad-graph-activitylogs
terrancedejesus 30046ce
azure/aadgraphactivitylogs: remove log YAML HBS file
terrancedejesus 0a4f697
azure/aadgraphactivitylogs: adding docs and updated README
terrancedejesus 20ba136
azure/aadgraphactivitylogs: adding separate manifest for aadgraphacti…
terrancedejesus 8e1dc86
azure/aadgraphactivitylogs: adjust data stream test pipeline events a…
terrancedejesus 97285cc
azure/aadgraphactivitylogs: fix vale linting warnings
terrancedejesus 7be1278
azure/aadgraphactivitylogs: adds shared pipeline link
terrancedejesus a62a1ac
azure/aadgraphactivitylogs: remove 8.14 workaround for dynamic field …
terrancedejesus 99758fd
azure/aadgraphactivitylogs: adds aadgraphactivitylogs schema referenc…
terrancedejesus 7f1b4b6
azure/aadgraphactivitylogs: added Azure AD Graph Activity Logs right …
terrancedejesus f7a11a1
Merge branch 'main' into enhancement/azure-ad-graph-activitylogs
terrancedejesus c6ba91a
azure/aadgraphactivitylogs: sample event and documentation fix
terrancedejesus 6575463
Update packages/azure/data_stream/aadgraphactivitylogs/elasticsearch/…
terrancedejesus 08110a7
Update packages/azure/data_stream/aadgraphactivitylogs/elasticsearch/…
terrancedejesus 8fae9c5
azure/aadgraphactivitylogs: url.original keep original from fingerprint
terrancedejesus f8eb858
Merge branch 'main' into enhancement/azure-ad-graph-activitylogs
terrancedejesus f995b6f
Merge branch 'main' into enhancement/azure-ad-graph-activitylogs
terrancedejesus bbc07f7
Merge branch 'main' into enhancement/azure-ad-graph-activitylogs
terrancedejesus File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
| # Azure AD Graph Activity Logs | ||
|
|
||
| Azure AD Graph Activity Logs provide an audit trail of all HTTP requests that the legacy Azure AD Graph service (`graph.windows.net`) has received and processed for a tenant. Although Microsoft has deprecated Azure AD Graph in favor of Microsoft Graph, the API is still actively used by Microsoft first-party tooling, older line-of-business applications, third-party SaaS connectors, and adversary tooling (for example ROADtools, AzureHound v1, AADInternals). Refer to the [AADGraphActivityLogs table reference](https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/aadgraphactivitylogs) for the canonical schema. | ||
|
|
||
| Tenant administrators can configure the collection and storage destinations of Azure AD Graph Activity Logs through Diagnostic Setting in the Entra Portal. This integration uses Azure Event Hubs destination to stream Azure AD Graph Activity Logs to Elastic. | ||
|
|
||
| ## Requirements and Setup | ||
|
|
||
| ### What do I need to use this integration? | ||
|
|
||
| The following privileges are required to collect Azure AD Graph Activity Logs: | ||
| - A Microsoft Entra ID P1 or P2 tenant license in your tenant. | ||
| - A `Security Administrator` or `Global Administrator` Microsoft Entra ID role to configure the diagnostic settings. | ||
|
|
||
| ### Setup | ||
|
|
||
| Refer to the [Azure Logs](https://docs.elastic.co/integrations/azure) page for more information about setting up and using this integration. | ||
|
|
||
| ### Limitations | ||
|
|
||
| - Activities of multi-tenant applications belonging to another tenant are not available. | ||
| - In rare cases, events might take up to 2 hours to be delivered to Event Hubs. | ||
| - Azure AD Graph is deprecated by Microsoft. New workloads should target Microsoft Graph; this dataset is intended for visibility into legacy traffic that still exists in the tenant. | ||
|
|
||
| ## Settings | ||
|
|
||
| `eventhub` : | ||
| _string_ | ||
| It is a fully managed, real-time data ingestion service. Elastic recommends using only letters, numbers, and the hyphen (-) character for Event Hub names to maximize compatibility. You _can_ use existing Event Hubs having underscores (_) in the Event Hub name. In this case, the integration will replace underscores with hyphens (-) when it uses the Event Hub name to create dependent Azure resources behind the scenes (for example, the storage account container to store Event Hub consumer offsets). Elastic also recommends using a separate event hub for each log type as the field mappings of each log type differ. | ||
| Default value `insights-operational-logs`. | ||
|
|
||
| `consumer_group` : | ||
| _string_ | ||
| The publish/subscribe mechanism of Event Hubs is enabled through consumer groups. A consumer group is a view (state, position, or offset) of an entire event hub. Consumer groups enable multiple consuming applications to each have a separate view of the event stream, and to read the stream independently at their own pace and with their own offsets. | ||
| Default value: `$Default` | ||
|
|
||
| `connection_string` : | ||
| _string_ | ||
| The connection string required to communicate with Event Hubs, steps [here](https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string). | ||
|
|
||
| A Blob Storage account is required to store, retrieve, and update the offset or state of the eventhub messages. This means that after stopping the filebeat azure module it can start back up at the spot that it stopped processing messages. | ||
|
|
||
| `storage_account` : | ||
| _string_ | ||
| The name of the storage account the state/offsets will be stored and updated. | ||
|
|
||
| `storage_account_key` : | ||
| _string_ | ||
| The storage account key, this key will be used to authorize access to data in your storage account. | ||
|
|
||
| `storage_account_container` : | ||
| _string_ | ||
| The storage account container where the integration stores the checkpoint data for the consumer group. It is an advanced option to use with extreme care. You MUST use a dedicated storage account container for each Azure log type (activity, sign-in, audit logs, and others). DO NOT REUSE the same container name for more than one Azure log type. See [Container Names](https://docs.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-containers--blobs--and-metadata#container-names) for details on naming rules from Microsoft. The integration generates a default container name if not specified. | ||
|
|
||
| `resource_manager_endpoint` : | ||
| _string_ | ||
| Optional. By default the integration uses the Azure public environment. To override, provide a specific resource manager endpoint to use a different Azure environment. | ||
|
|
||
| Resource manager endpoints: | ||
|
|
||
| ```text | ||
| # Azure ChinaCloud | ||
| https://management.chinacloudapi.cn/ | ||
|
|
||
| # Azure GermanCloud | ||
| https://management.microsoftazure.de/ | ||
|
|
||
| # Azure PublicCloud | ||
| https://management.azure.com/ | ||
|
|
||
| # Azure USGovernmentCloud | ||
| https://management.usgovcloudapi.net/ | ||
| ``` | ||
|
|
||
| ## Logs | ||
|
|
||
| ### aadgraphactivitylogs | ||
|
|
||
| The `aadgraphactivitylogs` data stream of the Azure Logs package collects Azure AD Graph activity events that have been streamed through an Azure event hub. The events ingest pipeline matches `category == "AzureADGraphActivityLogs"` and sets `event.dataset = azure.aadgraphactivitylogs`. The events data stream's routing rules then reroute the document from `logs-azure.events-*` directly to `logs-azure.aadgraphactivitylogs-*`, where this data stream's pipeline applies full ECS field extraction. | ||
|
|
||
| Before this data stream existed, AAD Graph events had no specific override in the events router and fell through to the `azure.platformlogs` catch-all, landing in `logs-azure.platformlogs-default` with only generic platform-log parsing. Those previously-indexed events are not backfilled. Only new events are routed to the dedicated dataset. | ||
|
|
||
| {{event "aadgraphactivitylogs"}} | ||
|
|
||
| **ECS Field Reference** | ||
|
|
||
| Refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. | ||
|
|
||
| {{fields "aadgraphactivitylogs"}} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5 changes: 5 additions & 0 deletions
5
...ages/azure/data_stream/aadgraphactivitylogs/_dev/test/pipeline/test-aadgraph-activity.log
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| {"category":"AzureADGraphActivityLogs","location":"WestUS","operationName":"AAD Graph Activity","properties":{"__UDI_RequiredFields_EventTime":639140000000000000,"__UDI_RequiredFields_RegionScope":"NA","__UDI_RequiredFields_TenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","__UDI_RequiredFields_UniqueId":"00000001-0001-0001-0001-000000000001","actorType":"User","apiVersion":"1.6","appId":"04b07795-8ddb-461a-bbee-02f9e1bf7b46","callerIpAddress":"81.2.69.143","clientAuthMethod":0,"deviceId":"","directAccessSource":"Gateway","durationMs":59,"env_cloud_role":"restdirectoryservice","httpMethod":"GET","httpStatusCode":200,"identityProvider":"https://sts.windows.net/ab30785b-417f-42a4-b5dc-8f9051718acb/","issuedAt":"5/7/2026 1:50:39 PM","location":"WestUS","requestId":"00000001-0001-0001-0001-000000000001","requestUri":"/v2/ab30785b-417f-42a4-b5dc-8f9051718acb/users","responseSizeBytes":54662,"roles":"","scopes":"62e90394-69f5-4237-9190-012177145e10","servicePrincipalId":"","sessionId":"5a5a5a5a-5a5a-5a5a-5a5a-5a5a5a5a5a5a","signInActivityId":"AAAAAAAAAAAAAAAAAAAAAA==","tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","timeGenerated":"2026-05-07T15:19:33.5368860Z","userAgent":"azure-graph-test-client/1.0","userId":"b37ec517-0a34-4266-b627-f7bb0d679d70","wids":""},"tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb"} | ||
| {"category":"AzureADGraphActivityLogs","location":"WestUS","operationName":"AAD Graph Activity","properties":{"__UDI_RequiredFields_EventTime":639140000000000001,"__UDI_RequiredFields_RegionScope":"NA","__UDI_RequiredFields_TenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","__UDI_RequiredFields_UniqueId":"00000002-0002-0002-0002-000000000002","actorType":"User","apiVersion":"1.61-internal","appId":"1b730954-1685-4b74-9bfd-dac224a7b894","callerIpAddress":"81.2.69.143","clientAuthMethod":0,"deviceId":"","directAccessSource":"Gateway","durationMs":25,"env_cloud_role":"restdirectoryservice","httpMethod":"GET","httpStatusCode":403,"identityProvider":"","issuedAt":"5/8/2026 4:11:19 PM","location":"WestUS","requestId":"00000002-0002-0002-0002-000000000002","requestUri":"/v2/ab30785b-417f-42a4-b5dc-8f9051718acb/tenantDetails","responseSizeBytes":-1,"roles":"","scopes":"user_impersonation","servicePrincipalId":"","sessionId":"004d8baa-891d-139e-633f-6c194e566b78","signInActivityId":"BCF50s3Ik0q7Ke-mIBAGAA==","tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","timeGenerated":"2026-05-08T16:37:21.2823997Z","userAgent":"AADInternals","userId":"b37ec517-0a34-4266-b627-f7bb0d679d70","wids":""},"tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb"} | ||
| {"category":"AzureADGraphActivityLogs","location":"WestUS","operationName":"AAD Graph Activity","properties":{"__UDI_RequiredFields_EventTime":639141000000000000,"__UDI_RequiredFields_RegionScope":"NA","__UDI_RequiredFields_TenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","__UDI_RequiredFields_UniqueId":"00000003-0003-0003-0003-000000000003","actorType":"Application","apiVersion":"AGSbeta-internal","appId":"65d91a3d-ab74-42e6-8a2f-0add61688c74","callerIpAddress":"81.2.69.144","clientAuthMethod":0,"deviceId":"","directAccessSource":"Gateway","durationMs":44,"env_cloud_role":"restdirectoryservice","httpMethod":"GET","httpStatusCode":200,"identityProvider":"https://sts.windows.net/ab30785b-417f-42a4-b5dc-8f9051718acb/","issuedAt":"5/11/2026 12:35:58 PM","location":"WestUS","requestId":"00000003-0003-0003-0003-000000000003","requestUri":"/v2/ab30785b-417f-42a4-b5dc-8f9051718acb/groups","responseSizeBytes":695,"roles":" Group.Read.All,","scopes":"","servicePrincipalId":"fa5cc200-f464-4068-88b9-c6b8335c8c70","sessionId":"","signInActivityId":"Rr_bbVOVu0WHOTzl0BhgAA==","tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","timeGenerated":"2026-05-11T12:41:10.9430757Z","userAgent":"","userId":"","wids":" 0997a1d0-0d1d-4acb-b408-d5ca73121e90,"},"tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb"} | ||
| {"category":"AzureADGraphActivityLogs","location":"SouthCentralUS","operationName":"AAD Graph Activity","properties":{"__UDI_RequiredFields_EventTime":639141000000000001,"__UDI_RequiredFields_RegionScope":"NA","__UDI_RequiredFields_TenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","__UDI_RequiredFields_UniqueId":"00000004-0004-0004-0004-000000000004","actorType":"Application","apiVersion":"1.61-internal","appId":"66244124-575c-4284-92bc-fdd00e669cea","callerIpAddress":"89.160.20.156","clientAuthMethod":0,"deviceId":"","directAccessSource":"Gateway","durationMs":152,"env_cloud_role":"restdirectoryservice","httpMethod":"POST","httpStatusCode":202,"identityProvider":"https://sts.windows.net/ab30785b-417f-42a4-b5dc-8f9051718acb/","issuedAt":"5/9/2026 12:28:05 PM","location":"SouthCentralUS","requestId":"00000004-0004-0004-0004-000000000004","requestUri":"/ab30785b-417f-42a4-b5dc-8f9051718acb/$batch","responseSizeBytes":9580,"roles":"","scopes":"","servicePrincipalId":"4ed697af-4af9-43f7-8bf2-7b2172300068","sessionId":"","signInActivityId":"-TZsVeDWj0e-jamRtuUnAA==","tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","timeGenerated":"2026-05-09T12:33:05.9908603Z","userAgent":"","userId":"","wids":""},"tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb"} | ||
| {"category":"AzureADGraphActivityLogs","location":"WestUS","operationName":"AAD Graph Activity","properties":{"__UDI_RequiredFields_EventTime":639142000000000000,"__UDI_RequiredFields_RegionScope":"NA","__UDI_RequiredFields_TenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","__UDI_RequiredFields_UniqueId":"00000005-0005-0005-0005-000000000005","actorType":"User","apiVersion":"1.6","appId":"04b07795-8ddb-461a-bbee-02f9e1bf7b46","callerIpAddress":"81.2.69.143","clientAuthMethod":0,"deviceId":"","directAccessSource":"Gateway","durationMs":71,"env_cloud_role":"restdirectoryservice","httpMethod":"PATCH","httpStatusCode":204,"identityProvider":"https://sts.windows.net/ab30785b-417f-42a4-b5dc-8f9051718acb/","issuedAt":"5/7/2026 1:50:39 PM","location":"WestUS","requestId":"00000005-0005-0005-0005-000000000005","requestUri":"/v2/ab30785b-417f-42a4-b5dc-8f9051718acb/applications/fda2d70f-7df3-4556-8bcb-30acc0a45082","responseSizeBytes":0,"roles":"","scopes":"62e90394-69f5-4237-9190-012177145e10","servicePrincipalId":"","sessionId":"5a5a5a5a-5a5a-5a5a-5a5a-5a5a5a5a5a5a","signInActivityId":"AAAAAAAAAAAAAAAAAAAAAA==","tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb","timeGenerated":"2026-05-07T15:15:44.4909654Z","userAgent":"azure-graph-test-client/1.0","userId":"b37ec517-0a34-4266-b627-f7bb0d679d70","wids":""},"tenantId":"ab30785b-417f-42a4-b5dc-8f9051718acb"} |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.