Deleting Application Insights telemetry data on-demand with Powershell

I think that everyone of you knows that telemetry is an important aspect to take into consideration when working with a Dynamics 365 Business Central SaaS tenant and I continuously encourage you to activate the integration with Application Insights for your production environments.

In a post about this topic that I’ve written months ago, I’ve provided you a way to analyze telemetry data of a tenant programmatically by using Powershell (you can find the post here). This post described a quick and efficient way of executing a KUSTO (KQL) query inside the Application Insights telemetry store directly by using a Powershell script.

Linked to this topic, last week I had a request to delete from the Azure Application Insights telemetry a set of custom events ingested from some Dynamics 365 Business central apps. Unfortunately there’s not an easy way to do that and this is the reason for this post.

To delete Application Insights data, you’ve essentially 3 possible ways:

1. Delete your Application Insight instance (but this is not what do you want to achieve I think)
2. Use data retention policy: this deletes all data over a defined time period (default = 90 days). I suggest to have a policy like this in place in order to avoid paying too much for a large amount of old data that maybe you’re not using anymore but this is way does not satisfy the requirement (we cannot delete the telemetry data we want selectively).
3. Use the REST API for purging data stored in Application Insights. This API (docs available here) permits you to purge data in an Application Insights component by applying a set of user-defined filters.

Solution at point 3 is what fits our needs. From Powershell, we can call the following endpoint:

POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Insights/components/{resourceName}/purge?api-version=2015-05-01

We can send a POST request with a body that contains the Application Insights telemetry table we want to purge and the filter that we want to apply on those records and data will be deleted.

How to do that?

First, to use this API you need to authenticate by using the Azure Active Directory OAuth2 implicit Flow. To do that, you need to register an application into your Azure AD:

When the app is created, take note of the Application (client) ID:

The next step is to create a key for the application registration. Select your AD application, click on Certificates and Secrets and then create a new client secret. Take note of its value:

Then you need to assign a role to your app in order to access the Application Insights resource. Select the Application Insight resource (or directly select the Subscription object if you want), click on Access Control (IAM) and then add a role assignment:

Here create a new role assignment with Data Purger as the role and select your app registration (search for the app name):

Now the setup for authorization is finished.

The Powershell script that performs the authorization is the following:

Import-Module AzureRM.Profile

#Application ID of the registered app
$appId = "YOR_APPLICATION_ID"
#Client secret value for the app registration
$key = "YOR_CLIENT_SECRET"
#AAD tenant ID
$tenantId = "YOUR_AAD_TENANT_ID"
#Azure Subscription ID
$subscriptionId = "YOUR_SUBSCRIPTION_ID"
#Resource group where Application Insight Instance is assigned to
$resourceGroupName = "YOUR_APPINSIGHTS_RESOURCE_GROUP_NAME"
#Name of the Application Insight instance
$resourceName = "YOUR_APPINSIGHTS_INSTANCE_NAME"

# Create the authentication URL and get the authentication context
$authUrl = "https://login.windows.net/${tenantId}"
$AuthContext = [Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext]$authUrl

# Build the credential object and get the token form AAD
$credentials = New-Object Microsoft.IdentityModel.Clients.ActiveDirectory.ClientCredential -ArgumentList $appId,$key
$result = $AuthContext.AcquireToken("https://management.core.windows.net/",$credentials)
# Build the authorization header JSON object
$authHeader = @{
'Content-Type'='application/json'
'Authorization'=$result.CreateAuthorizationHeader()
}

When the authorization process is completed, you can call the purge REST API in the following way:

#Creates the API URI
$URI = "https://management.azure.com/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.Insights/components/${resourceName}/purge?api-version=2015-05-01"

$body = @"
{
   "table": "customEvents",
   "filters": [
     {
       "column": "timestamp",
       "operator": "<",
       "value": "2021-01-01T00:00:00.000"
     }
   ]
}
"@

#Invoke the REST API to purge the data on Application Insights
$purgeID=Invoke-RestMethod -Uri $URI -Method POST -Headers $authHeader -Body $body

As you can see, here I’m deleting the customEvents telemetry table for events with timestamp < 01/01/2021.

When the API is called, a purge task is started and you can see its ID:

Please remember that a purge task of Application Insights data is not immediatly executed. You can see the status of the task by sending a GET request as follows:

$uri="https://management.azure.com/subscriptions/$subscriptionId/resourceGroups/${resourceGroupName}/providers/Microsoft.Insights/components/${resourceName}/operations/$($purgeID.operationId)?api-version=2015-05-01"
Invoke-RestMethod -Uri $uri -Method GET -Headers $authHeader

As you can see, initially the status of the ongoing purge operation is pending:

and to be completed the task could require also some days (it has an heavy impact on the data platform used by Log Analytics). Microsoft has officially a SLA for purge operations set at 30 days (but on all my tests this was completed in 1 day), so act accordingly (don’t expect an immediate data deletion).

Please also remember this Microsoft’s official statement:

Purge is a highly privileged operation that no app or user in Azure (including even the resource owner) will have permissions to execute without explicitly being granted a role in Azure Resource Manager. This role is Data Purger and should be cautiously delegated due to the potential for data loss.

The complete script code is available on my Github here.