Timer Trigger Azure Functions: handling retry policies

In my last post about serverless architectures for decoupling integrations between Dynamics 365 Business Central and external applications, I’ve talked about the usage of queues and Azure Functions for exchanging messages between different cloud applications.

One of the quite common and simplest schema presented in the post was the following:

In this schema there’s a Timer Trigger Azure Function responsible for retrieving the top N messages from an Azure Queue, process them and then calling the required Dynamics 365 Business Central APIs. The Timer Trigger Azure Function usage is a possible approach to avoid API throttling in Dynamics 365 Business Central (you can process the incoming messages at the desired rate).

But if something fails during the message processing? You have two choices:

• process the messages at the next iteration of the timer
• handle retries

Starting with version 3.x of the Azure Functions runtime, you can define a retry policy for Timer and Event Hubs triggers that are enforced by the Functions runtime. The retry policy tells the runtime to rerun a failed execution until either successful completion occurs or the maximum number of retries is reached. This feature is in GA from July 2022.

You can define two types of retry policies:

• Fixed delay: fixed amount of time between each retry.
• Exponential backoff: the first retry waits for the minimum delay. On subsequent retries, time is added exponentially to the initial duration for each retry, until the maximum delay is reached.

You can also specify the maximum number of times function execution is retried before eventual failure. If you want an infinite retry, you can set this value to -1.

A retry policy is evaluated when a Timer or Event Hubs triggered function raises an uncaught exception. As a best practice, you should then catch all exceptions in your code and rethrow any errors that you want to result in a retry.

To show you an example on how this new policy works, let’s go into some details of the previous post and consider the part in red in the previous architecture diagram:

Here I have a function app that exposes two functions:

• an HTTP Trigger function for writing messages into an Azure Queue
• a Timer Trigger function for processing messages from the Azure Queue

The code of the HTTP Trigger function is as follows (simplified):

public static class HttpTriggerPostMessage
    {
        [FunctionName("PostMessage")]
        public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequest req,
            ILogger log)
        {
            log.LogInformation("C# HTTP trigger function processed a request.");

            string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
            dynamic data = JsonConvert.DeserializeObject(requestBody);
            string message = data?.message;

            if (!string.IsNullOrEmpty(message))
            {
                // Add a message to the output collection.
                string connectionString = "YOURSTORAGEACCOUNTCONNECTIONSTRING";
                string queueName = "d365bcqueue";
                QueueClient queue = new QueueClient(connectionString, queueName);
                queue.SendMessage(message);
            }
            return message != null
                ? (ActionResult)new OkObjectResult($"Message, {message} written to the queue.")
                : new BadRequestObjectResult("Please pass a message parameter in the request body");
        }
    }

This function takes a JSON object from the request body and saves it to the d365bcqueue Azure Queue.

The code of the Timer Trigger function (responsible for reading the messages from the Azure Queue, process them and performing the required actions in Dynamics 365 Business Central via APIs is defined as follows:

public class TimerTriggerGetMessages
    {
        [FunctionName("TimerTriggerGetMessages")]
        [FixedDelayRetry(-1, "00:00:10")]  //-1 for infinite retry
        public void Run([TimerTrigger("0 */1 * * * *")]TimerInfo myTimer, ILogger log)  //executes every 1 minute
        {
            log.LogInformation($"C# Timer trigger function executed at: {DateTime.Now}");

            string connectionString = "YOURSTORAGEACCOUNTCONNECTIONSTRING"; 
            string queueName = "d365bcqueue";
            QueueClient queue = new QueueClient(connectionString, queueName);            
            
            foreach(QueueMessage message in queue.ReceiveMessages(maxMessages: 10).Value)
            {
                //Here I throw a custom exception for showing the retry policy in action
                throw new Exception("Custom exception thrown");

                //Write your code here to process the messages
                //...

                //Call Dynamics 365 Business Central APIs to perform the required actions
                //...

                //Delete the message once it has been processed
                queue.DeleteMessage(message.MessageId, message.PopReceipt);
            }
        }
    }

Let’s check the Timer Trigger definition. It executes every 1 minute and I have define a FixedDelayRetry policy with infinite retry (-1) and 10 seconds on every retry as delay interval. I don’t recommend to use and indefinite retry on production honestly, so please use a fixed number.

In the function code, I’ve placed an exception into the piece of code that process a queue message. If the function fails, the retry policy should start and re-execute the trigger.

Let’s start by calling the HTTP trigger function and place two messages into the queue:

Now when the Timer Trigger function that polls the queue starts, the exception is thrown and the retry policy automatically starts. This is what I can see in the Application Insights instance connected to my function app:

As you can see, only the function that writes the messages into the queue is successfully executed, while the function that process the queue messages fails. In the Failures log, there are an indefinite set of retries.

The details of each exception shows that it’s exactly my custom exception:

If I remove the custom exception, the Timer Trigger function is correctly executed and queue messages are processed (no more retries):

Hoping that with this post you can understand how the new Retry Policy feature (personally I was waiting for it for a long time) can be helpful on your real world serverless projects. Exponential Backoff policy for example could be helpful if you want to handle HTTP Status Code 429 – Too Many Requests in Dynamics 365 Business Central APIs.

Please note that at the time of writing this post, retry policies are not supported when running in an isolated process (new Azure Function model).

I plan to write more posts about tips for improving your serverless architectures with Business Central, so stay tuned if you’re interested on this topic.