Handling the “HTTP 429 – Too Many Requests” error when calling external services

I think that many of you that creates integrations with external services could have the pleasure to know the HTTP 429 error: too many requests.

The HTTP 429 - Too Many Requests response status code indicates the user has sent too many requests in a given amount of time (rate limit) to a given endpoint (API or web service).

When you receive this error, normally a Retry-After header might be included to this response indicating how long to wait before making a new request.

Receiving an HTTP status 429 is not an error in the request, but it’s the other server that “kindly” asking you to please stop spamming requests. Obviously, your rate of requests has been too high and the server is not willing to accept this.

Many external services have a rate limit for incoming requests. Dynamics 365 Business Central APIs have the following rate limits:

ODataSOAP
Sandbox300 requests/min300 requests/min
Production600 requests/min600 requests/min

The maximum number of simultaneous OData or SOAP requests is actually 100.

For more details, you can check this link.

Why this blog post?

Because I received a message days ago from a big Microsoft partner in Italy that has local applications for retail shops (POS) and this application (installed locally on every shop floor) is calling Dynamics 365 Business Central APIs for sending data to the headquarter’s ERP in the cloud. On certain hours of the day, the HTTP 429 error appears, blocking the data transmission. Their solution works like in the following schema:

As you can imagine, if the number of clients increases, the Dynamics 365 Business Central API endpoints could be flooded of requests.

In this post I don’t want to talk about the architecture of this solution (not reliable and scalable for a cloud environment in my opinion, but this will be maybe another topic) but I want to cover only how to avoid this HTTP 429 error.

Their solution (C# application) does something like this (pseudo-code here):

string json = JsonConvert.SerializeObject(SalesOrder); 
var data = new StringContent(json, Encoding.UTF8, "application/json"); 
string url = "https://api.businesscentraldynamics.com/..."; 
using var client = new HttpClient(); 
var response = await client.PostAsync(url, data);

They send a SalesOrder JSON object in the body of a POST request to the Dynamics 365 Business Central API endpoint. This is ok. But what happens here? If the POST request fails for the HTTP 429 – Too many requests error, the transmission to the API endpoint of this data is interrupted (Exception). To better handling this situation, the solution should implement a Retry pattern for sending requests. A retry pattern enables an application to handle transient failures when it tries to connect to a service or network resource, by transparently retrying a failed operation. This can improve the entire stability of the application.

The Retry pattern can be described with the following diagram:

When applying this pattern to our scenario described above, we want that when the client (POS) calls the Dynamics 365 Business Central API, if the request fails for HTTP 429 error, a retry must be automatically executed (no interruption in the transmission flow):

A Retry pattern can be implemented on different ways. Here I want to propose a possible solution written in C#, that you can use on your applications, with support for asynchronous programming (async/await). This code implements a RetryHelper class that handles the retry logic for an API call:

public static class RetryHelper
    {
        public static async Task RetryOnExceptionAsync(int maxRetryAttempts, Func<Task> operation)
        {
            await RetryOnExceptionAsync<Exception>(maxRetryAttempts, operation);
        }

        public static async Task RetryOnExceptionAsync<TException>(int maxRetryAttempts, Func<Task> operation) where TException : Exception
        {
            if (maxRetryAttempts <= 0)
                throw new ArgumentOutOfRangeException(nameof(maxRetryAttempts));

            var retryattempts = 0;
            do
            {
                try
                {
                    retryattempts++;
                    await operation();
                    break;
                }
                catch (TException ex)
                {
                    if (retryattempts == maxRetryAttempts)
                        throw;

                    await CreateRetryDelayForException(maxRetryAttempts, retryattempts, ex);
                }
            } while (true);
        }

        private static Task CreateRetryDelayForException(int maxRetryAttempts, int attempts, Exception ex)
        {
            int delay = IncreasingDelayInSeconds(attempts);
            Console.WriteLine("Attempt {0} of {1} failed. New retry after {2} seconds.", attempts.ToString(), maxRetryAttempts.ToString(), delay.ToString());
            return Task.Delay(delay);
        }

        internal static int[] DelayPerAttemptInSeconds =
        {
            (int) TimeSpan.FromSeconds(5).TotalSeconds,
            (int) TimeSpan.FromSeconds(30).TotalSeconds,
            (int) TimeSpan.FromMinutes(3).TotalSeconds,
            (int) TimeSpan.FromMinutes(10).TotalSeconds,
            (int) TimeSpan.FromMinutes(30).TotalSeconds
        };

        static int IncreasingDelayInSeconds(int failedAttempts)
        {
            if (failedAttempts <= 0) throw new ArgumentOutOfRangeException();

            return failedAttempts >= DelayPerAttemptInSeconds.Length ? DelayPerAttemptInSeconds.Last() : DelayPerAttemptInSeconds[failedAttempts];
        }
    }

This class defines the number of attempts and the delay for every attempt in the internal DelayPerAttemptsInSeconds integer array (you can set the values you want here, in this sample I’m doing 5 retries after 5 seconds, 30 seconds, 3 minutes, 10 minutes and 30 minutes).

When you call an API endpoint (or an external web service) and an exception occours, the method that handles the retry logic is the RetryOnExceptionAsync function. The logic here calculates the time to wait before raising a retry and then starts a new API call. If all retries are failed, an exception is thrown (and you need to handle this).

The Main method calls the API endpoint by using a lambda expression and wrapping the action that we want to retry when failed. The code is as follows:

static async Task Main(string[] args)
        {
            var maxRetryAttempts = 5;
            var client = new HttpClient();
            string json = "{Here you should have a valid JSON object to POST}";
            
            try
            {
                await RetryHelper.RetryOnExceptionAsync<HttpRequestException>
                    (maxRetryAttempts, async () =>
                    {
                        var data = new StringContent(json, Encoding.UTF8, "application/json");
                        var response = await client.PostAsync("https://api.businesscentral.dynamics.com/...", data);
                        response.EnsureSuccessStatusCode();
                    });
            }
            catch(Exception ex)
            {
                //All retries here are failed.
                Console.WriteLine("Exception: " + ex.Message);
            }
        }

The POST HTTP request to the endpoint is executed for N times (with an increasing delay for each attempts). If all the attempts are failed, you should handle the exception and you’re forced to restart the transaction.

This is a code that you can absolutely improve, but it shows how you should handle the retry logic on your applications when calling external services. Your integrations will be more reliable if you apply it.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.