Handling the “HTTP 429 – Too Many Requests” error when calling external services

I think that many of you that creates integrations with external services could have the pleasure to know the HTTP 429 error: too many requests.

The HTTP 429 - Too Many Requests response status code indicates the user has sent too many requests in a given amount of time (rate limit) to a given endpoint (API or web service).

When you receive this error, normally a Retry-After header might be included to this response indicating how long to wait before making a new request.

Receiving an HTTP status 429 is not an error in the request, but it’s the other server that “kindly” asking you to please stop spamming requests. Obviously, your rate of requests has been too high and the server is not willing to accept this.

Many external services have a rate limit for incoming requests. Dynamics 365 Business Central APIs have the following rate limits:

	OData	SOAP
Sandbox	300 requests/min	300 requests/min
Production	600 requests/min	600 requests/min

The maximum number of simultaneous OData or SOAP requests is actually 100.

For more details, you can check this link.

Why this blog post?

Because I received a message days ago from a big Microsoft partner in Italy that has local applications for retail shops (POS) and this application (installed locally on every shop floor) is calling Dynamics 365 Business Central APIs for sending data to the headquarter’s ERP in the cloud. On certain hours of the day, the HTTP 429 error appears, blocking the data transmission. Their solution works like in the following schema:

As you can imagine, if the number of clients increases, the Dynamics 365 Business Central API endpoints could be flooded of requests.

In this post I don’t want to talk about the architecture of this solution (not reliable and scalable for a cloud environment in my opinion, but this will be maybe another topic) but I want to cover only how to avoid this HTTP 429 error.

Their solution (C# application) does something like this (pseudo-code here):

string json = JsonConvert.SerializeObject(SalesOrder); 
var data = new StringContent(json, Encoding.UTF8, "application/json"); 
string url = "https://api.businesscentraldynamics.com/..."; 
using var client = new HttpClient(); 
var response = await client.PostAsync(url, data);

A Retry pattern can be implemented on different ways. Here I want to propose a possible solution written in C#, that you can use on your applications, with support for asynchronous programming (async/await). This code implements a RetryHelper class that handles the retry logic for an API call:

public static class RetryHelper
    {
        public static async Task RetryOnExceptionAsync(int maxRetryAttempts, Func<Task> operation)
        {
            await RetryOnExceptionAsync<Exception>(maxRetryAttempts, operation);
        }

        public static async Task RetryOnExceptionAsync<TException>(int maxRetryAttempts, Func<Task> operation) where TException : Exception
        {
            if (maxRetryAttempts <= 0)
                throw new ArgumentOutOfRangeException(nameof(maxRetryAttempts));

            var retryattempts = 0;
            do
            {
                try
                {
                    retryattempts++;
                    await operation();
                    break;
                }
                catch (TException ex)
                {
                    if (retryattempts == maxRetryAttempts)
                        throw;

                    await CreateRetryDelayForException(maxRetryAttempts, retryattempts, ex);
                }
            } while (true);
        }

        private static Task CreateRetryDelayForException(int maxRetryAttempts, int attempts, Exception ex)
        {
            int delay = IncreasingDelayInSeconds(attempts);
            Console.WriteLine("Attempt {0} of {1} failed. New retry after {2} seconds.", attempts.ToString(), maxRetryAttempts.ToString(), delay.ToString());
            return Task.Delay(delay);
        }

        internal static int[] DelayPerAttemptInSeconds =
        {
            (int) TimeSpan.FromSeconds(5).TotalSeconds,
            (int) TimeSpan.FromSeconds(30).TotalSeconds,
            (int) TimeSpan.FromMinutes(3).TotalSeconds,
            (int) TimeSpan.FromMinutes(10).TotalSeconds,
            (int) TimeSpan.FromMinutes(30).TotalSeconds
        };

        static int IncreasingDelayInSeconds(int failedAttempts)
        {
            if (failedAttempts <= 0) throw new ArgumentOutOfRangeException();

            return failedAttempts >= DelayPerAttemptInSeconds.Length ? DelayPerAttemptInSeconds.Last() : DelayPerAttemptInSeconds[failedAttempts];
        }
    }

The Main method calls the API endpoint by using a lambda expression and wrapping the action that we want to retry when failed. The code is as follows:

static async Task Main(string[] args)
        {
            var maxRetryAttempts = 5;
            var client = new HttpClient();
            string json = "{Here you should have a valid JSON object to POST}";
            
            try
            {
                await RetryHelper.RetryOnExceptionAsync<HttpRequestException>
                    (maxRetryAttempts, async () =>
                    {
                        var data = new StringContent(json, Encoding.UTF8, "application/json");
                        var response = await client.PostAsync("https://api.businesscentral.dynamics.com/...", data);
                        response.EnsureSuccessStatusCode();
                    });
            }
            catch(Exception ex)
            {
                //All retries here are failed.
                Console.WriteLine("Exception: " + ex.Message);
            }
        }

The POST HTTP request to the endpoint is executed for N times (with an increasing delay for each attempts). If all the attempts are failed, you should handle the exception and you’re forced to restart the transaction.

This is a code that you can absolutely improve, but it shows how you should handle the retry logic on your applications when calling external services. Your integrations will be more reliable if you apply it.