Throttling HTTP requests

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Throttling HTTP requests

Frank Muller
Hi guys

We have an HTTP web service based Cowboy which is receiving more and more traffic.

We would like to implement a throttling mechanism per IP address to limit how many request/sec the same IP address can make.

Is there any library out there for that?
Or any (simple yet efficient) algorithm to implement?

Ideas & feedabacks are very welcome. 

Thanks in advance.

/Frank

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Throttling HTTP requests

Tobias Schlager-2
Hi Frank,

I've implemented a simple form of rate limiting in this [1] little project (module http2smtp_rate). The module is probably not what you need but the idea behind it, could be used as a starting point. The basic idea is the following:

Everytime a request comes in, the request process makes a gen_server call to this server (you could also spawn the limiter processes per source IP address for sharding). This request will increment and return a counter (specific for this IP address). If the counter exceeds the limit, the request gets terminated immediately. The server has an internal timer that fires once per desired interval and resets all counters.

Regards
Tobias

[1] https://github.com/lindenbaum/http2smtp


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Throttling HTTP requests

Dmitry Kolesnikov-2
In reply to this post by Frank Muller
Hi,

Global throttling might be challenging. You would need to run a proxy and define throttling policies there. I would discard this option unless you are talking about large service.

Let’s consider local throttling policies, where each api nodes maintain counters independently each other.
It was not clear do you want to reject request or put it into queue so that customers are still served but they latency grows.

There are few easy solutions, which does not require any Erlang
* Nginx
* Linux traffic shaping

If you keen to implement this feature in Erlang then please consider a circuit breaker pattern. You can create a fuse per IP address or sub network to reject a traffic.

Best Regards,
Dmitry


> On 21 Sep 2018, at 9.40, Frank Muller <[hidden email]> wrote:
>
> Hi guys
>
> We have an HTTP web service based Cowboy which is receiving more and more traffic.
>
> We would like to implement a throttling mechanism per IP address to limit how many request/sec the same IP address can make.
>
> Is there any library out there for that?
> Or any (simple yet efficient) algorithm to implement?
>
> Ideas & feedabacks are very welcome.
>
> Thanks in advance.
>
> /Frank
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Throttling HTTP requests

Roger Lipscombe-2
In reply to this post by Frank Muller
I've not tried, but it looks like you could use https://github.com/uwiger/jobs and write some cowboy middleware that returns 429.

Or do what we do: since our HTTP-stuff is fronted by nginx (for SSL termination and upstream service routing), we used https://www.nginx.com/blog/rate-limiting-nginx/

On 21 September 2018 at 07:40, Frank Muller <[hidden email]> wrote:
Hi guys

We have an HTTP web service based Cowboy which is receiving more and more traffic.

We would like to implement a throttling mechanism per IP address to limit how many request/sec the same IP address can make.

Is there any library out there for that?
Or any (simple yet efficient) algorithm to implement?

Ideas & feedabacks are very welcome. 

Thanks in advance.

/Frank

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Throttling HTTP requests

Karolis Petrauskas-2
I was trying to use https://github.com/uwiger/jobs for rate limiting in my recent project.
With the spike pattern, this library was not performing. I had to accept 50k requests in ~10-20s, queue them and then handle them at a limited rate (10tps).
The rate was limited way more, than configured (1tps instead of 10tps, etc).
Then I switched to a custom gen_server with timer:sleep/1 and queue in the process inbox.

Karolis

On Fri, Sep 21, 2018 at 10:19 AM Roger Lipscombe <[hidden email]> wrote:
I've not tried, but it looks like you could use https://github.com/uwiger/jobs and write some cowboy middleware that returns 429.

Or do what we do: since our HTTP-stuff is fronted by nginx (for SSL termination and upstream service routing), we used https://www.nginx.com/blog/rate-limiting-nginx/

On 21 September 2018 at 07:40, Frank Muller <[hidden email]> wrote:
Hi guys

We have an HTTP web service based Cowboy which is receiving more and more traffic.

We would like to implement a throttling mechanism per IP address to limit how many request/sec the same IP address can make.

Is there any library out there for that?
Or any (simple yet efficient) algorithm to implement?

Ideas & feedabacks are very welcome. 

Thanks in advance.

/Frank

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Throttling HTTP requests

Gerhard Lazu
In reply to this post by Frank Muller
Last time I had to do something similar I went with Cloudflare. It's been a few years now and it's still the cheaper option with no management overhad. If delegating this functionality to a SaaS is an option for you, it's worth considering.
On 21 Sep 2018, 07:40 +0100, Frank Muller <[hidden email]>, wrote:
Hi guys

We have an HTTP web service based Cowboy which is receiving more and more traffic.

We would like to implement a throttling mechanism per IP address to limit how many request/sec the same IP address can make.

Is there any library out there for that?
Or any (simple yet efficient) algorithm to implement?

Ideas & feedabacks are very welcome. 

Thanks in advance.

/Frank
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Throttling HTTP requests

Ruan Pienaar
In reply to this post by Frank Muller
spent a few hours writing a erlang library for rate limiting within the vm.

On Fri, Sep 21, 2018 at 7:40 AM Frank Muller <[hidden email]> wrote:
Hi guys

We have an HTTP web service based Cowboy which is receiving more and more traffic.

We would like to implement a throttling mechanism per IP address to limit how many request/sec the same IP address can make.

Is there any library out there for that?
Or any (simple yet efficient) algorithm to implement?

Ideas & feedabacks are very welcome. 

Thanks in advance.

/Frank
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


--
Erlang Solutions cares about your data and privacy; please find all details about the basis for communicating with you and the way we process your data in our Privacy Policy.You can update your email preferences or opt-out from receiving Marketing emails here.

Ruan Pienaar
+44 7500 708 814

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Throttling HTTP requests

Paul Peregud-2

On Fri, Sep 21, 2018 at 9:14 AM Ruan Pienaar <[hidden email]> wrote:
spent a few hours writing a erlang library for rate limiting within the vm.

On Fri, Sep 21, 2018 at 7:40 AM Frank Muller <[hidden email]> wrote:
Hi guys

We have an HTTP web service based Cowboy which is receiving more and more traffic.

We would like to implement a throttling mechanism per IP address to limit how many request/sec the same IP address can make.

Is there any library out there for that?
Or any (simple yet efficient) algorithm to implement?

Ideas & feedabacks are very welcome. 

Thanks in advance.

/Frank
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


--
Erlang Solutions cares about your data and privacy; please find all details about the basis for communicating with you and the way we process your data in our Privacy Policy.You can update your email preferences or opt-out from receiving Marketing emails here.

Ruan Pienaar
+44 7500 708 814
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


--
Best regards,
Paul Peregud
+48602112091

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Throttling HTTP requests

Fred Hebert-2
In reply to this post by Frank Muller
The tricky question is about how much overload you might be expecting. The thing is that accepting an HTTP request and parsing it in order to know where it goes is a relatively costly operation (compared to just paying $2, getting a few thousand AWS instances, and throwing pre-built garbage queries at a server somewhere).

If what you're trying to do is absorb and track HTTP traffic to put a cap on even costlier operations at the back-end (or on servers upstream) such as costly processing or DB queries, then you don't necessarily need to be super fast; you just need to be cheaper than the big costly operations you're trying to protect against.

There are various strategies there depending on how slow you can afford to be, many of which have been described here:
  • using a gen_server to track and count (riskier on large servers as it may represent a single point of contention)
  • Jobs, which can be augmented with a lot of probes
  • Locks taken and handled with ETS
  • Circuit breakers to look at various metrics to force input to stall when becoming unstable
I've written a blog post that tracks most mechanism available for overload handling and flow control there: https://ferd.ca/handling-overload.html

And at Heroku, we also developed a locking library (also in the blog post above) named canal_lock (https://github.com/heroku/canal_lock) which was aimed at being able to do lock-management for backend services where the number of available nodes changes and adapts over time. Still, if you can afford it, circuit breakers or frameworks like jobs do a good there since they can adapt with a lot of local-only information (and approaches such as https://en.wikipedia.org/wiki/Additive_increase/multiplicative_decrease for self-regulation. Also look at CoDel and other algorithms)

But all of this won't necessarily go far if what you have is overload traffic that is too costly to handle directly in terms of "parsing all the HTTP Requests is already too much CPU". Then you need to move to cheaper means: track connections or per-domain traffic through SNI data rather than plain HTTP requests, and impose limits there. Turn on options such as TCP_DEFER_ACCEPT which tell the kernel not to return you an 'accepted' socket until it has data waiting on the line, which saves on context switching and prevents idle connections from going up to your app (but won't do much if you're using TLS or the PROXY protocol since they expect to send info down the line right away).

When that won't work, you'll need to go down to the OS-level. You can tweak the size of ACK and SYN queues for indiscriminate filtering, or use packet filtering rules where the kernel itself can do quick inspection and dropping of data you cannot accept based on some data they contain. If you're at that level though, you may need a second NIC that is privately accessible just in order to be able to go set and modify these rules while the public NIC is under stress. If that doesn't work, then you need to look at a broader scope in terms of what your data center can do, but I figure this gets far from what you needed here.

In any case, the critical bit is that your system can be modeled as a queuing system; if you want to be able to take control over its stability, you must have the ability to handle validation and expulsion of overflow tasks faster than they can arrive. If you can't do that, you will not control what it is that you drop in terms of overflow. Any point of asynchrony in your system can act as an open valve where a task is reported to go fast when it goes sit into a growing queue down somewhere, so you have to be careful and make sure that your control and filtering at the edge reflects your actual bottlenecks and loads down the line.

On Fri, Sep 21, 2018 at 2:40 AM Frank Muller <[hidden email]> wrote:
Hi guys

We have an HTTP web service based Cowboy which is receiving more and more traffic.

We would like to implement a throttling mechanism per IP address to limit how many request/sec the same IP address can make.

Is there any library out there for that?
Or any (simple yet efficient) algorithm to implement?

Ideas & feedabacks are very welcome. 

Thanks in advance.

/Frank
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Throttling HTTP requests

Guilherme Andrade
In reply to this post by Frank Muller
Hello Frank,

If you're looking at imposing some amount of fairness besides a rate limit, aequitas might be useful for you:

https://github.com/g-andrade/aequitas

It attempts to detect outliers in consumers of resources and throttle them (in your use case, these could be IP addresses performing an unusually large number of requests.) It can also enforce a collective rate limit.

On Fri, 21 Sep 2018 at 07:40, Frank Muller <[hidden email]> wrote:
Hi guys

We have an HTTP web service based Cowboy which is receiving more and more traffic.

We would like to implement a throttling mechanism per IP address to limit how many request/sec the same IP address can make.

Is there any library out there for that?
Or any (simple yet efficient) algorithm to implement?

Ideas & feedabacks are very welcome. 

Thanks in advance.

/Frank
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


--
Guilherme

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions