Erlang HTTP client libraries- pros/cons

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Erlang HTTP client libraries- pros/cons

Derek Brown-2
Looking for thoughts on the various Erlang HTTP client libraries, for a use case involving, say, 3000-5000 calls per second (from my server, as a client, to HTTP servers). I'm thinking of httpc, Gun, Hackney, any others... Performance, reliability, other pros/cons. Involves both GET and POST requests, with a hard roundtrip timeout of ~100 ms. Thanks for any thoughts.

Derek


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang HTTP client libraries- pros/cons

Paul Oliver-2
Hi Derek,

You could also take a look at https://github.com/puzza007/katipo (disclaimer - I'm the author). It uses libcurl/libevent, so is highly compatible and very fast (coming second in https://github.com/lpgauth/httpc_bench.

Cheers,
Paul.

On Thu, Aug 31, 2017 at 8:44 AM Derek Brown <[hidden email]> wrote:
Looking for thoughts on the various Erlang HTTP client libraries, for a use case involving, say, 3000-5000 calls per second (from my server, as a client, to HTTP servers). I'm thinking of httpc, Gun, Hackney, any others... Performance, reliability, other pros/cons. Involves both GET and POST requests, with a hard roundtrip timeout of ~100 ms. Thanks for any thoughts.

Derek

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang HTTP client libraries- pros/cons

Martin Karlsson-3
In reply to this post by Derek Brown-2
I have done some benchmarking for our specific use case.

* Main web server with high number of concurrent requests coming in
  (~1000 concurrent requests per second but benchmarked up to 10,000
  requests)
* Most requests requires a call out to an back-end HTTP server (only
  GET)
* There are few back-end endpoints (3).

It is not clear how many different kind of servers you are going to call
out to and this will likely affect your choice of library.

For our use case  I benchmarked httpc, gun, hackney, dlhttpc, ibrowse.

All of them behaves well when the endpoint replies quickly. However,
once the latency of the response goes up we have problems with all "traditional
clients". The only one that worked for us in this use case was dlhttpc
(https://github.com/ferd/dlhttpc) and this is likely because of the
dispcount "router".

I know hackney has experimental dispcount pooling so that would likely
work as well.

* httpc is known to have a few quirks. For our use case which only do HTTP
  get requests, no cookies, pipelines or other it works well but there is
  a gen_server bottleneck in there somewhere when calling only few
  endpoints (have not tried with many endpoints). The biggest pro is that
  it is built-in, but behaves a bit strange under load.

* gun - was discarded quickly as I couldn't get anywhere near the
  performance I was after and I didn't have the time to trouble-shoot
  with so many other alternatives.

* hackney and ibrowse - seemed like the most stable clients in term of
  response times. hackney is also used lots in the elixir community so
  it should have some good production usage. ibrowse has been around
  for a long time.

* dlhttpc - works best for us. Not updated in a while so can't say
  anything for more advanced usage. Again, this is becuase of dispcount
  routing, not the actual http client in it self.


Cheers
Martin
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang HTTP client libraries- pros/cons

Frank Muller
The above mentioned clients are also slow compared to "buoy":


Watch the author's video at:


/Frank

Le mer. 30 août 2017 à 23:47, Martin Karlsson <[hidden email]> wrote:
I have done some benchmarking for our specific use case.

* Main web server with high number of concurrent requests coming in
  (~1000 concurrent requests per second but benchmarked up to 10,000
  requests)
* Most requests requires a call out to an back-end HTTP server (only
  GET)
* There are few back-end endpoints (3).

It is not clear how many different kind of servers you are going to call
out to and this will likely affect your choice of library.

For our use case  I benchmarked httpc, gun, hackney, dlhttpc, ibrowse.

All of them behaves well when the endpoint replies quickly. However,
once the latency of the response goes up we have problems with all "traditional
clients". The only one that worked for us in this use case was dlhttpc
(https://github.com/ferd/dlhttpc) and this is likely because of the
dispcount "router".

I know hackney has experimental dispcount pooling so that would likely
work as well.

* httpc is known to have a few quirks. For our use case which only do HTTP
  get requests, no cookies, pipelines or other it works well but there is
  a gen_server bottleneck in there somewhere when calling only few
  endpoints (have not tried with many endpoints). The biggest pro is that
  it is built-in, but behaves a bit strange under load.

* gun - was discarded quickly as I couldn't get anywhere near the
  performance I was after and I didn't have the time to trouble-shoot
  with so many other alternatives.

* hackney and ibrowse - seemed like the most stable clients in term of
  response times. hackney is also used lots in the elixir community so
  it should have some good production usage. ibrowse has been around
  for a long time.

* dlhttpc - works best for us. Not updated in a while so can't say
  anything for more advanced usage. Again, this is becuase of dispcount
  routing, not the actual http client in it self.


Cheers
Martin
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang HTTP client libraries- pros/cons

Benoit Chesneau-2
hackney may also fit your needs. it tries to handle much of the specs. and http2/wbsockets are coming, release is planned on 25/09 since it had to be postponed due to some others priorities.

how slow it is, isrelative. most of the time your backend on the same host will accept less connections than you except anyway. Hence the need of pools. And most of the time the speed is more a matter of your application internals . hackney is used in prod since a long time on large applications.

There are some speedup coming in hackney when you disable some options. no figures yet. otherwise patches are also welcome.

If you have some questions feel free to contact me.

Benoit

On 31 August 2017 at 03:34:34, Frank Muller ([hidden email]) wrote:

The above mentioned clients are also slow compared to "buoy":


Watch the author's video at:


/Frank

Le mer. 30 août 2017 à 23:47, Martin Karlsson <[hidden email]> wrote:
I have done some benchmarking for our specific use case.

* Main web server with high number of concurrent requests coming in
  (~1000 concurrent requests per second but benchmarked up to 10,000
  requests)
* Most requests requires a call out to an back-end HTTP server (only
  GET)
* There are few back-end endpoints (3).

It is not clear how many different kind of servers you are going to call
out to and this will likely affect your choice of library.

For our use case  I benchmarked httpc, gun, hackney, dlhttpc, ibrowse.

All of them behaves well when the endpoint replies quickly. However,
once the latency of the response goes up we have problems with all "traditional
clients". The only one that worked for us in this use case was dlhttpc
(https://github.com/ferd/dlhttpc) and this is likely because of the
dispcount "router".

I know hackney has experimental dispcount pooling so that would likely
work as well.

* httpc is known to have a few quirks. For our use case which only do HTTP
  get requests, no cookies, pipelines or other it works well but there is
  a gen_server bottleneck in there somewhere when calling only few
  endpoints (have not tried with many endpoints). The biggest pro is that
  it is built-in, but behaves a bit strange under load.

* gun - was discarded quickly as I couldn't get anywhere near the
  performance I was after and I didn't have the time to trouble-shoot
  with so many other alternatives.

* hackney and ibrowse - seemed like the most stable clients in term of
  response times. hackney is also used lots in the elixir community so
  it should have some good production usage. ibrowse has been around
  for a long time.

* dlhttpc - works best for us. Not updated in a while so can't say
  anything for more advanced usage. Again, this is becuase of dispcount
  routing, not the actual http client in it self.


Cheers
Martin
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang HTTP client libraries- pros/cons

Max Lapshin-2
In reply to this post by Paul Oliver-2
I've looked into your katipo and I do not understand what for is it.

You launch curl as a separate program and communicate with it from erlang. What for?

Erlang has excellent network communication system and you have selected most inefficient way to fetch http.



We in Flussonic use lhttpc and it works for us.  Some fixes like adding timing and internal protocol support and it works for us.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang HTTP client libraries- pros/cons

Paul Oliver-2


On Thu, Aug 31, 2017 at 5:01 PM Max Lapshin <[hidden email]> wrote:
I've looked into your katipo and I do not understand what for is it.

It's a way to use libcurl from Erlang. You can find more information about libcurl here https://curl.haxx.se/libcurl/.
 

You launch curl as a separate program and communicate with it from erlang. What for?

HTTP compatibility (curl) and safety (port program).
 

Erlang has excellent network communication system and you have selected most inefficient way to fetch http.

As mentioned in my earlier email, the httpc_benchmark ranks katipo fairly well so any inefficiency of using an external port doesn't seem to be taking too much of a toll. I've considered using some sort of NIF but haven't found the need. Pull requests welcome though.
 



We in Flussonic use lhttpc and it works for us.  Some fixes like adding timing and internal protocol support and it works for us.



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang HTTP client libraries- pros/cons

Taras Halturin
I think, Max means that you choose most expensive way to deal with it and it's not about efficiency of http handling but about efficiency of aim achieving :)

On Thu, Aug 31, 2017 at 9:04 AM, Paul Oliver <[hidden email]> wrote:


On Thu, Aug 31, 2017 at 5:01 PM Max Lapshin <[hidden email]> wrote:
I've looked into your katipo and I do not understand what for is it.

It's a way to use libcurl from Erlang. You can find more information about libcurl here https://curl.haxx.se/libcurl/.
 

You launch curl as a separate program and communicate with it from erlang. What for?

HTTP compatibility (curl) and safety (port program).
 

Erlang has excellent network communication system and you have selected most inefficient way to fetch http.

As mentioned in my earlier email, the httpc_benchmark ranks katipo fairly well so any inefficiency of using an external port doesn't seem to be taking too much of a toll. I've considered using some sort of NIF but haven't found the need. Pull requests welcome though.
 



We in Flussonic use lhttpc and it works for us.  Some fixes like adding timing and internal protocol support and it works for us.



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions




--
Best Regards.
Taras Halturin

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang HTTP client libraries- pros/cons

Paul Oliver-2
On Thu, Aug 31, 2017 at 6:20 PM Taras Halturin <[hidden email]> wrote:
I think, Max means that you choose most expensive way to deal with it and it's not about efficiency of http handling but about efficiency of aim achieving :)


The most expensive way in terms of what? If not speed do you mean development effort? Given that the aim is to use libcurl then the choice is a port executable or some sort of NIF. When using a port executable I don't have to worry about it crashing my VM and all I pay is the price of port communications. If I use a NIF I have to concern myself with making sure my NIF code and the code in libcurl doesn't crash my VM. That's a lot more development time and risk.

Cheers,
Paul.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang HTTP client libraries- pros/cons

Loïc Hoguin-3
In reply to this post by Derek Brown-2
On 08/30/2017 10:30 PM, Derek Brown wrote:
> Looking for thoughts on the various Erlang HTTP client libraries, for a
> use case involving, say, 3000-5000 calls per second (from my server, as
> a client, to HTTP servers). I'm thinking of httpc, Gun, Hackney, any
> others... Performance, reliability, other pros/cons. Involves both GET
> and POST requests, with a hard roundtrip timeout of ~100 ms. Thanks for
> any thoughts.

Gun is not a general purpose client, it is made exclusively for long
running connections (for example, connect when the node starts, and stay
connected forever). It requires a lot more setup efforts than others
because of this (pooling, circuit breakers and other are for you to do).
Sameroom.io is an example service using it.

Regardless of clients, I would recommend using HTTP/2 as much as
possible if performance is a concern and you're doing more than one
request per connection. The gains are really significant.

--
Loïc Hoguin
https://ninenines.eu
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang HTTP client libraries- pros/cons

Max Lapshin-2
In reply to this post by Paul Oliver-2
HTTP client over external port is the most expensive way in all terms:

1) programming. It is REALLY hard to debug it.  Was it launched under valgrind? If no, then there are at least 5 horrible memory leaks and memory corruptions per each screen of code that haven't been discovered yet

2) deploying. Deploying NIF is a pain because you need to have build farm for each architecture that you target to.  For example, we deploy flussonic under suse 10, debian 7, debian8/ubuntu16, arm7, arm8, windows, elbrus 2k.   All these platforms are different and you cannot rely on cross compiler. Good luck with building repeatable infrastructure for compiling under windows.

If you have erlang code, you can compile under mac and launch under windows because guys from OTP team have already done all dirty job. Just read their manual and follow it.

3) speed. It will be slow in all terms. High latency due to multiple OS process scheduling: read in one process, then write to pipe and send further.  Do you think that linux pipe is a "good and optimized" thing? It is not.
What if we speak about high traffic?  1 gigabit of input will become several gigabits of _useless_ copying data.




So I do not understand what can give libcurl that cannot be achieved in plain erlang.  It is definitely not about high traffic speed because plain vanilla lhttpc can download 10 Gbit/s over fiber without any extra modifications.


On Thu, Aug 31, 2017 at 10:22 AM, Paul Oliver <[hidden email]> wrote:
On Thu, Aug 31, 2017 at 6:20 PM Taras Halturin <[hidden email]> wrote:
I think, Max means that you choose most expensive way to deal with it and it's not about efficiency of http handling but about efficiency of aim achieving :)


The most expensive way in terms of what? If not speed do you mean development effort? Given that the aim is to use libcurl then the choice is a port executable or some sort of NIF. When using a port executable I don't have to worry about it crashing my VM and all I pay is the price of port communications. If I use a NIF I have to concern myself with making sure my NIF code and the code in libcurl doesn't crash my VM. That's a lot more development time and risk.

Cheers,
Paul.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang HTTP client libraries- pros/cons

Paul Oliver-2


On Sat, Sep 2, 2017 at 5:22 PM Max Lapshin <[hidden email]> wrote:
HTTP client over external port is the most expensive way in all terms:

1) programming. It is REALLY hard to debug it.  Was it launched under valgrind? If no, then there are at least 5 horrible memory leaks and memory corruptions per each screen of code that haven't been discovered yet

And yet you wrote https://github.com/maxlapshin/csv_reader (which I use btw). Running port executables protects the VM from any bugs in the C code, at least. It's almost like Erlang in that regard! Valgrind was used during Katipo's development.
 

2) deploying. Deploying NIF is a pain because you need to have build farm for each architecture that you target to.  For example, we deploy flussonic under suse 10, debian 7, debian8/ubuntu16, arm7, arm8, windows, elbrus 2k.   All these platforms are different and you cannot rely on cross compiler. Good luck with building repeatable infrastructure for compiling under windows.

If you have erlang code, you can compile under mac and launch under windows because guys from OTP team have already done all dirty job. Just read their manual and follow it.

This I will concede can be an issue with NIFs. It's not an issue for my use case, however, and can be overcome by using things like Docker.
 

3) speed. It will be slow in all terms. High latency due to multiple OS process scheduling: read in one process, then write to pipe and send further.  Do you think that linux pipe is a "good and optimized" thing? It is not.
What if we speak about high traffic?  1 gigabit of input will become several gigabits of _useless_ copying data.


So I do not understand what can give libcurl that cannot be achieved in plain erlang.  It is definitely not about high traffic speed because plain vanilla lhttpc can download 10 Gbit/s over fiber without any extra modifications.


I think perhaps you have tunnel vision based on your specific HTTP use case. I'm not sure what you mean by "in all terms". In https://github.com/lpgauth/httpc_bench (d)lhttpc peaks at 93178 reqs/s whereas Katipo peaks at 107900. You may not require the HTTP compatibility that *22K* commits to libcurl provides. Your custom version of lhttpc may well be able to sustain 10Gbit/s over fiber - that is not my use case so I've no idea if Katipo could do the same.

 

On Thu, Aug 31, 2017 at 10:22 AM, Paul Oliver <[hidden email]> wrote:
On Thu, Aug 31, 2017 at 6:20 PM Taras Halturin <[hidden email]> wrote:
I think, Max means that you choose most expensive way to deal with it and it's not about efficiency of http handling but about efficiency of aim achieving :)


The most expensive way in terms of what? If not speed do you mean development effort? Given that the aim is to use libcurl then the choice is a port executable or some sort of NIF. When using a port executable I don't have to worry about it crashing my VM and all I pay is the price of port communications. If I use a NIF I have to concern myself with making sure my NIF code and the code in libcurl doesn't crash my VM. That's a lot more development time and risk.

Cheers,
Paul.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang HTTP client libraries- pros/cons

scott ribe
In reply to this post by Max Lapshin-2
On Sep 1, 2017, at 11:22 PM, Max Lapshin <[hidden email]> wrote:
>
> So I do not understand what can give libcurl that cannot be achieved in plain erlang.  It is definitely not about high traffic speed because plain vanilla lhttpc can download 10 Gbit/s over fiber without any extra modifications.

From fast servers. It bogs down with lots of connections, such that when pointed at a wide selection of real-world servers, it's quite slow.

--
Scott Ribe
[hidden email]
(303) 722-0567

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang HTTP client libraries- pros/cons

scott ribe
On Sep 2, 2017, at 5:21 AM, scott ribe <[hidden email]> wrote:
>
> On Sep 1, 2017, at 11:22 PM, Max Lapshin <[hidden email]> wrote:
>>
>> So I do not understand what can give libcurl that cannot be achieved in plain erlang.  It is definitely not about high traffic speed because plain vanilla lhttpc can download 10 Gbit/s over fiber without any extra modifications.
>
> From fast servers. It bogs down with lots of connections, such that when pointed at a wide selection of real-world servers, it's quite slow.

Replying to myself... It's 5AM here, and I was talking about httpc, not lhttpc.

--
Scott Ribe
[hidden email]
(303) 722-0567

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions