Erlang & TLS Termination

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Erlang & TLS Termination

Frank Muller
Hi guys

Would like to hear from your experience(s) on using Erlang with TLS/SSL. The default Erlang stack doesn't perform well, Google says.

Does anyone use Erlang in production behind:

. ???

Or directly using:
. gen_ssl: how to make it scale?
. ???

Some of the solutions above are fast, some scale well ... but all with downsides (ex. Hitch adds  200kiB overhead per connection).

We’re planning to go live with a fairly large number of secure connections: 45k - 75k. The sessions should be short (few seconds). Transferred data should range between 50KiB to 1miB max.

Insights and/or feedbacks are very welcome. 

/Frank

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang & TLS Termination

Fred Hebert-2
During my time at Heroku, we managed to make the Erlang TLS implementation (as a server) perform on par with Amazon's ELB stack -- it was something like 1-2ms (out of like 13-15ms) slower on the median case, but several seconds (if not minutes!) faster in the worst case, per connection, overall much more stable. We needed to patch a few things, but since around OTP-20, all of these have made it to upstream in OTP.

The type of configuration used is described in a gist at https://gist.github.com/ferd/af9abf6b3600d2d7f08dba58fdfb514a -- the format changed in OTP-21, but the most important parts are all there.

In short, it relies mostly on:
- good configuration of cache (mostly disabling a bunch of them when you don't need disk cache)
- good configuration of preferred cipher suites and ECCs (picking more complex ones than required slows things down -- i.e. picking a secp512 ECC instead of secp256 one iirc almost doubled the handshake time compared to AWS until we replicated their configurations)


On Wed, Sep 5, 2018 at 9:19 AM, Frank Muller <[hidden email]> wrote:
Hi guys

Would like to hear from your experience(s) on using Erlang with TLS/SSL. The default Erlang stack doesn't perform well, Google says.

Does anyone use Erlang in production behind:

. ???

Or directly using:
. gen_ssl: how to make it scale?
. ???

Some of the solutions above are fast, some scale well ... but all with downsides (ex. Hitch adds  200kiB overhead per connection).

We’re planning to go live with a fairly large number of secure connections: 45k - 75k. The sessions should be short (few seconds). Transferred data should range between 50KiB to 1miB max.

Insights and/or feedbacks are very welcome. 

/Frank

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang & TLS Termination

Frank Muller
Thanks Fred. Forgot to mention that I’ve already read your blog post about this. 

/Frank

<[hidden email]> a écrit :
During my time at Heroku, we managed to make the Erlang TLS implementation (as a server) perform on par with Amazon's ELB stack -- it was something like 1-2ms (out of like 13-15ms) slower on the median case, but several seconds (if not minutes!) faster in the worst case, per connection, overall much more stable. We needed to patch a few things, but since around OTP-20, all of these have made it to upstream in OTP.

The type of configuration used is described in a gist at https://gist.github.com/ferd/af9abf6b3600d2d7f08dba58fdfb514a -- the format changed in OTP-21, but the most important parts are all there.

In short, it relies mostly on:
- good configuration of cache (mostly disabling a bunch of them when you don't need disk cache)
- good configuration of preferred cipher suites and ECCs (picking more complex ones than required slows things down -- i.e. picking a secp512 ECC instead of secp256 one iirc almost doubled the handshake time compared to AWS until we replicated their configurations)


On Wed, Sep 5, 2018 at 9:19 AM, Frank Muller <[hidden email]> wrote:
Hi guys

Would like to hear from your experience(s) on using Erlang with TLS/SSL. The default Erlang stack doesn't perform well, Google says.

Does anyone use Erlang in production behind:

. ???

Or directly using:
. gen_ssl: how to make it scale?
. ???

Some of the solutions above are fast, some scale well ... but all with downsides (ex. Hitch adds  200kiB overhead per connection).

We’re planning to go live with a fairly large number of secure connections: 45k - 75k. The sessions should be short (few seconds). Transferred data should range between 50KiB to 1miB max.

Insights and/or feedbacks are very welcome. 

/Frank

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang & TLS Termination

Dave Cottlehuber-5
In reply to this post by Frank Muller
On Wed, 5 Sep 2018, at 15:19, Frank Muller wrote:
> Hi guys
>
> Would like to hear from your experience(s) on using Erlang with TLS/SSL.
> The default Erlang stack doesn't perform well, Google says.
>
> Does anyone use Erlang in production behind:
>
> . Hitch: https://hitch-tls.org/
> . Envoy: https://www.envoyproxy.io/
> . HAProxy: http://www.haproxy.org/
> . ???

I’ve used haproxy to great effect handling all those messy acme/let’s encrypt renewals and load balancing connections across multiple servers for plain https, rabbitmq and socketio traffic. Roughly 50-80k https txns  in and out per hour. The major win was matching up ibrowse connection settings with http1.1 pipelined connections to a 3rd party API. I think ferd’s tuning is a very similar outcome.

the most important things in hindsight were

- getting pipelining working to reuse TLS connections
- observabilty
- good logging
- predictable failure modes

I moved all services including DB and message brokers behind haproxy for these reasons.

You should look very carefully at service discovery, and http2 support, in all of the situations you intend to proxy, not all of these projects have the same level of support.

Developing your own heroku style LB would be interest g for a bigger organisation I think I needed something I could forget about.

A+
Dave







_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang & TLS Termination

Ingela Andin
There are always lots of trade offs between security and efficiency. And we can never make all of them for you.
Basic TLS Session handling I think is kind of flawed by design, we have tried to mitigate it. It seems for really powerful machines it is not enough and session table can be filled quicker than cleaned. We where thinking session tickets could be the solution and a workaround was to have a own session table callback as Fred suggested not saving any sessions. But TLS 1.3 deprecates both these ways and introduces a new way so TLS-1.2 session tickets are not so tempting any more. If people think they have no or a low benefit from the TLS-session tables maybe we could change it somehow.

Regards Ingela Erlang(OTP team - Ericsson AB

Den ons 5 sep. 20able18 kl 17:17 skrev Dave Cottlehuber <[hidden email]>:
On Wed, 5 Sep 2018, at 15:19, Frank Muller wrote:
> Hi guys
>
> Would like to hear from your experience(s) on using Erlang with TLS/SSL.
> The default Erlang stack doesn't perform well, Google says.
>
> Does anyone use Erlang in production behind:
>
> . ???

I’ve used haproxy to great effect handling all those messy acme/let’s encrypt renewals and load balancing connections across multiple servers for plain https, rabbitmq and socketio traffic. Roughly 50-80k https txns  in and out per hour. The major win was matching up ibrowse connection settings with http1.1 pipelined connections to a 3rd party API. I think ferd’s tuning is a very similar outcome.

the most important things in hindsight were

- getting pipelining working to reuse TLS connections
- observabilty
- good logging
- predictable failure modes

I moved all services including DB and message brokers behind haproxy for these reasons.

You should look very carefully at service discovery, and http2 support, in all of the situations you intend to proxy, not all of these projects have the same level of support.

Developing your own heroku style LB would be interest g for a bigger organisation I think I needed something I could forget about.

A+
Dave






_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions