Memory leak in SSL

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory leak in SSL

Max Treskin
Hello,

I have two different HTTPS-servers (different purposes, code, etc) built on top of cowboy/ranch, and both have memory leaks.
ETS table server_ssl_otp_session_cache has millions of records just after hours of work and consumes gigabytes of RAM.
What should I do to eliminate this? Is it bug or intended behaviour?

Thanks



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Memory leak in SSL

Dmitry Kolesnikov-2
Hello,

Have you tried to disable reuse of ssl sessions?
{ssl_options, [{reuse_sessions, false}]}

And reduce the session time
-ssl session_lifetime 120

Best Regards,
Dmitry

> On 8 Aug 2017, at 11.59, Max Treskin <[hidden email]> wrote:
>
> Hello,
>
> I have two different HTTPS-servers (different purposes, code, etc) built on top of cowboy/ranch, and both have memory leaks.
> ETS table server_ssl_otp_session_cache has millions of records just after hours of work and consumes gigabytes of RAM.
> What should I do to eliminate this? Is it bug or intended behaviour?
>
> Thanks
>
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Memory leak in SSL

Ingela Andin
Hi!

Depending on the version of the ssl application you may also set these values. I guess you have an older version as they default to 1000.

From doc:

"session_cache_client_max = integer() <optional>

Limits the growth of the clients session cache, that is how many sessions towards servers that are cached to be used by new client connections. If the maximum number of sessions is reached, the current cache entries will be invalidated regardless of their remaining lifetime. Defaults to 1000.

session_cache_server_max = integer() <optional>

Limits the growth of the servers session cache, that is how many client sessions are cached by the server. If the maximum number of sessions is reached, the current cache entries will be invalidated regardless of their remaining lifetime. Defaults to 1000."


Regards Ingela Erlang/OTP Team - Ericsson AB




2017-08-08 11:25 GMT+02:00 Dmitry Kolesnikov <[hidden email]>:
Hello,

Have you tried to disable reuse of ssl sessions?
{ssl_options, [{reuse_sessions, false}]}

And reduce the session time
-ssl session_lifetime 120

Best Regards,
Dmitry

> On 8 Aug 2017, at 11.59, Max Treskin <[hidden email]> wrote:
>
> Hello,
>
> I have two different HTTPS-servers (different purposes, code, etc) built on top of cowboy/ranch, and both have memory leaks.
> ETS table server_ssl_otp_session_cache has millions of records just after hours of work and consumes gigabytes of RAM.
> What should I do to eliminate this? Is it bug or intended behaviour?
>
> Thanks
>
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Memory leak in SSL

Max Treskin
In reply to this post by Dmitry Kolesnikov-2
Thank you Dmitry! It works


On 8 August 2017 at 12:25:12, Dmitry Kolesnikov ([hidden email]) wrote:

Hello,

Have you tried to disable reuse of ssl sessions?
{ssl_options, [{reuse_sessions, false}]}

And reduce the session time
-ssl session_lifetime 120

Best Regards,
Dmitry

> On 8 Aug 2017, at 11.59, Max Treskin <[hidden email]> wrote:
>
> Hello,
>
> I have two different HTTPS-servers (different purposes, code, etc) built on top of cowboy/ranch, and both have memory leaks.
> ETS table server_ssl_otp_session_cache has millions of records just after hours of work and consumes gigabytes of RAM.
> What should I do to eliminate this? Is it bug or intended behaviour?
>
> Thanks
>
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Memory leak in SSL

Danil Zagoskin-2
In reply to this post by Ingela Andin
Hi Ingela!

We've hit presumably the same bug on OTP 20.
That's what I found:
  1. This patch https://github.com/erlang/otp/commit/256e01ce80b3aadd63f303b9bda5722ad313220f
     -- we start invalidation only on Size == Max
  2. ([hidden email])33> (sys:get_state(ssl_manager))#state.session_cache_server_max.
     > 1000
  3. ([hidden email])37> redbug:start("ssl_session_cache:size/1 -> return", [{msgs, 2}]).
     ...
     ssl_session_cache:size/1 -> 8610379

Seems like on high load if we accept some new sessions while session_validation does its job, cache size limiting stops working.

On Tue, Aug 8, 2017 at 4:55 PM, Ingela Andin <[hidden email]> wrote:
Hi!

Depending on the version of the ssl application you may also set these values. I guess you have an older version as they default to 1000.

From doc:

"session_cache_client_max = integer() <optional>

Limits the growth of the clients session cache, that is how many sessions towards servers that are cached to be used by new client connections. If the maximum number of sessions is reached, the current cache entries will be invalidated regardless of their remaining lifetime. Defaults to 1000.

session_cache_server_max = integer() <optional>

Limits the growth of the servers session cache, that is how many client sessions are cached by the server. If the maximum number of sessions is reached, the current cache entries will be invalidated regardless of their remaining lifetime. Defaults to 1000."


Regards Ingela Erlang/OTP Team - Ericsson AB




2017-08-08 11:25 GMT+02:00 Dmitry Kolesnikov <[hidden email]>:
Hello,

Have you tried to disable reuse of ssl sessions?
{ssl_options, [{reuse_sessions, false}]}

And reduce the session time
-ssl session_lifetime 120

Best Regards,
Dmitry

> On 8 Aug 2017, at 11.59, Max Treskin <[hidden email]> wrote:
>
> Hello,
>
> I have two different HTTPS-servers (different purposes, code, etc) built on top of cowboy/ranch, and both have memory leaks.
> ETS table server_ssl_otp_session_cache has millions of records just after hours of work and consumes gigabytes of RAM.
> What should I do to eliminate this? Is it bug or intended behaviour?
>
> Thanks
>
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions




--
Danil Zagoskin | [hidden email]

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Memory leak in SSL

Ingela Andin
Hi!

2017-08-21 13:36 GMT+02:00 Danil Zagoskin <[hidden email]>:
Hi Ingela!

We've hit presumably the same bug on OTP 20.
That's what I found:
  1. This patch https://github.com/erlang/otp/commit/256e01ce80b3aadd63f303b9bda5722ad313220f
     -- we start invalidation only on Size == Max
  2. ([hidden email])33> (sys:get_state(ssl_manager))#state.session_cache_server_max.
     > 1000
  3. ([hidden email])37> redbug:start("ssl_session_cache:size/1 -> return", [{msgs, 2}]).
     ...
     ssl_session_cache:size/1 -> 8610379

Seems like on high load if we accept some new sessions while session_validation does its job, cache size limiting stops working.


Humm... maybe something was changed, that changed the criteria breaking the  assumption, let me ponder that some more.
Would it work for you if the test was changed to Size >= Max ?

Regards Ingela Erlang/OTP Team - Ericsson AB

 
On Tue, Aug 8, 2017 at 4:55 PM, Ingela Andin <[hidden email]> wrote:
Hi!

Depending on the version of the ssl application you may also set these values. I guess you have an older version as they default to 1000.

From doc:

"session_cache_client_max = integer() <optional>

Limits the growth of the clients session cache, that is how many sessions towards servers that are cached to be used by new client connections. If the maximum number of sessions is reached, the current cache entries will be invalidated regardless of their remaining lifetime. Defaults to 1000.

session_cache_server_max = integer() <optional>

Limits the growth of the servers session cache, that is how many client sessions are cached by the server. If the maximum number of sessions is reached, the current cache entries will be invalidated regardless of their remaining lifetime. Defaults to 1000."


Regards Ingela Erlang/OTP Team - Ericsson AB




2017-08-08 11:25 GMT+02:00 Dmitry Kolesnikov <[hidden email]>:
Hello,

Have you tried to disable reuse of ssl sessions?
{ssl_options, [{reuse_sessions, false}]}

And reduce the session time
-ssl session_lifetime 120

Best Regards,
Dmitry

> On 8 Aug 2017, at 11.59, Max Treskin <[hidden email]> wrote:
>
> Hello,
>
> I have two different HTTPS-servers (different purposes, code, etc) built on top of cowboy/ranch, and both have memory leaks.
> ETS table server_ssl_otp_session_cache has millions of records just after hours of work and consumes gigabytes of RAM.
> What should I do to eliminate this? Is it bug or intended behaviour?
>
> Thanks
>
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions




--
Danil Zagoskin | [hidden email]


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Memory leak in SSL

Danil Zagoskin-2
Would it work for you if the test was changed to Size >= Max ?

Yes. Yesterday I hot-loaded ssl_manager on affected system. Cache size slowly decreased to 1000 and stays in a range of 200-1100.

On Mon, Aug 21, 2017 at 4:23 PM, Ingela Andin <[hidden email]> wrote:
Hi!

2017-08-21 13:36 GMT+02:00 Danil Zagoskin <[hidden email]>:
Hi Ingela!

We've hit presumably the same bug on OTP 20.
That's what I found:
  1. This patch https://github.com/erlang/otp/commit/256e01ce80b3aadd63f303b9bda5722ad313220f
     -- we start invalidation only on Size == Max
  2. ([hidden email])33> (sys:get_state(ssl_manager))#state.session_cache_server_max.
     > 1000
  3. ([hidden email])37> redbug:start("ssl_session_cache:size/1 -> return", [{msgs, 2}]).
     ...
     ssl_session_cache:size/1 -> 8610379

Seems like on high load if we accept some new sessions while session_validation does its job, cache size limiting stops working.


Humm... maybe something was changed, that changed the criteria breaking the  assumption, let me ponder that some more.
Would it work for you if the test was changed to Size >= Max ?

Regards Ingela Erlang/OTP Team - Ericsson AB

 
On Tue, Aug 8, 2017 at 4:55 PM, Ingela Andin <[hidden email]> wrote:
Hi!

Depending on the version of the ssl application you may also set these values. I guess you have an older version as they default to 1000.

From doc:

"session_cache_client_max = integer() <optional>

Limits the growth of the clients session cache, that is how many sessions towards servers that are cached to be used by new client connections. If the maximum number of sessions is reached, the current cache entries will be invalidated regardless of their remaining lifetime. Defaults to 1000.

session_cache_server_max = integer() <optional>

Limits the growth of the servers session cache, that is how many client sessions are cached by the server. If the maximum number of sessions is reached, the current cache entries will be invalidated regardless of their remaining lifetime. Defaults to 1000."


Regards Ingela Erlang/OTP Team - Ericsson AB




2017-08-08 11:25 GMT+02:00 Dmitry Kolesnikov <[hidden email]>:
Hello,

Have you tried to disable reuse of ssl sessions?
{ssl_options, [{reuse_sessions, false}]}

And reduce the session time
-ssl session_lifetime 120

Best Regards,
Dmitry

> On 8 Aug 2017, at 11.59, Max Treskin <[hidden email]> wrote:
>
> Hello,
>
> I have two different HTTPS-servers (different purposes, code, etc) built on top of cowboy/ranch, and both have memory leaks.
> ETS table server_ssl_otp_session_cache has millions of records just after hours of work and consumes gigabytes of RAM.
> What should I do to eliminate this? Is it bug or intended behaviour?
>
> Thanks
>
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions




--
Danil Zagoskin | [hidden email]




--
Danil Zagoskin | [hidden email]

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Memory leak in SSL

Ingela Andin
Hi!

2017-08-22 12:50 GMT+02:00 Danil Zagoskin <[hidden email]>:
Would it work for you if the test was changed to Size >= Max ?

Yes. Yesterday I hot-loaded ssl_manager on affected system. Cache size slowly decreased to 1000 and stays in a range of 200-1100.


And it now makes sense to me, I have included the fix in our builds. It probably will fix a wobbling test case.  Scheduled for inclusion in
OTP 20.1

Regards Ingela Erlang/OTP team - Ericsson AB

 
On Mon, Aug 21, 2017 at 4:23 PM, Ingela Andin <[hidden email]> wrote:
Hi!

2017-08-21 13:36 GMT+02:00 Danil Zagoskin <[hidden email]>:
Hi Ingela!

We've hit presumably the same bug on OTP 20.
That's what I found:
  1. This patch https://github.com/erlang/otp/commit/256e01ce80b3aadd63f303b9bda5722ad313220f
     -- we start invalidation only on Size == Max
  2. ([hidden email])33> (sys:get_state(ssl_manager))#state.session_cache_server_max.
     > 1000
  3. ([hidden email])37> redbug:start("ssl_session_cache:size/1 -> return", [{msgs, 2}]).
     ...
     ssl_session_cache:size/1 -> 8610379

Seems like on high load if we accept some new sessions while session_validation does its job, cache size limiting stops working.


Humm... maybe something was changed, that changed the criteria breaking the  assumption, let me ponder that some more.
Would it work for you if the test was changed to Size >= Max ?

Regards Ingela Erlang/OTP Team - Ericsson AB

 
On Tue, Aug 8, 2017 at 4:55 PM, Ingela Andin <[hidden email]> wrote:
Hi!

Depending on the version of the ssl application you may also set these values. I guess you have an older version as they default to 1000.

From doc:

"session_cache_client_max = integer() <optional>

Limits the growth of the clients session cache, that is how many sessions towards servers that are cached to be used by new client connections. If the maximum number of sessions is reached, the current cache entries will be invalidated regardless of their remaining lifetime. Defaults to 1000.

session_cache_server_max = integer() <optional>

Limits the growth of the servers session cache, that is how many client sessions are cached by the server. If the maximum number of sessions is reached, the current cache entries will be invalidated regardless of their remaining lifetime. Defaults to 1000."


Regards Ingela Erlang/OTP Team - Ericsson AB




2017-08-08 11:25 GMT+02:00 Dmitry Kolesnikov <[hidden email]>:
Hello,

Have you tried to disable reuse of ssl sessions?
{ssl_options, [{reuse_sessions, false}]}

And reduce the session time
-ssl session_lifetime 120

Best Regards,
Dmitry

> On 8 Aug 2017, at 11.59, Max Treskin <[hidden email]> wrote:
>
> Hello,
>
> I have two different HTTPS-servers (different purposes, code, etc) built on top of cowboy/ranch, and both have memory leaks.
> ETS table server_ssl_otp_session_cache has millions of records just after hours of work and consumes gigabytes of RAM.
> What should I do to eliminate this? Is it bug or intended behaviour?
>
> Thanks
>
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions




--
Danil Zagoskin | [hidden email]




--
Danil Zagoskin | [hidden email]


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions