SSL Client memory usage

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

SSL Client memory usage

Jamie McClymont
Hi all

I'm building an app which needs to be able to make a good number of concurrent HTTPS requests (ideally I could keep high-hundreds/low-thousands of idle keepalive connections open at a time with modest memory usage).

Poking around (by making concurrent requests to 50 domains), I find that holding HTTP connections is effectively free memory-wise, while holding HTTPS connections costs around 2.5MB per connection! All this memory usage is in the tls_connection:init/1 processes, according to observer.

Looking at the state of these processes, they each hold a lot of information: the extracted/decoded certificate files for all the CA certs on my system (of which there are 134!). From some cursory googling about erlang's memory model, it seems that because this data is not a binary blob but a big tree of terms, it's going to be copied to every tls_connection process.

Any pointers on if there is an existing way to mitigate this (while keeping a full set of root certs available)? Otherwise, it seems to me the reasonable solution would be to keep the CA store in a central process (ETS?) and query that for the specific root cert that the server says it is signed by. If I'll be implementing it, could someone sanity-check this approach:
* provide an empty list of CAs to the ssl module
* provide a verify_fun which matches on {bad_cert, unknown_ca} and then gets the correct CA and performs all the remaining necessary checks (whatever those are...) before returning {valid, UserState} (or {valid_peer, UserState}?)


Thanks
- Jamie

P.s. sorry if this email is a duplicate: I think my last one was silently dropped since I sent it from a different address to the one I subscribed with, but not certain.

Appendix - one of the 134 decoded certs stored by every process:

     {decoded,
      {{#Ref<0.577909663.1295777797.255240>,
        106100277556486529736699587978573607008,
        {rdnSequence,
         [[{'AttributeTypeAndValue',{2,5,4,6},"CN"}],
          [{'AttributeTypeAndValue',
            {2,5,4,10},
            {utf8String,<<"UniTrust">>}}],
          [{'AttributeTypeAndValue',
            {2,5,4,3},
            {utf8String,
             <<"UCA Extend"...(28 B)>>}}]]}},
       {<<48,130,5,90,48,130,3,66,160,3...(1 kB)>>,
        {'OTPCertificate',
         {'OTPTBSCertificate',v3,106100277556486529736699587978573607008,
          {'SignatureAlgorithm',{1,2,840,113549,1,1,11},'NULL'},
          {rdnSequence,
           [[{'AttributeTypeAndValue',{2,5,4,6},"CN"}],
            [{'AttributeTypeAndValue',
              {2,5,4,10},
              {utf8String,<<"UniTrust">>}}],
            [{'AttributeTypeAndValue',
              {2,5,4,3},
              {utf8String,
               <<"UCA Extend"...(28 B)>>}}]]},
          {'Validity',{utcTime,"150313000000Z"},{utcTime,"381231000000Z"}},
          {rdnSequence,
           [[{'AttributeTypeAndValue',{2,5,4,6},"CN"}],
            [{'AttributeTypeAndValue',
              {2,5,4,10},
              {utf8String,<<"UniTrust">>}}],
            [{'AttributeTypeAndValue',
              {2,5,4,3},
              {utf8String,
               <<"UCA Extend"...(28 B)>>}}]]},
          {'OTPSubjectPublicKeyInfo',
           {'PublicKeyAlgorithm',{1,2,840,113549,1,1,1},'NULL'},
           {'RSAPublicKey',
            689603717979852636831081426524261160465705539154429034072886964586226830541206711367761253770112725458465044497361739659482435934368052465546153546798794021727234724250491430601994902725573258440903769572247434719710136763019389835974660379560646189661431052777862620577580627735499519427983731365873646101116220703443112119094192330629089538925644308340292126955780669276375525797477380635249258541277353787220035677250408659616695990328739419944306797836041926101961617154636604738870383637100044978431121318943912528365357866046388476749437076957867974020718613998841641003305315809368105957269212277478816744375233680463674812053558530520195020911324127390436183137612053158831455696931169855471108960460199610930021503675868841756093916690737925568854144895439113023618139270733541586805299082563244530623575372402602522085830629158312024768857183218701826386841425816972120556588392270337545276474712543306659887836405319075385033422367187135702233624763183702955559196400
 695312060899684142798408078723091214153237781607055313585627048184929762700635100359534649156945631489915475941635189136663455081054904713071171644045263631869342706234148072080520374089220391876551645591673277270360119934713372188503089344323356273673087,
            65537}},
          asn1_NOVALUE,asn1_NOVALUE,
          [{'Extension',
            {2,5,29,14},
            false,
            <<217,116,58,228,48,61,13,247,18,220...(20 B)>>},
           {'Extension',
            {2,5,29,19},
            true,
            {'BasicConstraints',true,asn1_NOVALUE}},
           {'Extension',
            {2,5,29,15},
            true,
            [digitalSignature,keyCertSign,cRLSign]}]},
         {'SignatureAlgorithm',{1,2,840,113549,1,1,11},'NULL'},
         <<54,141,151,204,66,21,100,41,55,155...(512 B)>>}}}},
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: SSL Client memory usage

Ingela Andin
Hi!

How do you provide your certificates to ssl? If you provide your CA-certs as binary DER blobs to the connection they
will be stored by the connection and be local to the connection. If you provide them through a file they will be stored
in a ETS table and maybe referenced by other connections as there is a way to refer to them.

Regards Ingela Erlang/OTP Team

Den mån 10 juni 2019 kl 05:10 skrev Jamie McClymont <[hidden email]>:
Hi all

I'm building an app which needs to be able to make a good number of concurrent HTTPS requests (ideally I could keep high-hundreds/low-thousands of idle keepalive connections open at a time with modest memory usage).

Poking around (by making concurrent requests to 50 domains), I find that holding HTTP connections is effectively free memory-wise, while holding HTTPS connections costs around 2.5MB per connection! All this memory usage is in the tls_connection:init/1 processes, according to observer.

Looking at the state of these processes, they each hold a lot of information: the extracted/decoded certificate files for all the CA certs on my system (of which there are 134!). From some cursory googling about erlang's memory model, it seems that because this data is not a binary blob but a big tree of terms, it's going to be copied to every tls_connection process.

Any pointers on if there is an existing way to mitigate this (while keeping a full set of root certs available)? Otherwise, it seems to me the reasonable solution would be to keep the CA store in a central process (ETS?) and query that for the specific root cert that the server says it is signed by. If I'll be implementing it, could someone sanity-check this approach:
* provide an empty list of CAs to the ssl module
* provide a verify_fun which matches on {bad_cert, unknown_ca} and then gets the correct CA and performs all the remaining necessary checks (whatever those are...) before returning {valid, UserState} (or {valid_peer, UserState}?)


Thanks
- Jamie

P.s. sorry if this email is a duplicate: I think my last one was silently dropped since I sent it from a different address to the one I subscribed with, but not certain.

Appendix - one of the 134 decoded certs stored by every process:

     {decoded,
      {{#Ref<0.577909663.1295777797.255240>,
        106100277556486529736699587978573607008,
        {rdnSequence,
         [[{'AttributeTypeAndValue',{2,5,4,6},"CN"}],
          [{'AttributeTypeAndValue',
            {2,5,4,10},
            {utf8String,<<"UniTrust">>}}],
          [{'AttributeTypeAndValue',
            {2,5,4,3},
            {utf8String,
             <<"UCA Extend"...(28 B)>>}}]]}},
       {<<48,130,5,90,48,130,3,66,160,3...(1 kB)>>,
        {'OTPCertificate',
         {'OTPTBSCertificate',v3,106100277556486529736699587978573607008,
          {'SignatureAlgorithm',{1,2,840,113549,1,1,11},'NULL'},
          {rdnSequence,
           [[{'AttributeTypeAndValue',{2,5,4,6},"CN"}],
            [{'AttributeTypeAndValue',
              {2,5,4,10},
              {utf8String,<<"UniTrust">>}}],
            [{'AttributeTypeAndValue',
              {2,5,4,3},
              {utf8String,
               <<"UCA Extend"...(28 B)>>}}]]},
          {'Validity',{utcTime,"150313000000Z"},{utcTime,"381231000000Z"}},
          {rdnSequence,
           [[{'AttributeTypeAndValue',{2,5,4,6},"CN"}],
            [{'AttributeTypeAndValue',
              {2,5,4,10},
              {utf8String,<<"UniTrust">>}}],
            [{'AttributeTypeAndValue',
              {2,5,4,3},
              {utf8String,
               <<"UCA Extend"...(28 B)>>}}]]},
          {'OTPSubjectPublicKeyInfo',
           {'PublicKeyAlgorithm',{1,2,840,113549,1,1,1},'NULL'},
           {'RSAPublicKey',
            689603717979852636831081426524261160465705539154429034072886964586226830541206711367761253770112725458465044497361739659482435934368052465546153546798794021727234724250491430601994902725573258440903769572247434719710136763019389835974660379560646189661431052777862620577580627735499519427983731365873646101116220703443112119094192330629089538925644308340292126955780669276375525797477380635249258541277353787220035677250408659616695990328739419944306797836041926101961617154636604738870383637100044978431121318943912528365357866046388476749437076957867974020718613998841641003305315809368105957269212277478816744375233680463674812053558530520195020911324127390436183137612053158831455696931169855471108960460199610930021503675868841756093916690737925568854144895439113023618139270733541586805299082563244530623575372402602522085830629158312024768857183218701826386841425816972120556588392270337545276474712543306659887836405319075385033422367187135702233624763183702955559196400
 695312060899684142798408078723091214153237781607055313585627048184929762700635100359534649156945631489915475941635189136663455081054904713071171644045263631869342706234148072080520374089220391876551645591673277270360119934713372188503089344323356273673087,
            65537}},
          asn1_NOVALUE,asn1_NOVALUE,
          [{'Extension',
            {2,5,29,14},
            false,
            <<217,116,58,228,48,61,13,247,18,220...(20 B)>>},
           {'Extension',
            {2,5,29,19},
            true,
            {'BasicConstraints',true,asn1_NOVALUE}},
           {'Extension',
            {2,5,29,15},
            true,
            [digitalSignature,keyCertSign,cRLSign]}]},
         {'SignatureAlgorithm',{1,2,840,113549,1,1,11},'NULL'},
         <<54,141,151,204,66,21,100,41,55,155...(512 B)>>}}}},
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: SSL Client memory usage

Jamie McClymont
Ahh that did it, thanks!

I'm using hackney, which in turn uses certifi:cacerts() (binary blobs of cert data).

Using hackney's ssl options to instead pass certifi:cacertfile() as an argument brings the memory usage of each keepalive'd tls connection (once the process is GCed) down to <30KB -- that's excellent :)



On Mon, 10 Jun 2019, at 9:51 PM, Ingela Andin wrote:

> Hi!
>
> How do you provide your certificates to ssl? If you provide your
> CA-certs as binary DER blobs to the connection they
> will be stored by the connection and be local to the connection. If you
> provide them through a file they will be stored
> in a ETS table and maybe referenced by other connections as there is a
> way to refer to them.
>
> Regards Ingela Erlang/OTP Team
>
> Den mån 10 juni 2019 kl 05:10 skrev Jamie McClymont
> <[hidden email]
> <mailto:jamie%[hidden email]>>:
> > Hi all
> >
> >  I'm building an app which needs to be able to make a good number of concurrent HTTPS requests (ideally I could keep high-hundreds/low-thousands of idle keepalive connections open at a time with modest memory usage).
> >
> >  Poking around (by making concurrent requests to 50 domains), I find that holding HTTP connections is effectively free memory-wise, while holding HTTPS connections costs around 2.5MB per connection! All this memory usage is in the tls_connection:init/1 processes, according to observer.
> >
> >  Looking at the state of these processes, they each hold a lot of information: the extracted/decoded certificate files for all the CA certs on my system (of which there are 134!). From some cursory googling about erlang's memory model, it seems that because this data is not a binary blob but a big tree of terms, it's going to be copied to every tls_connection process.
> >
> >  Any pointers on if there is an existing way to mitigate this (while keeping a full set of root certs available)? Otherwise, it seems to me the reasonable solution would be to keep the CA store in a central process (ETS?) and query that for the specific root cert that the server says it is signed by. If I'll be implementing it, could someone sanity-check this approach:
> >  * provide an empty list of CAs to the ssl module
> >  * provide a verify_fun which matches on {bad_cert, unknown_ca} and then gets the correct CA and performs all the remaining necessary checks (whatever those are...) before returning {valid, UserState} (or {valid_peer, UserState}?)
> >
> >
> >  Thanks
> >  - Jamie
> >
> >  P.s. sorry if this email is a duplicate: I think my last one was silently dropped since I sent it from a different address to the one I subscribed with, but not certain.
> >
> >  Appendix - one of the 134 decoded certs stored by every process:
> >
> >  {decoded,
> >  {{#Ref<0.577909663.1295777797.255240>,
> >  106100277556486529736699587978573607008,
> >  {rdnSequence,
> >  [[{'AttributeTypeAndValue',{2,5,4,6},"CN"}],
> >  [{'AttributeTypeAndValue',
> >  {2,5,4,10},
> >  {utf8String,<<"UniTrust">>}}],
> >  [{'AttributeTypeAndValue',
> >  {2,5,4,3},
> >  {utf8String,
> >  <<"UCA Extend"...(28 B)>>}}]]}},
> >  {<<48,130,5,90,48,130,3,66,160,3...(1 kB)>>,
> >  {'OTPCertificate',
> >  {'OTPTBSCertificate',v3,106100277556486529736699587978573607008,
> >  {'SignatureAlgorithm',{1,2,840,113549,1,1,11},'NULL'},
> >  {rdnSequence,
> >  [[{'AttributeTypeAndValue',{2,5,4,6},"CN"}],
> >  [{'AttributeTypeAndValue',
> >  {2,5,4,10},
> >  {utf8String,<<"UniTrust">>}}],
> >  [{'AttributeTypeAndValue',
> >  {2,5,4,3},
> >  {utf8String,
> >  <<"UCA Extend"...(28 B)>>}}]]},
> >  {'Validity',{utcTime,"150313000000Z"},{utcTime,"381231000000Z"}},
> >  {rdnSequence,
> >  [[{'AttributeTypeAndValue',{2,5,4,6},"CN"}],
> >  [{'AttributeTypeAndValue',
> >  {2,5,4,10},
> >  {utf8String,<<"UniTrust">>}}],
> >  [{'AttributeTypeAndValue',
> >  {2,5,4,3},
> >  {utf8String,
> >  <<"UCA Extend"...(28 B)>>}}]]},
> >  {'OTPSubjectPublicKeyInfo',
> >  {'PublicKeyAlgorithm',{1,2,840,113549,1,1,1},'NULL'},
> >  {'RSAPublicKey',
> >  689603717979852636831081426524261160465705539154429034072886964586226830541206711367761253770112725458465044497361739659482435934368052465546153546798794021727234724250491430601994902725573258440903769572247434719710136763019389835974660379560646189661431052777862620577580627735499519427983731365873646101116220703443112119094192330629089538925644308340292126955780669276375525797477380635249258541277353787220035677250408659616695990328739419944306797836041926101961617154636604738870383637100044978431121318943912528365357866046388476749437076957867974020718613998841641003305315809368105957269212277478816744375233680463674812053558530520195020911324127390436183137612053158831455696931169855471108960460199610930021503675868841756093916690737925568854144895439113023618139270733541586805299082563244530623575372402602522085830629158312024768857183218701826386841425816972120556588392270337545276474712543306659887836405319075385033422367187135702233624763183702955559196400
> >  695312060899684142798408078723091214153237781607055313585627048184929762700635100359534649156945631489915475941635189136663455081054904713071171644045263631869342706234148072080520374089220391876551645591673277270360119934713372188503089344323356273673087,
> >  65537}},
> >  asn1_NOVALUE,asn1_NOVALUE,
> >  [{'Extension',
> >  {2,5,29,14},
> >  false,
> >  <<217,116,58,228,48,61,13,247,18,220...(20 B)>>},
> >  {'Extension',
> >  {2,5,29,19},
> >  true,
> >  {'BasicConstraints',true,asn1_NOVALUE}},
> >  {'Extension',
> >  {2,5,29,15},
> >  true,
> >  [digitalSignature,keyCertSign,cRLSign]}]},
> >  {'SignatureAlgorithm',{1,2,840,113549,1,1,11},'NULL'},
> >  <<54,141,151,204,66,21,100,41,55,155...(512 B)>>}}}},
> >  _______________________________________________
> >  erlang-questions mailing list
> > [hidden email]
> > http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: SSL Client memory usage

Bram Verburg
Please make sure to also pass {verify, verify_peer}, since any ssl_options you pass will overwrite the hackney defaults, and therefore pick up the ssl default of verify_none.

Eshell V10.2.5  (abort with ^G)
1> hackney:request(get, <<"https://self-signed.badssl.com/">>, [], <<>>, [{ssl_options, [{cacertfile, certifi:cacertfile()}]}]).
{ok,200,[...],#Ref<...>} // Not good!
2> hackney:request(get, <<"https://self-signed.badssl.com/">>, [], <<>>, [{ssl_options, [{cacertfile, certifi:cacertfile()}, {verify, verify_peer}]}]).
{error,{tls_alert,"bad certificate"}}

Bram



On Tue, 11 Jun 2019 at 09:06, Jamie McClymont <[hidden email]> wrote:
Ahh that did it, thanks!

I'm using hackney, which in turn uses certifi:cacerts() (binary blobs of cert data).

Using hackney's ssl options to instead pass certifi:cacertfile() as an argument brings the memory usage of each keepalive'd tls connection (once the process is GCed) down to <30KB -- that's excellent :)



On Mon, 10 Jun 2019, at 9:51 PM, Ingela Andin wrote:
> Hi!
>
> How do you provide your certificates to ssl? If you provide your
> CA-certs as binary DER blobs to the connection they
> will be stored by the connection and be local to the connection. If you
> provide them through a file they will be stored
> in a ETS table and maybe referenced by other connections as there is a
> way to refer to them.
>
> Regards Ingela Erlang/OTP Team
>
> Den mån 10 juni 2019 kl 05:10 skrev Jamie McClymont
> <[hidden email]
> <mailto:[hidden email]>>:
> > Hi all
> >
> >  I'm building an app which needs to be able to make a good number of concurrent HTTPS requests (ideally I could keep high-hundreds/low-thousands of idle keepalive connections open at a time with modest memory usage).
> >
> >  Poking around (by making concurrent requests to 50 domains), I find that holding HTTP connections is effectively free memory-wise, while holding HTTPS connections costs around 2.5MB per connection! All this memory usage is in the tls_connection:init/1 processes, according to observer.
> >
> >  Looking at the state of these processes, they each hold a lot of information: the extracted/decoded certificate files for all the CA certs on my system (of which there are 134!). From some cursory googling about erlang's memory model, it seems that because this data is not a binary blob but a big tree of terms, it's going to be copied to every tls_connection process.
> >
> >  Any pointers on if there is an existing way to mitigate this (while keeping a full set of root certs available)? Otherwise, it seems to me the reasonable solution would be to keep the CA store in a central process (ETS?) and query that for the specific root cert that the server says it is signed by. If I'll be implementing it, could someone sanity-check this approach:
> >  * provide an empty list of CAs to the ssl module
> >  * provide a verify_fun which matches on {bad_cert, unknown_ca} and then gets the correct CA and performs all the remaining necessary checks (whatever those are...) before returning {valid, UserState} (or {valid_peer, UserState}?)
> >
> >
> >  Thanks
> >  - Jamie
> >
> >  P.s. sorry if this email is a duplicate: I think my last one was silently dropped since I sent it from a different address to the one I subscribed with, but not certain.
> >
> >  Appendix - one of the 134 decoded certs stored by every process:
> >
> >  {decoded,
> >  {{#Ref<0.577909663.1295777797.255240>,
> >  106100277556486529736699587978573607008,
> >  {rdnSequence,
> >  [[{'AttributeTypeAndValue',{2,5,4,6},"CN"}],
> >  [{'AttributeTypeAndValue',
> >  {2,5,4,10},
> >  {utf8String,<<"UniTrust">>}}],
> >  [{'AttributeTypeAndValue',
> >  {2,5,4,3},
> >  {utf8String,
> >  <<"UCA Extend"...(28 B)>>}}]]}},
> >  {<<48,130,5,90,48,130,3,66,160,3...(1 kB)>>,
> >  {'OTPCertificate',
> >  {'OTPTBSCertificate',v3,106100277556486529736699587978573607008,
> >  {'SignatureAlgorithm',{1,2,840,113549,1,1,11},'NULL'},
> >  {rdnSequence,
> >  [[{'AttributeTypeAndValue',{2,5,4,6},"CN"}],
> >  [{'AttributeTypeAndValue',
> >  {2,5,4,10},
> >  {utf8String,<<"UniTrust">>}}],
> >  [{'AttributeTypeAndValue',
> >  {2,5,4,3},
> >  {utf8String,
> >  <<"UCA Extend"...(28 B)>>}}]]},
> >  {'Validity',{utcTime,"150313000000Z"},{utcTime,"381231000000Z"}},
> >  {rdnSequence,
> >  [[{'AttributeTypeAndValue',{2,5,4,6},"CN"}],
> >  [{'AttributeTypeAndValue',
> >  {2,5,4,10},
> >  {utf8String,<<"UniTrust">>}}],
> >  [{'AttributeTypeAndValue',
> >  {2,5,4,3},
> >  {utf8String,
> >  <<"UCA Extend"...(28 B)>>}}]]},
> >  {'OTPSubjectPublicKeyInfo',
> >  {'PublicKeyAlgorithm',{1,2,840,113549,1,1,1},'NULL'},
> >  {'RSAPublicKey',
> >  689603717979852636831081426524261160465705539154429034072886964586226830541206711367761253770112725458465044497361739659482435934368052465546153546798794021727234724250491430601994902725573258440903769572247434719710136763019389835974660379560646189661431052777862620577580627735499519427983731365873646101116220703443112119094192330629089538925644308340292126955780669276375525797477380635249258541277353787220035677250408659616695990328739419944306797836041926101961617154636604738870383637100044978431121318943912528365357866046388476749437076957867974020718613998841641003305315809368105957269212277478816744375233680463674812053558530520195020911324127390436183137612053158831455696931169855471108960460199610930021503675868841756093916690737925568854144895439113023618139270733541586805299082563244530623575372402602522085830629158312024768857183218701826386841425816972120556588392270337545276474712543306659887836405319075385033422367187135702233624763183702955559196400
> >  695312060899684142798408078723091214153237781607055313585627048184929762700635100359534649156945631489915475941635189136663455081054904713071171644045263631869342706234148072080520374089220391876551645591673277270360119934713372188503089344323356273673087,
> >  65537}},
> >  asn1_NOVALUE,asn1_NOVALUE,
> >  [{'Extension',
> >  {2,5,29,14},
> >  false,
> >  <<217,116,58,228,48,61,13,247,18,220...(20 B)>>},
> >  {'Extension',
> >  {2,5,29,19},
> >  true,
> >  {'BasicConstraints',true,asn1_NOVALUE}},
> >  {'Extension',
> >  {2,5,29,15},
> >  true,
> >  [digitalSignature,keyCertSign,cRLSign]}]},
> >  {'SignatureAlgorithm',{1,2,840,113549,1,1,11},'NULL'},
> >  <<54,141,151,204,66,21,100,41,55,155...(512 B)>>}}}},
> >  _______________________________________________
> >  erlang-questions mailing list
> > [hidden email]
> > http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions