Patch Package OTP 21.2 Released

[Henrik Nord X]

Erlang/OTP 21.2 is the second service release for the 21st major
release with, improvements as well as a few features!

Highlights:

SSH:

* Public key methods: ssh-ed25519 and ssh-ed448 added. Requires OpenSSL
1.1.1 or later as cryptolib under the OTP application

SSL:

* ssl now uses active n internally to boost performance. Old active
once behaviour can be restored by setting a application variable.

ERTS, Kernel:

* New counters and atomics modules supplies access to highly efficient
operations on mutable fixed word sized variables.

* New module persistent_term!. Lookups are in constant time! No copying
the terms!

* New pollset made to handle sockets that use {active, true} or
{active, N}. Polled by a normal scheduler!

* No more ONESHOT mechanism overhead on fds! Only on Linux and BSD

For a full list of details see:
http://erlang.org/download/otp_src_21.2.readme

Pre built versions for Windows can be fetched here:
http://erlang.org/download/otp_win32_21.2.exe
http://erlang.org/download/otp_win64_21.2.exe

Online documentation can be browsed here:
http://erlang.org/documentation/doc-10.2/doc

The Erlang/OTP source can also be found at GitHub on the official
Erlang repository, Here: 
https://github.com/erlang/otp/releases/tag/OTP-21.2

Please report any new issues via Erlang/OTPs public issue tracker
https://bugs.erlang.org

We want to thank all of those who sent us patches, suggestions and bug
reports!

Thank you!

The Erlang/OTP Team at Ericsson
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

[Edmond Begumisa]

On Wed, 12 Dec 2018 00:09:11 +1000, Henrik Nord X  
<[hidden email]> wrote:

> Erlang/OTP 21.2 is the second service release for the 21st major
> release with, improvements as well as a few features!
>
> Highlights:
>
> SSH:
>
> * Public key methods: ssh-ed25519 and ssh-ed448 added. Requires OpenSSL
> 1.1.1 or later as cryptolib under the OTP application
>
> SSL:
>
> * ssl now uses active n internally to boost performance. Old active
> once behaviour can be restored by setting a application variable.
>
> ERTS, Kernel:
>
> * New counters and atomics modules supplies access to highly efficient
> operations on mutable fixed word sized variables.
>
> * New module persistent_term!. Lookups are in constant time! No copying
> the terms!
>

WOW! This persistent_term module is very interesting [1], and already see  
many places in both my private and work codebase where it will be a  
godsend.

However, I fear that there is going to be a lot of abuse.

The temptation to use this as a global shared memory quick-and-dirty  
alternative to more elaborate better thought through designs is going to  
be too irresistible to many. The downside to many projects these days  
using rebar3 and the resulting deep dependencies, is that it's  
increasingly more difficult to avoid abuse instigated by others being  
introduced into your system (as we're seeing with quick-and-dirty NIFs).

I foresee many projects looking at those warnings in the persistent_term  
docs and thinking "I've weighed it, and I think my project is 'special'  
enough to call put/1 and erase/1 a little more frequently than the  
documentation suggests, or have more entries than is recommended, and I  
have my reasons for not using ETS instead". With deep dependencies pulled  
down from github, this kind of thinking gets compounded and infectious.

I wish there was a way of providing protections against that sort of  
thing. I can't think of one.

Other than that, bravo! This is a very useful feature.

[1] My immediate thought is a replacement for runtime re-compiled and  
re-loaded configuration modules which make use the ERTS constant pool --  
I've been using these a lot. persistent_term would solve the lingering  
purge process-killing problem that the hot-loading approach to config  
suffers from.

> * New pollset made to handle sockets that use {active, true} or
> {active, N}. Polled by a normal scheduler!
>
> * No more ONESHOT mechanism overhead on fds! Only on Linux and BSD
>
> For a full list of details see:
> http://erlang.org/download/otp_src_21.2.readme
>
> Pre built versions for Windows can be fetched here:
> http://erlang.org/download/otp_win32_21.2.exe
> http://erlang.org/download/otp_win64_21.2.exe
>
> Online documentation can be browsed here:
> http://erlang.org/documentation/doc-10.2/doc
>
> The Erlang/OTP source can also be found at GitHub on the official
> Erlang repository, Here:
> https://github.com/erlang/otp/releases/tag/OTP-21.2
>
> Please report any new issues via Erlang/OTPs public issue tracker
> https://bugs.erlang.org
>
> We want to thank all of those who sent us patches, suggestions and bug
> reports!
>
> Thank you!
>
> The Erlang/OTP Team at Ericsson
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

--
Using Opera's mail client: http://www.opera.com/mail/
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

[Michael Truog]

On 12/14/18 7:52 AM, Edmond Begumisa wrote:

>
> WOW! This persistent_term module is very interesting [1], and already
> see many places in both my private and work codebase where it will be
> a godsend.
>
> However, I fear that there is going to be a lot of abuse.
>
> The temptation to use this as a global shared memory quick-and-dirty
> alternative to more elaborate better thought through designs is going
> to be too irresistible to many. The downside to many projects these
> days using rebar3 and the resulting deep dependencies, is that it's
> increasingly more difficult to avoid abuse instigated by others being
> introduced into your system (as we're seeing with quick-and-dirty NIFs).
>
> I foresee many projects looking at those warnings in the
> persistent_term docs and thinking "I've weighed it, and I think my
> project is 'special' enough to call put/1 and erase/1 a little more
> frequently than the documentation suggests, or have more entries than
> is recommended, and I have my reasons for not using ETS instead". With
> deep dependencies pulled down from github, this kind of thinking gets
> compounded and infectious.
>
> I wish there was a way of providing protections against that sort of
> thing. I can't think of one.

For avoiding NIFs/port-drivers and other problems that relate to
security, I created https://github.com/okeuday/pest .  PEST is a static
analysis tool that looks for function calls it knows are problematic in
Erlang source code and provides information about where the problems
occur.  You could also use the PEST source code to identify other
function calls that should be avoided if you are concerned about
persistent_term.

Best Regards,
Michael
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

[Anthony Ramine-4]

Are you saying you don't do due diligence and just pull dependencies in your project without reviewing them?

> Le 14 déc. 2018 à 16:52, Edmond Begumisa <[hidden email]> a écrit :
>
> I foresee many projects looking at those warnings in the persistent_term docs and thinking "I've weighed it, and I think my project is 'special' enough to call put/1 and erase/1 a little more frequently than the documentation suggests, or have more entries than is recommended, and I have my reasons for not using ETS instead". With deep dependencies pulled down from github, this kind of thinking gets compounded and infectious.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

[Edmond Begumisa]

On Sun, 16 Dec 2018 16:44:13 +1000, Michael Truog <[hidden email]>  
wrote:

> On 12/14/18 7:52 AM, Edmond Begumisa wrote:
>>
>> WOW! This persistent_term module is very interesting [1], and already  
>> see many places in both my private and work codebase where it will be a  
>> godsend.
>>
>> However, I fear that there is going to be a lot of abuse.
>>
>> The temptation to use this as a global shared memory quick-and-dirty  
>> alternative to more elaborate better thought through designs is going  
>> to be too irresistible to many. The downside to many projects these  
>> days using rebar3 and the resulting deep dependencies, is that it's  
>> increasingly more difficult to avoid abuse instigated by others being  
>> introduced into your system (as we're seeing with quick-and-dirty NIFs).
>>
>> I foresee many projects looking at those warnings in the  
>> persistent_term docs and thinking "I've weighed it, and I think my  
>> project is 'special' enough to call put/1 and erase/1 a little more  
>> frequently than the documentation suggests, or have more entries than  
>> is recommended, and I have my reasons for not using ETS instead". With  
>> deep dependencies pulled down from github, this kind of thinking gets  
>> compounded and infectious.
>>
>> I wish there was a way of providing protections against that sort of  
>> thing. I can't think of one.
>
> For avoiding NIFs/port-drivers and other problems that relate to  
> security, I created https://github.com/okeuday/pest .  PEST is a static  
> analysis tool that looks for function calls it knows are problematic in  
> Erlang source code and provides information about where the problems  
> occur.  You could also use the PEST source code to identify other  
> function calls that should be avoided if you are concerned about  
> persistent_term.
>
> Best Regards,
> Michael
>

This is very helpful tool. You might want to drop the "security" in the  
description because it could be used as part of general code review.  
Thanks for this. I'm going to start using it.

I was thinking of something along the lines of persistent_term internally  
doing some sort of tracking at runtime and warning when it's being called  
in such a way that would degrade performance. And perhaps the NIF API too.  
Neither seem practical though.

- Edmond -

--
Using Opera's mail client: http://www.opera.com/mail/
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

[Edmond Begumisa]

On Mon, 17 Dec 2018 02:21:17 +1000, Anthony Ramine <[hidden email]>  
wrote:

> Are you saying you don't do due diligence and just pull dependencies in  
> your project without reviewing them?
>

Certainly not.

I'm saying that when you've reviewed both your rebar3 direct dependencies  
and their deep dependencies right down to the leaves (every .erl file for  
every git URL in all the rebar.config files), it becomes increasingly more  
difficult to perform this review task when you upgrade your direct  
dependencies and they introduce new indirect dependencies and/or when you  
introduce new direct dependencies. As both the width and depth grows, at  
some point, it becomes impractical as anyone who's used Node.js with  
npm/yarn can attest (half the time you've no idea what all those js files  
NPM has pulled down are doing).

The route Anthony Ramine suggests I think is more practical. A tool that  
points you to which parts of which dependencies you need to take (another)  
look at.

- Edmond -

>> Le 14 déc. 2018 à 16:52, Edmond Begumisa <[hidden email]>  
>> a écrit :
>>
>> I foresee many projects looking at those warnings in the  
>> persistent_term docs and thinking "I've weighed it, and I think my  
>> project is 'special' enough to call put/1 and erase/1 a little more  
>> frequently than the documentation suggests, or have more entries than  
>> is recommended, and I have my reasons for not using ETS instead". With  
>> deep dependencies pulled down from github, this kind of thinking gets  
>> compounded and infectious.
>

--
Using Opera's mail client: http://www.opera.com/mail/
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

[Edmond Begumisa]

On Mon, 17 Dec 2018 05:44:22 +1000, Edmond Begumisa  
<[hidden email]> wrote:

> On Mon, 17 Dec 2018 02:21:17 +1000, Anthony Ramine <[hidden email]>  
> wrote:
>
>> Are you saying you don't do due diligence and just pull dependencies in  
>> your project without reviewing them?
>>
>
> Certainly not.
>
> I'm saying that when you've reviewed both your rebar3 direct  
> dependencies and their deep dependencies right down to the leaves (every  
> .erl file for every git URL in all the rebar.config files), it becomes  
> increasingly more difficult to perform this review task when you upgrade  
> your direct dependencies and they introduce new indirect dependencies  
> and/or when you introduce new direct dependencies. As both the width and  
> depth grows, at some point, it becomes impractical as anyone who's used  
> Node.js with npm/yarn can attest (half the time you've no idea what all  
> those js files NPM has pulled down are doing).
>
> The route Anthony Ramine suggests I think is more practical.

Correction: The route Michael Truog suggests!

> A tool that points you to which parts of which dependencies you need to  
> take (another) look at.
>
> - Edmond -
>
>>> Le 14 déc. 2018 à 16:52, Edmond Begumisa <[hidden email]>  
>>> a écrit :
>>>
>>> I foresee many projects looking at those warnings in the  
>>> persistent_term docs and thinking "I've weighed it, and I think my  
>>> project is 'special' enough to call put/1 and erase/1 a little more  
>>> frequently than the documentation suggests, or have more entries than  
>>> is recommended, and I have my reasons for not using ETS instead". With  
>>> deep dependencies pulled down from github, this kind of thinking gets  
>>> compounded and infectious.
>>
>
>

--
Using Opera's mail client: http://www.opera.com/mail/
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

[Jesper Louis Andersen-2]

On Fri, Dec 14, 2018 at 4:52 PM Edmond Begumisa <[hidden email]> wrote:

However, I fear that there is going to be a lot of abuse.

I'm guessing there is going to be a lot of abuse as well. But I don't think that is different from using a large list where a map would do for instance. So lots of code has performance problems, uses excessive amounts of memory or otherwise slows down your system.

As for the discussion on reading through your dependencies:

There is a security aspect and a performance aspect. Both are important. Currently I think the best tool wrt. security is to have libraries define capabilities on what they do, and then verify those capabilities with a crash in the system. See e.g., OpenBSDs pledge(2) system call. Once a process pledges itself to a subset of all syscalls, it will be terminated if it ever calls one of the illegal targets. But as for the slowness, the best bet is to run benchmarks on your system and profile the code base extensively. People tend to make innocuously looking changes to a system---only for it to have catastrophic consequences on your software.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions