Erlang/OTP 20.0-rc2 is available for testing

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Erlang/OTP 20.0-rc2 is available for testing

Kenneth Lundin-5

OTP 20 Release Candidate 2

This is the second of two release candidates before the OTP 20 release. The intention with this release is that you as users try it and give us feedback if something does not work as expected. Could be a bug, an unexpected incompatibility, a significant change of characteristics in negative direction, etc.

There are only minor changes compared to the first release candidate, some of them listed below:

  • erts: ./configure --enable-lock-counter will enable building of an additional emulator that has support for
    lock counting. (The option previously existed, but would turn on lock counting in the default emulator
    being built.) To start the lock-counting emulator, use erl -emu_type lcnt.
  • kernel: Added the process_flag message_queue_data = off_heap to the code_server process in order to improve characteristics during code upgrade, which can generate a huge amount of messages.

Here are some of the most important news in OTP 20 (same as in RC1):

Potential Incompatibilities

  • ERTS:

    • The non SMP Erlang VM is deprecated and not built by default
    • Remove deprecated erlang:hash/2
    • erlang:statistics/1 with scheduler_wall_time now also includes info about dirty CPU schedulers.
    • The new purge strategy introduced in OTP 19.1 is mandatory and slightly incompatible for processes holding funs
      see erlang:check_process_code/3.
    • The NIF library reload is not supported anymore.
  • Asn1: Deprecated module and functions removed (asn1rt, asn1ct:encode/3 and decode/3)

  • Ssh: client only option in a call to start a daemon will now fail

Highlights

Erts:

  • Dirty schedulers enabled and supported on VM with SMP support.
  • support for “dirty” BIFs and “dirty” GC.
  • erlang:garbage_collect/2 for control of minor or major GC
  • Erlang literals are no longer copied when sending messages.
  • Improved performance for large ETS tables, >256 entries (except ordered_set)
  • erlang:system_info/1 atom_count and atom_limit
  • Reduced memory pressure by converting sub-binaries to heap-binaries during GC
  • enif_select, map an external event to message

Compiler:

  • Code generation for complicated guards is improved.
  • Warnings for repeated identical map keys. #{'a'=>1, 'b'=>2, 'a'=>3} will warn for the repeated key a.
  • By default there is now a warning when export_all is used. Can be disabled
  • Pattern matching for maps is optimized
  • New option deterministic to omit path to source + options info the BEAM file.
  • Atoms may now contain arbitrary unicode characters.
  • compile:file/2 has an option to include extra chunks in the BEAM file.

Misc other applications

  • Unnamed ets tables optimized
  • A new event manager to handle a subset of OS signals in Erlang
  • Optimized sets add_element, del_element and union
  • Added rand:jump/0-1
  • When a gen_server crashes, the stacktrace for the client will be printed to facilitate debugging.
  • take/2 has been added to dict, orddict, and gb_trees.
  • take_any/2 has been added to gb_trees
  • Significantly updated string module with unicode support
  • erl_tar support for long path names and new file formats
  • Dtls: Documented API, experimental
  • SSH: improving security, removing and adding algorithms
  • New math:fmod/2

For more details see
http://erlang.org/download/otp_src_20.0-rc2.readme

Pre built versions for Windows can be fetched here:
http://erlang.org/download/otp_win32_20.0-rc2.exe
http://erlang.org/download/otp_win64_20.0-rc2.exe

On line documentation can be browsed here:
www.erlang.org/documentation/doc-9.0-rc2/doc/

Thanks to all contributors.

/The Erlang/OTP team at Ericsson

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang/OTP 20.0-rc2 is available for testing

Lukas Larsson-8
Hello Everyone,

As some of you have seen, the multiple-pollset implementation is still not present in OTP 20-RC2.

The implementation that we did that was almost ready by OTP 20-RC1 has turned out to have un-acceptable performance regressions for systems that mainly consist of Erlang processes communicating with each other. There is no quick fix to this problem, so we have had to rethink our original approach to polling. We are continuing to work on the problem and plan to have a solution ready "soon", it will however not be part OTP 20.

Technical details: Our original approach was to simply spread the I/O polling to each scheduler so that all schedulers were responsible for polling of fd's of ports and nifs that executed on them. This had the side-effect of making it much slower to wake up schedulers to do work (because now all scheduler sleep in poll rather than on a futex). Because of this we have decided to move the I/O polling from the scheduler threads to separate thread(s).

Lukas


On Wed, May 31, 2017 at 8:16 PM, Kenneth Lundin <[hidden email]> wrote:

OTP 20 Release Candidate 2

This is the second of two release candidates before the OTP 20 release. The intention with this release is that you as users try it and give us feedback if something does not work as expected. Could be a bug, an unexpected incompatibility, a significant change of characteristics in negative direction, etc.

There are only minor changes compared to the first release candidate, some of them listed below:

  • erts: ./configure --enable-lock-counter will enable building of an additional emulator that has support for
    lock counting. (The option previously existed, but would turn on lock counting in the default emulator
    being built.) To start the lock-counting emulator, use erl -emu_type lcnt.
  • kernel: Added the process_flag message_queue_data = off_heap to the code_server process in order to improve characteristics during code upgrade, which can generate a huge amount of messages.

Here are some of the most important news in OTP 20 (same as in RC1):

Potential Incompatibilities

  • ERTS:

    • The non SMP Erlang VM is deprecated and not built by default
    • Remove deprecated erlang:hash/2
    • erlang:statistics/1 with scheduler_wall_time now also includes info about dirty CPU schedulers.
    • The new purge strategy introduced in OTP 19.1 is mandatory and slightly incompatible for processes holding funs
      see erlang:check_process_code/3.
    • The NIF library reload is not supported anymore.
  • Asn1: Deprecated module and functions removed (asn1rt, asn1ct:encode/3 and decode/3)

  • Ssh: client only option in a call to start a daemon will now fail

Highlights

Erts:

  • Dirty schedulers enabled and supported on VM with SMP support.
  • support for “dirty” BIFs and “dirty” GC.
  • erlang:garbage_collect/2 for control of minor or major GC
  • Erlang literals are no longer copied when sending messages.
  • Improved performance for large ETS tables, >256 entries (except ordered_set)
  • erlang:system_info/1 atom_count and atom_limit
  • Reduced memory pressure by converting sub-binaries to heap-binaries during GC
  • enif_select, map an external event to message

Compiler:

  • Code generation for complicated guards is improved.
  • Warnings for repeated identical map keys. #{'a'=>1, 'b'=>2, 'a'=>3} will warn for the repeated key a.
  • By default there is now a warning when export_all is used. Can be disabled
  • Pattern matching for maps is optimized
  • New option deterministic to omit path to source + options info the BEAM file.
  • Atoms may now contain arbitrary unicode characters.
  • compile:file/2 has an option to include extra chunks in the BEAM file.

Misc other applications

  • Unnamed ets tables optimized
  • A new event manager to handle a subset of OS signals in Erlang
  • Optimized sets add_element, del_element and union
  • Added rand:jump/0-1
  • When a gen_server crashes, the stacktrace for the client will be printed to facilitate debugging.
  • take/2 has been added to dict, orddict, and gb_trees.
  • take_any/2 has been added to gb_trees
  • Significantly updated string module with unicode support
  • erl_tar support for long path names and new file formats
  • Dtls: Documented API, experimental
  • SSH: improving security, removing and adding algorithms
  • New math:fmod/2

For more details see
http://erlang.org/download/otp_src_20.0-rc2.readme

Pre built versions for Windows can be fetched here:
http://erlang.org/download/otp_win32_20.0-rc2.exe
http://erlang.org/download/otp_win64_20.0-rc2.exe

On line documentation can be browsed here:
www.erlang.org/documentation/doc-9.0-rc2/doc/

Thanks to all contributors.

/The Erlang/OTP team at Ericsson

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang/OTP 20.0-rc2 is available for testing

Adam Jensen
In reply to this post by Kenneth Lundin-5
Hi,

I am attempting to build Erlang/OTP 20.0-rc2 with clang/llvm Release 4.0 on Scientific Linux 7.3 (a RHEL clone). "$ERL_TOP/HOWTO/INSTALL.md" seems to suggest that compiling with clang is supported/possible but "configure" dies fairly early [for me].

[hanzer@minerva otp]$ ./configure --prefix=$HOME/.local/otp-20.0-rc2
Ignoring the --cache-file argument since it can cause the system to be erroneously configured
Disabling caching
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for gcc... /home/hanzer/.local/llvm/bin/clang
checking whether the C compiler works... no
configure: error: in `/home/hanzer/build/otp':
configure: error: C compiler cannot create executables
See `config.log' for more details

[hanzer@minerva otp]$ cat config.log | nc termbin.com 9999
http://termbin.com/74uzy

<http://www.scientificlinux.org/>
<http://releases.llvm.org/4.0.0/docs/index.html>


If anyone wants to reproduce this, the entire process so far is:

----
sudo rpm -Uvh http://ftp.scientificlinux.org/linux/scientific/7x/external_products/softwarecollections/yum-conf-softwarecollections-2.0-1.el7.noarch.rpm
# See <http://linux.web.cern.ch/linux/centos7/docs/softwarecollections.shtml>
scl enable {devtoolset-6,python27} bash

mkdir build
export BASE=$HOME/build
cd $BASE
svn co http://llvm.org/svn/llvm-project/llvm/tags/RELEASE_400/final/ llvm_Rel400
cd $BASE/llvm_Rel400/tools
svn co http://llvm.org/svn/llvm-project/cfe/tags/RELEASE_400/final/ clang
svn co http://llvm.org/svn/llvm-project/lld/tags/RELEASE_400/final/ lld
svn co http://llvm.org/svn/llvm-project/polly/tags/RELEASE_400/final/ polly
cd $BASE/llvm_Rel400/tools/clang/tools
svn co http://llvm.org/svn/llvm-project/clang-tools-extra/tags/RELEASE_400/final/ extra
cd $BASE/llvm_Rel400/projects
svn co http://llvm.org/svn/llvm-project/compiler-rt/tags/RELEASE_400/final/ compiler-rt
svn co http://llvm.org/svn/llvm-project/openmp/tags/RELEASE_400/final/ openmp
svn co http://llvm.org/svn/llvm-project/libcxx/tags/RELEASE_400/final/ libcxx
svn co http://llvm.org/svn/llvm-project/libcxxabi/tags/RELEASE_400/final/ libcxxabi
svn co http://llvm.org/svn/llvm-project/test-suite/tags/RELEASE_400/final/ test-suite
mkdir $BASE/build_llvm
cd $BASE/build_llvm

cmake3 -G "Unix Makefiles" \
  -C $BASE/llvm_Rel400/tools/clang/cmake/caches/PGO-stage2-instrumented.cmake \
  -DLLVM_TARGETS_TO_BUILD=all \
  -DLLVM_BUILD_TOOLS=on \
  -DLLVM_BUILD_EXAMPLES=on \
  -DLLVM_BUILD_TESTS=on \
  -DCMAKE_INSTALL_PREFIX=$HOME/.local/llvm \
  -DCMAKE_BUILD_TYPE=Release $BASE/llvm_Rel400 \
  $BASE/llvm_Rel400

make -j4 2>&1 | tee build.log
make -j4 check-llvm 2>&1 | tee check-llvm.log
make -j4 check-clang 2>&1 | tee check-clang.log
make install

export PATH=$PATH:$HOME/.local/llvm/bin

export LANG=C
export CC=clang
export CXX=clang++
export CPP=clang-cpp
export LD=lld
export RANLIB=llvm-ranlib
export AR=llvm-ar
export GETCONF=llvm-config
export CFLAGS="-O4"
export CXXFLAGS="-O4"
export CPPFLAGS="-I/usr/include -I /usr/local/include -I$HOME/.local/llvm/include"
export LDFLAGS="-L/usr/lib -L/usr/lib64 -L/usr/local/lib -L/usr/local/lib64 -L$HOME/.local/llvm/lib"
export LIBS=/usr/lib:/usr/lib64:/usr/local/lib:/usr/local/lib64:$HOME/.local/llvm/lib/

cd $BASE
git clone https://github.com/erlang/otp.git
cd $BASE/otp
git checkout OTP-20.0-rc2
export ERL_TOP=`pwd`
./otp_build autoconf
export MAKEFLAGS=-j4
./configure --prefix=$HOME/.local/otp-20.0-rc2
make 2>&1 | tee build.log

make release_tests
cd release/tests/test_server
$ERL_TOP/bin/erl -s ts install -s ts smoke_test batch -s init stop
make install
cd $ERL_TOP
export PATH=$ERL_TOP/bin:$PATH
make docs
make install-docs

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang/OTP 20.0-rc2 is available for testing

Jachym Holecek
Hi,

# Adam Jensen 2017-06-05:
> [hanzer@minerva otp]$ cat config.log | nc termbin.com 9999
> http://termbin.com/74uzy

The failing command was:

        configure:2617: /home/hanzer/.local/llvm/bin/clang -O4 -I/home/hanzer/.local/llvm/include -L/home/hanzer/.local/llvm/lib conftest.c /usr/lib:/usr/lib64:/usr/local/lib:/usr/local/lib64:/home/hanzer/.local/llvm/lib/ >&5
        clang-4.0: error: no such file or directory: '/usr/lib:/usr/lib64:/usr/local/lib:/usr/local/lib64:/home/hanzer/.local/llvm/lib/'
        clang-4.0: warning: -O4 is equivalent to -O3 [-Wdeprecated]

Meaning the problem most likely is:

> If anyone wants to reproduce this, the entire process so far is:
>
> ----
>
> [...]
>
> export PATH=$PATH:$HOME/.local/llvm/bin
>
> export LANG=C
> export CC=clang
> export CXX=clang++
> export CPP=clang-cpp
> export LD=lld
> export RANLIB=llvm-ranlib
> export AR=llvm-ar
> export GETCONF=llvm-config
> export CFLAGS="-O4"
> export CXXFLAGS="-O4"
> export CPPFLAGS="-I/usr/include -I /usr/local/include -I$HOME/.local/llvm/include"
> export LDFLAGS="-L/usr/lib -L/usr/lib64 -L/usr/local/lib -L/usr/local/lib64 -L$HOME/.local/llvm/lib"
> export LIBS=/usr/lib:/usr/lib64:/usr/local/lib:/usr/local/lib64:$HOME/.local/llvm/lib/

The LIBS line above -- IIRC that's supposed to hold list of additional libraries to link
against, in the form of readily digestable linker arguments. Instead you're giving it
an unintelligible colon-separated thing. So keeping LIBS empty should sort it out.

BR,
        -- Jachym
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang/OTP 20.0-rc2 is available for testing

Ulf Wiger-2
In reply to this post by Kenneth Lundin-5
I like the ets:select_replace/2 function.

Of course, in the first case where I thought it might be useful, I ended up wishing for an extended version. ;-)


The 'claim' operation performs a chunked select of workers in a given pool, and for each chunk of workers, tries to claim one by incrementing a counter value - if the value went from 0 to 1, the claim succeeded.

A select_replace() function could handle the atomic claim operation, but what I would want to do is to limit it to only one replace and to return the resulting object - not just the count. This could potentially be supported by a select_replace/3 function.

As far as I can recall, gproc_pool is competitive in terms of performance. Such an extended select_replace() function _should_ give a significant speedup, especially in the case of large busy pools (which possibly haven't been benchmarked.)

BR,
Ulf W

2017-05-31 19:16 GMT+01:00 Kenneth Lundin <[hidden email]>:

OTP 20 Release Candidate 2

This is the second of two release candidates before the OTP 20 release. [...]


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang/OTP 20.0-rc2 is available for testing

Adam Jensen
In reply to this post by Adam Jensen
On Mon, 5 Jun 2017 17:22:18 -0400
Adam Jensen <[hidden email]> wrote:

> Hi,
>
> I am attempting to build Erlang/OTP 20.0-rc2 with clang/llvm Release 4.0 on Scientific Linux 7.3 (a RHEL clone). "$ERL_TOP/HOWTO/INSTALL.md" seems to suggest that compiling with clang is supported/possible but "configure" dies fairly early [for me].
>

[snip]

> export LIBS=/usr/lib:/usr/lib64:/usr/local/lib:/usr/local/lib64:$HOME/.local/llvm/lib/

Someone described the problem (outside of the mailing list) which was my misuse/misunderstanding of the "LIBS" environment variable. When "LIBS" is left unset, the compilation completes and passes the tests as expected.

Thanks!

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions