Patch Package OTP 22.0.6 Released

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Patch Package OTP 22.0.6 Released

Erlang/OTP
Patch Package:           OTP 22.0.6
Git Tag:                 OTP-22.0.6
Date:                    2019-07-10
Trouble Report Id:       OTP-15943, OTP-15945, OTP-15946, OTP-15949,
                         OTP-15952
Seq num:                
System:                  OTP
Release:                 22
Application:             compiler-7.4.3, dialyzer-4.0.3, hipe-3.19.1,
                         ssl-9.3.5
Predecessor:             OTP 22.0.5

 Check out the git tag OTP-22.0.6, and build a full OTP system
 including documentation. Apply one or more applications from this
 build as patches to your installation using the 'otp_patch_apply'
 tool. For information on install requirements, see descriptions for
 each application version below.

 ---------------------------------------------------------------------
 --- POTENTIAL INCOMPATIBILITIES -------------------------------------
 ---------------------------------------------------------------------

  OTP-15949    Application(s): dialyzer, hipe

               The HiPE compiler would badly miscompile certain
               try/catch expressions, so it will now refuse to compile
               modules containing try or catch.

               As a consequence of this, dialyzer will no longer
               compile key modules to native code.


 ---------------------------------------------------------------------
 --- compiler-7.4.3 --------------------------------------------------
 ---------------------------------------------------------------------

 The compiler-7.4.3 application can be applied independently of other
 applications on a full OTP 22 installation.

 --- Fixed Bugs and Malfunctions ---

  OTP-15945    Application(s): compiler

               Fixed an unsafe optimization when matching tuple_size/1
               outside of guards, which could crash the emulator if
               the argument was not a tuple.


  OTP-15946    Application(s): compiler

               Fixed a rare bug that could cause the wrong kind of
               exception to be thrown when a BIF failed in a function
               that matched bitstrings.


  OTP-15952    Application(s): compiler

               Fixed a bug where receive statements inside try/catch
               blocks could return incorrect results.


 Full runtime dependencies of compiler-7.4.3: crypto-3.6, erts-9.0,
 hipe-3.12, kernel-4.0, stdlib-2.5


 ---------------------------------------------------------------------
 --- dialyzer-4.0.3 --------------------------------------------------
 ---------------------------------------------------------------------

 The dialyzer-4.0.3 application can be applied independently of other
 applications on a full OTP 22 installation.

 --- Fixed Bugs and Malfunctions ---

  OTP-15949    Application(s): dialyzer, hipe

               *** POTENTIAL INCOMPATIBILITY ***

               The HiPE compiler would badly miscompile certain
               try/catch expressions, so it will now refuse to compile
               modules containing try or catch.

               As a consequence of this, dialyzer will no longer
               compile key modules to native code.


 Full runtime dependencies of dialyzer-4.0.3: compiler-7.0, erts-9.0,
 hipe-3.16.1, kernel-5.3, stdlib-3.4, syntax_tools-2.0, wx-1.2


 ---------------------------------------------------------------------
 --- hipe-3.19.1 -----------------------------------------------------
 ---------------------------------------------------------------------

 The hipe-3.19.1 application can be applied independently of other
 applications on a full OTP 22 installation.

 --- Fixed Bugs and Malfunctions ---

  OTP-15949    Application(s): dialyzer, hipe

               *** POTENTIAL INCOMPATIBILITY ***

               The HiPE compiler would badly miscompile certain
               try/catch expressions, so it will now refuse to compile
               modules containing try or catch.

               As a consequence of this, dialyzer will no longer
               compile key modules to native code.


 Full runtime dependencies of hipe-3.19.1: compiler-5.0, erts-9.3,
 kernel-5.3, stdlib-3.4, syntax_tools-1.6.14


 ---------------------------------------------------------------------
 --- ssl-9.3.5 -------------------------------------------------------
 ---------------------------------------------------------------------

 The ssl-9.3.5 application can be applied independently of other
 applications on a full OTP 22 installation.

 --- Improvements and New Features ---

  OTP-15943    Application(s): ssl

               Enhance error handling for erroneous alerts from the
               peer.


 Full runtime dependencies of ssl-9.3.5: crypto-4.2, erts-10.0,
 inets-5.10.7, kernel-6.0, public_key-1.5, stdlib-3.5


 ---------------------------------------------------------------------
 ---------------------------------------------------------------------
 ---------------------------------------------------------------------

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Patch Package OTP 22.0.6 Released

Loïc Hoguin-3
Hello,

On 10/07/2019 10:18, Erlang/OTP wrote:

>   ---------------------------------------------------------------------
>   --- POTENTIAL INCOMPATIBILITIES -------------------------------------
>   ---------------------------------------------------------------------
>
>    OTP-15949    Application(s): dialyzer, hipe
>
>                 The HiPE compiler would badly miscompile certain
>                 try/catch expressions, so it will now refuse to compile
>                 modules containing try or catch.
>
>                 As a consequence of this, dialyzer will no longer
>                 compile key modules to native code.

This means --enable-native-libs is now broken. Results in:

hipe.erl: native-code compilation failed because of an unimplemented
instruction (catch).

That's a bit rough.

Is HiPE just going to slowly rot or are there plans to revive it somehow
(official or not)? Wondering if I should just give up on testing against it.

Cheers,

--
Loïc Hoguin
https://ninenines.eu
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Patch Package OTP 22.0.6 Released

Frank Muller
Same issue here. With or without HIPE I’m getting this error (Ubuntu 16.04/18.04 and CentOs7):

Build failed.
make[3]: *** [../ebin/hipe.beam] Error 1
make[3]: Leaving directory '/home/frankm/.kerl/builds/22.0.6/otp_src_git/lib/hipe/main'
/home/frankm/.kerl/builds/22.0.6/otp_src_git/make/otp_subdir.mk:29: recipe for target 'opt' failed
make[2]: *** [opt] Error 2
make[2]: Leaving directory '/home/frankm/.kerl/builds/22.0.6/otp_src_git/lib/hipe'
/home/frankm/.kerl/builds/22.0.6/otp_src_git/make/otp_subdir.mk:29: recipe for target 'opt' failed
make[1]: *** [opt] Error 2
make[1]: Leaving directory '/home/frankm/.kerl/builds/22.0.6/otp_src_git/lib'
Makefile:490: recipe for target 'libs' failed
make: *** [libs] Error 2

Please see /home/frankm/.kerl/builds/22.0.6/otp_build_git.log for full details.


while 22.0.5 compiles perfectly. 

/Frank

Hello,

On 10/07/2019 10:18, Erlang/OTP wrote:
>   ---------------------------------------------------------------------
>   --- POTENTIAL INCOMPATIBILITIES -------------------------------------
>   ---------------------------------------------------------------------
>
>    OTP-15949    Application(s): dialyzer, hipe
>
>                 The HiPE compiler would badly miscompile certain
>                 try/catch expressions, so it will now refuse to compile
>                 modules containing try or catch.
>
>                 As a consequence of this, dialyzer will no longer
>                 compile key modules to native code.

This means --enable-native-libs is now broken. Results in:

hipe.erl: native-code compilation failed because of an unimplemented
instruction (catch).

That's a bit rough.

Is HiPE just going to slowly rot or are there plans to revive it somehow
(official or not)? Wondering if I should just give up on testing against it.

Cheers,

--
Loïc Hoguin
https://ninenines.eu
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Patch Package OTP 22.0.6 Released

Mikael Pettersson-5
In reply to this post by Loïc Hoguin-3
On Wed, Jul 10, 2019 at 6:08 PM Loïc Hoguin <[hidden email]> wrote:

> >    OTP-15949    Application(s): dialyzer, hipe
> >
> >                 The HiPE compiler would badly miscompile certain
> >                 try/catch expressions, so it will now refuse to compile
> >                 modules containing try or catch.
> >
> >                 As a consequence of this, dialyzer will no longer
> >                 compile key modules to native code.
>
> This means --enable-native-libs is now broken. Results in:
>
> hipe.erl: native-code compilation failed because of an unimplemented
> instruction (catch).
>
> That's a bit rough.
>
> Is HiPE just going to slowly rot or are there plans to revive it somehow

I had this discussion with Björn from the OTP team at the latest EUC
(er, "Code BEAM Sto").

The issue is that the original implementers of HiPE have long since
moved on to other things
and can't spend time on it any more.  At the same time the OTP team is only able
to support things their customers care about and pay for.  HiPE isn't
one of those things.

So it's up to the community as a whole to support HiPE, if it wants
it.  This could take the
form of one or more dedicated individuals taking on maintenance of
HiPE and fixing issues
as they are discovered.  (There's at least one such individual working
on the bit syntax
compilation issue in OTP-22 right now.)  Another solution could be for
interested parties to
fund maintenance via the Erlang Ecosystem Foundation.

The try/catch issue is news to me, but since the bit syntax
compilation issue was known
as OTP-22.0 was released, my view is that OTP should have deprecated
HiPE at that point
with the understanding that OTP-23 would delete HiPE unless the
community stepped up
and fixed the issues.  (And by that I don't mean "silently don't
compile things we can't handle
any more".)

/Mikael, ex-HiPE developer
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Patch Package OTP 22.0.6 Released

Mikael Pettersson-5
In reply to this post by Frank Muller
On Wed, Jul 10, 2019 at 6:23 PM Frank Muller <[hidden email]> wrote:

>
> Same issue here. With or without HIPE I’m getting this error (Ubuntu 16.04/18.04 and CentOs7):
>
> Build failed.
> make[3]: *** [../ebin/hipe.beam] Error 1
> make[3]: Leaving directory '/home/frankm/.kerl/builds/22.0.6/otp_src_git/lib/hipe/main'
> /home/frankm/.kerl/builds/22.0.6/otp_src_git/make/otp_subdir.mk:29: recipe for target 'opt' failed
> make[2]: *** [opt] Error 2
> make[2]: Leaving directory '/home/frankm/.kerl/builds/22.0.6/otp_src_git/lib/hipe'
> /home/frankm/.kerl/builds/22.0.6/otp_src_git/make/otp_subdir.mk:29: recipe for target 'opt' failed
> make[1]: *** [opt] Error 2
> make[1]: Leaving directory '/home/frankm/.kerl/builds/22.0.6/otp_src_git/lib'
> Makefile:490: recipe for target 'libs' failed
> make: *** [libs] Error 2

OTP-22.0.6 builds fine for me, with or without --enable-hipe, only
--enable-native-libs doesn't work so don't do that.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Patch Package OTP 22.0.6 Released

Frank Muller
Mickael

What —-enable-native-libs helps for?

/Frank

On Wed, Jul 10, 2019 at 6:23 PM Frank Muller <[hidden email]> wrote:
>
> Same issue here. With or without HIPE I’m getting this error (Ubuntu 16.04/18.04 and CentOs7):
>
> Build failed.
> make[3]: *** [../ebin/hipe.beam] Error 1
> make[3]: Leaving directory '/home/frankm/.kerl/builds/22.0.6/otp_src_git/lib/hipe/main'
> /home/frankm/.kerl/builds/22.0.6/otp_src_git/make/otp_subdir.mk:29: recipe for target 'opt' failed
> make[2]: *** [opt] Error 2
> make[2]: Leaving directory '/home/frankm/.kerl/builds/22.0.6/otp_src_git/lib/hipe'
> /home/frankm/.kerl/builds/22.0.6/otp_src_git/make/otp_subdir.mk:29: recipe for target 'opt' failed
> make[1]: *** [opt] Error 2
> make[1]: Leaving directory '/home/frankm/.kerl/builds/22.0.6/otp_src_git/lib'
> Makefile:490: recipe for target 'libs' failed
> make: *** [libs] Error 2

OTP-22.0.6 builds fine for me, with or without --enable-hipe, only
--enable-native-libs doesn't work so don't do that.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Patch Package OTP 22.0.6 Released

Mikael Pettersson-5
On Thu, Jul 11, 2019 at 8:12 AM Frank Muller <[hidden email]> wrote:
>
> Mickael
>
> What —-enable-native-libs helps for?

If you have HiPE enabled in the VM, this option also compiles (most
of) the standard Erlang libraries to native code.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Patch Package OTP 22.0.6 Released

Frank Muller
Even after disabling native-libs it still won’t compile on Ubuntu 16.04/18.04LTS.

I tried to disable HIPE as well but without luck.

And I’m facing the same problem with 22.0.7

Strange that the compilation broke after 22.0.5 

/Frank


On Thu, Jul 11, 2019 at 8:12 AM <[hidden email]> wrote:
>
> Mickael
>
> What —-enable-native-libs helps for?

If you have HiPE enabled in the VM, this option also compiles (most
of) the standard Erlang libraries to native code.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Patch Package OTP 22.0.6 Released

Frank Muller
Mikæl

I don’t use HIPE in my case. Can i disable it and set back —enable-native-libs?

It seems like —enable-native-libs makes the VM faster and I don’t want to lose that. Am i right or wrong?

/Frank

Le jeu. 11 juil. 2019 à 13:21, Frank Muller <[hidden email]> a écrit :
Even after disabling native-libs it still won’t compile on Ubuntu 16.04/18.04LTS.

I tried to disable HIPE as well but without luck.

And I’m facing the same problem with 22.0.7

Strange that the compilation broke after 22.0.5 

/Frank


On Thu, Jul 11, 2019 at 8:12 AM <[hidden email]> wrote:
>
> Mickael
>
> What —-enable-native-libs helps for?

If you have HiPE enabled in the VM, this option also compiles (most
of) the standard Erlang libraries to native code.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Patch Package OTP 22.0.6 Released

Rickard Green-2

On Thu, Jul 11, 2019 at 7:53 PM Frank Muller <[hidden email]> wrote:
Mikæl

I don’t use HIPE in my case. Can i disable it and set back —enable-native-libs?


--enable-native-libs needs hipe, and you don't want to use --enable-native-libs as it is now.

--
Rickard Green, Erlang/OTP, Ericsson AB

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Patch Package OTP 22.0.6 Released

Frank Muller
Got it, thanks!


On Thu, Jul 11, 2019 at 7:53 PM Frank Muller <[hidden email]> wrote:
Mikæl

I don’t use HIPE in my case. Can i disable it and set back —enable-native-libs?


--enable-native-libs needs hipe, and you don't want to use --enable-native-libs as it is now.


--
Rickard Green, Erlang/OTP, Ericsson AB

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Patch Package OTP 22.0.6 Released

Grzegorz Junka
In reply to this post by Mikael Pettersson-5
On 10/07/2019 22:00, Mikael Pettersson wrote:

> On Wed, Jul 10, 2019 at 6:08 PM Loïc Hoguin <[hidden email]> wrote:
>>>     OTP-15949    Application(s): dialyzer, hipe
>>>
>>>                  The HiPE compiler would badly miscompile certain
>>>                  try/catch expressions, so it will now refuse to compile
>>>                  modules containing try or catch.
>>>
>>>                  As a consequence of this, dialyzer will no longer
>>>                  compile key modules to native code.
>> This means --enable-native-libs is now broken. Results in:
>>
>> hipe.erl: native-code compilation failed because of an unimplemented
>> instruction (catch).
>>
>> That's a bit rough.
>>
>> Is HiPE just going to slowly rot or are there plans to revive it somehow
> I had this discussion with Björn from the OTP team at the latest EUC
> (er, "Code BEAM Sto").
>
> The issue is that the original implementers of HiPE have long since
> moved on to other things
> and can't spend time on it any more.  At the same time the OTP team is only able
> to support things their customers care about and pay for.  HiPE isn't
> one of those things.
>
> So it's up to the community as a whole to support HiPE, if it wants
> it.  This could take the
> form of one or more dedicated individuals taking on maintenance of
> HiPE and fixing issues
> as they are discovered.  (There's at least one such individual working
> on the bit syntax
> compilation issue in OTP-22 right now.)  Another solution could be for
> interested parties to
> fund maintenance via the Erlang Ecosystem Foundation.
>
> The try/catch issue is news to me, but since the bit syntax
> compilation issue was known
> as OTP-22.0 was released, my view is that OTP should have deprecated
> HiPE at that point
> with the understanding that OTP-23 would delete HiPE unless the
> community stepped up
> and fixed the issues.  (And by that I don't mean "silently don't
> compile things we can't handle
> any more".)
>
> /Mikael, ex-HiPE developer


Mikael, would it be possible for you or someone else from the OTP team
to provide more details about the minimal amount of work required to
bring HiPE back to the working order? I am looking for things like:

1. Any test suites available for HiPE (I would expect some tests not
passing due to recent changes)
2. Documentation detailing the HiPE implementation (if available)
3. New features that the new compiler no longer compiles for HiPE
4. Planned features that will require support in the compiler in the
foreseeable future

Also, is the HiPE code tightly integrated with OTP and its compiler or
it could be extracted into a separate repo and maintained at its own peace?

Thanks
GrzegorzJ

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Patch Package OTP 22.0.6 Released

Mikael Pettersson-5
On Fri, Jul 12, 2019 at 12:57 PM Grzegorz Junka <[hidden email]> wrote:

>
> On 10/07/2019 22:00, Mikael Pettersson wrote:
> > On Wed, Jul 10, 2019 at 6:08 PM Loïc Hoguin <[hidden email]> wrote:
> >>>     OTP-15949    Application(s): dialyzer, hipe
> >>>
> >>>                  The HiPE compiler would badly miscompile certain
> >>>                  try/catch expressions, so it will now refuse to compile
> >>>                  modules containing try or catch.
> >>>
> >>>                  As a consequence of this, dialyzer will no longer
> >>>                  compile key modules to native code.
> >> This means --enable-native-libs is now broken. Results in:
> >>
> >> hipe.erl: native-code compilation failed because of an unimplemented
> >> instruction (catch).
> >>
> >> That's a bit rough.
> >>
> >> Is HiPE just going to slowly rot or are there plans to revive it somehow
> > I had this discussion with Björn from the OTP team at the latest EUC
> > (er, "Code BEAM Sto").
> >
> > The issue is that the original implementers of HiPE have long since
> > moved on to other things
> > and can't spend time on it any more.  At the same time the OTP team is only able
> > to support things their customers care about and pay for.  HiPE isn't
> > one of those things.
> >
> > So it's up to the community as a whole to support HiPE, if it wants
> > it.  This could take the
> > form of one or more dedicated individuals taking on maintenance of
> > HiPE and fixing issues
> > as they are discovered.  (There's at least one such individual working
> > on the bit syntax
> > compilation issue in OTP-22 right now.)  Another solution could be for
> > interested parties to
> > fund maintenance via the Erlang Ecosystem Foundation.
> >
> > The try/catch issue is news to me, but since the bit syntax
> > compilation issue was known
> > as OTP-22.0 was released, my view is that OTP should have deprecated
> > HiPE at that point
> > with the understanding that OTP-23 would delete HiPE unless the
> > community stepped up
> > and fixed the issues.  (And by that I don't mean "silently don't
> > compile things we can't handle
> > any more".)
> >
> > /Mikael, ex-HiPE developer
>
>
> Mikael, would it be possible for you or someone else from the OTP team

Note: drop "else", I'm not in the OTP team, just a long-time contributor
to Erlang/OTP.

> to provide more details about the minimal amount of work required to
> bring HiPE back to the working order? I am looking for things like:
>
> 1. Any test suites available for HiPE (I would expect some tests not
> passing due to recent changes)

HiPE used to maintain its own internal test suite.  The HiPE compiler
application in OTP includes some tests, but I have no idea how
complete it is; a quick look indicates it doesn't include everything
from the old internal test suite.  A snapshot of that test suite as of early
2011 is available in <https://github.com/andysan/hipe_tests>, but there
are a few later additions that AFAIK are only available in a 2014 CVS
snapshot I have locally.

> 2. Documentation detailing the HiPE implementation (if available)

Go to <https://www.it.uu.se/research/group/hipe/> and read the
various publications.  Those and the source code (erts/emulator/hipe/,
other parts of erts/ under #ifdef HIPE, lib/hipe/ (the compiler), and a
few other bits under lib/) are the documentation.

> 3. New features that the new compiler no longer compiles for HiPE
> 4. Planned features that will require support in the compiler in the
> foreseeable future
>
> Also, is the HiPE code tightly integrated with OTP and its compiler or
> it could be extracted into a separate repo and maintained at its own peace?

HiPE and OTP are mutually dependent:
- the HiPE compiler's input is the BEAM code output by the OTP compiler,
  so whenever BEAM instructions are added or the OTP compiler changes
  drastically, the HiPE compiler needs to be updated
- the OTP compiler dispatches to the HiPE compiler if you compile with the
  native flag enabled
- the OTP code loader dispatches to the HiPE code loader for modules
  compiled to native code
- the OTP VM (aka runtime system) has extensions to
  + load native code
  + allow Erlang processes to call native code from BEAM and vice versa
  + likewise handle throwing exceptions across the two execution modes
  + garbage collection of the native code's stack
  + handling dynamic (re-)loading of modules containing native code

You can't distribute this as just another "application".

As for the concrete problems right now:
1. The binary compilation issue is documented in both the bug tracker
    and as a PR on github, so I won't expand on that.
2. The "try/catch miscompilation" issue is mentioned in the OTP-22.0.6
    release notes, but the only reference is to an OTP-internal bug entry
    (OTP-15949) and there's no test case added, making it difficult to
    understand what the issue is or how to fix it.

Finally, a major part of the problem is that OTP is a moving target.  The
VM keeps changing (new BEAM instructions, new conventions for
memory management or BIF calls), and the OTP compiler also changes,
and each such change has the potential for breaking HiPE.

/Mikael
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Patch Package OTP 22.0.6 Released

Grzegorz Junka

On 14/07/2019 15:48, Mikael Pettersson wrote:

> On Fri, Jul 12, 2019 at 12:57 PM Grzegorz Junka <[hidden email]> wrote:
>> On 10/07/2019 22:00, Mikael Pettersson wrote:
>>> On Wed, Jul 10, 2019 at 6:08 PM Loïc Hoguin <[hidden email]> wrote:
>>>>>      OTP-15949    Application(s): dialyzer, hipe
>>>>>
>>>>>                   The HiPE compiler would badly miscompile certain
>>>>>                   try/catch expressions, so it will now refuse to compile
>>>>>                   modules containing try or catch.
>>>>>
>>>>>                   As a consequence of this, dialyzer will no longer
>>>>>                   compile key modules to native code.
>>>> This means --enable-native-libs is now broken. Results in:
>>>>
>>>> hipe.erl: native-code compilation failed because of an unimplemented
>>>> instruction (catch).
>>>>
>>>> That's a bit rough.
>>>>
>>>> Is HiPE just going to slowly rot or are there plans to revive it somehow
>>> I had this discussion with Björn from the OTP team at the latest EUC
>>> (er, "Code BEAM Sto").
>>>
>>> The issue is that the original implementers of HiPE have long since
>>> moved on to other things
>>> and can't spend time on it any more.  At the same time the OTP team is only able
>>> to support things their customers care about and pay for.  HiPE isn't
>>> one of those things.
>>>
>>> So it's up to the community as a whole to support HiPE, if it wants
>>> it.  This could take the
>>> form of one or more dedicated individuals taking on maintenance of
>>> HiPE and fixing issues
>>> as they are discovered.  (There's at least one such individual working
>>> on the bit syntax
>>> compilation issue in OTP-22 right now.)  Another solution could be for
>>> interested parties to
>>> fund maintenance via the Erlang Ecosystem Foundation.
>>>
>>> The try/catch issue is news to me, but since the bit syntax
>>> compilation issue was known
>>> as OTP-22.0 was released, my view is that OTP should have deprecated
>>> HiPE at that point
>>> with the understanding that OTP-23 would delete HiPE unless the
>>> community stepped up
>>> and fixed the issues.  (And by that I don't mean "silently don't
>>> compile things we can't handle
>>> any more".)
>>>
>>> /Mikael, ex-HiPE developer
>>
>> Mikael, would it be possible for you or someone else from the OTP team
> Note: drop "else", I'm not in the OTP team, just a long-time contributor
> to Erlang/OTP.
>
>> to provide more details about the minimal amount of work required to
>> bring HiPE back to the working order? I am looking for things like:
>>
>> 1. Any test suites available for HiPE (I would expect some tests not
>> passing due to recent changes)
> HiPE used to maintain its own internal test suite.  The HiPE compiler
> application in OTP includes some tests, but I have no idea how
> complete it is; a quick look indicates it doesn't include everything
> from the old internal test suite.  A snapshot of that test suite as of early
> 2011 is available in <https://github.com/andysan/hipe_tests>, but there
> are a few later additions that AFAIK are only available in a 2014 CVS
> snapshot I have locally.


Thanks for all the info. Is that CVS snapshot available anywhere apart
from your local comp? Is it possible to upload it for example to Github?

GrzegorzJ
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Patch Package OTP 22.0.6 Released

Grzegorz Junka
In reply to this post by Mikael Pettersson-5

On 14/07/2019 15:48, Mikael Pettersson wrote:

> On Fri, Jul 12, 2019 at 12:57 PM Grzegorz Junka <[hidden email]> wrote:
>> On 10/07/2019 22:00, Mikael Pettersson wrote:
>>> On Wed, Jul 10, 2019 at 6:08 PM Loïc Hoguin <[hidden email]> wrote:
>>>>>      OTP-15949    Application(s): dialyzer, hipe
>>>>>
>>>>>                   The HiPE compiler would badly miscompile certain
>>>>>                   try/catch expressions, so it will now refuse to compile
>>>>>                   modules containing try or catch.
>>>>>
>>>>>                   As a consequence of this, dialyzer will no longer
>>>>>                   compile key modules to native code.
>>>> This means --enable-native-libs is now broken. Results in:
>>>>
>>>> hipe.erl: native-code compilation failed because of an unimplemented
>>>> instruction (catch).
>>>>
>>>> That's a bit rough.
>>>>
>>>> Is HiPE just going to slowly rot or are there plans to revive it somehow
>>> I had this discussion with Björn from the OTP team at the latest EUC
>>> (er, "Code BEAM Sto").
>>>
>>> The issue is that the original implementers of HiPE have long since
>>> moved on to other things
>>> and can't spend time on it any more.  At the same time the OTP team is only able
>>> to support things their customers care about and pay for.  HiPE isn't
>>> one of those things.
>>>
>>> So it's up to the community as a whole to support HiPE, if it wants
>>> it.  This could take the
>>> form of one or more dedicated individuals taking on maintenance of
>>> HiPE and fixing issues
>>> as they are discovered.  (There's at least one such individual working
>>> on the bit syntax
>>> compilation issue in OTP-22 right now.)  Another solution could be for
>>> interested parties to
>>> fund maintenance via the Erlang Ecosystem Foundation.
>>>
>>> The try/catch issue is news to me, but since the bit syntax
>>> compilation issue was known
>>> as OTP-22.0 was released, my view is that OTP should have deprecated
>>> HiPE at that point
>>> with the understanding that OTP-23 would delete HiPE unless the
>>> community stepped up
>>> and fixed the issues.  (And by that I don't mean "silently don't
>>> compile things we can't handle
>>> any more".)
>>>
>>> /Mikael, ex-HiPE developer
>>
>> Mikael, would it be possible for you or someone else from the OTP team
> Note: drop "else", I'm not in the OTP team, just a long-time contributor
> to Erlang/OTP.
>
>> to provide more details about the minimal amount of work required to
>> bring HiPE back to the working order? I am looking for things like:
>>
>> 1. Any test suites available for HiPE (I would expect some tests not
>> passing due to recent changes)
> HiPE used to maintain its own internal test suite.  The HiPE compiler
> application in OTP includes some tests, but I have no idea how
> complete it is; a quick look indicates it doesn't include everything
> from the old internal test suite.  A snapshot of that test suite as of early
> 2011 is available in <https://github.com/andysan/hipe_tests>, but there
> are a few later additions that AFAIK are only available in a 2014 CVS
> snapshot I have locally.
>
>> 2. Documentation detailing the HiPE implementation (if available)
> Go to <https://www.it.uu.se/research/group/hipe/> and read the
> various publications.  Those and the source code (erts/emulator/hipe/,
> other parts of erts/ under #ifdef HIPE, lib/hipe/ (the compiler), and a
> few other bits under lib/) are the documentation.
>
>> 3. New features that the new compiler no longer compiles for HiPE
>> 4. Planned features that will require support in the compiler in the
>> foreseeable future
>>
>> Also, is the HiPE code tightly integrated with OTP and its compiler or
>> it could be extracted into a separate repo and maintained at its own peace?
> HiPE and OTP are mutually dependent:
> - the HiPE compiler's input is the BEAM code output by the OTP compiler,
>    so whenever BEAM instructions are added or the OTP compiler changes
>    drastically, the HiPE compiler needs to be updated
> - the OTP compiler dispatches to the HiPE compiler if you compile with the
>    native flag enabled
> - the OTP code loader dispatches to the HiPE code loader for modules
>    compiled to native code
> - the OTP VM (aka runtime system) has extensions to
>    + load native code
>    + allow Erlang processes to call native code from BEAM and vice versa
>    + likewise handle throwing exceptions across the two execution modes
>    + garbage collection of the native code's stack
>    + handling dynamic (re-)loading of modules containing native code


Sorry, forgot to ask one more question. If HiPE compiler's input is the
BEAM code then I understand interested in HiPE may not only be the
Erlang community but also other communities that use the BEAM VM, e.g.
Elixir and some other listed here
https://github.com/llaisdy/beam_languages ?

GrzegorzJ

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Patch Package OTP 22.0.6 Released

Mikael Pettersson-5
In reply to this post by Grzegorz Junka
On Mon, Jul 15, 2019 at 1:55 PM Grzegorz Junka <[hidden email]> wrote:

>
>
> On 14/07/2019 15:48, Mikael Pettersson wrote:
> > On Fri, Jul 12, 2019 at 12:57 PM Grzegorz Junka <[hidden email]> wrote:
> >> On 10/07/2019 22:00, Mikael Pettersson wrote:
> >>> On Wed, Jul 10, 2019 at 6:08 PM Loïc Hoguin <[hidden email]> wrote:
> >>>>>      OTP-15949    Application(s): dialyzer, hipe
> >>>>>
> >>>>>                   The HiPE compiler would badly miscompile certain
> >>>>>                   try/catch expressions, so it will now refuse to compile
> >>>>>                   modules containing try or catch.
> >>>>>
> >>>>>                   As a consequence of this, dialyzer will no longer
> >>>>>                   compile key modules to native code.
> >>>> This means --enable-native-libs is now broken. Results in:
> >>>>
> >>>> hipe.erl: native-code compilation failed because of an unimplemented
> >>>> instruction (catch).
> >>>>
> >>>> That's a bit rough.
> >>>>
> >>>> Is HiPE just going to slowly rot or are there plans to revive it somehow
> >>> I had this discussion with Björn from the OTP team at the latest EUC
> >>> (er, "Code BEAM Sto").
> >>>
> >>> The issue is that the original implementers of HiPE have long since
> >>> moved on to other things
> >>> and can't spend time on it any more.  At the same time the OTP team is only able
> >>> to support things their customers care about and pay for.  HiPE isn't
> >>> one of those things.
> >>>
> >>> So it's up to the community as a whole to support HiPE, if it wants
> >>> it.  This could take the
> >>> form of one or more dedicated individuals taking on maintenance of
> >>> HiPE and fixing issues
> >>> as they are discovered.  (There's at least one such individual working
> >>> on the bit syntax
> >>> compilation issue in OTP-22 right now.)  Another solution could be for
> >>> interested parties to
> >>> fund maintenance via the Erlang Ecosystem Foundation.
> >>>
> >>> The try/catch issue is news to me, but since the bit syntax
> >>> compilation issue was known
> >>> as OTP-22.0 was released, my view is that OTP should have deprecated
> >>> HiPE at that point
> >>> with the understanding that OTP-23 would delete HiPE unless the
> >>> community stepped up
> >>> and fixed the issues.  (And by that I don't mean "silently don't
> >>> compile things we can't handle
> >>> any more".)
> >>>
> >>> /Mikael, ex-HiPE developer
> >>
> >> Mikael, would it be possible for you or someone else from the OTP team
> > Note: drop "else", I'm not in the OTP team, just a long-time contributor
> > to Erlang/OTP.
> >
> >> to provide more details about the minimal amount of work required to
> >> bring HiPE back to the working order? I am looking for things like:
> >>
> >> 1. Any test suites available for HiPE (I would expect some tests not
> >> passing due to recent changes)
> > HiPE used to maintain its own internal test suite.  The HiPE compiler
> > application in OTP includes some tests, but I have no idea how
> > complete it is; a quick look indicates it doesn't include everything
> > from the old internal test suite.  A snapshot of that test suite as of early
> > 2011 is available in <https://github.com/andysan/hipe_tests>, but there
> > are a few later additions that AFAIK are only available in a 2014 CVS
> > snapshot I have locally.
>
>
> Thanks for all the info. Is that CVS snapshot available anywhere apart
> from your local comp? Is it possible to upload it for example to Github?

Not yet, but I can fork andysan's repo and add the missing bits on my github.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Patch Package OTP 22.0.6 Released

Mikael Pettersson-5
In reply to this post by Grzegorz Junka
On Mon, Jul 15, 2019 at 2:00 PM Grzegorz Junka <[hidden email]> wrote:

>
>
> On 14/07/2019 15:48, Mikael Pettersson wrote:
> > On Fri, Jul 12, 2019 at 12:57 PM Grzegorz Junka <[hidden email]> wrote:
> >> On 10/07/2019 22:00, Mikael Pettersson wrote:
> >>> On Wed, Jul 10, 2019 at 6:08 PM Loïc Hoguin <[hidden email]> wrote:
> >>>>>      OTP-15949    Application(s): dialyzer, hipe
> >>>>>
> >>>>>                   The HiPE compiler would badly miscompile certain
> >>>>>                   try/catch expressions, so it will now refuse to compile
> >>>>>                   modules containing try or catch.
> >>>>>
> >>>>>                   As a consequence of this, dialyzer will no longer
> >>>>>                   compile key modules to native code.
> >>>> This means --enable-native-libs is now broken. Results in:
> >>>>
> >>>> hipe.erl: native-code compilation failed because of an unimplemented
> >>>> instruction (catch).
> >>>>
> >>>> That's a bit rough.
> >>>>
> >>>> Is HiPE just going to slowly rot or are there plans to revive it somehow
> >>> I had this discussion with Björn from the OTP team at the latest EUC
> >>> (er, "Code BEAM Sto").
> >>>
> >>> The issue is that the original implementers of HiPE have long since
> >>> moved on to other things
> >>> and can't spend time on it any more.  At the same time the OTP team is only able
> >>> to support things their customers care about and pay for.  HiPE isn't
> >>> one of those things.
> >>>
> >>> So it's up to the community as a whole to support HiPE, if it wants
> >>> it.  This could take the
> >>> form of one or more dedicated individuals taking on maintenance of
> >>> HiPE and fixing issues
> >>> as they are discovered.  (There's at least one such individual working
> >>> on the bit syntax
> >>> compilation issue in OTP-22 right now.)  Another solution could be for
> >>> interested parties to
> >>> fund maintenance via the Erlang Ecosystem Foundation.
> >>>
> >>> The try/catch issue is news to me, but since the bit syntax
> >>> compilation issue was known
> >>> as OTP-22.0 was released, my view is that OTP should have deprecated
> >>> HiPE at that point
> >>> with the understanding that OTP-23 would delete HiPE unless the
> >>> community stepped up
> >>> and fixed the issues.  (And by that I don't mean "silently don't
> >>> compile things we can't handle
> >>> any more".)
> >>>
> >>> /Mikael, ex-HiPE developer
> >>
> >> Mikael, would it be possible for you or someone else from the OTP team
> > Note: drop "else", I'm not in the OTP team, just a long-time contributor
> > to Erlang/OTP.
> >
> >> to provide more details about the minimal amount of work required to
> >> bring HiPE back to the working order? I am looking for things like:
> >>
> >> 1. Any test suites available for HiPE (I would expect some tests not
> >> passing due to recent changes)
> > HiPE used to maintain its own internal test suite.  The HiPE compiler
> > application in OTP includes some tests, but I have no idea how
> > complete it is; a quick look indicates it doesn't include everything
> > from the old internal test suite.  A snapshot of that test suite as of early
> > 2011 is available in <https://github.com/andysan/hipe_tests>, but there
> > are a few later additions that AFAIK are only available in a 2014 CVS
> > snapshot I have locally.
> >
> >> 2. Documentation detailing the HiPE implementation (if available)
> > Go to <https://www.it.uu.se/research/group/hipe/> and read the
> > various publications.  Those and the source code (erts/emulator/hipe/,
> > other parts of erts/ under #ifdef HIPE, lib/hipe/ (the compiler), and a
> > few other bits under lib/) are the documentation.
> >
> >> 3. New features that the new compiler no longer compiles for HiPE
> >> 4. Planned features that will require support in the compiler in the
> >> foreseeable future
> >>
> >> Also, is the HiPE code tightly integrated with OTP and its compiler or
> >> it could be extracted into a separate repo and maintained at its own peace?
> > HiPE and OTP are mutually dependent:
> > - the HiPE compiler's input is the BEAM code output by the OTP compiler,
> >    so whenever BEAM instructions are added or the OTP compiler changes
> >    drastically, the HiPE compiler needs to be updated
> > - the OTP compiler dispatches to the HiPE compiler if you compile with the
> >    native flag enabled
> > - the OTP code loader dispatches to the HiPE code loader for modules
> >    compiled to native code
> > - the OTP VM (aka runtime system) has extensions to
> >    + load native code
> >    + allow Erlang processes to call native code from BEAM and vice versa
> >    + likewise handle throwing exceptions across the two execution modes
> >    + garbage collection of the native code's stack
> >    + handling dynamic (re-)loading of modules containing native code
>
>
> Sorry, forgot to ask one more question. If HiPE compiler's input is the
> BEAM code then I understand interested in HiPE may not only be the
> Erlang community but also other communities that use the BEAM VM, e.g.
> Elixir and some other listed here
> https://github.com/llaisdy/beam_languages ?

HiPE should work with any language that compiles to Erlang.
It may also work with languages that compile via Core Erlang, but
that path is less tested.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions