NIF + HiPE - can it works together?

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

NIF + HiPE - can it works together?

Radosław Bułat
I'm playing with some erlang stuffs (still learning). Lately I wanted
to compile erlang module with HiPE and also with NIF extension.
But when I try to load nif (erlang:load_nif function) I get error:

{"init terminating in do_boot",{{badmatch,{error,{bad_lib,"Library
module name 'gim_crypto' does not match calling module
'lists'"}}},[{gim_crypto,load_nif,0},{gim_benchmark,bm,1},{lists,flatten,1}]}}

When compiling without HiPE everything works fine. Am I doing
something wrong or HiPE doesn't work with NIF?

--
Pozdrawiam

Radosław Bułat
http://radarek.jogger.pl - mój blog
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: NIF + HiPE - can it works together?

Mikael Pettersson-3
On Mon, 6 Jun 2011 22:35:50 +0200, Rados³aw Bu³at <[hidden email]> wrote:

> I'm playing with some erlang stuffs (still learning). Lately I wanted
> to compile erlang module with HiPE and also with NIF extension.
> But when I try to load nif (erlang:load_nif function) I get error:
>
> {"init terminating in do_boot",{{badmatch,{error,{bad_lib,"Library
> module name 'gim_crypto' does not match calling module
> 'lists'"}}},[{gim_crypto,load_nif,0},{gim_benchmark,bm,1},{lists,flatten,1}]}}
>
> When compiling without HiPE everything works fine. Am I doing
> something wrong or HiPE doesn't work with NIF?

Please post a standalone and minimal test case.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: NIF + HiPE - can it works together?

Radosław Bułat
2011/6/6 Mikael Pettersson <[hidden email]>:
> Please post a standalone and minimal test case.
>
Yeah, sure.

$ git clone https://github.com/radarek/nif_with_hipe
$ cd nif_with_hipe
$ less README

$ cat rebar.config
$ ./rebar compile # compile with hipe support (see rebar.config)
$ erl -pa ../nif_with_hipe/ebin/ -s nif_with_hipe start -s init stop
Erlang R14B03 (erts-5.8.4) [source] [64-bit] [smp:2:2] [rq:2]
[async-threads:0] [hipe] [kernel-poll:false]

{"init terminating in do_boot",{{badmatch,{error,{bad_lib,"Library
module name 'nif_with_hipe' does not match calling module
'init'"}}},[{nif_with_hipe,load_nif,0},{nif_with_hipe,start,0},{code,lib_dir,1}]}}

Crash dump was written to: erl_crash.dump
init terminating in do_boot ()

$ # now remove rebar.config and run again
$ mv rebar.config rebar.config.deleted
$ ./rebar clean compile
$ erl -pa ../nif_with_hipe/ebin/ -s nif_with_hipe start -s init stop
'atom from nif!'
Eshell V5.8.4  (abort with ^G)
1>


--
Pozdrawiam

Radosław Bułat
http://radarek.jogger.pl - mój blog
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: NIF + HiPE - can it works together?

Radosław Bułat
2011/6/7 Radosław Bułat <[hidden email]>:
> 2011/6/6 Mikael Pettersson <[hidden email]>:
>> Please post a standalone and minimal test case.
>>
> Yeah, sure.
>

One more thing. When module is compiled with HiPE and try to use
on_load attribute it ends with error:

<HiPE (v 3.8)> EXITED with reason {'trans_fun/2',on_load}
@hipe_beam_to_icode:1148

=ERROR REPORT==== 7-Jun-2011::01:18:36 ===
Error in process <0.50.0> with exit value:
{{badmatch,{'EXIT',{{hipe_beam_to_icode,1148,{'trans_fun/2',on_load}},[{hipe_beam_to_icode,trans_fun,2},{hipe_beam_to_icode,trans_fun,2},{hipe_beam_to_icode,trans_mfa_code,5},{hipe_beam_to_icode,trans_beam_function_chunk...

src/nif_with_hipe.erl:none: internal error in native_compile;
crash reason: {{badmatch,
                   {'EXIT',
                       {{hipe_beam_to_icode,1148,{'trans_fun/2',on_load}},
                        [{hipe_beam_to_icode,trans_fun,2},
                         {hipe_beam_to_icode,trans_fun,2},
                         {hipe_beam_to_icode,trans_mfa_code,5},
                         {hipe_beam_to_icode,trans_beam_function_chunk,2},
                         {hipe_beam_to_icode,'-module/2-lc$^1/1-1-',2},
                         {hipe,get_beam_icode,4},
                         {hipe,'-run_compiler_1/3-fun-0-',4}]}}},
               [{hipe,get_beam_icode,4},{hipe,'-run_compiler_1/3-fun-0-',4}]}

--
Pozdrawiam

Radosław Bułat
http://radarek.jogger.pl - mój blog
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: NIF + HiPE - can it works together?

Paul Davis-3
I'm fairly certain that I saw somewhere that HiPE is incompatible with
NIF's but I'm unable to actually Google a reference so I could be
making that up based on that on_init error that I've seen before.

If so, I'd probably just shove the code you want HiPE compiled into a
lib module and have the nif stubs call out to that as necessary.

HTH,
Paul Davis

2011/6/6 Radosław Bułat <[hidden email]>:

> 2011/6/7 Radosław Bułat <[hidden email]>:
>> 2011/6/6 Mikael Pettersson <[hidden email]>:
>>> Please post a standalone and minimal test case.
>>>
>> Yeah, sure.
>>
>
> One more thing. When module is compiled with HiPE and try to use
> on_load attribute it ends with error:
>
> <HiPE (v 3.8)> EXITED with reason {'trans_fun/2',on_load}
> @hipe_beam_to_icode:1148
>
> =ERROR REPORT==== 7-Jun-2011::01:18:36 ===
> Error in process <0.50.0> with exit value:
> {{badmatch,{'EXIT',{{hipe_beam_to_icode,1148,{'trans_fun/2',on_load}},[{hipe_beam_to_icode,trans_fun,2},{hipe_beam_to_icode,trans_fun,2},{hipe_beam_to_icode,trans_mfa_code,5},{hipe_beam_to_icode,trans_beam_function_chunk...
>
> src/nif_with_hipe.erl:none: internal error in native_compile;
> crash reason: {{badmatch,
>                   {'EXIT',
>                       {{hipe_beam_to_icode,1148,{'trans_fun/2',on_load}},
>                        [{hipe_beam_to_icode,trans_fun,2},
>                         {hipe_beam_to_icode,trans_fun,2},
>                         {hipe_beam_to_icode,trans_mfa_code,5},
>                         {hipe_beam_to_icode,trans_beam_function_chunk,2},
>                         {hipe_beam_to_icode,'-module/2-lc$^1/1-1-',2},
>                         {hipe,get_beam_icode,4},
>                         {hipe,'-run_compiler_1/3-fun-0-',4}]}}},
>               [{hipe,get_beam_icode,4},{hipe,'-run_compiler_1/3-fun-0-',4}]}
>
> --
> Pozdrawiam
>
> Radosław Bułat
> http://radarek.jogger.pl - mój blog
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: NIF + HiPE - can it works together?

Sverker Eriksson
I don't know if it has been stated, but HiPE and NIFs do not play nice
together *within the same module*.
Using HiPE and NIFs on different modules within the same VM will
hopefully work fine.


There seem to be a number of problems here.

1. erlang:load_nif fails as reported by Radosław Bułat. This seems to be
a "simple" bug that can be fixed, but even if so...

2. The NIF overriding mechanism is not implemented for hipe and would
probably yield unpredictable results.

3. Hipe compiler fails on on_load as reported.

I will look into the first two.  Not sure if #3 was expected not to
work? (Mikael? Kostis?)


/Sverker, Erlang/OTP

Paul Davis wrote:

> I'm fairly certain that I saw somewhere that HiPE is incompatible with
> NIF's but I'm unable to actually Google a reference so I could be
> making that up based on that on_init error that I've seen before.
>
> If so, I'd probably just shove the code you want HiPE compiled into a
> lib module and have the nif stubs call out to that as necessary.
>
> HTH,
> Paul Davis
>
> 2011/6/6 Radosław Bułat <[hidden email]>:
>  
>> 2011/6/7 Radosław Bułat <[hidden email]>:
>>    
>>> 2011/6/6 Mikael Pettersson <[hidden email]>:
>>>      
>>>> Please post a standalone and minimal test case.
>>>>
>>>>        
>>> Yeah, sure.
>>>
>>>      
>> One more thing. When module is compiled with HiPE and try to use
>> on_load attribute it ends with error:
>>
>> <HiPE (v 3.8)> EXITED with reason {'trans_fun/2',on_load}
>> @hipe_beam_to_icode:1148
>>
>> =ERROR REPORT==== 7-Jun-2011::01:18:36 ===
>> Error in process <0.50.0> with exit value:
>> {{badmatch,{'EXIT',{{hipe_beam_to_icode,1148,{'trans_fun/2',on_load}},[{hipe_beam_to_icode,trans_fun,2},{hipe_beam_to_icode,trans_fun,2},{hipe_beam_to_icode,trans_mfa_code,5},{hipe_beam_to_icode,trans_beam_function_chunk...
>>
>> src/nif_with_hipe.erl:none: internal error in native_compile;
>> crash reason: {{badmatch,
>>                   {'EXIT',
>>                       {{hipe_beam_to_icode,1148,{'trans_fun/2',on_load}},
>>                        [{hipe_beam_to_icode,trans_fun,2},
>>                         {hipe_beam_to_icode,trans_fun,2},
>>                         {hipe_beam_to_icode,trans_mfa_code,5},
>>                         {hipe_beam_to_icode,trans_beam_function_chunk,2},
>>                         {hipe_beam_to_icode,'-module/2-lc$^1/1-1-',2},
>>                         {hipe,get_beam_icode,4},
>>                         {hipe,'-run_compiler_1/3-fun-0-',4}]}}},
>>               [{hipe,get_beam_icode,4},{hipe,'-run_compiler_1/3-fun-0-',4}]}
>>
>> --
>> Pozdrawiam
>>
>> Radosław Bułat
>> http://radarek.jogger.pl - mój blog
>> _______________________________________________
>> erlang-questions mailing list
>> [hidden email]
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>    
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
>  


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: NIF + HiPE - can it works together?

Mikael Pettersson-3
On Wed, 8 Jun 2011 15:55:35 +0200, Sverker Eriksson <[hidden email]> wrote:

> I don't know if it has been stated, but HiPE and NIFs do not play nice=20
> together *within the same module*.
> Using HiPE and NIFs on different modules within the same VM will=20
> hopefully work fine.
>
>
> There seem to be a number of problems here.
>
> 1. erlang:load_nif fails as reported by Rados=C5=82aw Bu=C5=82at. This se=
> ems to be=20
> a "simple" bug that can be fixed, but even if so...
>
> 2. The NIF overriding mechanism is not implemented for hipe and would=20
> probably yield unpredictable results.
>
> 3. Hipe compiler fails on on_load as reported.
>
> I will look into the first two.  Not sure if #3 was expected not to=20
> work? (Mikael? Kostis?)

Last time I checked on_load was unsupportable by HiPE.  The BEAM loader
did some hacks behind the scenes that it could get away with, but which
were unexpected by and incompatible with the HiPE loader.

Keeping HiPE and NIFs separate seems to be the only viable solution currently.

Alas there are effectively no compiler/vm developers left in the HiPE team;
we've all left to pursue other career paths.

/Mikael

>
>
> /Sverker, Erlang/OTP
>
> Paul Davis wrote:
> > I'm fairly certain that I saw somewhere that HiPE is incompatible with
> > NIF's but I'm unable to actually Google a reference so I could be
> > making that up based on that on_init error that I've seen before.
> >
> > If so, I'd probably just shove the code you want HiPE compiled into a
> > lib module and have the nif stubs call out to that as necessary.
> >
> > HTH,
> > Paul Davis
> >
> > 2011/6/6 Rados=C5=82aw Bu=C5=82at <[hidden email]>:
> >  =20
> >> 2011/6/7 Rados=C5=82aw Bu=C5=82at <[hidden email]>:
> >>    =20
> >>> 2011/6/6 Mikael Pettersson <[hidden email]>:
> >>>      =20
> >>>> Please post a standalone and minimal test case.
> >>>>
> >>>>        =20
> >>> Yeah, sure.
> >>>
> >>>      =20
> >> One more thing. When module is compiled with HiPE and try to use
> >> on_load attribute it ends with error:
> >>
> >> <HiPE (v 3.8)> EXITED with reason {'trans_fun/2',on_load}
> >> @hipe_beam_to_icode:1148
> >>
> >> =3DERROR REPORT=3D=3D=3D=3D 7-Jun-2011::01:18:36 =3D=3D=3D
> >> Error in process <0.50.0> with exit value:
> >> {{badmatch,{'EXIT',{{hipe_beam_to_icode,1148,{'trans_fun/2',on_load}},=
> [{hipe_beam_to_icode,trans_fun,2},{hipe_beam_to_icode,trans_fun,2},{hipe_=
> beam_to_icode,trans_mfa_code,5},{hipe_beam_to_icode,trans_beam_function_c=
> hunk...
> >>
> >> src/nif_with_hipe.erl:none: internal error in native_compile;
> >> crash reason: {{badmatch,
> >>                   {'EXIT',
> >>                       {{hipe_beam_to_icode,1148,{'trans_fun/2',on_load=
> }},
> >>                        [{hipe_beam_to_icode,trans_fun,2},
> >>                         {hipe_beam_to_icode,trans_fun,2},
> >>                         {hipe_beam_to_icode,trans_mfa_code,5},
> >>                         {hipe_beam_to_icode,trans_beam_function_chunk,=
> 2},
> >>                         {hipe_beam_to_icode,'-module/2-lc$^1/1-1-',2},=
>
> >>                         {hipe,get_beam_icode,4},
> >>                         {hipe,'-run_compiler_1/3-fun-0-',4}]}}},
> >>               [{hipe,get_beam_icode,4},{hipe,'-run_compiler_1/3-fun-0-=
> ',4}]}
> >>
> >> --
> >> Pozdrawiam
> >>
> >> Rados=C5=82aw Bu=C5=82at
> >> http://radarek.jogger.pl - m=C3=B3j blog
> >> _______________________________________________
> >> erlang-questions mailing list
> >> [hidden email]
> >> http://erlang.org/mailman/listinfo/erlang-questions
> >>
> >>    =20
> > _______________________________________________
> > erlang-questions mailing list
> > [hidden email]
> > http://erlang.org/mailman/listinfo/erlang-questions
> >
> >  =20
>
>
>
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: NIF + HiPE - can it works together?

Joel Reymont


On Wednesday, June 8, 2011 7:08:58 PM UTC+4, Mikael Pettersson wrote:

Alas there are effectively no compiler/vm developers left in the HiPE team;
we've all left to pursue other career paths.


Does this mean that HiPE is up for adoption?

It wasn't very cuddly last time I looked at it but...


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: NIF + HiPE - can it works together?

Kenny Stone
Is HiPE generally kept up with beam?  Is there a set of automated tests you can point to to say "HiPE has 98% coverage of BEAM"?   

On Thu, Jun 9, 2011 at 8:25 AM, Joel Reymont <[hidden email]> wrote:


On Wednesday, June 8, 2011 7:08:58 PM UTC+4, Mikael Pettersson wrote:

Alas there are effectively no compiler/vm developers left in the HiPE team;
we've all left to pursue other career paths.


Does this mean that HiPE is up for adoption?

It wasn't very cuddly last time I looked at it but...


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

NIF + HiPE - can it works together?

Mikael Pettersson
In reply to this post by Joel Reymont
Joel Reymont writes:
 >
 >
 > On Wednesday, June 8, 2011 7:08:58 PM UTC+4, Mikael Pettersson wrote:
 > >
 > > Alas there are effectively no compiler/vm developers left in the HiPE team;
 > > we've all left to pursue other career paths.
 > >
 >
 > Does this mean that HiPE is up for adoption?

Not quite.  Kostis still has a claim on both the name and the code,
and he usually oversees a couple of (undergraduate) student projects
each year that do some experiments in the compiler parts.

But if other people submit high-quality patches that fix bugs or add
useful improvements then I'm sure those patches will eventually find
their way into OTP.


Reply | Threaded
Open this post in threaded view
|

Re: NIF + HiPE - can it works together?

Mikael Pettersson-3
In reply to this post by Kenny Stone
Kenny Stone writes:
 > Is HiPE generally kept up with beam?

Yes.  I'm only aware of three omissions:
- apply with {M,F} tuple; that's a legacy feature we don't like
- on_load, because of unresolved implementation difficulties
- stack dumps in exceptions look different; that's a WONTFIX

 >  Is there a set of automated tests you
 > can point to to say "HiPE has 98% coverage of BEAM"?

No.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: NIF + HiPE - can it works together?

Robert Virding-2
----- "Mikael Pettersson" <[hidden email]> wrote:

> Kenny Stone writes:
>  > Is HiPE generally kept up with beam?
>
> Yes.  I'm only aware of three omissions:
> - apply with {M,F} tuple; that's a legacy feature we don't like

Please don't support this feature, please! It should have dies an ignominious death long ago.

> - on_load, because of unresolved implementation difficulties
> - stack dumps in exceptions look different; that's a WONTFIX
>
>  >  Is there a set of automated tests you
>  > can point to to say "HiPE has 98% coverage of BEAM"?
>
> No.

Robert
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions