Quantcast

Problem with binary_to_term on different machines

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Problem with binary_to_term on different machines

Michel Boaventura
Hi all,

I've build erlang 19 from scratch on my gentoo machine but can't use dialyzer, because it complains my beam files weren't compiled with debug_info.

I've read and debugged a log of dialyzer and erlang code and find out the issue is that for some reason erlang can't parse the abstract part of the files.

To check this I save the abstract part on a file and tried to parse it on my gentoo machine and on a Ubuntu server I have access. For some reason I don't know it reads with no problem Ubuntu but gives me an invalid arg on gentoo.

I'm very new to erlang and have no clue of how to debug further. The binary is attached and I'm reading it like:

{ok, [Binary]} = file:consult("Abstr").
Term = binary_to_term(Binary).

ps: I've sent a message similar to this one on this list, but since it was my first one I think it wasn't accepted (and I do apologize for the double post if you received it). But after it I've discovered the problem with binary_to_term, so that message isn't so useful anyway.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

Abstr (219K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Problem with binary_to_term on different machines

Sverker Eriksson-4

This is indeed a problem in Erlang VM code (shallow copy of inflate state)
that has existed since R16B03, but not caused actual problem until zlib v1.2.9.

Fix coming up. Here is a preliminary patch for the impatient.

diff --git a/erts/emulator/beam/external.c b/erts/emulator/beam/external.c
index beed847..1c4fff5 100644
--- a/erts/emulator/beam/external.c
+++ b/erts/emulator/beam/external.c
@@ -1431,6 +1431,10 @@ static B2TContext* b2t_export_context(Process* p, B2TContext* src)
     if (ctx->state >= B2TDecode && ctx->u.dc.next == &src->u.dc.res) {
         ctx->u.dc.next = &ctx->u.dc.res;
     }
+    else if (ctx->state == B2TUncompressChunk) {
+        int cres = inflateCopy(&ctx->u.uc.stream, &src->u.uc.stream);
+        ASSERT(cres == Z_OK); (void)cres;
+    }
     hp = HAlloc(p, PROC_BIN_SIZE);
     ctx->trap_bin = erts_mk_magic_binary_term(&hp, &MSO(p), context_b);
     return ctx;


/Sverker, Erlang/OTP


On 01/19/2017 04:14 AM, Michel Boaventura wrote:
Hi all,

I've build erlang 19 from scratch on my gentoo machine but can't use
dialyzer, because it complains my beam files weren't compiled with
debug_info.

I've read and debugged a log of dialyzer and erlang code and find out the
issue is that for some reason erlang can't parse the abstract part of the
files.

To check this I save the abstract part on a file and tried to parse it on
my gentoo machine and on a Ubuntu server I have access. For some reason I
don't know it reads with no problem Ubuntu but gives me an invalid arg on
gentoo.

I'm very new to erlang and have no clue of how to debug further. The binary
is attached and I'm reading it like:

{ok, [Binary]} = file:consult("Abstr").
Term = binary_to_term(Binary).

ps: I've sent a message similar to this one on this list, but since it was
my first one I think it wasn't accepted (and I do apologize for the double
post if you received it). But after it I've discovered the problem with
binary_to_term, so that message isn't so useful anyway.



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Problem with binary_to_term on different machines

Sverker Eriksson-4

Correction: Bug exists since OTP-17.0.

(and i tags R16B02_yielding_binary_to_term and OTP_R16B03_yielding_binary_to_term)

/Sverker


On 01/20/2017 05:17 PM, Sverker Eriksson wrote:
This is indeed a problem in Erlang VM code (shallow copy of inflate state)
that has existed since R16B03, but not caused actual problem until zlib v1.2.9.

Fix coming up. Here is a preliminary patch for the impatient.

diff --git a/erts/emulator/beam/external.c b/erts/emulator/beam/external.c
index beed847..1c4fff5 100644
--- a/erts/emulator/beam/external.c
+++ b/erts/emulator/beam/external.c
@@ -1431,6 +1431,10 @@ static B2TContext* b2t_export_context(Process* p, B2TContext* src)
     if (ctx->state >= B2TDecode && ctx->u.dc.next == &src->u.dc.res) {
         ctx->u.dc.next = &ctx->u.dc.res;
     }
+    else if (ctx->state == B2TUncompressChunk) {
+        int cres = inflateCopy(&ctx->u.uc.stream, &src->u.uc.stream);
+        ASSERT(cres == Z_OK); (void)cres;
+    }
     hp = HAlloc(p, PROC_BIN_SIZE);
     ctx->trap_bin = erts_mk_magic_binary_term(&hp, &MSO(p), context_b);
     return ctx;


/Sverker, Erlang/OTP


On 01/19/2017 04:14 AM, Michel Boaventura wrote:
Hi all,

I've build erlang 19 from scratch on my gentoo machine but can't use
dialyzer, because it complains my beam files weren't compiled with
debug_info.

I've read and debugged a log of dialyzer and erlang code and find out the
issue is that for some reason erlang can't parse the abstract part of the
files.

To check this I save the abstract part on a file and tried to parse it on
my gentoo machine and on a Ubuntu server I have access. For some reason I
don't know it reads with no problem Ubuntu but gives me an invalid arg on
gentoo.

I'm very new to erlang and have no clue of how to debug further. The binary
is attached and I'm reading it like:

{ok, [Binary]} = file:consult("Abstr").
Term = binary_to_term(Binary).

ps: I've sent a message similar to this one on this list, but since it was
my first one I think it wasn't accepted (and I do apologize for the double
post if you received it). But after it I've discovered the problem with
binary_to_term, so that message isn't so useful anyway.



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Problem with binary_to_term on different machines

Michel Boaventura
Thank you Sverker!

On 20 January 2017 at 14:45, Sverker Eriksson <[hidden email]> wrote:

Correction: Bug exists since OTP-17.0.

(and i tags R16B02_yielding_binary_to_term and OTP_R16B03_yielding_binary_to_term)

/Sverker


On 01/20/2017 05:17 PM, Sverker Eriksson wrote:
This is indeed a problem in Erlang VM code (shallow copy of inflate state)
that has existed since R16B03, but not caused actual problem until zlib v1.2.9.

Fix coming up. Here is a preliminary patch for the impatient.

diff --git a/erts/emulator/beam/external.c b/erts/emulator/beam/external.c
index beed847..1c4fff5 100644
--- a/erts/emulator/beam/external.c
+++ b/erts/emulator/beam/external.c
@@ -1431,6 +1431,10 @@ static B2TContext* b2t_export_context(Process* p, B2TContext* src)
     if (ctx->state >= B2TDecode && ctx->u.dc.next == &src->u.dc.res) {
         ctx->u.dc.next = &ctx->u.dc.res;
     }
+    else if (ctx->state == B2TUncompressChunk) {
+        int cres = inflateCopy(&ctx->u.uc.stream, &src->u.uc.stream);
+        ASSERT(cres == Z_OK); (void)cres;
+    }
     hp = HAlloc(p, PROC_BIN_SIZE);
     ctx->trap_bin = erts_mk_magic_binary_term(&hp, &MSO(p), context_b);
     return ctx;


/Sverker, Erlang/OTP


On 01/19/2017 04:14 AM, Michel Boaventura wrote:
Hi all,

I've build erlang 19 from scratch on my gentoo machine but can't use
dialyzer, because it complains my beam files weren't compiled with
debug_info.

I've read and debugged a log of dialyzer and erlang code and find out the
issue is that for some reason erlang can't parse the abstract part of the
files.

To check this I save the abstract part on a file and tried to parse it on
my gentoo machine and on a Ubuntu server I have access. For some reason I
don't know it reads with no problem Ubuntu but gives me an invalid arg on
gentoo.

I'm very new to erlang and have no clue of how to debug further. The binary
is attached and I'm reading it like:

{ok, [Binary]} = file:consult("Abstr").
Term = binary_to_term(Binary).

ps: I've sent a message similar to this one on this list, but since it was
my first one I think it wasn't accepted (and I do apologize for the double
post if you received it). But after it I've discovered the problem with
binary_to_term, so that message isn't so useful anyway.



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions




--
Michel Almada de Castro Boaventura
Analista de Sistemas
Laboratório de Software Livre - LSL

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Problem with binary_to_term on different machines

Mikael Pettersson-5
In reply to this post by Sverker Eriksson-4
Sverker Eriksson writes:
 > Correction: Bug exists since OTP-17.0.
 >
 > (and i tags R16B02_yielding_binary_to_term and
 > OTP_R16B03_yielding_binary_to_term)
 >
 > /Sverker
 >
 >
 > On 01/20/2017 05:17 PM, Sverker Eriksson wrote:
 > > This is indeed a problem in Erlang VM code (shallow copy of inflate
 > > state)
 > > that has existed since R16B03, but not caused actual problem until
 > > zlib v1.2.9.
 > >
 > > Fix coming up. Here is a preliminary patch for the impatient.

Has this landed in OTP master yet?  I've been keeping a look out for it,
but haven't seen it (I may have missed it of course).

I'd like to backport it to our internal OTP-18 and OTP-19.

/Mikael



 > >
 > > diff --git a/erts/emulator/beam/external.c
 > > b/erts/emulator/beam/external.c
 > > index beed847..1c4fff5 100644
 > > --- a/erts/emulator/beam/external.c
 > > +++ b/erts/emulator/beam/external.c
 > > @@ -1431,6 +1431,10 @@ static B2TContext* b2t_export_context(Process*
 > > p, B2TContext* src)
 > >      if (ctx->state >= B2TDecode && ctx->u.dc.next == &src->u.dc.res) {
 > >          ctx->u.dc.next = &ctx->u.dc.res;
 > >      }
 > > +    else if (ctx->state == B2TUncompressChunk) {
 > > +        int cres = inflateCopy(&ctx->u.uc.stream, &src->u.uc.stream);
 > > +        ASSERT(cres == Z_OK); (void)cres;
 > > +    }
 > >      hp = HAlloc(p, PROC_BIN_SIZE);
 > >      ctx->trap_bin = erts_mk_magic_binary_term(&hp, &MSO(p), context_b);
 > >      return ctx;
 > >
 > >
 > > /Sverker, Erlang/OTP
 > >
 > >
 > > On 01/19/2017 04:14 AM, Michel Boaventura wrote:
 > >> Hi all,
 > >>
 > >> I've build erlang 19 from scratch on my gentoo machine but can't use
 > >> dialyzer, because it complains my beam files weren't compiled with
 > >> debug_info.
 > >>
 > >> I've read and debugged a log of dialyzer and erlang code and find out
 > >> the
 > >> issue is that for some reason erlang can't parse the abstract part of
 > >> the
 > >> files.
 > >>
 > >> To check this I save the abstract part on a file and tried to parse
 > >> it on
 > >> my gentoo machine and on a Ubuntu server I have access. For some
 > >> reason I
 > >> don't know it reads with no problem Ubuntu but gives me an invalid
 > >> arg on
 > >> gentoo.
 > >>
 > >> I'm very new to erlang and have no clue of how to debug further. The
 > >> binary
 > >> is attached and I'm reading it like:
 > >>
 > >> {ok, [Binary]} = file:consult("Abstr").
 > >> Term = binary_to_term(Binary).
 > >>
 > >> ps: I've sent a message similar to this one on this list, but since
 > >> it was
 > >> my first one I think it wasn't accepted (and I do apologize for the
 > >> double
 > >> post if you received it). But after it I've discovered the problem with
 > >> binary_to_term, so that message isn't so useful anyway.
 > >>
 > >>
 > >>
 > >> _______________________________________________
 > >> erlang-questions mailing list
 > >> [hidden email]
 > >> http://erlang.org/mailman/listinfo/erlang-questions
 > >
 > >
 > >
 > >
 > > _______________________________________________
 > > erlang-questions mailing list
 > > [hidden email]
 > > http://erlang.org/mailman/listinfo/erlang-questions
 >
 >
 > ----------------------------------------------------------------------
 > _______________________________________________
 > erlang-questions mailing list
 > [hidden email]
 > http://erlang.org/mailman/listinfo/erlang-questions

--
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Problem with binary_to_term on different machines

Sverker Eriksson-4
https://bugs.erlang.org/browse/ERL-340

I just merged e27119948fc6ab28bea81019720bddaac5b655a7
and pushed to maint and master.

Thanks for the reminder, Mikael.

WARNING: The preliminary source patch I published
earlier (seen below) will leak memory. The fix I just merged
has a different solution.

/Sverker



On 02/01/2017 02:31 PM, Mikael Pettersson wrote:

> Sverker Eriksson writes:
>   > Correction: Bug exists since OTP-17.0.
>   >
>   > (and i tags R16B02_yielding_binary_to_term and
>   > OTP_R16B03_yielding_binary_to_term)
>   >
>   > /Sverker
>   >
>   >
>   > On 01/20/2017 05:17 PM, Sverker Eriksson wrote:
>   > > This is indeed a problem in Erlang VM code (shallow copy of inflate
>   > > state)
>   > > that has existed since R16B03, but not caused actual problem until
>   > > zlib v1.2.9.
>   > >
>   > > Fix coming up. Here is a preliminary patch for the impatient.
>
> Has this landed in OTP master yet?  I've been keeping a look out for it,
> but haven't seen it (I may have missed it of course).
>
> I'd like to backport it to our internal OTP-18 and OTP-19.
>
> /Mikael
>
>
>
>   > >
>   > > diff --git a/erts/emulator/beam/external.c
>   > > b/erts/emulator/beam/external.c
>   > > index beed847..1c4fff5 100644
>   > > --- a/erts/emulator/beam/external.c
>   > > +++ b/erts/emulator/beam/external.c
>   > > @@ -1431,6 +1431,10 @@ static B2TContext* b2t_export_context(Process*
>   > > p, B2TContext* src)
>   > >      if (ctx->state >= B2TDecode && ctx->u.dc.next == &src->u.dc.res) {
>   > >          ctx->u.dc.next = &ctx->u.dc.res;
>   > >      }
>   > > +    else if (ctx->state == B2TUncompressChunk) {
>   > > +        int cres = inflateCopy(&ctx->u.uc.stream, &src->u.uc.stream);
>   > > +        ASSERT(cres == Z_OK); (void)cres;
>   > > +    }
>   > >      hp = HAlloc(p, PROC_BIN_SIZE);
>   > >      ctx->trap_bin = erts_mk_magic_binary_term(&hp, &MSO(p), context_b);
>   > >      return ctx;
>   > >
>   > >
>   > > /Sverker, Erlang/OTP
>   > >
>   > >
>   > > On 01/19/2017 04:14 AM, Michel Boaventura wrote:
>   > >> Hi all,
>   > >>
>   > >> I've build erlang 19 from scratch on my gentoo machine but can't use
>   > >> dialyzer, because it complains my beam files weren't compiled with
>   > >> debug_info.
>   > >>
>   > >> I've read and debugged a log of dialyzer and erlang code and find out
>   > >> the
>   > >> issue is that for some reason erlang can't parse the abstract part of
>   > >> the
>   > >> files.
>   > >>
>   > >> To check this I save the abstract part on a file and tried to parse
>   > >> it on
>   > >> my gentoo machine and on a Ubuntu server I have access. For some
>   > >> reason I
>   > >> don't know it reads with no problem Ubuntu but gives me an invalid
>   > >> arg on
>   > >> gentoo.
>   > >>
>   > >> I'm very new to erlang and have no clue of how to debug further. The
>   > >> binary
>   > >> is attached and I'm reading it like:
>   > >>
>   > >> {ok, [Binary]} = file:consult("Abstr").
>   > >> Term = binary_to_term(Binary).
>   > >>
>   > >> ps: I've sent a message similar to this one on this list, but since
>   > >> it was
>   > >> my first one I think it wasn't accepted (and I do apologize for the
>   > >> double
>   > >> post if you received it). But after it I've discovered the problem with
>   > >> binary_to_term, so that message isn't so useful anyway.
>   > >>
>   > >>
>   > >>
>   > >> _______________________________________________
>   > >> erlang-questions mailing list
>   > >> [hidden email]
>   > >> http://erlang.org/mailman/listinfo/erlang-questions
>   > >
>   > >
>   > >
>   > >
>   > > _______________________________________________
>   > > erlang-questions mailing list
>   > > [hidden email]
>   > > http://erlang.org/mailman/listinfo/erlang-questions
>   >
>   >
>   > ----------------------------------------------------------------------
>   > _______________________________________________
>   > erlang-questions mailing list
>   > [hidden email]
>   > http://erlang.org/mailman/listinfo/erlang-questions
>

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Loading...