High Sierra and binary_to_term (compressed)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

High Sierra and binary_to_term (compressed)

Martin Sumner
Rather foolishly I accepted the upgrade to High Sierra on my Macbook last night.  This has led to an interesting problem with a binary_to_term line on my project.

Following the upgrade, tests have started failing on this line:


with error:badarg being thrown.

However if I copy and paste the binary on which the badarg has been thrown out of the crash report, and into a erlang console - binary_to_term works just fine.


This failure is intermittent, and indeed wrapping the binary_to_term like this is enough to make the problem seemingly disappear:

try binary_to_term(Bin) of
    T -> 
        T
catch
    error:badarg ->
        binary_to_term(Bin)
end.

Re-running the tests on another OSX device on a release previous to High Sierra shows no issues with/without the wrapper.

The problem appears to consistently exist in OTP 17.5 and 18.3.  However, running the tests in 19.3 and R16B02-basho10 doesn't show the problem.

The terms in question have all been compressed using term_to_binary's zlib compression.

If I build 18.3 on High Sierra with the  "--enable-builtin-zlib" option, the problem does *not* re-occur.  This switch appears to be a reliable delta between having and not having the problem, and I believe may have become the default in OTP 17. 

It appears to be specific to v17/v18 Erlang with compressed terms, on High Sierra where the OS zlib implementation is used, not the inbuilt OTP zlib.

There appears to be internet chatter of other issues with Mac OSX High Sierra and changes to the zlib implementation - https://github.com/madler/zlib/issues/305.

So I think this doesn't look like it is an Erlang issue, but I suspect it may not be resolved in High Sierra any time soon, and may impact other Erlang users using zlib functionality.  Does this it seem reasonable to pass this off as a High Sierra zlib bug, and use the --enable-builtin-zlib workaround?  If this is the case I'm not sure why I don't get the issue on 19.3, any ideas as to why?

Martin 



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: High Sierra and binary_to_term (compressed)

Jan Chochol-2
Hi Martin,

This is bug with combination of new zlib and old Erlang - https://github.com/erlang/otp/commit/e27119948fc6ab28bea81019720bddaac5b655a7 .
It is fixed in OTP-18.3.4.5 and newer.
"--enable-builtin-zlib" works, because it uses older bundled zlib.

Jan

On Mon, Nov 13, 2017 at 10:01 PM, Martin Sumner <[hidden email]> wrote:
Rather foolishly I accepted the upgrade to High Sierra on my Macbook last night.  This has led to an interesting problem with a binary_to_term line on my project.

Following the upgrade, tests have started failing on this line:


with error:badarg being thrown.

However if I copy and paste the binary on which the badarg has been thrown out of the crash report, and into a erlang console - binary_to_term works just fine.


This failure is intermittent, and indeed wrapping the binary_to_term like this is enough to make the problem seemingly disappear:

try binary_to_term(Bin) of
    T -> 
        T
catch
    error:badarg ->
        binary_to_term(Bin)
end.

Re-running the tests on another OSX device on a release previous to High Sierra shows no issues with/without the wrapper.

The problem appears to consistently exist in OTP 17.5 and 18.3.  However, running the tests in 19.3 and R16B02-basho10 doesn't show the problem.

The terms in question have all been compressed using term_to_binary's zlib compression.

If I build 18.3 on High Sierra with the  "--enable-builtin-zlib" option, the problem does *not* re-occur.  This switch appears to be a reliable delta between having and not having the problem, and I believe may have become the default in OTP 17. 

It appears to be specific to v17/v18 Erlang with compressed terms, on High Sierra where the OS zlib implementation is used, not the inbuilt OTP zlib.

There appears to be internet chatter of other issues with Mac OSX High Sierra and changes to the zlib implementation - https://github.com/madler/zlib/issues/305.

So I think this doesn't look like it is an Erlang issue, but I suspect it may not be resolved in High Sierra any time soon, and may impact other Erlang users using zlib functionality.  Does this it seem reasonable to pass this off as a High Sierra zlib bug, and use the --enable-builtin-zlib workaround?  If this is the case I'm not sure why I don't get the issue on 19.3, any ideas as to why?

Martin 



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: High Sierra and binary_to_term (compressed)

Martin Sumner
Jan,

Thank-you, that does explain things.

Martin

On 14 November 2017 at 07:30, Jan Chochol <[hidden email]> wrote:
Hi Martin,

This is bug with combination of new zlib and old Erlang - https://github.com/erlang/otp/commit/e27119948fc6ab28bea81019720bddaac5b655a7 .
It is fixed in OTP-18.3.4.5 and newer.
"--enable-builtin-zlib" works, because it uses older bundled zlib.

Jan

On Mon, Nov 13, 2017 at 10:01 PM, Martin Sumner <[hidden email]> wrote:
Rather foolishly I accepted the upgrade to High Sierra on my Macbook last night.  This has led to an interesting problem with a binary_to_term line on my project.

Following the upgrade, tests have started failing on this line:


with error:badarg being thrown.

However if I copy and paste the binary on which the badarg has been thrown out of the crash report, and into a erlang console - binary_to_term works just fine.


This failure is intermittent, and indeed wrapping the binary_to_term like this is enough to make the problem seemingly disappear:

try binary_to_term(Bin) of
    T -> 
        T
catch
    error:badarg ->
        binary_to_term(Bin)
end.

Re-running the tests on another OSX device on a release previous to High Sierra shows no issues with/without the wrapper.

The problem appears to consistently exist in OTP 17.5 and 18.3.  However, running the tests in 19.3 and R16B02-basho10 doesn't show the problem.

The terms in question have all been compressed using term_to_binary's zlib compression.

If I build 18.3 on High Sierra with the  "--enable-builtin-zlib" option, the problem does *not* re-occur.  This switch appears to be a reliable delta between having and not having the problem, and I believe may have become the default in OTP 17. 

It appears to be specific to v17/v18 Erlang with compressed terms, on High Sierra where the OS zlib implementation is used, not the inbuilt OTP zlib.

There appears to be internet chatter of other issues with Mac OSX High Sierra and changes to the zlib implementation - https://github.com/madler/zlib/issues/305.

So I think this doesn't look like it is an Erlang issue, but I suspect it may not be resolved in High Sierra any time soon, and may impact other Erlang users using zlib functionality.  Does this it seem reasonable to pass this off as a High Sierra zlib bug, and use the --enable-builtin-zlib workaround?  If this is the case I'm not sure why I don't get the issue on 19.3, any ideas as to why?

Martin 



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: High Sierra and binary_to_term (compressed)

Sergey Elin
In reply to this post by Jan Chochol-2
The `--enable-builtin-zlib` flag also fixes strange xref/dialyzer bugs in one of our projects built on top of Erlang 17.5 on High Sierra.

Thanks for sharing! :)
---
Best regards,
Sergey Yelin.



On 14 Nov 2017, at 10:30, Jan Chochol <[hidden email]> wrote:

Hi Martin,

This is bug with combination of new zlib and old Erlang - https://github.com/erlang/otp/commit/e27119948fc6ab28bea81019720bddaac5b655a7 .
It is fixed in OTP-18.3.4.5 and newer.
"--enable-builtin-zlib" works, because it uses older bundled zlib.

Jan

On Mon, Nov 13, 2017 at 10:01 PM, Martin Sumner <[hidden email]> wrote:
Rather foolishly I accepted the upgrade to High Sierra on my Macbook last night.  This has led to an interesting problem with a binary_to_term line on my project.

Following the upgrade, tests have started failing on this line:


with error:badarg being thrown.

However if I copy and paste the binary on which the badarg has been thrown out of the crash report, and into a erlang console - binary_to_term works just fine.


This failure is intermittent, and indeed wrapping the binary_to_term like this is enough to make the problem seemingly disappear:

try binary_to_term(Bin) of
    T -> 
        T
catch
    error:badarg ->
        binary_to_term(Bin)
end.

Re-running the tests on another OSX device on a release previous to High Sierra shows no issues with/without the wrapper.

The problem appears to consistently exist in OTP 17.5 and 18.3.  However, running the tests in 19.3 and R16B02-basho10 doesn't show the problem.

The terms in question have all been compressed using term_to_binary's zlib compression.

If I build 18.3 on High Sierra with the  "--enable-builtin-zlib" option, the problem does *not* re-occur.  This switch appears to be a reliable delta between having and not having the problem, and I believe may have become the default in OTP 17. 

It appears to be specific to v17/v18 Erlang with compressed terms, on High Sierra where the OS zlib implementation is used, not the inbuilt OTP zlib.

There appears to be internet chatter of other issues with Mac OSX High Sierra and changes to the zlib implementation - https://github.com/madler/zlib/issues/305.

So I think this doesn't look like it is an Erlang issue, but I suspect it may not be resolved in High Sierra any time soon, and may impact other Erlang users using zlib functionality.  Does this it seem reasonable to pass this off as a High Sierra zlib bug, and use the --enable-builtin-zlib workaround?  If this is the case I'm not sure why I don't get the issue on 19.3, any ideas as to why?

Martin 



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions