Erlang 18.0, hanging erlang:memory(total)?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Erlang 18.0, hanging erlang:memory(total)?

Nico Kruber-5
Hi folks,
is it possible that erlang:memory/1 can hang?

Today I had a unit test failing in the middle of its execution (it runs the
same test over and over again and it was like the 20th time it was run). There
was a timeout (>50s) in one process not being able to finish its message
handling and erlang:process_info(Pid, current_stacktrace) is repeatedly
showing the following stacktrace during these 50s:

{current_stacktrace,
             [{erlang,receive_emd,3,[]},
              {erlang,get_mem_data,3,[]},
              {erlang,memory,1,[]},
              {dht_node_state,details,1,
                  [{file,"src/dht_node_state.erl"},{line,510}]},
              {dht_node,on,2,[{file,"src/dht_node.erl"},{line,384}]},
              {mockup_dht_node,on,2,
                  [{file,"test/mockup_dht_node.erl"},{line,67}]},
              {gen_component,on,3,
                  [{file,"src/gen_component.erl"},{line,614}]}]}

The line in the dht_node_state module literally only calls
erlang:memory(total) and creates a record with this info. Apparently, it never
gets out of erlang:memory/1 :(


Thanks for your help
Nico
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

signature.asc (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Erlang 18.0, hanging erlang:memory(total)?

Rickard Green-2
On Thu, Jul 2, 2015 at 12:29 AM, Nico Kruber <[hidden email]> wrote:

> Hi folks,
> is it possible that erlang:memory/1 can hang?
>
> Today I had a unit test failing in the middle of its execution (it runs the
> same test over and over again and it was like the 20th time it was run). There
> was a timeout (>50s) in one process not being able to finish its message
> handling and erlang:process_info(Pid, current_stacktrace) is repeatedly
> showing the following stacktrace during these 50s:
>
> {current_stacktrace,
>              [{erlang,receive_emd,3,[]},
>               {erlang,get_mem_data,3,[]},
>               {erlang,memory,1,[]},
>               {dht_node_state,details,1,
>                   [{file,"src/dht_node_state.erl"},{line,510}]},
>               {dht_node,on,2,[{file,"src/dht_node.erl"},{line,384}]},
>               {mockup_dht_node,on,2,
>                   [{file,"test/mockup_dht_node.erl"},{line,67}]},
>               {gen_component,on,3,
>                   [{file,"src/gen_component.erl"},{line,614}]}]}
>
> The line in the dht_node_state module literally only calls
> erlang:memory(total) and creates a record with this info. Apparently, it never
> gets out of erlang:memory/1 :(
>
>
> Thanks for your help
> Nico
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>

The bug fixed in OTP-18.0.1 (and OTP-17.5.6.1) can cause such a hang.

Regards,
Rickard
--
Rickard Green, Erlang/OTP, Ericsson AB
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang 18.0, hanging erlang:memory(total)?

Nico Kruber-5
On Thursday 02 Jul 2015 14:23:01 Rickard Green wrote:

> On Thu, Jul 2, 2015 at 12:29 AM, Nico Kruber <[hidden email]> wrote:
> > Hi folks,
> > is it possible that erlang:memory/1 can hang?
> >
> > Today I had a unit test failing in the middle of its execution (it runs
> > the
> > same test over and over again and it was like the 20th time it was run).
> > There was a timeout (>50s) in one process not being able to finish its
> > message handling and erlang:process_info(Pid, current_stacktrace) is
> > repeatedly showing the following stacktrace during these 50s:
> >
> > {current_stacktrace,
> >
> >              [{erlang,receive_emd,3,[]},
> >              
> >               {erlang,get_mem_data,3,[]},
> >               {erlang,memory,1,[]},
> >               {dht_node_state,details,1,
> >              
> >                   [{file,"src/dht_node_state.erl"},{line,510}]},
> >              
> >               {dht_node,on,2,[{file,"src/dht_node.erl"},{line,384}]},
> >               {mockup_dht_node,on,2,
> >              
> >                   [{file,"test/mockup_dht_node.erl"},{line,67}]},
> >              
> >               {gen_component,on,3,
> >              
> >                   [{file,"src/gen_component.erl"},{line,614}]}]}
> >
> > The line in the dht_node_state module literally only calls
> > erlang:memory(total) and creates a record with this info. Apparently, it
> > never gets out of erlang:memory/1 :(
> >
> >
> > Thanks for your help
> > Nico
> > _______________________________________________
> > erlang-questions mailing list
> > [hidden email]
> > http://erlang.org/mailman/listinfo/erlang-questions
>
> The bug fixed in OTP-18.0.1 (and OTP-17.5.6.1) can cause such a hang.
>
> Regards,
> Rickard
ok, I'll keep an eye on 18.0.1 on our test systems
-> note however, that I encountered the bug not "just after emulator start" as
stated in the release log. Also since I ran several tests before the failing
one, several calls to erlang:memory/1 succeeded before the hang!


Nico
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

signature.asc (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Erlang 18.0, hanging erlang:memory(total)?

Sverker Eriksson-4

On 07/02/2015 10:42 PM, Nico Kruber wrote:

> On Thursday 02 Jul 2015 14:23:01 Rickard Green wrote:
>> On Thu, Jul 2, 2015 at 12:29 AM, Nico Kruber<[hidden email]>  wrote:
>>> Hi folks,
>>> is it possible that erlang:memory/1 can hang?
>>>
>>> Today I had a unit test failing in the middle of its execution (it runs
>>> the
>>> same test over and over again and it was like the 20th time it was run).
>>> There was a timeout (>50s) in one process not being able to finish its
>>> message handling and erlang:process_info(Pid, current_stacktrace) is
>>> repeatedly showing the following stacktrace during these 50s:
>>>
>>> {current_stacktrace,
>>>
>>>               [{erlang,receive_emd,3,[]},
>>>              
>>>                {erlang,get_mem_data,3,[]},
>>>                {erlang,memory,1,[]},
>>>                {dht_node_state,details,1,
>>>                
>>>                    [{file,"src/dht_node_state.erl"},{line,510}]},
>>>                
>>>                {dht_node,on,2,[{file,"src/dht_node.erl"},{line,384}]},
>>>                {mockup_dht_node,on,2,
>>>                
>>>                    [{file,"test/mockup_dht_node.erl"},{line,67}]},
>>>                
>>>                {gen_component,on,3,
>>>                
>>>                    [{file,"src/gen_component.erl"},{line,614}]}]}
>>>
>>> The line in the dht_node_state module literally only calls
>>> erlang:memory(total) and creates a record with this info. Apparently, it
>>> never gets out of erlang:memory/1 :(
>>>
>>>
>>> Thanks for your help
>>> Nico
>>> _______________________________________________
>>> erlang-questions mailing list
>>> [hidden email]
>>> http://erlang.org/mailman/listinfo/erlang-questions
>> The bug fixed in OTP-18.0.1 (and OTP-17.5.6.1) can cause such a hang.
>>
>> Regards,
>> Rickard
> ok, I'll keep an eye on 18.0.1 on our test systems
> -> note however, that I encountered the bug not "just after emulator start" as
> stated in the release log. Also since I ran several tests before the failing
> one, several calls to erlang:memory/1 succeeded before the hang!
>
>
> Nico
>
The bug fixed in 18.0.1 (OTP-12859) is not specific to emulator start,
but that is only where we have seen it so far, and we think the VM might be
extra vulnerable for this bug at start.
The bug is in the general logic to wake up scheduler threads
that is blocking for IO events, and can happen "anytime". It's a very
specific
race between threads however, so you have to be quite unlucky for it to
happen.

The symptom is actually never an eternal hanging as the polling thread will
eventually be released by its next finite timeout, which can be anything
from
a couple of milliseconds to several hours.

/Sverker, Erlang/OTP

...
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Erlang 18.0, hanging erlang:memory(total)?

Nico Kruber-5
On Friday 03 Jul 2015 15:32:52 Sverker Eriksson wrote:
> On 07/02/2015 10:42 PM, Nico Kruber wrote:
> > On Thursday 02 Jul 2015 14:23:01 Rickard Green wrote:
> >> On Thu, Jul 2, 2015 at 12:29 AM, Nico Kruber<[hidden email]>  
wrote:

> >>> Hi folks,
> >>> is it possible that erlang:memory/1 can hang?
> >>>
> >>> Today I had a unit test failing in the middle of its execution (it runs
> >>> the
> >>> same test over and over again and it was like the 20th time it was run).
> >>> There was a timeout (>50s) in one process not being able to finish its
> >>> message handling and erlang:process_info(Pid, current_stacktrace) is
> >>> repeatedly showing the following stacktrace during these 50s:
> >>>
> >>> {current_stacktrace,
> >>>
> >>>               [{erlang,receive_emd,3,[]},
> >>>              
> >>>                {erlang,get_mem_data,3,[]},
> >>>                {erlang,memory,1,[]},
> >>>                {dht_node_state,details,1,
> >>>                
> >>>                    [{file,"src/dht_node_state.erl"},{line,510}]},
> >>>                
> >>>                {dht_node,on,2,[{file,"src/dht_node.erl"},{line,384}]},
> >>>                {mockup_dht_node,on,2,
> >>>                
> >>>                    [{file,"test/mockup_dht_node.erl"},{line,67}]},
> >>>                
> >>>                {gen_component,on,3,
> >>>                
> >>>                    [{file,"src/gen_component.erl"},{line,614}]}]}
> >>>
> >>> The line in the dht_node_state module literally only calls
> >>> erlang:memory(total) and creates a record with this info. Apparently, it
> >>> never gets out of erlang:memory/1 :(
> >>>
> >>>
> >>> Thanks for your help
> >>> Nico
> >>> _______________________________________________
> >>> erlang-questions mailing list
> >>> [hidden email]
> >>> http://erlang.org/mailman/listinfo/erlang-questions
> >>
> >> The bug fixed in OTP-18.0.1 (and OTP-17.5.6.1) can cause such a hang.
> >>
> >> Regards,
> >> Rickard
> >
> > ok, I'll keep an eye on 18.0.1 on our test systems
> > -> note however, that I encountered the bug not "just after emulator
> > start" as stated in the release log. Also since I ran several tests
> > before the failing one, several calls to erlang:memory/1 succeeded before
> > the hang!
> >
> >
> > Nico
>
> The bug fixed in 18.0.1 (OTP-12859) is not specific to emulator start,
> but that is only where we have seen it so far, and we think the VM might be
> extra vulnerable for this bug at start.
> The bug is in the general logic to wake up scheduler threads
> that is blocking for IO events, and can happen "anytime". It's a very
> specific
> race between threads however, so you have to be quite unlucky for it to
> happen.
>
> The symptom is actually never an eternal hanging as the polling thread will
> eventually be released by its next finite timeout, which can be anything
> from
> a couple of milliseconds to several hours.
>
> /Sverker, Erlang/OTP
>
> ...
Thanks for your explanation, Sverker,
I've actually only seen this error this one time with Erlang 18.0 and no other
Erlang versions so far. We changed these tests to using 18.0.1 now and I'll
see what happens.

Our "unlucky" test machine has been quite a gift though since through that
machine we found several bugs in our code as well ;)


Regards
Nico

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

signature.asc (188 bytes) Download Attachment