beam is crashed due to out of memory & it is not started automatically after that even used heart

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

beam is crashed due to out of memory & it is not started automatically after that even used heart

Sanath Prasanna-2
Hi all,
My system is going down due to out of memory & it is not auto started even set -heart command in starting. What is the reason for this ? Can someone explain how to overcome this problem. (not starting system automatically using heart) 
Br,
Robert

/var/log/message 

aa_hub kernel: [60952995.574233] Out of memory: Kill process 2480 (beam.smp) score 498 or sacrifice child
Jun 27 13:31:07 aa_hub kernel: [60952995.577670] Killed process 2565 (heart) total-vm:8408kB, anon-rss:92kB, file-rss:0kB
Jun 27 13:31:07 aa_hub kernel: [60952995.584754] epmd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Jun 27 13:31:07 aa_hub kernel: [60952995.587996] epmd cpuset=/ mems_allowed=0

startup command

/system/bin/run_erl -daemon /system/test/pipe/ /tmp "/system/bin/start_erl /system /system /system/bin/start_erl  -smp auto -heart -sname test +64 +P10000 +B i -setcookie test"

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: beam is crashed due to out of memory & it is not started automatically after that even used heart

Magnus Henoch-6
Sanath Prasanna <[hidden email]> writes:

> Hi all, My system is going down due to out of memory & it is not
> auto started even set -heart command in starting. What is the
> reason for this ? Can someone explain how to overcome this
> problem. (not starting system automatically using heart) Br,
> Robert
>
> /var/log/message
>
> aa_hub kernel: [60952995.574233] Out of memory: Kill process
> 2480 (beam.smp) score 498 or sacrifice child Jun 27 13:31:07
> aa_hub kernel: [60952995.577670] Killed process 2565 (heart)
> total-vm:8408kB, anon-rss:92kB, file-rss:0kB Jun 27 13:31:07
> aa_hub kernel: [60952995.584754] epmd invoked oom-killer:
> gfp_mask=0x201da, order=0, oom_score_adj=0 Jun 27 13:31:07
> aa_hub kernel: [60952995.587996] epmd cpuset=/ mems_allowed=0
>
> startup command
>
> /system/bin/run_erl -daemon /system/test/pipe/ /tmp
> "/system/bin/start_erl /system /system /system/bin/start_erl
> -smp auto* -heart -*sname test +64 +P10000 +B i -setcookie test"

I've seen the same issue.  The problem is that the OOM killer
identifies a process to kill, and then kills all its child
processes before killing the chosen process itself.  Since heart
is a child process of the beam process, it doesn't have a chance
to restart the Erlang VM.

I tried to fix it with the patch below, but it didn't work for
some reason - I don't remember the details.  I ended up solving
the problem by addressing the memory usage itself.

https://github.com/legoscia/otp/commit/bc19687664697dcff9d5472e14da231cf830abe5

Regards,
Magnus

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: beam is crashed due to out of memory & it is not started automatically after that even used heart

Éric Pailleau

Hi,

you can tell OOM to not kill some process . for heart, somewhere in your start script do :

 

pgrep -f "heart -pid" | while read PID; do echo -17 > /proc/$PID/oom_adj; done

 

-17 is the value to let OOM ignore the process.

 

regards

 

 

 

 

 

> Message du 28/06/17 12:58

> De : "Magnus Henoch" <[hidden email]>
> A : "Sanath Prasanna" <[hidden email]>
> Copie à : "erlang-questions" <[hidden email]>
> Objet : Re: [erlang-questions] beam is crashed due to out of memory & it is not started automatically after that even used heart
>
> Sanath Prasanna <[hidden email]> writes:
>
> > Hi all, My system is going down due to out of memory & it is not
> > auto started even set -heart command in starting. What is the
> > reason for this ? Can someone explain how to overcome this
> > problem. (not starting system automatically using heart) Br,
> > Robert
> >
> > /var/log/message
> >
> > aa_hub kernel: [60952995.574233] Out of memory: Kill process
> > 2480 (beam.smp) score 498 or sacrifice child Jun 27 13:31:07
> > aa_hub kernel: [60952995.577670] Killed process 2565 (heart)
> > total-vm:8408kB, anon-rss:92kB, file-rss:0kB Jun 27 13:31:07
> > aa_hub kernel: [60952995.584754] epmd invoked oom-killer:
> > gfp_mask=0x201da, order=0, oom_score_adj=0 Jun 27 13:31:07
> > aa_hub kernel: [60952995.587996] epmd cpuset=/ mems_allowed=0
> >
> > startup command
> >
> > /system/bin/run_erl -daemon /system/test/pipe/ /tmp
> > "/system/bin/start_erl /system /system /system/bin/start_erl
> > -smp auto* -heart -*sname test +64 +P10000 +B i -setcookie test"
>
> I've seen the same issue. The problem is that the OOM killer
> identifies a process to kill, and then kills all its child
> processes before killing the chosen process itself. Since heart
> is a child process of the beam process, it doesn't have a chance
> to restart the Erlang VM.
>
> I tried to fix it with the patch below, but it didn't work for
> some reason - I don't remember the details. I ended up solving
> the problem by addressing the memory usage itself.
>
> https://github.com/legoscia/otp/commit/bc19687664697dcff9d5472e14da231cf830abe5
>
> Regards,
> Magnus
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Loading...