Beam.smp killed by Linux

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Beam.smp killed by Linux

Frank Muller
Hi guys,

We just found this in our systemd’s logs:

Nov 02 15:17:01 ns342284.ip-91-121-153.eu CROND[24089]: (root) CMD (/usr/local/rtm/bin/rtm 55 > /dev/null 2> /dev/null)
Nov 02 15:17:53 ns342284.ip-91-121-153.eu kernel: beam.smp[24236]: segfault at 1c ip 000000000051be70 sp 00007f2d3e0bfea0 error 6 in beam.smp[400000+22f000]
Nov 02 15:17:53 ns342284.ip-91-121-153.eu systemd[1]: zeta.service: main process exited, code=killed, status=11/SEGV
Nov 02 15:17:53 ns342284.ip-91-121-153.eu systemd[1]: zeta.service: control process exited, code=exited status=1

Anyone knows what may cause this?

Machine: CentOS 7.2.1511 Kernel: 3.10.0.el7.x86_64
Erlang: 16B03-1
CPU: 4 Intel Xeon(R) E5-2690 @2.60GHz
RAM: 16GB
Disk: 2TB

Best
/Frank

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Beam.smp killed by Linux

Luke Bakken-2
Hi Frank,

That message means that beam.smp died from a segmentation fault, and
systemd is reporting it in the log. There are many things that can
cause this and that have been fixed in later Erlang/OTP releases.
You're using a very old release so my first recommendation would be to
upgrade if possible.

Tracking down the root cause of this requires building a VM with debug
symbols and getting a core file from the next crash.

Good luck -
Luke

On Thu, Nov 2, 2017 at 7:48 AM, Frank Muller <[hidden email]> wrote:

> Hi guys,
>
> We just found this in our systemd’s logs:
>
> Nov 02 15:17:01 ns342284.ip-91-121-153.eu CROND[24089]: (root) CMD
> (/usr/local/rtm/bin/rtm 55 > /dev/null 2> /dev/null)
> Nov 02 15:17:53 ns342284.ip-91-121-153.eu kernel: beam.smp[24236]: segfault
> at 1c ip 000000000051be70 sp 00007f2d3e0bfea0 error 6 in
> beam.smp[400000+22f000]
> Nov 02 15:17:53 ns342284.ip-91-121-153.eu systemd[1]: zeta.service: main
> process exited, code=killed, status=11/SEGV
> Nov 02 15:17:53 ns342284.ip-91-121-153.eu systemd[1]: zeta.service: control
> process exited, code=exited status=1
>
> Anyone knows what may cause this?
>
> Machine: CentOS 7.2.1511 Kernel: 3.10.0.el7.x86_64
> Erlang: 16B03-1
> CPU: 4 Intel Xeon(R) E5-2690 @2.60GHz
> RAM: 16GB
> Disk: 2TB
>
> Best
> /Frank
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Beam.smp killed by Linux

Guilherme Andrade


On 2 November 2017 at 15:02, Luke Bakken <[hidden email]> wrote:
Tracking down the root cause of this requires building a VM with debug
symbols and getting a core file from the next crash.

In case that's unpractical, objdump'ing the beam.smp binary and navigating to the location of the crash might also help if the misbehaving code is listed under a routine with a readable name and that name can be traced back to the ERTS C code.

--
Guilherme

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Beam.smp killed by Linux

Frank Muller
Thanks guys.

I’ll try to recompile with 20.1 + debug symbols on and I’ll leg you know. 

/Frank

On 2 November 2017 at 15:02, Luke Bakken <[hidden email]> wrote:
Tracking down the root cause of this requires building a VM with debug
symbols and getting a core file from the next crash.

In case that's unpractical, objdump'ing the beam.smp binary and navigating to the location of the crash might also help if the misbehaving code is listed under a routine with a readable name and that name can be traced back to the ERTS C code.

--
Guilherme

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Beam.smp killed by Linux

Jesper Louis Andersen-2
Typical errors include:

* Bug in the VM
* Hardware error, especially if running on older hardware
* NIF has a bug
* Linked in driver has a bug



On Fri, Nov 3, 2017 at 9:42 AM Frank Muller <[hidden email]> wrote:
Thanks guys.

I’ll try to recompile with 20.1 + debug symbols on and I’ll leg you know. 

/Frank

On 2 November 2017 at 15:02, Luke Bakken <[hidden email]> wrote:
Tracking down the root cause of this requires building a VM with debug
symbols and getting a core file from the next crash.

In case that's unpractical, objdump'ing the beam.smp binary and navigating to the location of the crash might also help if the misbehaving code is listed under a routine with a readable name and that name can be traced back to the ERTS C code.

--
Guilherme
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Beam.smp killed by Linux

Frank Muller
Hi Jesper,

We do not use any NIF nor linked in driver.

But I agree R16B03-1 is quite old (porting our app now to 20.1).

I’ll try to change to a newer hardware if 20.1 will not solve the issue.

Thanks
/Frank

Typical errors include:

* Bug in the VM
* Hardware error, especially if running on older hardware
* NIF has a bug
* Linked in driver has a bug



On Fri, Nov 3, 2017 at 9:42 AM Frank Muller <[hidden email]> wrote:
Thanks guys.

I’ll try to recompile with 20.1 + debug symbols on and I’ll leg you know. 

/Frank

On 2 November 2017 at 15:02, Luke Bakken <[hidden email]> wrote:
Tracking down the root cause of this requires building a VM with debug
symbols and getting a core file from the next crash.

In case that's unpractical, objdump'ing the beam.smp binary and navigating to the location of the crash might also help if the misbehaving code is listed under a routine with a readable name and that name can be traced back to the ERTS C code.

--
Guilherme
_______________________________________________

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions