How to view the content of memory using gdb

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

How to view the content of memory using gdb

Monk Boy
Hi all:
    My cowboy server occasionally crashe, I suspect that the data of the query is too large, but I don't know how to view the content of memory using gdb to find the detail data.

```
sudo gdb /home/q/erlang1705/lib/erlang/erts-6.4/bin/beam.smp core.18269
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/q/erlang1705/lib/erlang/erts-6.4/bin/beam.smp...done.
[New LWP 18288]
[New LWP 18274]
[New LWP 18276]
[New LWP 18289]
[New LWP 18281]
[New LWP 18273]
[New LWP 18280]
[New LWP 18278]
[New LWP 18277]
[New LWP 18284]
[New LWP 18282]
[New LWP 18275]
[New LWP 18287]
[New LWP 18285]
[New LWP 18291]
[New LWP 18269]
[New LWP 18279]
[New LWP 18286]
[New LWP 18292]
[New LWP 18290]
[New LWP 18271]
[New LWP 18272]
[New LWP 18283]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/home/q/erlang1705/lib/erlang/erts-6.4/bin/beam.smp -- -root /home/q/erlang1705'.
Program terminated with signal 6, Aborted.
#0  0x00007fda423b55f7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.17-106.el7_2.8.x86_64 ncurses-libs-5.9-13.20130511.el7.x86_64 zlib-1.2.7-15.el7.x86_64
(gdb) bt
#0  0x00007fda423b55f7 in raise () from /lib64/libc.so.6
#1  0x00007fda423b6ce8 in abort () from /lib64/libc.so.6
#2  0x000000000045806e in erl_exit_vv (n=1, flush_async=flush_async@entry=0, fmt=fmt@entry=0x5fbce8 "%s: Cannot %s %lu bytes of memory (of type \"%s\", thread %d).\n", args1=args1@entry=0x7fda361ba830, args2=args2@entry=0x7fda361ba848)
    at beam/erl_init.c:2123
#3  0x0000000000458308 in erl_exit (n=n@entry=1, fmt=fmt@entry=0x5fbce8 "%s: Cannot %s %lu bytes of memory (of type \"%s\", thread %d).\n") at beam/erl_init.c:2133
#4  0x000000000043c4c1 in erts_alc_fatal_error (error=error@entry=1, func=<optimized out>, func@entry=0, n=n@entry=155) at beam/erl_alloc.c:1875
#5  0x000000000043c507 in erts_alloc_n_enomem (n=n@entry=155, size=size@entry=4723591984) at beam/erl_alloc.c:1900
#6  0x0000000000529023 in erts_alloc (size=4723591984, type=19862) at beam/erl_alloc.h:228
#7  minor_collection (recl=<synthetic pointer>, nobj=2, objv=0x7fda415d0380, need=0, p=0x7fda40a46a58) at beam/erl_gc.c:852
#8  erts_garbage_collect (p=p@entry=0x7fda40a46a58, need=need@entry=0, objv=0x7fda415d0380, nobj=2) at beam/erl_gc.c:450
#9  0x00000000005291d5 in erts_gc_after_bif_call (p=0x7fda40a46a58, result=0, regs=<optimized out>, arity=<optimized out>) at beam/erl_gc.c:362
#10 0x000000000056e99c in process_main () at beam/beam_emu.c:3578
#11 0x00000000004a734a in sched_thread_func (vesdp=0x7fda398acd40) at beam/erl_process.c:7695
#12 0x00000000005f6eb5 in thr_wrapper (vtwd=0x7ffcbf9b3b70) at pthread/ethread.c:106
#13 0x00007fda42951dc5 in start_thread () from /lib64/libpthread.so.0
#14 0x00007fda42476ced in clone () from /lib64/libc.so.6
(gdb) 9
Undefined command: "9".  Try "help".
(gdb) f 9
#9  0x00000000005291d5 in erts_gc_after_bif_call (p=0x7fda40a46a58, result=0, regs=<optimized out>, arity=<optimized out>) at beam/erl_gc.c:362
362                 cost = erts_garbage_collect(p, 0, regs, p->arity);
(gdb) source /home/q/lffan.liu/download/otp_src_17.5/erts/etc/unix/etp-commands.in
%---------------------------------------------------------------------------
% Use etp-help for a command overview and general help.
%
% To use the Erlang support module, the environment variable ROOTDIR
% must be set to the toplevel installation directory of Erlang/OTP,
% so the etp-commands file becomes:
%     $ROOTDIR/erts/etc/unix/etp-commands
% Also, erl and erlc must be in the path.
%---------------------------------------------------------------------------
etp-set-max-depth 20
etp-set-max-string-length 100
--------------- System Information ---------------
OTP release: 17
ERTS version: 6.4
Compile date: Thu Apr 11 12:23:01 2019
Arch: x86_64-unknown-linux-gnu
Endianness: Little
Word size: 64-bit
Halfword: no
HiPE support: yes
SMP support: yes
Thread support: yes
Kernel poll: Supported but not used
Debug compiled: no
Lock checking: no
Lock counting: no
Node name: 'ejb_http_server@localhost'
Number of schedulers: 8
Number of async-threads: 10
--------------------------------------------------
(gdb) etp-stackdump p
% Stackdump (21): #Cp<lists:reverse/1+0x120>.
#Cp<rfc4627:encode/1+0x48>.
#Cp<http_utils:gen_result/4+0x130>.
#Cp<http_get_history:post_echo/3+0x770>.
#Cp<http_get_history:do_handle/2+0x1c8>.
{http_req,#Port<0.101737>,ranch_tcp,keepalive,<0.15468.5>,#SubBinary<0x4,0,0xde010000:0x13fffcd2>,'HTTP/1.1',{{10,88,193,24},13758},#SubBinary<0xf,0,0x10000:0x13fffd2a>,undefined,80,#SubBinary<0x13,0,0xde000000:0x13fffd5a>,undefined,#SubBinary<0x48,0,0xde010000:0x13fffd8a>,undefined,[],[{#SubBinary<0x4,0,0xde010000:0x13fffffa>,#SubBinary<0xf,0,0x10000:0x1400002a>},{0.000000,#HeapBinary<0x2,0xffff3839>},{#SubBinary<0xf,0,0xd5010000:0x1400026a>,#SubBinary<0xd,0,0x10000:0x1400029a>},{#SubBinary<0xf,0,0xde010000:0x14000382>,#SubBinary<0xa,0,0x10000:0x140003b2>},{#SubBinary<0xa,0,0x10000:0x140004c2>,#SubBinary<0xb,0,0x10000:0x140004f2>},{#SubBinary<0xc,0,0x10000:0x14000612>,#SubBinary<0x21,0,0x10000:0x14000642>},{#SubBinary<0x6,0,0xde010000:0x14000792>,#SubBinary<0x18,0,0x10000:0x140007c2>},{#SubBinary<0x9,0,0xde010000:0x14000912>,#SubBinary<0xe,0,0xde010000:0x14000942>},{#SubBinary<0xd,0,0xde010000:0x14000a72>,#SubBinary<0x5,0,0x10000:0x14000aa2>},{#SubBinary<0xf,0,0xde010000:0x14000bba>,#SubBinary<0xe,0,0x10000:0x14000bea>}],[{0.000000,98},{0.000000,undefined},{0.000000,98}],undefined,[],...}.
#Cp<cowboy_handler:handler_handle/4+0xb8>.
[].
undefined.
[].
#Cp<cowboy_protocol:execute/4+0xb0>.
[].
[].
undefined.
http_get_history.
{state,[{handler,http_get_history},{handler_opts,[]},{listener,http},{dispatch,[{'_',[],[{[#RefcBinary<0x8,0x13fffcd0,0x311dfa20,0x311dfa38,0>],[],http_getusers,[]},{[#RefcBinary<0x7,0x140006f8,0x311c3288,0x311c32a0,0>],[],http_getdeps,[]},{[#RefcBinary<0xb,0x14000878,0x311e9fc0,0x311e9fd8,0>],[],http_getmsgs,[]},{[#RefcBinary<0xd,0x140009d8,0x311c2d10,0x311c2d28,0>],[],http_getonlineuser,[]},{[#RefcBinary<0xe,0x14000b20,0x311c2d50,0x311c2d68,0>],[],http_getonlineuser2,[]},{[#RefcBinary<0xe,0x14000c18,0x311e9e40,0x311e9e58,0>],[],http_get_adjacentmsg,...},{[...],[],...},{[...],...},{...},...]}]},{max_connections,infinity}],false,0,5000,infinity,undefined,false}.
{http_req,#Port<0.101737>,ranch_tcp,keepalive,<0.15468.5>,#SubBinary<0x4,0,0xde010000:0x13fffcd2>,'HTTP/1.1',{{10,88,193,24},13758},#SubBinary<0xf,0,0x10000:0x13fffd2a>,undefined,80,#SubBinary<0x13,0,0xde000000:0x13fffd5a>,undefined,#SubBinary<0x48,0,0xde010000:0x13fffd8a>,undefined,[],[{#SubBinary<0x4,0,0xde010000:0x13fffffa>,#SubBinary<0xf,0,0x10000:0x1400002a>},{#SubBinary<0xe,0,0xde010000:0x140001ba>,#SubBinary<0x2,0,0xde010000:0x140001ea>},{#SubBinary<0xf,0,0xd5010000:0x1400026a>,#SubBinary<0xd,0,0x10000:0x1400029a>},{#SubBinary<0xf,0,0xde010000:0x14000382>,#SubBinary<0xa,0,0x10000:0x140003b2>},{#SubBinary<0xa,0,0x10000:0x140004c2>,#SubBinary<0xb,0,0x10000:0x140004f2>},{#SubBinary<0xc,0,0x10000:0x14000612>,#SubBinary<0x21,0,0x10000:0x14000642>},{#SubBinary<0x6,0,0xde010000:0x14000792>,#SubBinary<0x18,0,0x10000:0x140007c2>},{#SubBinary<0x9,0,0xde010000:0x14000912>,#SubBinary<0xe,0,0xde010000:0x14000942>},{#SubBinary<0xd,0,0xde010000:0x14000a72>,#SubBinary<0x5,0,0x10000:0x14000aa2>},{#SubBinary<0xf,0,0xde010000:0x14000bba>,#SubBinary<0xe,0,0x10000:0x14000bea>}],[{0.000000,98},{0.000000,[Cannot access memory at address 0x18
(gdb) etp-process
etp-process-info         etp-process-memory-info  etp-processes            etp-processes-memory    
(gdb) etp-process-info p
  Pid: <0.15468.5>
  State: garbage-collecting | running | active | prq-prio-normal | usr-prio-normal | act-prio-normal
  Current function: lists:reverse/2
  CP: #Cp<lists:reverse/1+0x120>
  I: #Cp<0x3ff50a70>
  Heap size: 492040832
  Mbuf size: 160000
  Msgq len: 0 (inner=0, outer=0)
  Parent: <0.600.0>
  Pointer: (Process *) 0x7fda40a46a58
```

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: How to view the content of memory using gdb

Marc Worrell
Do you have some other logs?
Preferably the one where the message below is printed.

On 21 May 2019, at 05:39, Monk Boy <[hidden email]> wrote:

#2  0x000000000045806e in erl_exit_vv (n=1, flush_async=flush_async@entry=0, fmt=fmt@entry=0x5fbce8 "%s: Cannot %s %lu bytes of memory (of type \"%s\", thread %d).\n", args1=args1@entry=0x7fda361ba830, args2=args2@entry=0x7fda361ba848)
    at beam/erl_init.c:2123

This could be out of memory, but also something else “Cannot %s” …

If it is OOM then you can easily check the memory usage of beam.smp whilst it is running.
That would give enough hints.

Also check your running server with ‘etop’. 

Cowboy has configurations max request content length, it will refuse the request with a 4xx error.
Afaik it defaults to 64K for urlencoded bodies and 8M for other bodies.

- Marc


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: How to view the content of memory using gdb

Eckard Brauer
Hi,

stack frames 6..4 seem to point to a OOM condition:

#4  0x000000000043c4c1 in erts_alc_fatal_error (error=error@entry=1, func=<optimized out>, func@entry=0, n=n@entry=155) at beam/erl_alloc.c:1875
#5  0x000000000043c507 in erts_alloc_n_enomem (n=n@entry=155, size=size@entry=4723591984) at beam/erl_alloc.c:1900
#6  0x0000000000529023 in erts_alloc (size=4723591984, type=19862) at beam/erl_alloc.h:228

as first memory seems to be requested by erts_alloc() in #6, then it's
detected that no more memory is available and erts_alloc_n_enomem() is
called because of that (#5; still at C level) what is turned into a
fatal error at Erlang level too in #4, erts_alc_fatal_error().

For viewing memory content, it's good to know both the currently executed
code and the involved C structs before, to be able to jump along that IIRC.
Look at gdb's "print" command ("help print"). If the binary/binaries is/are
not stripped, you can also use symbolic names, what makes things much
easier, and you can examine source code with "list" (help list). For the
rest, "disassemble" and some knowledge of the machine's/processor's
assembly language is helpful. That only for short, it's a while since, and
I'd have to try it on a "living body" again to clearer describe...

Indeed, best would be to have the correct Erlang log.

Cheers,
Eckard


Am Tue, 21 May 2019 08:50:16 +0200
schrieb Marc Worrell <[hidden email]>:

> Do you have some other logs?
> Preferably the one where the message below is printed.
>
> > On 21 May 2019, at 05:39, Monk Boy <[hidden email]> wrote:
> >
> > #2  0x000000000045806e in erl_exit_vv (n=1,
> > flush_async=flush_async@entry=0, fmt=fmt@entry=0x5fbce8 "%s: Cannot
> > %s %lu bytes of memory (of type \"%s\", thread %d).\n",
> > args1=args1@entry=0x7fda361ba830, args2=args2@entry=0x7fda361ba848)
> > at beam/erl_init.c:2123
>
> This could be out of memory, but also something else “Cannot %s” …
>
> If it is OOM then you can easily check the memory usage of beam.smp
> whilst it is running. That would give enough hints.
>
> Also check your running server with ‘etop’.
>
> Cowboy has configurations max request content length, it will refuse
> the request with a 4xx error. Afaik it defaults to 64K for urlencoded
> bodies and 8M for other bodies.
>
> - Marc
>


--
:)

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

attachment0 (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: How to view the content of memory using gdb

Monk Boy
In reply to this post by Marc Worrell
Because the memory suddenly rises , no time to run 'etop'.  I suspect that the responseof the request is too large, and  the flood-like keep-alive request caused such a result.

Marc Worrell <[hidden email]> 于2019年5月21日周二 下午2:50写道:
Do you have some other logs?
Preferably the one where the message below is printed.

On 21 May 2019, at 05:39, Monk Boy <[hidden email]> wrote:

#2  0x000000000045806e in erl_exit_vv (n=1, flush_async=flush_async@entry=0, fmt=fmt@entry=0x5fbce8 "%s: Cannot %s %lu bytes of memory (of type \"%s\", thread %d).\n", args1=args1@entry=0x7fda361ba830, args2=args2@entry=0x7fda361ba848)
    at beam/erl_init.c:2123

This could be out of memory, but also something else “Cannot %s” …

If it is OOM then you can easily check the memory usage of beam.smp whilst it is running.
That would give enough hints.

Also check your running server with ‘etop’. 

Cowboy has configurations max request content length, it will refuse the request with a 4xx error.
Afaik it defaults to 64K for urlencoded bodies and 8M for other bodies.

- Marc


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: How to view the content of memory using gdb

Lukas Larsson-8
In reply to this post by Monk Boy
Hello,

If you upgrade to Erlang/OTP vsn 20.3 or later this information is part of the crash dump so that you can use tools such as crash dump viewer to get this information.

If you want etp to not cut data you need to do:

  set $etp_max_depth = 20000
  set $etp_max_string_length = 10000

before you call any term printing function. You can also print the entire heap of the process using:

   etp-heapdump p

This will dump a lot of data that will need to dig through to try to find what is taking all of the memory.

Lukas

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: How to view the content of memory using gdb

Monk Boy
Thanks for the tip! I will do it now.

Lukas Larsson <[hidden email]> 于2019年5月21日周二 下午3:33写道:
Hello,

If you upgrade to Erlang/OTP vsn 20.3 or later this information is part of the crash dump so that you can use tools such as crash dump viewer to get this information.

If you want etp to not cut data you need to do:

  set $etp_max_depth = 20000
  set $etp_max_string_length = 10000

before you call any term printing function. You can also print the entire heap of the process using:

   etp-heapdump p

This will dump a lot of data that will need to dig through to try to find what is taking all of the memory.

Lukas

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions