Segmentation fault in erts_printf_char (R13B04)

[Kustarto Widoyo]

Hi All,

We found that our application was crashed and core dump file was
generated, but not found any erl_crash.dump file created.

* The application uses erlang distribution protocol that more than 50
nodes are involving in. All nodes run on Redhat 5.3.

* We're using Erlang R13B04 64 bit built from source and the following
patches have been applied:
        otp_src_R13B04-OTP-8475.patch
        otp_src_R13B04-OTP-8612.patch
        otp_src_R13B04-OTP-8643.patch
        otp_src_R13B04-OTP-8658.patch
        otp_src_R13B04-OTP-8661.patch
        otp_src_R13B04-OTP-8662.patch
        otp_src_R13B04-beam-break.patch
        otp_src_R13B04-emacs.patch
        otp_src_R13B04-erl_poll.patch
        otp_src_R13B04-erts_de_busy_limit.patch
        otp_src_R13B04-eunit.patch
        otp_src_R13B04-httpc-memoryleak.patch
        otp_src_R13B04-patch-etop.patch
        otp_src_R13B04-patch-odbc-oracleworkaround.patch
        otp_src_R13B04-supervisor.patch

* The erl command line we use:
$ /usr/local/gemini/ert/R13B04/lib/erlang/erts-5.7.5/bin/beam.smp -A 64
-K true -S 0 -- -root /usr/local/gemini/ert/R13B04/lib/erlang -progname
erl -- -home /export/home/mmssys -- -smp enable -noshell -noinput
-noshell -sname gdss1 -kernel net_ticktime 20 -boot
/usr/local/gemini/gdss/1.0.0/lib/app/gdss_all -config
/usr/local/gemini/gdss/1.0.0/../var/data/node1.config -pa
/usr/local/gemini/gdss/1.0.0/lib/app-patches -pz
/usr/local/gemini/gdss/1.0.0/lib/app -pz
/usr/local/gemini/gdss/1.0.0/lib -central_config
/usr/local/gemini/gdss/1.0.0/etc/central.conf -ticket_broker_config
/usr/local/gemini/gdss/1.0.0/etc/broker.conf


* The following is gdb and backtrace output:
------------------------
[root@gds001c data]# gdb
/usr/local/gemini/ert/R13B04/lib/erlang/erts-5.7.5/bin/beam.smp core.14720
GNU gdb Fedora (6.8-27.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
Reading symbols from /lib64/libutil.so.1...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /usr/lib64/libncurses.so.5...done.
Loaded symbols for /usr/lib64/libncurses.so.5
Reading symbols from /lib64/libpthread.so.0...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/librt.so.1...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from
/usr/local/gemini/ert/R13B04/lib/erlang/lib/crypto-1.6.4/priv/lib/crypto_drv.so...done.
Loaded symbols for
/usr/local/gemini/ert/R13B04/lib/erlang/lib/crypto-1.6.4/priv/lib/crypto_drv.so
Reading symbols from
/usr/local/gemini/ert/R13B04/openssl/lib/libcrypto.so.0.9.8...done.
Loaded symbols for
/usr/local/gemini/ert/R13B04/openssl/lib/libcrypto.so.0.9.8
Core was generated by
`/usr/local/gemini/ert/R13B04/lib/erlang/erts-5.7.5/bin/beam.smp -A 64
-K true -'.
Program terminated with signal 11, Segmentation fault.
[New process 14798]

snip ... snip ... snip ...

#0  0x0000000000585ea8 in erts_printf_char (fn=0x5880f0 <write_ds>,
arg=0xb750930, c=91 '[') at common/erl_printf_format.c:840
840     common/erl_printf_format.c: No such file or directory.
         in common/erl_printf_format.c
(gdb) bt
#0  0x0000000000585ea8 in erts_printf_char (fn=0x5880f0 <write_ds>,
arg=0xb750930, c=91 '[') at common/erl_printf_format.c:840
#1  0x000000000048c6d8 in print_term (fn=0x5880f0 <write_ds>,
arg=0xb750930, obj=5800176, dcount=0x458f2bc8)
     at beam/erl_printf_term.c:346
#2  0x000000000048c740 in print_term (fn=0x5880f0 <write_ds>,
arg=0xb750930, obj=<value optimized out>, dcount=0x458f2bc8)
     at beam/erl_printf_term.c:349
#3  0x000000000048c740 in print_term (fn=0x5880f0 <write_ds>,
arg=0xb750930, obj=<value optimized out>, dcount=0x458f2bc8)
     at beam/erl_printf_term.c:349

snip ... snip ... snip ...
------------------------

* And in syslog, we found:
Jan 27 23:48:21 gds001c kernel: beam.smp[14798]: segfault at
0000000044ef3ff8 rip 0000000000585ea8 rsp 0000000044ef4000 error 6

Please let me know, if there is anything else we have to provide.

Regards,
--
Kustarto Widoyo
Gemini Mobile Technologies

________________________________________________________________
erlang-bugs (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:[hidden email]

[Kustarto Widoyo]

Oops, forgot to put the url address for the patches.
https://github.com/hibari/patches

 > * We're using Erlang R13B04 64 bit built from source and the following
 > patches have been applied:
 > otp_src_R13B04-OTP-8475.patch
 > otp_src_R13B04-OTP-8612.patch
 > otp_src_R13B04-OTP-8643.patch
 > otp_src_R13B04-OTP-8658.patch
 > otp_src_R13B04-OTP-8661.patch
 > otp_src_R13B04-OTP-8662.patch
 > otp_src_R13B04-beam-break.patch
 > otp_src_R13B04-emacs.patch
 > otp_src_R13B04-erl_poll.patch
 > otp_src_R13B04-erts_de_busy_limit.patch
 > otp_src_R13B04-eunit.patch
 > otp_src_R13B04-httpc-memoryleak.patch
 > otp_src_R13B04-patch-etop.patch
 > otp_src_R13B04-patch-odbc-oracleworkaround.patch
 > otp_src_R13B04-supervisor.patch

Regards,
--
Kustarto Widoyo
Gemini Mobile Technologies

________________________________________________________________
erlang-bugs (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:[hidden email]

[Kustarto Widoyo]

Could someone help to take a look at this issue?

Regards,
Widoyo

________________________________________________________________
erlang-bugs (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:[hidden email]

[pan]

Not much to go on here, could be anything.

Could you supply the rest of the stack, i.e. who calls the print_term
function?

Also, by using the etp-commands gdb macros (in source tree,
$ERL_TOP/erts/etc/unix/), you could see the term that's being printed.
Is it corrupted?

I would also try increasing the schedulers stacksize (erl +sss Value), as
it could be a stack overrun (the parameters to the functions seem ok, but
that's a wild guess).

/Patrik

On Fri, 4 Feb 2011, Kustarto Widoyo wrote:

> Hi All,
>
> We found that our application was crashed and core dump file was generated,
> but not found any erl_crash.dump file created.
>
> * The application uses erlang distribution protocol that more than 50 nodes
> are involving in. All nodes run on Redhat 5.3.
>
> * We're using Erlang R13B04 64 bit built from source and the following
> patches have been applied:
> otp_src_R13B04-OTP-8475.patch
> otp_src_R13B04-OTP-8612.patch
> otp_src_R13B04-OTP-8643.patch
> otp_src_R13B04-OTP-8658.patch
> otp_src_R13B04-OTP-8661.patch
> otp_src_R13B04-OTP-8662.patch
> otp_src_R13B04-beam-break.patch
> otp_src_R13B04-emacs.patch
> otp_src_R13B04-erl_poll.patch
> otp_src_R13B04-erts_de_busy_limit.patch
> otp_src_R13B04-eunit.patch
> otp_src_R13B04-httpc-memoryleak.patch
> otp_src_R13B04-patch-etop.patch
> otp_src_R13B04-patch-odbc-oracleworkaround.patch
> otp_src_R13B04-supervisor.patch
>
> * The erl command line we use:
> $ /usr/local/gemini/ert/R13B04/lib/erlang/erts-5.7.5/bin/beam.smp -A 64 -K
> true -S 0 -- -root /usr/local/gemini/ert/R13B04/lib/erlang -progname erl --
> -home /export/home/mmssys -- -smp enable -noshell -noinput -noshell -sname
> gdss1 -kernel net_ticktime 20 -boot
> /usr/local/gemini/gdss/1.0.0/lib/app/gdss_all -config
> /usr/local/gemini/gdss/1.0.0/../var/data/node1.config -pa
> /usr/local/gemini/gdss/1.0.0/lib/app-patches -pz
> /usr/local/gemini/gdss/1.0.0/lib/app -pz /usr/local/gemini/gdss/1.0.0/lib
> -central_config /usr/local/gemini/gdss/1.0.0/etc/central.conf
> -ticket_broker_config /usr/local/gemini/gdss/1.0.0/etc/broker.conf
>
>
> * The following is gdb and backtrace output:
> ------------------------
> [root@gds001c data]# gdb
> /usr/local/gemini/ert/R13B04/lib/erlang/erts-5.7.5/bin/beam.smp core.14720
> GNU gdb Fedora (6.8-27.el5)
> Copyright (C) 2008 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu"...
> Reading symbols from /lib64/libutil.so.1...done.
> Loaded symbols for /lib64/libutil.so.1
> Reading symbols from /lib64/libdl.so.2...done.
> Loaded symbols for /lib64/libdl.so.2
> Reading symbols from /lib64/libm.so.6...done.
> Loaded symbols for /lib64/libm.so.6
> Reading symbols from /usr/lib64/libncurses.so.5...done.
> Loaded symbols for /usr/lib64/libncurses.so.5
> Reading symbols from /lib64/libpthread.so.0...done.
> Loaded symbols for /lib64/libpthread.so.0
> Reading symbols from /lib64/librt.so.1...done.
> Loaded symbols for /lib64/librt.so.1
> Reading symbols from /lib64/libc.so.6...done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from
> /usr/local/gemini/ert/R13B04/lib/erlang/lib/crypto-1.6.4/priv/lib/crypto_drv.so...done.
> Loaded symbols for
> /usr/local/gemini/ert/R13B04/lib/erlang/lib/crypto-1.6.4/priv/lib/crypto_drv.so
> Reading symbols from
> /usr/local/gemini/ert/R13B04/openssl/lib/libcrypto.so.0.9.8...done.
> Loaded symbols for
> /usr/local/gemini/ert/R13B04/openssl/lib/libcrypto.so.0.9.8
> Core was generated by
> `/usr/local/gemini/ert/R13B04/lib/erlang/erts-5.7.5/bin/beam.smp -A 64 -K
> true -'.
> Program terminated with signal 11, Segmentation fault.
> [New process 14798]
>
> snip ... snip ... snip ...
>
> #0  0x0000000000585ea8 in erts_printf_char (fn=0x5880f0 <write_ds>,
> arg=0xb750930, c=91 '[') at common/erl_printf_format.c:840
> 840     common/erl_printf_format.c: No such file or directory.
>        in common/erl_printf_format.c
> (gdb) bt
> #0  0x0000000000585ea8 in erts_printf_char (fn=0x5880f0 <write_ds>,
> arg=0xb750930, c=91 '[') at common/erl_printf_format.c:840
> #1  0x000000000048c6d8 in print_term (fn=0x5880f0 <write_ds>, arg=0xb750930,
> obj=5800176, dcount=0x458f2bc8)
>    at beam/erl_printf_term.c:346
> #2  0x000000000048c740 in print_term (fn=0x5880f0 <write_ds>, arg=0xb750930,
> obj=<value optimized out>, dcount=0x458f2bc8)
>    at beam/erl_printf_term.c:349
> #3  0x000000000048c740 in print_term (fn=0x5880f0 <write_ds>, arg=0xb750930,
> obj=<value optimized out>, dcount=0x458f2bc8)
>    at beam/erl_printf_term.c:349
>
> snip ... snip ... snip ...
> ------------------------
>
> * And in syslog, we found:
> Jan 27 23:48:21 gds001c kernel: beam.smp[14798]: segfault at 0000000044ef3ff8
> rip 0000000000585ea8 rsp 0000000044ef4000 error 6
>
> Please let me know, if there is anything else we have to provide.
>
> Regards,
> --
> Kustarto Widoyo
> Gemini Mobile Technologies
>
> ________________________________________________________________
> erlang-bugs (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:[hidden email]
>

________________________________________________________________
erlang-bugs (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:[hidden email]

[Kustarto Widoyo]

 > Could you supply the rest of the stack, i.e. who calls the print_term
 > function?

The following is about the rest of the stack　(result of thread apply
all bt).
...
Thread 1 (process 14798):
#0  0x0000000000585ea8 in erts_printf_char (fn=0x5880f0 <write_ds>,
     arg=0xb750930, c=91 '[') at common/erl_printf_format.c:840
#1  0x000000000048c6d8 in print_term (fn=0x5880f0 <write_ds>, arg=0xb750930,
     obj=5800176, dcount=0x458f2bc8) at beam/erl_printf_term.c:346
#2  0x000000000048c740 in print_term (fn=0x5880f0 <write_ds>, arg=0xb750930,
     obj=<value optimized out>, dcount=0x458f2bc8) at
beam/erl_printf_term.c:349
#3  0x000000000048c740 in print_term (fn=0x5880f0 <write_ds>, arg=0xb750930,
     obj=<value optimized out>, dcount=0x458f2bc8) at
beam/erl_printf_term.c:349
#4  0x000000000048c740 in print_term (fn=0x5880f0 <write_ds>, arg=0xb750930,
     obj=<value optimized out>, dcount=0x458f2bc8) at
beam/erl_printf_term.c:349

....snip....

#43669 0x000000000048c740 in print_term (fn=0x5880f0 <write_ds>,
     arg=0xb750930, obj=<value optimized out>, dcount=0x458f2bc8)
     at beam/erl_printf_term.c:349
#43670 0x000000000048cf48 in erts_printf_term (fn=0xb750930, arg=0x44ef4004,
     term=1, precision=56331) at beam/erl_printf_term.c:452
#43671 0x0000000000586b9c in erts_printf_format (fn=0x5880f0 <write_ds>,
     arg=0xb750930, fmt=<value optimized out>, ap=0x458f2cd0)
     at common/erl_printf_format.c:813
#43672 0x0000000000587df4 in erts_vdsprintf (dsbufp=0xb750930,
     format=0x591897 "   %T\n", arglist=0x458f2cd0) at
common/erl_printf.c:419
#43673 0x0000000000470fe0 in erts_print (to=<value optimized out>, arg=0x1,
     format=0x1c2510 <Address 0x1c2510 out of bounds>) at beam/utils.c:299
#43674 0x00000000004921a3 in erts_program_counter_info (to=-4,
     to_arg=0xb750930, p=0x2aaabb0ae630) at beam/erl_process.c:8387
#43675 0x00000000004921f5 in erts_stack_dump (to=-4, to_arg=0xb750930,
p=0x1)
     at beam/erl_process.c:8360
#43676 0x00000000004e9191 in print_process_info (to=-4, to_arg=0xb750930,
     p=0x2aaabb0ae630) at beam/break.c:343
#43677 0x00000000004e9384 in process_info (to=-4, to_arg=0xb750930)
     at beam/break.c:79
#43678 0x0000000000458744 in system_info_1 (A__p=0x2aaabb0b6ca0, A_1=24459)
     at beam/erl_bif_info.c:1973
#43679 0x000000000051279a in process_main () at beam/beam_emu.c:2087
#43680 0x000000000049fac2 in sched_thread_func (vesdp=<value optimized out>)
     at beam/erl_process.c:3060
#43681 0x0000000000585314 in thr_wrapper (vtwd=<value optimized out>)
     at common/ethread.c:480
#43682 0x0000003383006367 in start_thread () from /lib64/libpthread.so.0
#43683 0x00000033824d309d in clone () from /lib64/libc.so.6

 > Also, by using the etp-commands gdb macros (in source tree,
 > $ERL_TOP/erts/etc/unix/), you could see the term that's being printed.
 > Is it corrupted?

Sorry, I am still not able to use it.

Thanks,
Widoyo


________________________________________________________________
erlang-bugs (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:[hidden email]

[pan]

Hi!

Interesting, seems that the system_info bif barfs... It would be
interesting to see a printout of the parameters to system_info_1, the
first parameter should be a pointer to a process structure and the second
an erlang term (you print it with etp A__1).

Do you have any possibility to provide me with the core and your build of
the VM, together with the source?

Cheers,
/Patrik

On Tue, 8 Mar 2011, Kustarto Widoyo wrote:

>
>> Could you supply the rest of the stack, i.e. who calls the print_term
>> function?
>
> Please find the attached file. It's about the rest of the stack.
>
> Thread 1 (process 14798):
> #0  0x0000000000585ea8 in erts_printf_char (fn=0x5880f0 <write_ds>,
>    arg=0xb750930, c=91 '[') at common/erl_printf_format.c:840
> #1  0x000000000048c6d8 in print_term (fn=0x5880f0 <write_ds>, arg=0xb750930,
>    obj=5800176, dcount=0x458f2bc8) at beam/erl_printf_term.c:346
> #2  0x000000000048c740 in print_term (fn=0x5880f0 <write_ds>, arg=0xb750930,
>    obj=<value optimized out>, dcount=0x458f2bc8) at
> beam/erl_printf_term.c:349
> #3  0x000000000048c740 in print_term (fn=0x5880f0 <write_ds>, arg=0xb750930,
>    obj=<value optimized out>, dcount=0x458f2bc8) at
> beam/erl_printf_term.c:349
> #4  0x000000000048c740 in print_term (fn=0x5880f0 <write_ds>, arg=0xb750930,
>    obj=<value optimized out>, dcount=0x458f2bc8) at
> beam/erl_printf_term.c:349
>
> ....snip....
>
> #43669 0x000000000048c740 in print_term (fn=0x5880f0 <write_ds>,
>    arg=0xb750930, obj=<value optimized out>, dcount=0x458f2bc8)
>    at beam/erl_printf_term.c:349
> #43670 0x000000000048cf48 in erts_printf_term (fn=0xb750930, arg=0x44ef4004,
>    term=1, precision=56331) at beam/erl_printf_term.c:452
> #43671 0x0000000000586b9c in erts_printf_format (fn=0x5880f0 <write_ds>,
>    arg=0xb750930, fmt=<value optimized out>, ap=0x458f2cd0)
>    at common/erl_printf_format.c:813
> #43672 0x0000000000587df4 in erts_vdsprintf (dsbufp=0xb750930,
>    format=0x591897 "   %T\n", arglist=0x458f2cd0) at common/erl_printf.c:419
> #43673 0x0000000000470fe0 in erts_print (to=<value optimized out>, arg=0x1,
>    format=0x1c2510 <Address 0x1c2510 out of bounds>) at beam/utils.c:299
> #43674 0x00000000004921a3 in erts_program_counter_info (to=-4,
>    to_arg=0xb750930, p=0x2aaabb0ae630) at beam/erl_process.c:8387
> #43675 0x00000000004921f5 in erts_stack_dump (to=-4, to_arg=0xb750930, p=0x1)
>    at beam/erl_process.c:8360
> #43676 0x00000000004e9191 in print_process_info (to=-4, to_arg=0xb750930,
>    p=0x2aaabb0ae630) at beam/break.c:343
> #43677 0x00000000004e9384 in process_info (to=-4, to_arg=0xb750930)
>    at beam/break.c:79
> #43678 0x0000000000458744 in system_info_1 (A__p=0x2aaabb0b6ca0, A_1=24459)
>    at beam/erl_bif_info.c:1973
> #43679 0x000000000051279a in process_main () at beam/beam_emu.c:2087
> #43680 0x000000000049fac2 in sched_thread_func (vesdp=<value optimized out>)
>    at beam/erl_process.c:3060
> #43681 0x0000000000585314 in thr_wrapper (vtwd=<value optimized out>)
>    at common/ethread.c:480
> #43682 0x0000003383006367 in start_thread () from /lib64/libpthread.so.0
> #43683 0x00000033824d309d in clone () from /lib64/libc.so.6
>
>> Also, by using the etp-commands gdb macros (in source tree,
>> $ERL_TOP/erts/etc/unix/), you could see the term that's being printed.
>> Is it corrupted?
>
> Sorry, I am still not able to use it.
>
> Thanks,
> Widoyo
>
>
>

________________________________________________________________
erlang-bugs (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:[hidden email]

[Kustarto Widoyo]

Patrik did investigation for this issue, and as fyi, the following was
his comment.

 > The problem encountered is that the internal printf gets a really deep
 > structure to format for the call erlag:system_info(procs), called from a
 > fun declared in gmt_cinfo_basic:erlang_system_info. The call formats a
 > binary with debug information for every process in this huge system, of
 > which one has this reeeeally deep list structure.
 >
 > I shall limit the depth of the output in the erts_printf call, but you
 > should really not call erlang:system_info(procs) in such a huge system.
 > Using that kind of debug functionality in the system will cost *a lot*
 > in terms of memory and CPU. So a workaround for this would be to disable
 > this dumping of process debug information in your system.

Thank you very much Patrik.

Widoyo

________________________________________________________________
erlang-bugs (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:[hidden email]

[pan]

Hi!

A correction to erts_printf, that makes it not recurse on the C stack any
more, is on it's way. Expect to see it in the GitHub dev branch in a few
days.

On Fri, 18 Mar 2011, Kustarto Widoyo wrote:

> Patrik did investigation for this issue, and as fyi, the following was his
> comment.
>
>> The problem encountered is that the internal printf gets a really deep
>> structure to format for the call erlag:system_info(procs), called from a
>> fun declared in gmt_cinfo_basic:erlang_system_info. The call formats a
>> binary with debug information for every process in this huge system, of
>> which one has this reeeeally deep list structure.
>>
>> I shall limit the depth of the output in the erts_printf call, but you
>> should really not call erlang:system_info(procs) in such a huge system.
>> Using that kind of debug functionality in the system will cost *a lot*
>> in terms of memory and CPU. So a workaround for this would be to disable
>> this dumping of process debug information in your system.
>
> Thank you very much Patrik.

Thank you for the help in tracking this down!
>
> Widoyo

Cheers,
/Patrik
>
> ________________________________________________________________
> erlang-bugs (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:[hidden email]
>

________________________________________________________________
erlang-bugs (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:[hidden email]