erl.exe dies but werl.exe does not on both Windows XP and 2008R2 with R14B01

[Dave Cottlehuber]

Hallo,

There are 2 issues I've identified - VM crash & VM hang. Both occur
within a CouchDB build of erlang, on various windows variants. This
email covers the crash only. It is easy to reproduce:

Install CouchDB 1.0.1 or a more recent build from
https://github.com/dch/couchdb/downloads and curl.exe from
http://haxx.se/
open command prompt and change to couchdb/bin folder.
set erl=erl
couchdb.bat

& run this script until erlang hangs (watch the erl console scroll
by!). On my 2 testbeds this takes less than a minute to occur - just
25 curls.

::restart_couch.cmd
@echo off
:restart
for /l %%i in (1,1,100000000000) do @call :curl %%i
goto :eof
:curl
curl -v -H "Content-Type: application/json" -X POST
http://localhost:5984/_restart
:: check to see if couch died horribly
if exist erl_crash.dump echo Woops!!!! && move /y erl_crash.dump
erl_crash.dump.%1
goto :eof


=erl_crash_dump:0.1
Fri Mar 04 23:48:06 2011
Slogan: Kernel pid terminated (application_controller)
({application_start_failure,kernel,{shutdown,{kernel,start,[normal,[]]}}})
System version: Erlang R14B01 (erts-5.8.2) [source] [smp:2:2] [rq:2]
[async-threads:4]
Compiled: Sat Feb 12 23:25:21 2011

The full .dump can be found at http://friendpaste.com/1gMbN0i2zn3mlaHAC54D58

1>   [replicator] max_http_sessions="20"
1>   [replicator] ssl_certificate_max_depth="3"
1>   [replicator] verify_ssl_certificates="false"
1>   [stats] rate="1000"
1>   [stats] samples="[0, 60, 300, 900]"
1>   [uuids] algorithm="sequential"
1> Apache CouchDB has started. Time to relax.
1> [info] [<0.689.0>] Apache CouchDB has started on http://0.0.0.0:5984/
1> [debug] [<0.761.0>] 'POST' /_restart {1,1}
Headers: [{'Accept',"*/*"},
          {'Content-Type',"application/json"},
          {'Host',"localhost:5984"},
          {'User-Agent',"curl/7.19.0 (i586-pc-mingw32msvc)
libcurl/7.19.0 OpenSSL/1.0.0c zlib/1.2.3"
}]
1> [debug] [<0.761.0>] OAuth Params: []
1> [info] [<0.761.0>] 127.0.0.1 - - 'POST' /_restart 200
1> {error_logger,{{2011,3,4},{23,48,5}},crash_report,[[{initial_call,{supervisor_bridge,user_sup,['Argument__1']}},{pid,<0.785.0>},{registered_name,[]},{error_info,{exit,nouser,[{gen_server,init_it,6},{proc_lib,init_p_do_apply,3}]}},{ancestors,[kernel_sup,<0.773.0>]},{messages,[]},{links,[<0.774.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,610},{stack_size,24},{reductions,338}],[]]}
{error_logger,{{2011,3,4},{23,48,5}},supervisor_report,[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,nouser},{offender,[{pid,undefined},{name,user},{mfargs,{user_sup,start,[]}},{restart_type,temporary},{shutdown,2000},{child_type,supervisor}]}]}
{error_logger,{{2011,3,4},{23,48,5}},std_info,[{application,kernel},{exited,{shutdown,{kernel,start,[normal,[]]}}},{type,permanent}]}
{"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{shutdown,{kernel,start,[normal,[]]}}}"}

Crash dump was written to: erl_crash.dump
Kernel pid terminated (application_controller)
({application_start_failure,kernel,{shutdown,{kernel,start,[normal,[]]}}})


Can anybody help clarify why this happens, and what we can do about it?

Thanks
Dave

________________________________________________________________
erlang-bugs (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:[hidden email]

[Dave Cottlehuber]

On 4 March 2011 23:53, Dave Cottlehuber <[hidden email]> wrote:

> Hallo,
>
> There are 2 issues I've identified - VM crash & VM hang. Both occur
> within a CouchDB build of erlang, on various windows variants. This
> email covers the crash only. It is easy to reproduce:
>
> Install CouchDB 1.0.1 or a more recent build from
> https://github.com/dch/couchdb/downloads and curl.exe from
> http://haxx.se/
> open command prompt and change to couchdb/bin folder.
> set erl=erl
> couchdb.bat
>
> & run this script until erlang dies (watch the erl console scroll
> by!). On my 2 testbeds this takes less than a minute to occur - just
> 25 curls.

Sorry; key point is that running with werl.exe instead will run
successfully for days.
More info on the original issue is available at
https://issues.apache.org/jira/browse/COUCHDB-963.

Thanks again.
Dave

________________________________________________________________
erlang-bugs (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:[hidden email]

[Dave Cottlehuber]

On 4 March 2011 23:53, Dave Cottlehuber <[hidden email]> wrote:
> Hallo,
>
> There are 2 issues I've identified - VM crash & VM hang. Both occur
> within a CouchDB build of erlang, on various windows variants. This
> email covers the crash only. It is easy to reproduce:

The 2nd issue is related to VM hang, under erlsrv service account &
not batch file. I am using following erlsrv configurations; the debug
one provides a visible console so easier to work with but same issue
occurs with either service configuration.

debug: erlsrv.exe add "CouchDeBug" -workdir
"c:\couch\couchdb-1.0.2\bin" -onfail restart_always -debugtype console
-args "-sasl errlog_type error -s couch +A 4 +W w" -comment
"CouchDeBug" -machine "c:\couch\CouchDB-1.0.2\erts-5.8.2\bin\erl.exe"

new: erlsrv.exe add "NewCouch" -workdir "c:\couch\couchdb-1.0.2\bin"
-onfail restart_always -args "-sasl errlog_type error -s couch +A 4 +W
w" -comment "NewCouch" -machine
"c:\couch\CouchDB-1.0.2\erts-5.8.2\bin\erl.exe"

erlsrv starts erl.exe not werl.exe and so the issue noted in previous
email crops up and the application is down.

2 questions -

Why does erlsrv not use werl.exe ? it is technically possible to pass
erlsrv the -machine parameter with werl.exe, and it runs successfully
as a service avoiding the original issue. The erlang/OTP source
confirms that this is bad - but why?

2nd part. using erlsrv "-restart_always" couchdb restarts very
quickly. but after up to 8h of continuous curl _restart, the erl.exe
window definitely hangs - no input accepted - as if the REPL loop is
over.

Any ideas why erlang seems to hang around init:restart() and what we
can do about it? Do you want any more information?
Thanks
Dave

________________________________________________________________
erlang-bugs (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:[hidden email]

[pan]

Hi,

On Sat, 5 Mar 2011, Dave Cottlehuber wrote:

> On 4 March 2011 23:53, Dave Cottlehuber <[hidden email]> wrote:
>> Hallo,
>>
>> There are 2 issues I've identified - VM crash & VM hang. Both occur
>> within a CouchDB build of erlang, on various windows variants. This
>> email covers the crash only. It is easy to reproduce:
>
> The 2nd issue is related to VM hang, under erlsrv service account &
> not batch file. I am using following erlsrv configurations; the debug
> one provides a visible console so easier to work with but same issue
> occurs with either service configuration.
>
> debug: erlsrv.exe add "CouchDeBug" -workdir
> "c:\couch\couchdb-1.0.2\bin" -onfail restart_always -debugtype console
> -args "-sasl errlog_type error -s couch +A 4 +W w" -comment
> "CouchDeBug" -machine "c:\couch\CouchDB-1.0.2\erts-5.8.2\bin\erl.exe"
>
> new: erlsrv.exe add "NewCouch" -workdir "c:\couch\couchdb-1.0.2\bin"
> -onfail restart_always -args "-sasl errlog_type error -s couch +A 4 +W
> w" -comment "NewCouch" -machine
> "c:\couch\CouchDB-1.0.2\erts-5.8.2\bin\erl.exe"
>
> erlsrv starts erl.exe not werl.exe and so the issue noted in previous
> email crops up and the application is down.

I think it may be the same issue. We're investigating the batch file issue
to start with. The problem is easy to reproduce - very nice.

>
> 2 questions -
>
> Why does erlsrv not use werl.exe ? it is technically possible to pass
> erlsrv the -machine parameter with werl.exe, and it runs successfully
> as a service avoiding the original issue. The erlang/OTP source
> confirms that this is bad - but why?

Werl cannot handle everything the erlsrv program wants to do to the
machine, like stopactions, killing by signalling etc. Fixing erl so it
does not hang is the easiest and best thing to do here.

>
> 2nd part. using erlsrv "-restart_always" couchdb restarts very
> quickly. but after up to 8h of continuous curl _restart, the erl.exe
> window definitely hangs - no input accepted - as if the REPL loop is
> over.
>
> Any ideas why erlang seems to hang around init:restart() and what we
> can do about it? Do you want any more information?

When running erl on Windows you get the "old shell", meaning that another
io-server is running and also a special driver (the fd-driver) is used.
The fd-driver is emulatong Unix behaviour and might be the cause of all
the problems, but the actual user.erl code might also be broken. I'll
debug it, find the first problem and get back to you when I've narrowed it
down!


> Thanks
> Dave

Cheers,
/Patrik

>
> ________________________________________________________________
> erlang-bugs (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:[hidden email]
>

________________________________________________________________
erlang-bugs (at) erlang.org mailing list.
See http://www.erlang.org/faq.html
To unsubscribe; mailto:[hidden email]