{error,closed} vs. {error,econnreset}

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

{error,closed} vs. {error,econnreset}

Bekes, Andras G

Hi All,

 

Looks like I have a problem with the erroneous result of gen_tcp:recv/2, as it returns {error,closed} instead of my expectation of {error,econnreset}.

 

I can reproduce my problem by:

 

1, starting a server application and forcing it into a busy state when it is listening but not accepting connections

 

2,starting 1000 clients (in Erlang) that connect to the server, filling the TCP listen backlog of the server.

When the backlog is filled and after some time (~2 minutes), the Erlang connections return {error,closed}.

This is what I do in Erlang:

fun()->{ok,Socket}= TCPCONNECT(HOST,PORT),
       ok=gen_tcp:send(Socket,<<"SOMEDATA…”>>),
       io:format("~p:~p\n",[self(),gen_tcp:recv(Socket,100)])

end.

spawn this 1000 times.

 

3, a native client of the service however tells me that the connection was reset by peer.

According to strace, it executes a recvfrom which returns ECONNRESET:

recvfrom(...) = -1 ECONNRESET (Connection reset by peer)

I also straced the Erlang beam process and it indeed gets an ECONNRESET result for the recvfrom call, but the Erlang return value is {error,closed}.

 

Unfortunately it looks like I really need to separate these two results.

 

I also tried gen_tcp in active mode, but no difference, the result is {error,closed} instead of {error,econnreset}.

 

Can someone explain why the econnreset error is masked? Is there any way I can separate the two kinds of events?

 

Thanks,

Andras G. Bekes   
Morgan Stanley | ISG Technology   
Lechner Odon fasor 8 | Floor 06   
Budapest, 1095   
Phone: +36 1 881-3975   
[hidden email]   

 





NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: {error,closed} vs. {error,econnreset}

Zandra Hird
Hi,

yes this sounds confusing. We looked into it a little bit but couldn't find the reason for it right away. Have you tried tracing the resv0 in prim_inet? It might give you some more information about why this happens. We can look into it some more too when we get the time, and if you don't find out more before that :)

/ Zandra


On 2015-02-10 14:39, Bekes, Andras G wrote:

Hi All,

 

Looks like I have a problem with the erroneous result of gen_tcp:recv/2, as it returns {error,closed} instead of my expectation of {error,econnreset}.

 

I can reproduce my problem by:

 

1, starting a server application and forcing it into a busy state when it is listening but not accepting connections

 

2,starting 1000 clients (in Erlang) that connect to the server, filling the TCP listen backlog of the server.

When the backlog is filled and after some time (~2 minutes), the Erlang connections return {error,closed}.

This is what I do in Erlang:

fun()->{ok,Socket}= TCPCONNECT(HOST,PORT),
       ok=gen_tcp:send(Socket,<<"SOMEDATA…”>>),
       io:format("~p:~p\n",[self(),gen_tcp:recv(Socket,100)])

end.

spawn this 1000 times.

 

3, a native client of the service however tells me that the connection was reset by peer.

According to strace, it executes a recvfrom which returns ECONNRESET:

recvfrom(...) = -1 ECONNRESET (Connection reset by peer)

I also straced the Erlang beam process and it indeed gets an ECONNRESET result for the recvfrom call, but the Erlang return value is {error,closed}.

 

Unfortunately it looks like I really need to separate these two results.

 

I also tried gen_tcp in active mode, but no difference, the result is {error,closed} instead of {error,econnreset}.

 

Can someone explain why the econnreset error is masked? Is there any way I can separate the two kinds of events?

 

Thanks,

Andras G. Bekes   
Morgan Stanley | ISG Technology   
Lechner Odon fasor 8 | Floor 06   
Budapest, 1095   
Phone: +36 1 881-3975   
[hidden email]   

 





NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: {error,closed} vs. {error,econnreset}

Bekes, Andras G

Hi Zandra,

 

I tried tracing on all prim_inet module function calls, plus gen_tcp plus all messages, but did not find the econnreset or resv0 strings anywhere.

 

I used the following code:

dbg:tpl(prim_inet,[{'_',[],[{exception_trace}]}]).

dbg:tpl(gen_tcp,[{'_',[],[{exception_trace}]}]).

f(F),F=fun()->dbg:p(self(),[call,m,sos,sol]), {ok,Socket}=TCP_CONNECT("localhost",12345),

ok=gen_tcp:send(Socket,<<"a:b",0>>),io:format("~p:~p\n",[self(),gen_tcp:recv(Socket,100)]) end.

 

Please see the trace in the attached file. Do you have suggestions on what else shall I trace?

 

Thanks,

   Andras

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Zandra Hird
Sent: Wednesday, February 18, 2015 3:24 PM
To: [hidden email]
Subject: Re: [erlang-questions] {error,closed} vs. {error,econnreset}

 

Hi,

yes this sounds confusing. We looked into it a little bit but couldn't find the reason for it right away. Have you tried tracing the resv0 in prim_inet? It might give you some more information about why this happens. We can look into it some more too when we get the time, and if you don't find out more before that :)

/ Zandra

On 2015-02-10 14:39, Bekes, Andras G wrote:

Hi All,

 

Looks like I have a problem with the erroneous result of gen_tcp:recv/2, as it returns {error,closed} instead of my expectation of {error,econnreset}.

 

I can reproduce my problem by:

 

1, starting a server application and forcing it into a busy state when it is listening but not accepting connections

 

2,starting 1000 clients (in Erlang) that connect to the server, filling the TCP listen backlog of the server.

When the backlog is filled and after some time (~2 minutes), the Erlang connections return {error,closed}.

This is what I do in Erlang:

fun()->{ok,Socket}= TCPCONNECT(HOST,PORT),
       ok=gen_tcp:send(Socket,<<"SOMEDATA…”>>),
       io:format("~p:~p\n",[self(),gen_tcp:recv(Socket,100)])

end.

spawn this 1000 times.

 

3, a native client of the service however tells me that the connection was reset by peer.

According to strace, it executes a recvfrom which returns ECONNRESET:

recvfrom(...) = -1 ECONNRESET (Connection reset by peer)

I also straced the Erlang beam process and it indeed gets an ECONNRESET result for the recvfrom call, but the Erlang return value is {error,closed}.

 

Unfortunately it looks like I really need to separate these two results.

 

I also tried gen_tcp in active mode, but no difference, the result is {error,closed} instead of {error,econnreset}.

 

Can someone explain why the econnreset error is masked? Is there any way I can separate the two kinds of events?

 

Thanks,

Andras G. Bekes   
Morgan Stanley | ISG Technology   
Lechner Odon fasor 8 | Floor 06   
Budapest, 1095   
Phone: +36 1 881-3975   
[hidden email]   

 

 



NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

 





NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

econnreset_trace.txt (148K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: {error,closed} vs. {error,econnreset}

Bekes, Andras G
In reply to this post by Zandra Hird

I forgot to mention that I was using Erlang R16B03 (erts-5.10.4) and the OS process indeed received the ECONNRESET result, according to strace.

 

From: Bekes, Andras G (ICT)
Sent: Friday, May 01, 2015 1:53 PM
To: 'Zandra Hird'; [hidden email]
Subject: RE: [erlang-questions] {error,closed} vs. {error,econnreset}

 

Hi Zandra,

 

I tried tracing on all prim_inet module function calls, plus gen_tcp plus all messages, but did not find the econnreset or resv0 strings anywhere.

 

I used the following code:

dbg:tpl(prim_inet,[{'_',[],[{exception_trace}]}]).

dbg:tpl(gen_tcp,[{'_',[],[{exception_trace}]}]).

f(F),F=fun()->dbg:p(self(),[call,m,sos,sol]), {ok,Socket}=TCP_CONNECT("localhost",12345),

ok=gen_tcp:send(Socket,<<"a:b",0>>),io:format("~p:~p\n",[self(),gen_tcp:recv(Socket,100)]) end.

 

Please see the trace in the attached file. Do you have suggestions on what else shall I trace?

 

Thanks,

   Andras

 

From: [hidden email] [[hidden email]] On Behalf Of Zandra Hird
Sent: Wednesday, February 18, 2015 3:24 PM
To: [hidden email]
Subject: Re: [erlang-questions] {error,closed} vs. {error,econnreset}

 

Hi,

yes this sounds confusing. We looked into it a little bit but couldn't find the reason for it right away. Have you tried tracing the resv0 in prim_inet? It might give you some more information about why this happens. We can look into it some more too when we get the time, and if you don't find out more before that :)

/ Zandra

On 2015-02-10 14:39, Bekes, Andras G wrote:

Hi All,

 

Looks like I have a problem with the erroneous result of gen_tcp:recv/2, as it returns {error,closed} instead of my expectation of {error,econnreset}.

 

I can reproduce my problem by:

 

1, starting a server application and forcing it into a busy state when it is listening but not accepting connections

 

2,starting 1000 clients (in Erlang) that connect to the server, filling the TCP listen backlog of the server.

When the backlog is filled and after some time (~2 minutes), the Erlang connections return {error,closed}.

This is what I do in Erlang:

fun()->{ok,Socket}= TCPCONNECT(HOST,PORT),
       ok=gen_tcp:send(Socket,<<"SOMEDATA…”>>),
       io:format("~p:~p\n",[self(),gen_tcp:recv(Socket,100)])

end.

spawn this 1000 times.

 

3, a native client of the service however tells me that the connection was reset by peer.

According to strace, it executes a recvfrom which returns ECONNRESET:

recvfrom(...) = -1 ECONNRESET (Connection reset by peer)

I also straced the Erlang beam process and it indeed gets an ECONNRESET result for the recvfrom call, but the Erlang return value is {error,closed}.

 

Unfortunately it looks like I really need to separate these two results.

 

I also tried gen_tcp in active mode, but no difference, the result is {error,closed} instead of {error,econnreset}.

 

Can someone explain why the econnreset error is masked? Is there any way I can separate the two kinds of events?

 

Thanks,

Andras G. Bekes   
Morgan Stanley | ISG Technology   
Lechner Odon fasor 8 | Floor 06   
Budapest, 1095   
Phone: +36 1 881-3975   
[hidden email]   

 

 



NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

 





NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: {error,closed} vs. {error,econnreset}

Rory Byrne-3
In reply to this post by Bekes, Andras G
Hi Andras,

On Fri, May 01, 2015 at 11:53:13AM +0000, Bekes, Andras G wrote:
>
> Please see the trace in the attached file. Do you have suggestions on what else shall I trace?
>
>>> On 2015-02-10 14:39, Bekes, Andras G wrote:
>>>
>>> Looks like I have a problem with the erroneous result of gen_tcp:recv/2, as it returns {error,closed} instead of my expectation of {error,econnreset}.
>>>
...
>>>
>>> Unfortunately it looks like I really need to separate these two results.
>>>
>>> I also tried gen_tcp in active mode, but no difference, the result is {error,closed} instead of {error,econnreset}.
>>>
>>> Can someone explain why the econnreset error is masked? Is there any way I can separate the two kinds of events?
>>>

It seems it was a design decision to mask both ECONNRESET recv errors and
all send errors. You can see where this happens for recv errors at around
line 9988 in erts/emulator/drivers/common/inet_drv.c. Open that file and
search for ECONNRESET and you will find the following section of code.

    n = sock_recv(desc->inet.s, desc->i_ptr, nread, 0);

    if (IS_SOCKET_ERROR(n)) {
        int err = sock_errno();
        if (err == ECONNRESET) {
            DEBUGF((" => detected close (connreset)\r\n"));
            return tcp_recv_closed(desc);
        }
        if (err == ERRNO_BLOCK) {
            DEBUGF((" => would block\r\n"));
            return 0;
        }
        else {
            DEBUGF((" => error: %d\r\n", err));
            return tcp_recv_error(desc, err);
        }
    }
    else if (n == 0) {
        DEBUGF(("  => detected close\r\n"));
        return tcp_recv_closed(desc);
    }

As you can see from this, ECONNRESET is being treated as though it's an EOF
rather than an error: both end up calling tcp_recv_closed(). As a quick fix to
your problem, change the code to read:

        if (err == ECONNRESET) {
            DEBUGF((" => detected close (connreset)\r\n"));
            return tcp_recv_error(desc, err);
        }

However, this is an indiscriminate change which will effect all socket code,
including the networking code for distributed Erlang, and any third party
Erlang code you are using.

Also, be warned that this fix won't alert you to every incoming RST. For
example, if you have a large payload buffered in the socket driver in Erlang
(i.e. a payload that is too large for the kernel socket send buffer) and you
receive an RST, then your next call to gen_tcp:recv/2 will return
{error, closed} rather than {error, econnreset}. See below for example code
which shows this. The reason this happens is as follows:

 1. When there is outbound data buffered in the socket driver queue,
    gen_tcp:send/2 becomes an asynchronous call. Which is to say, the new data
    is added to the driver queue and the call returns 'ok'. Then the actual
    send syscall takes place at a later time, according to epoll, select or
    whatever.

 2. Then, when you are in passive mode and an error is detected on the send
    syscall there is no caller to return it to, so it is marked as a
    "delayed close error". This has two consequences:

      (a) it is masked to a generic {error, closed}; and
      (b) it is returned on the next call to gen_tcp:recv/2 or gen_tcp:send/2.

So, the send error is ultimately returned on a gen_tcp:recv/2 call, and all
send errors are masked as {error, closed}.

In active mode the problems are similar.

I've got a patch for the recv errors and I'm working on a patch for the send
errors. Both patches will allow the user to set a socket option that shows all
econnreset errors (changing some epipe errors to econnreset in the case of send
errors). With any luck, I'll release these patches over the next week or
two as part of a larger set of patches. No guarantee they'll be accepted
though.

Rory

%%--- Server Code ---
-module(server).
-export([run/0]).

-define(PORT, 7777).
-define(RECBUF, (4 * 1024)).

run() ->
    SockOpts = [{recbuf, ?RECBUF}, {reuseaddr, true}, {active, false}],
    {ok, LSock} = gen_tcp:listen(?PORT, SockOpts),
    {ok, Sock} = gen_tcp:accept(LSock),
    ok = gen_tcp:close(LSock),
    ok = inet:setopts(Sock, [{linger, {true, 0}}]),
    ok = gen_tcp:send(Sock, "Any payload"),
    timer:sleep(1000),
    ok = gen_tcp:close(Sock).

%%--- Client Code ---
-module(client).
-export([run/0]).

-define(PORT, 7777).
-define(SNDBUF, (4 * 1024)).
-define(PAYLOAD_SIZE, (30 * 1024)).

run() ->
    SockOpts = [{sndbuf, ?SNDBUF}, {active, false}],
    {ok, Sock} = gen_tcp:connect("localhost", ?PORT, SockOpts),
    Payload = lists:duplicate(?PAYLOAD_SIZE, $.),
    ok = gen_tcp:send(Sock, Payload),
    ok = timer:sleep(2000),
    Res = gen_tcp:recv(Sock, 0),
    io:format("Result: ~p~n", [Res]),
    gen_tcp:close(Sock).
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: {error,closed} vs. {error,econnreset}

Bekes, Andras G
Thank you very much for your efforts Rory.

The ability "to set a socket option that shows all econnreset errors" sounds like the right solution. I am wondering why hiding this detail is the default, but I believe there were good enough reasons to design it that way.

I accept that your solution will not notice the connection reset event in some corner cases. I think this will not apply in my case: I am sending a small amount of data (<1KB) and wait for the reply.

I am looking forward to see your patch in the next release of Erlang/OTP!

Thanks,
   Andras

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Rory Byrne
Sent: Saturday, May 02, 2015 12:32 PM
To: [hidden email]
Subject: Re: [erlang-questions] {error,closed} vs. {error,econnreset}

Hi Andras,

On Fri, May 01, 2015 at 11:53:13AM +0000, Bekes, Andras G wrote:
>
> Please see the trace in the attached file. Do you have suggestions on what else shall I trace?
>
>>> On 2015-02-10 14:39, Bekes, Andras G wrote:
>>>
>>> Looks like I have a problem with the erroneous result of gen_tcp:recv/2, as it returns {error,closed} instead of my expectation of {error,econnreset}.
>>>
...
>>>
>>> Unfortunately it looks like I really need to separate these two results.
>>>
>>> I also tried gen_tcp in active mode, but no difference, the result is {error,closed} instead of {error,econnreset}.
>>>
>>> Can someone explain why the econnreset error is masked? Is there any way I can separate the two kinds of events?
>>>

It seems it was a design decision to mask both ECONNRESET recv errors and
all send errors. You can see where this happens for recv errors at around
line 9988 in erts/emulator/drivers/common/inet_drv.c. Open that file and
search for ECONNRESET and you will find the following section of code.

    n = sock_recv(desc->inet.s, desc->i_ptr, nread, 0);

    if (IS_SOCKET_ERROR(n)) {
        int err = sock_errno();
        if (err == ECONNRESET) {
            DEBUGF((" => detected close (connreset)\r\n"));
            return tcp_recv_closed(desc);
        }
        if (err == ERRNO_BLOCK) {
            DEBUGF((" => would block\r\n"));
            return 0;
        }
        else {
            DEBUGF((" => error: %d\r\n", err));
            return tcp_recv_error(desc, err);
        }
    }
    else if (n == 0) {
        DEBUGF(("  => detected close\r\n"));
        return tcp_recv_closed(desc);
    }

As you can see from this, ECONNRESET is being treated as though it's an EOF
rather than an error: both end up calling tcp_recv_closed(). As a quick fix to
your problem, change the code to read:

        if (err == ECONNRESET) {
            DEBUGF((" => detected close (connreset)\r\n"));
            return tcp_recv_error(desc, err);
        }

However, this is an indiscriminate change which will effect all socket code,
including the networking code for distributed Erlang, and any third party
Erlang code you are using.

Also, be warned that this fix won't alert you to every incoming RST. For
example, if you have a large payload buffered in the socket driver in Erlang
(i.e. a payload that is too large for the kernel socket send buffer) and you
receive an RST, then your next call to gen_tcp:recv/2 will return
{error, closed} rather than {error, econnreset}. See below for example code
which shows this. The reason this happens is as follows:

 1. When there is outbound data buffered in the socket driver queue,
    gen_tcp:send/2 becomes an asynchronous call. Which is to say, the new data
    is added to the driver queue and the call returns 'ok'. Then the actual
    send syscall takes place at a later time, according to epoll, select or
    whatever.

 2. Then, when you are in passive mode and an error is detected on the send
    syscall there is no caller to return it to, so it is marked as a
    "delayed close error". This has two consequences:

      (a) it is masked to a generic {error, closed}; and
      (b) it is returned on the next call to gen_tcp:recv/2 or gen_tcp:send/2.

So, the send error is ultimately returned on a gen_tcp:recv/2 call, and all
send errors are masked as {error, closed}.

In active mode the problems are similar.

I've got a patch for the recv errors and I'm working on a patch for the send
errors. Both patches will allow the user to set a socket option that shows all
econnreset errors (changing some epipe errors to econnreset in the case of send
errors). With any luck, I'll release these patches over the next week or
two as part of a larger set of patches. No guarantee they'll be accepted
though.

Rory

%%--- Server Code ---
-module(server).
-export([run/0]).

-define(PORT, 7777).
-define(RECBUF, (4 * 1024)).

run() ->
    SockOpts = [{recbuf, ?RECBUF}, {reuseaddr, true}, {active, false}],
    {ok, LSock} = gen_tcp:listen(?PORT, SockOpts),
    {ok, Sock} = gen_tcp:accept(LSock),
    ok = gen_tcp:close(LSock),
    ok = inet:setopts(Sock, [{linger, {true, 0}}]),
    ok = gen_tcp:send(Sock, "Any payload"),
    timer:sleep(1000),
    ok = gen_tcp:close(Sock).

%%--- Client Code ---
-module(client).
-export([run/0]).

-define(PORT, 7777).
-define(SNDBUF, (4 * 1024)).
-define(PAYLOAD_SIZE, (30 * 1024)).

run() ->
    SockOpts = [{sndbuf, ?SNDBUF}, {active, false}],
    {ok, Sock} = gen_tcp:connect("localhost", ?PORT, SockOpts),
    Payload = lists:duplicate(?PAYLOAD_SIZE, $.),
    ok = gen_tcp:send(Sock, Payload),
    ok = timer:sleep(2000),
    Res = gen_tcp:recv(Sock, 0),
    io:format("Result: ~p~n", [Res]),
    gen_tcp:close(Sock).
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


--------------------------------------------------------------------------------

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: <Unverified Sender>Re: {error, closed} vs. {error, econnreset}

Musumeci, Antonio S
It feels like there really should be two code paths. One lower level which mimics traditional socket behavior and then something on top which may or may not hide some of the details on account of the abstraction. Masking the errors here can cause a lot of useful info to be lost.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Bekes, Andras G (ICT)
Sent: Tuesday, May 05, 2015 3:45 AM
To: Rory Byrne; [hidden email]
Subject: <Unverified Sender>Re: [erlang-questions] {error,closed} vs. {error,econnreset}

Note: This e-mail has been flagged because although the sender appears to be internal, the e-mail originated outside the Firm. The sender’s address may have been forged, which can be an indicator of malicious activity. If you have any concerns about the content or sender of this e-mail, please report it immediately at http://irespond.

Thank you very much for your efforts Rory.

The ability "to set a socket option that shows all econnreset errors" sounds like the right solution. I am wondering why hiding this detail is the default, but I believe there were good enough reasons to design it that way.

I accept that your solution will not notice the connection reset event in some corner cases. I think this will not apply in my case: I am sending a small amount of data (<1KB) and wait for the reply.

I am looking forward to see your patch in the next release of Erlang/OTP!

Thanks,
   Andras

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Rory Byrne
Sent: Saturday, May 02, 2015 12:32 PM
To: [hidden email]
Subject: Re: [erlang-questions] {error,closed} vs. {error,econnreset}

Hi Andras,

On Fri, May 01, 2015 at 11:53:13AM +0000, Bekes, Andras G wrote:
>
> Please see the trace in the attached file. Do you have suggestions on what else shall I trace?
>
>>> On 2015-02-10 14:39, Bekes, Andras G wrote:
>>>
>>> Looks like I have a problem with the erroneous result of gen_tcp:recv/2, as it returns {error,closed} instead of my expectation of {error,econnreset}.
>>>
...
>>>
>>> Unfortunately it looks like I really need to separate these two results.
>>>
>>> I also tried gen_tcp in active mode, but no difference, the result is {error,closed} instead of {error,econnreset}.
>>>
>>> Can someone explain why the econnreset error is masked? Is there any way I can separate the two kinds of events?
>>>

It seems it was a design decision to mask both ECONNRESET recv errors and
all send errors. You can see where this happens for recv errors at around
line 9988 in erts/emulator/drivers/common/inet_drv.c. Open that file and
search for ECONNRESET and you will find the following section of code.

    n = sock_recv(desc->inet.s, desc->i_ptr, nread, 0);

    if (IS_SOCKET_ERROR(n)) {
        int err = sock_errno();
        if (err == ECONNRESET) {
            DEBUGF((" => detected close (connreset)\r\n"));
            return tcp_recv_closed(desc);
        }
        if (err == ERRNO_BLOCK) {
            DEBUGF((" => would block\r\n"));
            return 0;
        }
        else {
            DEBUGF((" => error: %d\r\n", err));
            return tcp_recv_error(desc, err);
        }
    }
    else if (n == 0) {
        DEBUGF(("  => detected close\r\n"));
        return tcp_recv_closed(desc);
    }

As you can see from this, ECONNRESET is being treated as though it's an EOF
rather than an error: both end up calling tcp_recv_closed(). As a quick fix to
your problem, change the code to read:

        if (err == ECONNRESET) {
            DEBUGF((" => detected close (connreset)\r\n"));
            return tcp_recv_error(desc, err);
        }

However, this is an indiscriminate change which will effect all socket code,
including the networking code for distributed Erlang, and any third party
Erlang code you are using.

Also, be warned that this fix won't alert you to every incoming RST. For
example, if you have a large payload buffered in the socket driver in Erlang
(i.e. a payload that is too large for the kernel socket send buffer) and you
receive an RST, then your next call to gen_tcp:recv/2 will return
{error, closed} rather than {error, econnreset}. See below for example code
which shows this. The reason this happens is as follows:

 1. When there is outbound data buffered in the socket driver queue,
    gen_tcp:send/2 becomes an asynchronous call. Which is to say, the new data
    is added to the driver queue and the call returns 'ok'. Then the actual
    send syscall takes place at a later time, according to epoll, select or
    whatever.

 2. Then, when you are in passive mode and an error is detected on the send
    syscall there is no caller to return it to, so it is marked as a
    "delayed close error". This has two consequences:

      (a) it is masked to a generic {error, closed}; and
      (b) it is returned on the next call to gen_tcp:recv/2 or gen_tcp:send/2.

So, the send error is ultimately returned on a gen_tcp:recv/2 call, and all
send errors are masked as {error, closed}.

In active mode the problems are similar.

I've got a patch for the recv errors and I'm working on a patch for the send
errors. Both patches will allow the user to set a socket option that shows all
econnreset errors (changing some epipe errors to econnreset in the case of send
errors). With any luck, I'll release these patches over the next week or
two as part of a larger set of patches. No guarantee they'll be accepted
though.

Rory

%%--- Server Code ---
-module(server).
-export([run/0]).

-define(PORT, 7777).
-define(RECBUF, (4 * 1024)).

run() ->
    SockOpts = [{recbuf, ?RECBUF}, {reuseaddr, true}, {active, false}],
    {ok, LSock} = gen_tcp:listen(?PORT, SockOpts),
    {ok, Sock} = gen_tcp:accept(LSock),
    ok = gen_tcp:close(LSock),
    ok = inet:setopts(Sock, [{linger, {true, 0}}]),
    ok = gen_tcp:send(Sock, "Any payload"),
    timer:sleep(1000),
    ok = gen_tcp:close(Sock).

%%--- Client Code ---
-module(client).
-export([run/0]).

-define(PORT, 7777).
-define(SNDBUF, (4 * 1024)).
-define(PAYLOAD_SIZE, (30 * 1024)).

run() ->
    SockOpts = [{sndbuf, ?SNDBUF}, {active, false}],
    {ok, Sock} = gen_tcp:connect("localhost", ?PORT, SockOpts),
    Payload = lists:duplicate(?PAYLOAD_SIZE, $.),
    ok = gen_tcp:send(Sock, Payload),
    ok = timer:sleep(2000),
    Res = gen_tcp:recv(Sock, 0),
    io:format("Result: ~p~n", [Res]),
    gen_tcp:close(Sock).
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


--------------------------------------------------------------------------------

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


--------------------------------------------------------------------------------

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: {error,closed} vs. {error,econnreset}

Rory Byrne-3
In reply to this post by Bekes, Andras G
Hi Andras,

On Tue, May 05, 2015 at 07:44:47AM +0000, Bekes, Andras G wrote:
> Thank you very much for your efforts Rory.
>
> The ability "to set a socket option that shows all econnreset errors" sounds like the right solution. I am wondering why hiding this detail is the default, but I believe there were good enough reasons to design it that way.
>
> I accept that your solution will not notice the connection reset event in some corner cases. I think this will not apply in my case: I am sending a small amount of data (<1KB) and wait for the reply.
>
> I am looking forward to see your patch in the next release of Erlang/OTP!

The fix for this is in the 18.0 release. It should take care of the
corner cases too.

Use the socket option '{show_econnreset, true}' and you'll receive
{error, econnreset} in passive mode or {tcp_error, Socket, econnreset}
in active mode. See the docs [1] for more information.

Regards,

Rory

[1] http://www.erlang.org/doc/man/inet.html#setopts-2
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: {error,closed} vs. {error,econnreset}

Bekes, Andras G
Hi Rory,

I just tested this new feature in Erlang/OTP R18 and it works fine.

Thank you very much all for implementing it!

Regards,
   Andras

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Rory Byrne
Sent: Wednesday, June 24, 2015 2:48 PM
To: [hidden email]
Subject: Re: [erlang-questions] {error,closed} vs. {error,econnreset}

Hi Andras,

On Tue, May 05, 2015 at 07:44:47AM +0000, Bekes, Andras G wrote:
> Thank you very much for your efforts Rory.
>
> The ability "to set a socket option that shows all econnreset errors" sounds like the right solution. I am wondering why hiding this detail is the default, but I believe there were good enough reasons to design it that way.
>
> I accept that your solution will not notice the connection reset event in some corner cases. I think this will not apply in my case: I am sending a small amount of data (<1KB) and wait for the reply.
>
> I am looking forward to see your patch in the next release of Erlang/OTP!

The fix for this is in the 18.0 release. It should take care of the
corner cases too.

Use the socket option '{show_econnreset, true}' and you'll receive
{error, econnreset} in passive mode or {tcp_error, Socket, econnreset}
in active mode. See the docs [1] for more information.

Regards,

Rory

[1] http://www.erlang.org/doc/man/inet.html#setopts-2
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


--------------------------------------------------------------------------------

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: {error,closed} vs. {error,econnreset}

Bekes, Andras G
In reply to this post by Rory Byrne-3
Hi Rory, All,

Let me revive this old thread for a minute. The new gen_tcp option {show_econnreset, true} works well.

However, we still notice some cases when we observe {error,closed} on the Erlang side, but other signs suggest that the TCP connection wasn't intentionally closed by the peer, but was closed because of some error.
We suspect some packets being dropped by the OS due to various buffer overruns.
I am not very familiar with packet-level details of TCP. Can someone confirm if there are other erroneous terminations of a TCP connection (other than econnreset), reported simply as {error,closed} by Erlang?

I tried checking the code erts/emulator/drivers/common/inet_drv.c and it seems to me not, but can someone actually understanding that code also confirm?

Thank you very much,
   Andras

-----Original Message-----
From: Bekes, Andras G (ICT)
Sent: Tuesday, August 11, 2015 2:43 PM
To: 'Rory Byrne'; [hidden email]
Subject: RE: [erlang-questions] {error,closed} vs. {error,econnreset}

Hi Rory,

I just tested this new feature in Erlang/OTP R18 and it works fine.

Thank you very much all for implementing it!

Regards,
   Andras

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Rory Byrne
Sent: Wednesday, June 24, 2015 2:48 PM
To: [hidden email]
Subject: Re: [erlang-questions] {error,closed} vs. {error,econnreset}

Hi Andras,

On Tue, May 05, 2015 at 07:44:47AM +0000, Bekes, Andras G wrote:
> Thank you very much for your efforts Rory.
>
> The ability "to set a socket option that shows all econnreset errors" sounds like the right solution. I am wondering why hiding this detail is the default, but I believe there were good enough reasons to design it that way.
>
> I accept that your solution will not notice the connection reset event in some corner cases. I think this will not apply in my case: I am sending a small amount of data (<1KB) and wait for the reply.
>
> I am looking forward to see your patch in the next release of Erlang/OTP!

The fix for this is in the 18.0 release. It should take care of the corner cases too.

Use the socket option '{show_econnreset, true}' and you'll receive {error, econnreset} in passive mode or {tcp_error, Socket, econnreset} in active mode. See the docs [1] for more information.

Regards,

Rory

[1] http://www.erlang.org/doc/man/inet.html#setopts-2
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


--------------------------------------------------------------------------------

NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers  If you cannot access these links, please notify us by reply message and we will send the contents to you. By communicating with Morgan Stanley you consent to the foregoing and to the voice recording of conversations with personnel of Morgan Stanley.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: {error,closed} vs. {error,econnreset}

Rory Byrne-3
Hi Andras,

I'm answering this because you addressed my directly.

At the current moment I am not in a position to speak with any authority
regarding the code in inet_drv.c. It's been over a year since I last looked
at this code and it would take me a good few hours to get to a point where
I was comfortable to comment on it again.

If you sniff the wire and can confirm with absolute certainty that the
peer is sending an RST that is getting reported as {error, closed} in
Erlang, then I'll make the time to look into it. Needless to say, you
would need to provide significantly more information about the scenario
in which this occurred: OS where Erlang is running, OS on the peer, all
the socket options you set, the exact call(s) where you received
{error, closed}, etc. Ideally you'd be able to provide a tcpdump trace of
when the connection went down; it wouldn't need to be a deep trace, just
enough to see the details of the TCP traffic (flags, sequence numbers,
etc).

Regards,

Rory


On Thu, Sep 22, 2016 at 12:51:38PM +0000, Bekes, Andras G wrote:

> Hi Rory, All,
>
> Let me revive this old thread for a minute. The new gen_tcp option {show_econnreset, true} works well.
>
> However, we still notice some cases when we observe {error,closed} on the Erlang side, but other signs suggest that the TCP connection wasn't intentionally closed by the peer, but was closed because of some error.
> We suspect some packets being dropped by the OS due to various buffer overruns.
> I am not very familiar with packet-level details of TCP. Can someone confirm if there are other erroneous terminations of a TCP connection (other than econnreset), reported simply as {error,closed} by Erlang?
>
> I tried checking the code erts/emulator/drivers/common/inet_drv.c and it seems to me not, but can someone actually understanding that code also confirm?
>
> Thank you very much,
>    Andras
>
> -----Original Message-----
> From: Bekes, Andras G (ICT)
> Sent: Tuesday, August 11, 2015 2:43 PM
> To: 'Rory Byrne'; [hidden email]
> Subject: RE: [erlang-questions] {error,closed} vs. {error,econnreset}
>
> Hi Rory,
>
> I just tested this new feature in Erlang/OTP R18 and it works fine.
>
> Thank you very much all for implementing it!
>
> Regards,
>    Andras
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Rory Byrne
> Sent: Wednesday, June 24, 2015 2:48 PM
> To: [hidden email]
> Subject: Re: [erlang-questions] {error,closed} vs. {error,econnreset}
>
> Hi Andras,
>
> On Tue, May 05, 2015 at 07:44:47AM +0000, Bekes, Andras G wrote:
> > Thank you very much for your efforts Rory.
> >
> > The ability "to set a socket option that shows all econnreset errors" sounds like the right solution. I am wondering why hiding this detail is the default, but I believe there were good enough reasons to design it that way.
> >
> > I accept that your solution will not notice the connection reset event in some corner cases. I think this will not apply in my case: I am sending a small amount of data (<1KB) and wait for the reply.
> >
> > I am looking forward to see your patch in the next release of Erlang/OTP!
>
> The fix for this is in the 18.0 release. It should take care of the corner cases too.
>
> Use the socket option '{show_econnreset, true}' and you'll receive {error, econnreset} in passive mode or {tcp_error, Socket, econnreset} in active mode. See the docs [1] for more information.
>
> Regards,
>
> Rory
>
> [1] http://www.erlang.org/doc/man/inet.html#setopts-2
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
> --------------------------------------------------------------------------------
>
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers  If you cannot access these links, please notify us by reply message and we will send the contents to you. By communicating with Morgan Stanley you consent to the foregoing and to the voice recording of conversations with personnel of Morgan Stanley.

--
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: {error,closed} vs. {error,econnreset}

Bekes, Andras G
Hi All,

Reviving this old thread again, because I am getting more and more convinced that we need further changes.
We're still observing connection close events when an error should be reported to gen_tcp level.
It can be a reset error somehow still not reported as 'econnreset', but I suspect it must be some other error.

I checked the code in inet_drv.c. The function
static int tcp_recv(tcp_descriptor* desc, int request_len)
seems to work properly -- a reset is either reported as closed or econnreset, depending on show_econnreset, all other errors are reported as errors.

However, the function
static int tcp_send_or_shutdown_error(tcp_descriptor* desc, int err)
hides errors. Connection reset errors are properly handled (either reported as closed or econnreset, depending on show_econnreset), but all other errors are just reported as closed.
Active and passive modes have independent code paths, but I think both do the same: all errors are reported as normal close -- except for econnreset.

Apparently I need to detect all errors.
Is it possible to implement a show_errors (or show_all_errors) flag, too?

Actually, this new flag could replace the current show_econnreset flag.
Having two separate flags for econnreset & others requires more complex code, but having a single show_errors flag would simplify the current that provides special treatment for econnreset.
I am not sure if it makes much sense to expose connection reset errors but still mask all other errors as normal close events.

From a farther point of view, it seems there are network-programming tasks (there is at least one!), for which Erlang seems not suitable. This sounds rather sad.
Luckily the fix doesn't seem difficult.

What do you think?

Regards,
   Andras

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Rory Byrne
Sent: Monday, September 26, 2016 4:15 PM
To: [hidden email]
Subject: Re: [erlang-questions] {error,closed} vs. {error,econnreset}

Hi Andras,

I'm answering this because you addressed my directly.

At the current moment I am not in a position to speak with any authority
regarding the code in inet_drv.c. It's been over a year since I last looked
at this code and it would take me a good few hours to get to a point where
I was comfortable to comment on it again.

If you sniff the wire and can confirm with absolute certainty that the
peer is sending an RST that is getting reported as {error, closed} in
Erlang, then I'll make the time to look into it. Needless to say, you
would need to provide significantly more information about the scenario
in which this occurred: OS where Erlang is running, OS on the peer, all
the socket options you set, the exact call(s) where you received
{error, closed}, etc. Ideally you'd be able to provide a tcpdump trace of
when the connection went down; it wouldn't need to be a deep trace, just
enough to see the details of the TCP traffic (flags, sequence numbers,
etc).

Regards,

Rory


On Thu, Sep 22, 2016 at 12:51:38PM +0000, Bekes, Andras G wrote:

> Hi Rory, All,
>
> Let me revive this old thread for a minute. The new gen_tcp option {show_econnreset, true} works well.
>
> However, we still notice some cases when we observe {error,closed} on the Erlang side, but other signs suggest that the TCP connection wasn't intentionally closed by the peer, but was closed because of some error.
> We suspect some packets being dropped by the OS due to various buffer overruns.
> I am not very familiar with packet-level details of TCP. Can someone confirm if there are other erroneous terminations of a TCP connection (other than econnreset), reported simply as {error,closed} by Erlang?
>
> I tried checking the code erts/emulator/drivers/common/inet_drv.c and it seems to me not, but can someone actually understanding that code also confirm?
>
> Thank you very much,
>    Andras
>
> -----Original Message-----
> From: Bekes, Andras G (ICT)
> Sent: Tuesday, August 11, 2015 2:43 PM
> To: 'Rory Byrne'; [hidden email]
> Subject: RE: [erlang-questions] {error,closed} vs. {error,econnreset}
>
> Hi Rory,
>
> I just tested this new feature in Erlang/OTP R18 and it works fine.
>
> Thank you very much all for implementing it!
>
> Regards,
>    Andras
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Rory Byrne
> Sent: Wednesday, June 24, 2015 2:48 PM
> To: [hidden email]
> Subject: Re: [erlang-questions] {error,closed} vs. {error,econnreset}
>
> Hi Andras,
>
> On Tue, May 05, 2015 at 07:44:47AM +0000, Bekes, Andras G wrote:
> > Thank you very much for your efforts Rory.
> >
> > The ability "to set a socket option that shows all econnreset errors" sounds like the right solution. I am wondering why hiding this detail is the default, but I believe there were good enough reasons to design it that way.
> >
> > I accept that your solution will not notice the connection reset event in some corner cases. I think this will not apply in my case: I am sending a small amount of data (<1KB) and wait for the reply.
> >
> > I am looking forward to see your patch in the next release of Erlang/OTP!
>
> The fix for this is in the 18.0 release. It should take care of the corner cases too.
>
> Use the socket option '{show_econnreset, true}' and you'll receive {error, econnreset} in passive mode or {tcp_error, Socket, econnreset} in active mode. See the docs [1] for more information.
>
> Regards,
>
> Rory
>
> [1] http://www.erlang.org/doc/man/inet.html#setopts-2
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
> --------------------------------------------------------------------------------
>
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers  If you cannot access these links, please notify us by reply message and we will send the contents to you. By communicating with Morgan Stanley you consent to the foregoing and to the voice recording of conversations with personnel of Morgan Stanley.

--
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

--------------------------------------------------------------------------------
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers  If you cannot access these links, please notify us by reply message and we will send the contents to you. By communicating with Morgan Stanley you consent to the foregoing and to the voice recording of conversations with personnel of Morgan Stanley.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: {error,closed} vs. {error,econnreset}

Lukas Larsson-8
Hello,

On Mon, 6 Aug 2018, 10:49 Bekes, Andras G, <[hidden email]> wrote:
Hi All,

Reviving this old thread again, because I am getting more and more convinced that we need further changes.
We're still observing connection close events when an error should be reported to gen_tcp level.
It can be a reset error somehow still not reported as 'econnreset', but I suspect it must be some other error.

I suppose that you have not managed to catch the error in a tcp dump as Rory asked for?
 

I checked the code in inet_drv.c. The function
static int tcp_recv(tcp_descriptor* desc, int request_len)
seems to work properly -- a reset is either reported as closed or econnreset, depending on show_econnreset, all other errors are reported as errors.

However, the function
static int tcp_send_or_shutdown_error(tcp_descriptor* desc, int err)
hides errors. Connection reset errors are properly handled (either reported as closed or econnreset, depending on show_econnreset), but all other errors are just reported as closed.
Active and passive modes have independent code paths, but I think both do the same: all errors are reported as normal close -- except for econnreset.

There is a merge of error codes happening in tcp_send_error, so some other errors get mapped into econnreset before tcp_send_or_shutdown_error is called.
 

Apparently I need to detect all errors.
Is it possible to implement a show_errors (or show_all_errors) flag, too?

Actually, this new flag could replace the current show_econnreset flag.
Having two separate flags for econnreset & others requires more complex code, but having a single show_errors flag would simplify the current that provides special treatment for econnreset.
I am not sure if it makes much sense to expose connection reset errors but still mask all other errors as normal close events.

From a farther point of view, it seems there are network-programming tasks (there is at least one!), for which Erlang seems not suitable. This sounds rather sad.
Luckily the fix doesn't seem difficult.

What do you think?

I agree that it should be possible to get the original error from the tcp stack. Given the discussions here https://github.com/erlang/otp/pull/731, maybe it is time to reverse the options so that returning the original error becomes the default and you have to set an option to get the backwards compatible behaviour?

We are currently in the process of a major overhaul of gen_tcp and friends, so maybe this can be changed while doing that, as we are bound to break backwards compatibility in various ways during that rewrite anyways.

Lukas

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: {error,closed} vs. {error,econnreset}

Bekes, Andras G

Hi Lukas, All,

 

I do not have TCP dump, but according to strace, the error is:

<... writev resumed> ) = -1 ENOTCONN (Transport endpoint is not connected)

The error indeed happens on a write, and on the Erlang side this is reported as a normal close, at the next gen_tcp:recv call.

 

Let us not go into discussing what can cause this error on a  socket (unix domain BTW). It is irrelevant here.

The important here is that this socket error, and any other socket error shall not be confused with an orderly shutdown of the socket.

 

I support the below mentioned proposal of making the default behavior report all errors, and the masking behavior explicitly selected with an option.

 

Regards,

   Andras

 

From: Lukas Larsson [mailto:[hidden email]]
Sent: Thursday, August 16, 2018 2:37 PM
To: Bekes, Andras G (IST)
Cc: Erlang Questions
Subject: Re: [erlang-questions] {error,closed} vs. {error,econnreset}

 

Hello,

On Mon, 6 Aug 2018, 10:49 Bekes, Andras G, <[hidden email]> wrote:

Hi All,

Reviving this old thread again, because I am getting more and more convinced that we need further changes.
We're still observing connection close events when an error should be reported to gen_tcp level.
It can be a reset error somehow still not reported as 'econnreset', but I suspect it must be some other error.

 

I suppose that you have not managed to catch the error in a tcp dump as Rory asked for?

 


I checked the code in inet_drv.c. The function
static int tcp_recv(tcp_descriptor* desc, int request_len)
seems to work properly -- a reset is either reported as closed or econnreset, depending on show_econnreset, all other errors are reported as errors.

However, the function
static int tcp_send_or_shutdown_error(tcp_descriptor* desc, int err)
hides errors. Connection reset errors are properly handled (either reported as closed or econnreset, depending on show_econnreset), but all other errors are just reported as closed.
Active and passive modes have independent code paths, but I think both do the same: all errors are reported as normal close -- except for econnreset.

 

There is a merge of error codes happening in tcp_send_error, so some other errors get mapped into econnreset before tcp_send_or_shutdown_error is called.

 


Apparently I need to detect all errors.
Is it possible to implement a show_errors (or show_all_errors) flag, too?

Actually, this new flag could replace the current show_econnreset flag.
Having two separate flags for econnreset & others requires more complex code, but having a single show_errors flag would simplify the current that provides special treatment for econnreset.
I am not sure if it makes much sense to expose connection reset errors but still mask all other errors as normal close events.

From a farther point of view, it seems there are network-programming tasks (there is at least one!), for which Erlang seems not suitable. This sounds rather sad.
Luckily the fix doesn't seem difficult.

What do you think?

 

I agree that it should be possible to get the original error from the tcp stack. Given the discussions here https://github.com/erlang/otp/pull/731, maybe it is time to reverse the options so that returning the original error becomes the default and you have to set an option to get the backwards compatible behaviour?

 

We are currently in the process of a major overhaul of gen_tcp and friends, so maybe this can be changed while doing that, as we are bound to break backwards compatibility in various ways during that rewrite anyways.

 

Lukas



NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent required and/or permitted under applicable law, to monitor electronic communications, including telephone calls with Morgan Stanley personnel. This message is subject to the Morgan Stanley General Disclaimers available at the following link: http://www.morganstanley.com/disclaimers.  If you cannot access the links, please notify us by reply message and we will send the contents to you. By communicating with Morgan Stanley you acknowledge that you have read, understand and consent, (where applicable), to the foregoing and the Morgan Stanley General Disclaimers.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions