gen_udp:send/4

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

gen_udp:send/4

Steven
Hi,

We have a situation where gen_udp:send/4 does not return ok and it just hangs and the process is waiting indefinitely.

e.g.
 {current_function,{prim_inet,sendto,4}},
 {initial_call,{proc_lib,init_p,5}},
 {status,running},
 {message_queue_len,15363062},

The send buffer size on the socket is around 200KB.

3> inet:getopts(S, [sndbuf]).
{ok,[{sndbuf,212992}]}

The UDP socket doesn't receive any incoming packets so it is used for sending only. Running R21-3 with redhat 7.5. Would like to ask the group under which conditions will the port not come back with inet_reply? I assume if the sndbuf is full then it should come back with enobufs or some sort but it should be quick enough to flush the sndbuf to interface.

In prim_inet:sendTo/4, it always expect inet reply from OS but it never comes and never times out.

try erlang:port_command(S, PortCommandData) of
                        true ->
                            receive
                                {inet_reply,S,Reply} ->
                                    ?DBG_FORMAT(
                                       "prim_inet:sendto() -> ~p~n", [Reply]),
                                    Reply
                            end
                    catch

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: gen_udp:send/4

Ben Murphy-2
There used to be a bug with using the inet functions and having a large message queue. Because the receive couldn’t use the selective receive optimisation it would have to scan the whole of the message queue. It looks like you have a large message queue maybe this bug still exists and effects you.

Sent from my iPhone

> On 5 Dec 2019, at 16:39, Steven <[hidden email]> wrote:
>
> 
> Hi,
>
> We have a situation where gen_udp:send/4 does not return ok and it just hangs and the process is waiting indefinitely.
>
> e.g.
>  {current_function,{prim_inet,sendto,4}},
>  {initial_call,{proc_lib,init_p,5}},
>  {status,running},
>  {message_queue_len,15363062},
>
> The send buffer size on the socket is around 200KB.
>
> 3> inet:getopts(S, [sndbuf]).
> {ok,[{sndbuf,212992}]}
>
> The UDP socket doesn't receive any incoming packets so it is used for sending only. Running R21-3 with redhat 7.5. Would like to ask the group under which conditions will the port not come back with inet_reply? I assume if the sndbuf is full then it should come back with enobufs or some sort but it should be quick enough to flush the sndbuf to interface.
>
> In prim_inet:sendTo/4, it always expect inet reply from OS but it never comes and never times out.
>
> try erlang:port_command(S, PortCommandData) of
>                         true ->
>                             receive
>                                 {inet_reply,S,Reply} ->
>                                     ?DBG_FORMAT(
>                                        "prim_inet:sendto() -> ~p~n", [Reply]),
>                                     Reply
>                             end
>                     catch
>
> Thanks
Reply | Threaded
Open this post in threaded view
|

Re: gen_udp:send/4

Valentin Micic-6
In reply to this post by Steven

On 05 Dec 2019, at 18:39, Steven <[hidden email]> wrote:

Hi,

We have a situation where gen_udp:send/4 does not return ok and it just hangs and the process is waiting indefinitely.

e.g.
 {current_function,{prim_inet,sendto,4}},
 {initial_call,{proc_lib,init_p,5}},
 {status,running},
 {message_queue_len,15363062},

It appears that your server (e.g. process that is calling gen_udp:send/4) has way to many messages on its process queue — thus gen_udp:send/4 hanging is probably a symptom and not necessarily the cause.

Could it be that your server itself employs a selective receive for the incoming events/requests processing, e.g.

main_loop()
->
    receive
         {“Some pattern”, _}   -> gen_udp:send(…);
         “Some other pattern”  -> do_something else(…)
         _        -> ignore;
    after SomeTime             -> do_soemthing_different() 
    end,
   main_loop()
.

The construct above will slow message processing and that may account for more than 15,000,000 messages reported to be on your server's message queue.
In my view, selective receive is a bad idea for server-side implementation. Server should rather use something like this:

main_loop()
->
       receive
Msg      -> process_msg( Msg )
      after SomeTime -> do_something_different()
      end,
      main_loop()
.

process_msg( {“Some pattern”, _} )  -> gen_udp:send(…);
process_msg( “Some other pattern” ) -> do_something_else(…);
process_msg( _ )                    -> ignore. 

This will always keep server’s process queue reasonably empty, thus, even when server  uses a function that hides a selective receive, such function will come back reasonably quickly. 
Kind regards

V/


The send buffer size on the socket is around 200KB.

3> inet:getopts(S, [sndbuf]).
{ok,[{sndbuf,212992}]}

The UDP socket doesn't receive any incoming packets so it is used for sending only. Running R21-3 with redhat 7.5. Would like to ask the group under which conditions will the port not come back with inet_reply? I assume if the sndbuf is full then it should come back with enobufs or some sort but it should be quick enough to flush the sndbuf to interface.

In prim_inet:sendTo/4, it always expect inet reply from OS but it never comes and never times out.

try erlang:port_command(S, PortCommandData) of
                        true ->
                            receive
                                {inet_reply,S,Reply} ->
                                    ?DBG_FORMAT(
                                       "prim_inet:sendto() -> ~p~n", [Reply]),
                                    Reply
                            end
                    catch

Thanks

Reply | Threaded
Open this post in threaded view
|

Re: gen_udp:send/4

Steven
Thanks,

Seems like under CPU load and with a large volume of messages coming into the process,  the selective receive will slow it down and make it look like the udp send is taking long or not acknowledged meanwhile the acknowledgement is somewhere in the message queue.

Steven

On Thu, Dec 5, 2019 at 9:32 PM Valentin Micic <[hidden email]> wrote:

On 05 Dec 2019, at 18:39, Steven <[hidden email]> wrote:

Hi,

We have a situation where gen_udp:send/4 does not return ok and it just hangs and the process is waiting indefinitely.

e.g.
 {current_function,{prim_inet,sendto,4}},
 {initial_call,{proc_lib,init_p,5}},
 {status,running},
 {message_queue_len,15363062},

It appears that your server (e.g. process that is calling gen_udp:send/4) has way to many messages on its process queue — thus gen_udp:send/4 hanging is probably a symptom and not necessarily the cause.

Could it be that your server itself employs a selective receive for the incoming events/requests processing, e.g.

main_loop()
->
    receive
         {“Some pattern”, _}   -> gen_udp:send(…);
         “Some other pattern”  -> do_something else(…)
         _        -> ignore;
    after SomeTime             -> do_soemthing_different() 
    end,
   main_loop()
.

The construct above will slow message processing and that may account for more than 15,000,000 messages reported to be on your server's message queue.
In my view, selective receive is a bad idea for server-side implementation. Server should rather use something like this:

main_loop()
->
       receive
Msg      -> process_msg( Msg )
      after SomeTime -> do_something_different()
      end,
      main_loop()
.

process_msg( {“Some pattern”, _} )  -> gen_udp:send(…);
process_msg( “Some other pattern” ) -> do_something_else(…);
process_msg( _ )                    -> ignore. 

This will always keep server’s process queue reasonably empty, thus, even when server  uses a function that hides a selective receive, such function will come back reasonably quickly. 
Kind regards

V/


The send buffer size on the socket is around 200KB.

3> inet:getopts(S, [sndbuf]).
{ok,[{sndbuf,212992}]}

The UDP socket doesn't receive any incoming packets so it is used for sending only. Running R21-3 with redhat 7.5. Would like to ask the group under which conditions will the port not come back with inet_reply? I assume if the sndbuf is full then it should come back with enobufs or some sort but it should be quick enough to flush the sndbuf to interface.

In prim_inet:sendTo/4, it always expect inet reply from OS but it never comes and never times out.

try erlang:port_command(S, PortCommandData) of
                        true ->
                            receive
                                {inet_reply,S,Reply} ->
                                    ?DBG_FORMAT(
                                       "prim_inet:sendto() -> ~p~n", [Reply]),
                                    Reply
                            end
                    catch

Thanks

Reply | Threaded
Open this post in threaded view
|

Re: gen_udp:send/4

Mikael Pettersson-5
In reply to this post by Steven
On Thu, Dec 5, 2019 at 5:39 PM Steven <[hidden email]> wrote:

>
> Hi,
>
> We have a situation where gen_udp:send/4 does not return ok and it just hangs and the process is waiting indefinitely.
>
> e.g.
>  {current_function,{prim_inet,sendto,4}},
>  {initial_call,{proc_lib,init_p,5}},
>  {status,running},
>  {message_queue_len,15363062},
>
> The send buffer size on the socket is around 200KB.
>
> 3> inet:getopts(S, [sndbuf]).
> {ok,[{sndbuf,212992}]}
>
> The UDP socket doesn't receive any incoming packets so it is used for sending only. Running R21-3 with redhat 7.5. Would like to ask the group under which conditions will the port not come back with inet_reply? I assume if the sndbuf is full then it should come back with enobufs or some sort but it should be quick enough to flush the sndbuf to interface.
>
> In prim_inet:sendTo/4, it always expect inet reply from OS but it never comes and never times out.
>
> try erlang:port_command(S, PortCommandData) of
>                         true ->
>                             receive
>                                 {inet_reply,S,Reply} ->
>                                     ?DBG_FORMAT(
>                                        "prim_inet:sendto() -> ~p~n", [Reply]),
>                                     Reply
>                             end
>                     catch
>
> Thanks

I believe a (or the) problem is that the selective-receive in
prim_inet:sendto/4 doesn't use the ref trick, therefore it has to scan
the entire 15+ million entry message queue of the sender process
looking for that inet_reply message.  That is going to be very slow.
(Furthermore there are other performance penalties associated with
having very long message queues, see
erlang:process_flag(message_queue_data, off_heap) for one possible
remedy.)

It would be nice if the return signalling from calling port_command
could be fixed to either enable the ref trick or to not use messages
at all, but until that is the case, you know that making these calls
while having a long message queue is going to be expensive.  We've
found that it's often useful to offload such calls to temporary helper
processes, that by construction have very short message queues.  (The
signalling between the original process and the helper should of
course use the ref trick, this is non-trivial to get right.)
Reply | Threaded
Open this post in threaded view
|

Re: gen_udp:send/4

Steven
Hi,

It actually took a while to build to that kind of message queue size. On the upper level, there is no selective receive and incoming messages are accumulated and then send out via udp. Generally that should be relatively fast however under load, the message queue can slowly creep up leaving that prim inet sendto selective receive becoming problematic. 

Thank you for your suggestions as well, obviously keeping short message queue is the ideal situation

Steven

On Fri, 06 Dec 2019 at 22:00, Mikael Pettersson <[hidden email]> wrote:
On Thu, Dec 5, 2019 at 5:39 PM Steven <[hidden email]> wrote:
>
> Hi,
>
> We have a situation where gen_udp:send/4 does not return ok and it just hangs and the process is waiting indefinitely.
>
> e.g.
>  {current_function,{prim_inet,sendto,4}},
>  {initial_call,{proc_lib,init_p,5}},
>  {status,running},
>  {message_queue_len,15363062},
>
> The send buffer size on the socket is around 200KB.
>
> 3> inet:getopts(S, [sndbuf]).
> {ok,[{sndbuf,212992}]}
>
> The UDP socket doesn't receive any incoming packets so it is used for sending only. Running R21-3 with redhat 7.5. Would like to ask the group under which conditions will the port not come back with inet_reply? I assume if the sndbuf is full then it should come back with enobufs or some sort but it should be quick enough to flush the sndbuf to interface.
>
> In prim_inet:sendTo/4, it always expect inet reply from OS but it never comes and never times out.
>
> try erlang:port_command(S, PortCommandData) of
>                         true ->
>                             receive
>                                 {inet_reply,S,Reply} ->
>                                     ?DBG_FORMAT(
>                                        "prim_inet:sendto() -> ~p~n", [Reply]),
>                                     Reply
>                             end
>                     catch
>
> Thanks

I believe a (or the) problem is that the selective-receive in
prim_inet:sendto/4 doesn't use the ref trick, therefore it has to scan
the entire 15+ million entry message queue of the sender process
looking for that inet_reply message.  That is going to be very slow.
(Furthermore there are other performance penalties associated with
having very long message queues, see
erlang:process_flag(message_queue_data, off_heap) for one possible
remedy.)

It would be nice if the return signalling from calling port_command
could be fixed to either enable the ref trick or to not use messages
at all, but until that is the case, you know that making these calls
while having a long message queue is going to be expensive.  We've
found that it's often useful to offload such calls to temporary helper
processes, that by construction have very short message queues.  (The
signalling between the original process and the helper should of
course use the ref trick, this is non-trivial to get right.)