gen_udp:send blocks?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

gen_udp:send blocks?

Roger Lipscombe-2
I've got a gen_server that, in response to handle_cast, calls
gen_udp:send. On one server, the call to gen_udp:send takes about
1500us. On a different server, the call takes about 75000us.

Because of this, the process message queue is getting backed up, at
the rate of about 120-250 messages/minute.

The difference between the servers is obvious: one of them is in the
same AWS region as the destination host; the other is the other side
of the Atlantic (i.e. about 70ms away).

My question is more about understanding the semantics of gen_udp:send:
why is it blocking? If the buffer was full, I'd expect it to discard
the message and return immediately.

Regards,
Roger.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_udp:send blocks?

Valentin Micic-5
On 07 Sep 2018, at 12:11 PM, Roger Lipscombe wrote:

> I've got a gen_server that, in response to handle_cast, calls
> gen_udp:send. On one server, the call to gen_udp:send takes about
> 1500us. On a different server, the call takes about 75000us.
>
> Because of this, the process message queue is getting backed up, at
> the rate of about 120-250 messages/minute.
>
> The difference between the servers is obvious: one of them is in the
> same AWS region as the destination host; the other is the other side
> of the Atlantic (i.e. about 70ms away).
>
> My question is more about understanding the semantics of gen_udp:send:
> why is it blocking? If the buffer was full, I'd expect it to discard
> the message and return immediately.
>
> Regards,
> Roger.
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

UDP should behave the way you expected. May I ask if the caller of gen_udp:send is the owner (e.g. creator) of the socket?

V/
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_udp:send blocks?

Roger Lipscombe-2
On 7 September 2018 at 12:37, Valentin Micic <[hidden email]> wrote:
> On 07 Sep 2018, at 12:11 PM, Roger Lipscombe wrote:
>> My question is more about understanding the semantics of gen_udp:send:
>> why is it blocking? If the buffer was full, I'd expect it to discard
>> the message and return immediately.
>
> UDP should behave the way you expected. May I ask if the caller of gen_udp:send is the owner (e.g. creator) of the socket?

Yes it is. I call gen_udp:open in Module:init/1, and gen_udp:send in
Module:handle_cast/2.

Oh. Here's a thought: we use a hostname as the destination in
gen_udp:send. Could that be causing blocking?
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_udp:send blocks?

Dmitry Kolesnikov-2

On 7 Sep 2018, at 16.00, Roger Lipscombe <[hidden email]> wrote:

Oh. Here's a thought: we use a hostname as the destination in
gen_udp:send. Could that be causing blocking?

Usually, this causes DNS resolution, which takes some time. If you have short DNS TTL or disable cache it might cause visible delay. I think you need to check config of you DNS server and resolver as well. 

Best Regards, 
Dmitry


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: gen_udp:send blocks?

Roger Lipscombe-2
On 7 September 2018 at 14:13, Dmitry Kolesnikov <[hidden email]> wrote:

>
> On 7 Sep 2018, at 16.00, Roger Lipscombe <[hidden email]> wrote:
>
> Oh. Here's a thought: we use a hostname as the destination in
> gen_udp:send. Could that be causing blocking?
>
>
> Usually, this causes DNS resolution, which takes some time. If you have
> short DNS TTL or disable cache it might cause visible delay. I think you
> need to check config of you DNS server and resolver as well.

Yeah; this appears to be the problem. We're using consul, which has
zero TTL (by default?). This means that each call to gen_udp:send
resolves the name again. Based on the timings, it looks like this
involves a trip across the Atlantic. We want a fairly short TTL
(because otherwise how would health-check failover work?). I'll
explicitly cache the IP address and refresh it after each N messages
sent.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_udp:send blocks?

Valentin Micic-5
In reply to this post by Roger Lipscombe-2

On 07 Sep 2018, at 3:00 PM, Roger Lipscombe wrote:

> On 7 September 2018 at 12:37, Valentin Micic <[hidden email]> wrote:
>> On 07 Sep 2018, at 12:11 PM, Roger Lipscombe wrote:
>>> My question is more about understanding the semantics of gen_udp:send:
>>> why is it blocking? If the buffer was full, I'd expect it to discard
>>> the message and return immediately.
>>
>> UDP should behave the way you expected. May I ask if the caller of gen_udp:send is the owner (e.g. creator) of the socket?
>
> Yes it is. I call gen_udp:open in Module:init/1, and gen_udp:send in
> Module:handle_cast/2.
>
> Oh. Here's a thought: we use a hostname as the destination in
> gen_udp:send. Could that be causing blocking?

Well, if the name resolution is slow (e.g. you use some remote DNS server), this may cause some delay, but, as far as I know, local OS resolver would cache the name once it has been resolved.
HOWEVER, if that delay caused gen_server message process queue to build up due to constant incoming traffic, than all bets are off --  when this number grows above hundred, this may cause the whole run-time to underperform, which puts more pressure one the gen_server's process message queue and so on. This could explain the behavior you observed.
There's an easy way to check this -- just check the process message queue length for your gen_server (or any other process to that end), and you'll know :-)

V/




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: gen_udp:send blocks?

Dmitry Kolesnikov-2
In reply to this post by Roger Lipscombe-2


On 7 Sep 2018, at 16.20, Roger Lipscombe <[hidden email]> wrote:

 We want a fairly short TTL
(because otherwise how would health-check failover work?). I'll
explicitly cache the IP address and refresh it after each N messages
sent.

I would implement a side process that re-resolves DNS every N seconds and then notifies your sender process with IP address. 

or change DNS settings to few seconds to benefit from OS feature. Keep in-mind that OTP has custom resolver. I’ve not hacked and do not know all details but it might have an explicit cache feature. It worth of checking.

- Dmitry   


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: gen_udp:send blocks?

Roger Lipscombe-2
On 7 September 2018 at 14:28, Dmitry Kolesnikov <[hidden email]> wrote:
> I would implement a side process that re-resolves DNS every N seconds and
> then notifies your sender process with IP address.

That's a good suggestion, but we're already (as of 30 minutes ago)
using pobox, so it's easier just to wedge it in there. If we weren't
doing that, then your suggestion might be ideal. Thanks.

> or change DNS settings to few seconds to benefit from OS feature.

Ugh. That means talking to the ops team (just kidding!). I found
https://www.consul.io/docs/guides/dns-cache.html, so I'll talk with
them later about it.

> Keep
> in-mind that OTP has custom resolver. I’ve not hacked and do not know all
> details but it might have an explicit cache feature. It worth of checking.

Good point. I had a quick poke around in there a while ago. It's
probably worth taking another look.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions