inets {read_packets,N} option

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

inets {read_packets,N} option

Frank Muller
Hi guys,

Can someone explain me the semantic behind {read_packets, Integer} for UDP:
http://erlang.org/doc/man/inet.html

I understand the associated doc, but what i can’t get is how this option affects me if i set it for example with:
1. {read_packets, 20} + {active, once}
2. {read_packets, 20} + {active, 100}

How many packets my process will receive in each case?

And are these packets send as multiple messages one packet at a time, or as one message representing a list of N packets?

Thank you
/Frank

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: inets {read_packets,N} option

Alex S.
{read_packets, X} refers to number of UDP packets that are read whenever the socket is notified "ready". So if 100 UDP packets arrive at the same time, only 5 will be read until a new one arrives. There isn't really a way to know how many dgrams are waiting on a socket, so that's an anti-flood option.
{active, X} refers to number of UDP packets that are immediately sent to Erlang process instead of being read into an internal buffer.

2017-01-07 14:04 GMT+03:00 Frank Muller <[hidden email]>:
Hi guys,

Can someone explain me the semantic behind {read_packets, Integer} for UDP:
http://erlang.org/doc/man/inet.html

I understand the associated doc, but what i can’t get is how this option affects me if i set it for example with:
1. {read_packets, 20} + {active, once}
2. {read_packets, 20} + {active, 100}

How many packets my process will receive in each case?

And are these packets send as multiple messages one packet at a time, or as one message representing a list of N packets?

Thank you
/Frank

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: inets {read_packets,N} option

Frank Muller
Hi Alex

Still didn't get it. In your example 100 UDP packets arrive so the socket's ready for reading. Then, N=5 are read out of 100.

Why you said 5 will be read until new one arrives?
There's still 95 ready for reading right away after delivering the first 5. I'm right?

Thank you.
/Frank

Le sam. 7 janv. 2017 à 22:17, Alex S. <[hidden email]> a écrit :
{read_packets, X} refers to number of UDP packets that are read whenever the socket is notified "ready". So if 100 UDP packets arrive at the same time, only 5 will be read until a new one arrives. There isn't really a way to know how many dgrams are waiting on a socket, so that's an anti-flood option.
{active, X} refers to number of UDP packets that are immediately sent to Erlang process instead of being read into an internal buffer.

2017-01-07 14:04 GMT+03:00 Frank Muller <[hidden email]>:
Hi guys,

Can someone explain me the semantic behind {read_packets, Integer} for UDP:
http://erlang.org/doc/man/inet.html

I understand the associated doc, but what i can’t get is how this option affects me if i set it for example with:
1. {read_packets, 20} + {active, once}
2. {read_packets, 20} + {active, 100}

How many packets my process will receive in each case?

And are these packets send as multiple messages one packet at a time, or as one message representing a list of N packets?

Thank you
/Frank



_______________________________________________


erlang-questions mailing list


[hidden email]


http://erlang.org/mailman/listinfo/erlang-questions







_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: inets {read_packets,N} option

Jesper Louis Andersen-2
Packet delivery happens roughly as follows:

1. The network card gets the packet.
2. The network card stores the packet in a kernel buffer.
3. The Erlang VM reads packets out of the kernel buffer and into its own memory space. This opens up the kernel buffer for more UDP packets to arrive.
4. The Erlang VM sends a message to a mailbox of a process with the data.

Rules:

* If the kernel buffer is full, new arrivals are dropped on the floor.
* The kernel buffer is set separately, often with an operating system default.
* The VM won't flood a process unless it has {active, true} set.

{read_packets, K} controls what happens in 3. in the above. If set at 5, once the kernel tells the VM there are data available to read, the VM will read out 5 packets. It won't read more packets until a new packet arrives at the socket in which case it will read up to 5 packets more. Note that if the arrival rate is larger than the consumption rate then packets will eventually gets dropped on the floor. With UDP, your responsibility is to handle this situation.

{active, N} controls how many packets can pre-delivered into the mailbox with no signs of progress from the process itself. It corresponds to a limiter at 4. in the above. You have situations when N < K, N > K and N == K to think about here. In those situations, you may end up with lost packets if the system goes over the capacity it is able to systematically handle.

The kernel tracks stats for how many packets it has thrown away. I don't know if the Erlang VM does somewhere (it should, provided that it throws data on the floor).

On Sat, Jan 7, 2017 at 10:39 PM Frank Muller <[hidden email]> wrote:
Hi Alex

Still didn't get it. In your example 100 UDP packets arrive so the socket's ready for reading. Then, N=5 are read out of 100.

Why you said 5 will be read until new one arrives?
There's still 95 ready for reading right away after delivering the first 5. I'm right?

Thank you.
/Frank

Le sam. 7 janv. 2017 à 22:17, Alex S. <[hidden email]> a écrit :
{read_packets, X} refers to number of UDP packets that are read whenever the socket is notified "ready". So if 100 UDP packets arrive at the same time, only 5 will be read until a new one arrives. There isn't really a way to know how many dgrams are waiting on a socket, so that's an anti-flood option.
{active, X} refers to number of UDP packets that are immediately sent to Erlang process instead of being read into an internal buffer.

2017-01-07 14:04 GMT+03:00 Frank Muller <[hidden email]>:
Hi guys,

Can someone explain me the semantic behind {read_packets, Integer} for UDP:
http://erlang.org/doc/man/inet.html

I understand the associated doc, but what i can’t get is how this option affects me if i set it for example with:
1. {read_packets, 20} + {active, once}
2. {read_packets, 20} + {active, 100}

How many packets my process will receive in each case?

And are these packets send as multiple messages one packet at a time, or as one message representing a list of N packets?

Thank you
/Frank



_______________________________________________


erlang-questions mailing list


[hidden email]


http://erlang.org/mailman/listinfo/erlang-questions






_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: inets {read_packets,N} option

Ameretat Reith
> > 2017-01-07 14:04 GMT+03:00 Frank Muller
> > <[hidden email]>:
> >

> > I understand the associated doc, but what i can’t get is how this
> > option affects me if i set it for example with:
> > 1. {read_packets, 20} + {active, once}
> > 2. {read_packets, 20} + {active, 100}

1:
Once there is data in socket, you'll get it and then your socket will
not remain active anymore. you'll get another packet when you set socket
as active again and there is a more data.

2:
Once there is data in socket, up to twenty datagrams will be read and
will be passed to target process. If there is more data, It'll be
processed on next polling iteration over and over until socket is not
active anymore.


It's like this:

while true:
        if socket_is_active():

                // this is blocking polling function, a
                // select/poll/epoll_wait loop
                if there_is_data_for_reading_in_socket():

                        N = 0
                        while N < read_packets:

                                packet = non_blocking_read()

                                if socket_had_data(packet):
                                        N--

                                        pass_to_process(packet)

                                        update_active_status()
                                else:
                                        break


Basically there is a polling loop to select/poll/epoll on set of file
descriptors. There is also a number for packets that will be read if
polling loop notifies that socket has data, it is read_packets.
After certain number of packets is read, socket will be passive; it
means looping function won't poll for that socket anymore, that number
is X in {active, X}.

> > How many packets my process will receive in each case?
> >
> > And are these packets send as multiple messages one packet at a
> > time, or as one message representing a list of N packets?

One message for each datagram (packet).

> > Still didn't get it. In your example 100 UDP packets arrive so the
> > socket's ready for reading. Then, N=5 are read out of 100.
> >
> > Why you said 5 will be read until new one arrives?
> > There's still 95 ready for reading right away after delivering the
> > first 5. I'm right?

Assuming socket's read_packet is set to 5, 5 packets will be read and
on next polling iterations, buffered packets will be processed. There
is no need for more packets since there already are in kernel buffer.

> On Sun, 08 Jan 2017 15:31:49 +0000
> Jesper Louis Andersen <[hidden email]> wrote:

> Packet delivery happens roughly as follows:
> {read_packets, K} controls what happens in 3. in the above. If set at
> 5, once the kernel tells the VM there are data available to read, the
> VM will read out 5 packets. It won't read more packets until a new
> packet arrives at the socket in which case it will read up to 5
> packets more.

Actually It will read more packets if socket is still active and there
is data. Documentation is a little ambiguous too:

> {read_packets, Integer}(UDP sockets)
> Sets the max number of UDP packets to read without intervention from
> the socket when data is available. When this many packets have been
> read and delivered to the destination process, new packets are not
> read until a new notification of available data has arrived.

Here, *notification of available data* does not mean expectance for
*new* data.

I think read_packets is more useful to overcome blocking call overhead
of polling functions rather than being an flood protection solution. If
we are experiencing burst of packets, reads will be delayed a little but
will still affect and now with more polling syscalls too.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: inets {read_packets,N} option

Frank Muller
Thank you guys. Now I get it.

All this (or a summary) should go to the doc because it isn't so clear IMHO. 

/Frank

Le dim. 8 janv. 2017 à 19:44, Ameretat Reith <[hidden email]> a écrit :
> > 2017-01-07 14:04 GMT+03:00 Frank Muller

> > <[hidden email]>:

> >



> > I understand the associated doc, but what i can’t get is how this

> > option affects me if i set it for example with:

> > 1. {read_packets, 20} + {active, once}

> > 2. {read_packets, 20} + {active, 100}



1:

Once there is data in socket, you'll get it and then your socket will

not remain active anymore. you'll get another packet when you set socket

as active again and there is a more data.



2:

Once there is data in socket, up to twenty datagrams will be read and

will be passed to target process. If there is more data, It'll be

processed on next polling iteration over and over until socket is not

active anymore.





It's like this:



while true:

        if socket_is_active():



                // this is blocking polling function, a

                // select/poll/epoll_wait loop

                if there_is_data_for_reading_in_socket():



                        N = 0

                        while N < read_packets:



                                packet = non_blocking_read()



                                if socket_had_data(packet):

                                        N--



                                        pass_to_process(packet)



                                        update_active_status()

                                else:

                                        break





Basically there is a polling loop to select/poll/epoll on set of file

descriptors. There is also a number for packets that will be read if

polling loop notifies that socket has data, it is read_packets.

After certain number of packets is read, socket will be passive; it

means looping function won't poll for that socket anymore, that number

is X in {active, X}.



> > How many packets my process will receive in each case?

> >

> > And are these packets send as multiple messages one packet at a

> > time, or as one message representing a list of N packets?



One message for each datagram (packet).



> > Still didn't get it. In your example 100 UDP packets arrive so the

> > socket's ready for reading. Then, N=5 are read out of 100.

> >

> > Why you said 5 will be read until new one arrives?

> > There's still 95 ready for reading right away after delivering the

> > first 5. I'm right?



Assuming socket's read_packet is set to 5, 5 packets will be read and

on next polling iterations, buffered packets will be processed. There

is no need for more packets since there already are in kernel buffer.



> On Sun, 08 Jan 2017 15:31:49 +0000

> Jesper Louis Andersen <[hidden email]> wrote:



> Packet delivery happens roughly as follows:

> {read_packets, K} controls what happens in 3. in the above. If set at

> 5, once the kernel tells the VM there are data available to read, the

> VM will read out 5 packets. It won't read more packets until a new

> packet arrives at the socket in which case it will read up to 5

> packets more.



Actually It will read more packets if socket is still active and there

is data. Documentation is a little ambiguous too:



> {read_packets, Integer}(UDP sockets)

> Sets the max number of UDP packets to read without intervention from

> the socket when data is available. When this many packets have been

> read and delivered to the destination process, new packets are not

> read until a new notification of available data has arrived.



Here, *notification of available data* does not mean expectance for

*new* data.



I think read_packets is more useful to overcome blocking call overhead

of polling functions rather than being an flood protection solution. If

we are experiencing burst of packets, reads will be delayed a little but

will still affect and now with more polling syscalls too.

_______________________________________________

erlang-questions mailing list

[hidden email]

http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: inets {read_packets,N} option

Kenneth Lakin
On 01/08/2017 11:13 AM, Frank Muller wrote:
> All this (or a summary) should go to the doc because it isn't so clear
> IMHO.

If you get the itch to write up a correction, good pull requests are
accepted at <https://github.com/erlang/otp/pulls>. The file containing
the documentation that you're concerned about is at
$ERL_TOP/lib/kernel/doc/src/inet.xml

(But read <https://github.com/erlang/otp/wiki/Contribution-Guidelines>
and probably <https://github.com/erlang/otp/wiki/Documentation> first!)




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

signature.asc (836 bytes) Download Attachment