Quantcast

Message send guarantees

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Message send guarantees

Dmitry Kakurin-2

When I execute "pid ! msg" and it returns, what are the guarantees if pid and self are on the same node?
Can I assume that by the time it returns, msg is in the pid's queue? And as a result all messages sent to pid afterwards will be queued after msg?

The reason I'm asking is because I need a guarantee of proper message ordering in the following scenario when all processes involved are local to a node:
1. A --msg1--> C (process A sends msg1 to process C)
2. A --msg2--> B
3. B --msg2--> C (process B simply proxies msg2 from A to C)

I need a guarantee that msg2 will always appear in C's queue after msg1.

P.S. I've read the FAQ 10.8 and 10.9 :-)

Thank you, Dmitry.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Message send guarantees

Lukas Larsson-8
Hello,

On Wed, Jan 25, 2017 at 12:38 AM, Dmitry Kakurin <[hidden email]> wrote:

When I execute "pid ! msg" and it returns, what are the guarantees if pid and self are on the same node?
Can I assume that by the time it returns, msg is in the pid's queue? And as a result all messages sent to pid afterwards will be queued after msg?

No you cannot assume that. The message is considered "in transit" until the receiving process inspects it somehow.


The reason I'm asking is because I need a guarantee of proper message ordering in the following scenario when all processes involved are local to a node:
1. A --msg1--> C (process A sends msg1 to process C)
2. A --msg2--> B
3. B --msg2--> C (process B simply proxies msg2 from A to C)

I need a guarantee that msg2 will always appear in C's queue after msg1.

You have to do this using messages or some other sync mechanism.
 

P.S. I've read the FAQ 10.8 and 10.9 :-)

Please add to the FAQ if you think that you can make it clearer on this point. https://github.com/matthiasl/Erlang-FAQ

Lukas

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Message send guarantees

Dmitry Kakurin-2

Thanks Lukas, please see a couple of questions/comments below.

- Dmitry
 

From: Lukas Larsson <[hidden email]>
Sent: Wednesday, January 25, 2017 12:37 AM
 
Hello,

On Wed, Jan 25, 2017 at 12:38 AM, Dmitry Kakurin <[hidden email]> wrote:
When I execute "pid ! msg" and it returns, what are the guarantees if pid and self are on the same node?
Can I assume that by the time it returns, msg is in the pid's queue? And as a result all messages sent to pid afterwards will be queued after msg?

No you cannot assume that. The message is considered "in transit" until the receiving process inspects it somehow.

What aspect of implementation can cause such reordering? Could you give an example how this could happen? Maybe I'll be able to artificially create conditions where it would not happen. Or work around it in some other way. Please see below why.


The reason I'm asking is because I need a guarantee of proper message ordering in the following scenario when all processes involved are local to a node:
1. A --msg1--> C (process A sends msg1 to process C)
2. A --msg2--> B
3. B --msg2--> C (process B simply proxies msg2 from A to C)

I need a guarantee that msg2 will always appear in C's queue after msg1.

You have to do this using messages or some other sync mechanism.

You are right, in general it's easy to devise a simple scheme for that. 
Unfortunately I have a limitation. In my case A talks to C thru proxy B and would like to short-circuit the communication by getting rid of B and start talking to C directly. Process A needs to do that while messages thru B could be in flight. The limitation is that C is not aware of this proxy business, and cannot be changed (it's a regular GenServer that is not written by me).
With lack of cooperation from C I cannot think of a communication protocol to accomplish what I want.

Is there a way for processes A and B to observe C's queue in any way and derive some information about it to help me with my problem?
 

P.S. I've read the FAQ 10.8 and 10.9 :-)

Please add to the FAQ if you think that you can make it clearer on this point. https://github.com/matthiasl/Erlang-FAQ

Lukas


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Message send guarantees

Raimo Niskanen-2
On Wed, Jan 25, 2017 at 11:06:05PM +0000, Dmitry Kakurin wrote:

> Thanks Lukas, please see a couple of questions/comments below.
>
> - Dmitry
>
> ________________________________
> From: Lukas Larsson <[hidden email]>
> Sent: Wednesday, January 25, 2017 12:37 AM
>
> Hello,
>
> On Wed, Jan 25, 2017 at 12:38 AM, Dmitry Kakurin <[hidden email]<mailto:[hidden email]>> wrote:
> When I execute "pid ! msg" and it returns, what are the guarantees if pid and self are on the same node?
> Can I assume that by the time it returns, msg is in the pid's queue? And as a result all messages sent to pid afterwards will be queued after msg?
>
> No you cannot assume that. The message is considered "in transit" until the receiving process inspects it somehow.
>
> What aspect of implementation can cause such reordering? Could you give an example how this could happen? Maybe I'll be able to artificially create conditions where it would not happen. Or work around it in some other way. Please see below why.

On a multi-threade system it is very hard to know in which order things
happen, so there is no "reordering" since there is no temporal order
to begin with.

Using the non-SMP virtual machine and erlang:yield/0 could do the trick,
but depending on the computer you might loose boatloads of performance.

It is also _very_ bad programming practice to pretend that asynchronous
message passing is not asynchronous.  It is solving a problem the
wrong way...

>
>
> The reason I'm asking is because I need a guarantee of proper message ordering in the following scenario when all processes involved are local to a node:
> 1. A --msg1--> C (process A sends msg1 to process C)
> 2. A --msg2--> B
> 3. B --msg2--> C (process B simply proxies msg2 from A to C)
>
> I need a guarantee that msg2 will always appear in C's queue after msg1.
>
> You have to do this using messages or some other sync mechanism.
>
> You are right, in general it's easy to devise a simple scheme for that.
> Unfortunately I have a limitation. In my case A talks to C thru proxy B and would like to short-circuit the communication by getting rid of B and start talking to C directly. Process A needs to do that while messages thru B could be in flight. The limitation is that C is not aware of this proxy business, and cannot be changed (it's a regular GenServer that is not written by me).
> With lack of cooperation from C I cannot think of a communication protocol to accomplish what I want.

If B can be modified as needed; you can get an ack from B so both A and B
know about the bypass e.g to delay bypass until no messages are in flight,
provided that the messages from B to C are acked.

Or, maybe you can only use messages where the order does not matter.

But without modifying your requirements as you state them I see no solution.

Why do you really need to bypass some messages and still keep message
ordering - it is impossible!

>
> Is there a way for processes A and B to observe C's queue in any way and derive some information about it to help me with my problem?

From a process on the same node as C it is possible to inspect C's queue
through erlang:process_info/2, but then you have started to dig a deep hole
for yourself.  You will get whole queue of C and have to traverse all of it
to see if the message you sent is there.  But just to inspect the queue will
hurt message passing performance due to the locking involved.

We have had customers that have used something like that in an attemt to do
overload protection, only resulting in bad high load behaviour...

>
>
> P.S. I've read the FAQ 10.8 and 10.9 :-)
>
> Please add to the FAQ if you think that you can make it clearer on this point. https://github.com/matthiasl/Erlang-FAQ
>
> Lukas
>


--

/ Raimo Niskanen, Erlang/OTP, Ericsson AB
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Message send guarantees

Alex S.

26 янв. 2017 г., в 10:14, Raimo Niskanen <[hidden email]> написал(а):

On Wed, Jan 25, 2017 at 11:06:05PM +0000, Dmitry Kakurin wrote:
Thanks Lukas, please see a couple of questions/comments below.

- Dmitry

________________________________
From: Lukas Larsson <[hidden email]>
Sent: Wednesday, January 25, 2017 12:37 AM

Hello,

On Wed, Jan 25, 2017 at 12:38 AM, Dmitry Kakurin <[hidden email]<[hidden email]>> wrote:
When I execute "pid ! msg" and it returns, what are the guarantees if pid and self are on the same node?
Can I assume that by the time it returns, msg is in the pid's queue? And as a result all messages sent to pid afterwards will be queued after msg?

No you cannot assume that. The message is considered "in transit" until the receiving process inspects it somehow.

What aspect of implementation can cause such reordering? Could you give an example how this could happen? Maybe I'll be able to artificially create conditions where it would not happen. Or work around it in some other way. Please see below why.

On a multi-threade system it is very hard to know in which order things
happen, so there is no "reordering" since there is no temporal order
to begin with.

Using the non-SMP virtual machine and erlang:yield/0 could do the trick,
but depending on the computer you might loose boatloads of performance.

It is also _very_ bad programming practice to pretend that asynchronous
message passing is not asynchronous.  It is solving a problem the
wrong way…
I’d like to clarify that the order of messages *sent from A to B* is kept intact indeed, and later messages will arrive later.




The reason I'm asking is because I need a guarantee of proper message ordering in the following scenario when all processes involved are local to a node:
1. A --msg1--> C (process A sends msg1 to process C)
2. A --msg2--> B
3. B --msg2--> C (process B simply proxies msg2 from A to C)

I need a guarantee that msg2 will always appear in C's queue after msg1.

You have to do this using messages or some other sync mechanism.

You are right, in general it's easy to devise a simple scheme for that.
Unfortunately I have a limitation. In my case A talks to C thru proxy B and would like to short-circuit the communication by getting rid of B and start talking to C directly. Process A needs to do that while messages thru B could be in flight. The limitation is that C is not aware of this proxy business, and cannot be changed (it's a regular GenServer that is not written by me).
With lack of cooperation from C I cannot think of a communication protocol to accomplish what I want.

If B can be modified as needed; you can get an ack from B so both A and B
know about the bypass e.g to delay bypass until no messages are in flight,
provided that the messages from B to C are acked.

Or, maybe you can only use messages where the order does not matter.

But without modifying your requirements as you state them I see no solution.

Why do you really need to bypass some messages and still keep message
ordering - it is impossible!
I would recommend using synchronous calls, perhaps with a proxy «queue» process or whatnot.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Message send guarantees

Fred Hebert-2
In reply to this post by Dmitry Kakurin-2
On 01/25, Dmitry Kakurin wrote:

>
>The reason I'm asking is because I need a guarantee of proper message ordering in the following scenario when all processes involved are local to a node:
>1. A --msg1--> C (process A sends msg1 to process C)
>2. A --msg2--> B
>3. B --msg2--> C (process B simply proxies msg2 from A to C)
>
>I need a guarantee that msg2 will always appear in C's queue after msg1.
>
> [...]
>
>Unfortunately I have a limitation. In my case A talks to C thru proxy B
>and would like to short-circuit the communication by getting rid of B
>and start talking to C directly. Process A needs to do that while
>messages thru B could be in flight. The limitation is that C is not
>aware of this proxy business, and cannot be changed (it's a regular
>GenServer that is not written by me).
>With lack of cooperation from C I cannot think of a communication protocol to accomplish what I want.
>
>Is there a way for processes A and B to observe C's queue in any way and derive some information about it to help me with my problem?
>

As others have mentioned, this is not really possible. You *could*
observe the queue, but it is not practical.

The interesting thing with Erlang there is that a lot of these design
decisions work the same whether the processes are local or remote. The
problem you see here of message ordering is very similar to the one you
get of packet ordering when delivering data over a network.

Generally, the way to solve the problem is to label every message you
send with a sequence number that increments. On every message received,
the final receiver can then look at the sequence numbers and see if any
message is missing. It can then block, ask for a retransmission, or
disregard duplicated messages.

Because Erlang's guarantees for message passing only involve any given
pair of processes (A -> B, or A -> C) but not more, we get a case where
the following race conditions are very possible:

  A            B           C
  | --- 1 ---> |           |
  | --- 2 ---------------> |
  |            | --- 1 --> |

The stop at B means that through scheduling, the message can be delayed
for unkown periods of time and the delays can mess up the ordering.

The only surefire way of getting the messages to C in order is to either
a) send them directly from a single sender every time, or b) to let C
know if the messages are not in order so it can do the reordering
itself.

Anything else will be very hacky (implementing locks in userland and
synchronizing everything) and would usually be more of a code smell than
anything.

Note: The usage of sequential number is a more specific implementation
of causal ordering, where a logical 'stamp' is used to denote a
'happens-before' relationship between events. Vector clocks, version
vectors, and interval tree clocks are other data structures that let you
get more general ordering guarantees, but that much power isn't
necessarily required here.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Message send guarantees

Joe Armstrong-2
In reply to this post by Dmitry Kakurin-2
On Wed, Jan 25, 2017 at 12:38 AM, Dmitry Kakurin
<[hidden email]> wrote:

> When I execute "pid ! msg" and it returns, what are the guarantees if pid
> and self are on the same node?
> Can I assume that by the time it returns, msg is in the pid's queue? And as
> a result all messages sent to pid afterwards will be queued after msg?
>
> The reason I'm asking is because I need a guarantee of proper message
> ordering in the following scenario when all processes involved are local to
> a node:
> 1. A --msg1--> C (process A sends msg1 to process C)
> 2. A --msg2--> B
> 3. B --msg2--> C (process B simply proxies msg2 from A to C)
>
> I need a guarantee that msg2 will always appear in C's queue after msg1.

This cannot be guaranteed

In the absence of an error you can assume the following:

    That sequences of messages between pairs of processes are ordered (ie they
     arrive in the same order they were sent)
    You will never know if messages are processed by the destination process

Note I said "in the absence of an error" - to take care of the error case
we use links - links tell you if the linked processes have crashed,
nothing else.

Even if a message arrives at a process you will never know, you will also never
know if the processing it triggered worked - if you want to know send a message
back and wait for it.

If you want to know if a process works send it a message asking it to do
something, wait for a reply and check the reply is OK.

All this will tell you is that the process WAS working they last time you
asked it - and NOT that it IS working now.

Also remember that messages take time to propagate - and processes
take time to do
things

In your A -> B followed by B->C B might take a long time between relaying the
messages (due to garbage collection, or something weird)

The above behaviour does not break any laws of physics - ie messages
take time, causality is obeyed, these is no spooky action at a distance etc.

Cheers

/Joe








>
> P.S. I've read the FAQ 10.8 and 10.9 :-)
>
> Thank you, Dmitry.
>
>
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Message send guarantees

Jesper Louis Andersen-2
In reply to this post by Dmitry Kakurin-2
On Thu, Jan 26, 2017 at 5:21 AM Dmitry Kakurin <[hidden email]> wrote:
What aspect of implementation can cause such reordering? Could you give an example how this could happen? Maybe I'll be able to artificially create conditions where it would not happen. Or work around it in some other way. Please see below why.



A straightforward implementation would lock the mailbox of the receiving process, put a message there, unlock the mailbox and proceed. In such a system, the ordering is guaranteed because you have a happens-before relationship on the message send. If the process is remote, TCP buffering can reorder message arrival, so in general it is not good to structure systems under the assumption that there is a happens-before relationship between messages like this.

However, there is also the need to satisfy future possible implementations. I could delay sending messages for a while and then send them as a large batch later on. This is allowed in the semantics since the ! operator works asynchronously. I could implement it as:

Allocate a new message.
Try a CAS operation on the target mailbox.
If the CAS operation fails, delay the attempt to later due to contention on the target mailbox.

In this setting, there is no happens-before on '!', and thus messages may be reordered.

If you start relying on internal implementation structure, chances are your program subtly breaks later on. A good example was when the OTP team split up the single timer wheel into multiple timer wheels. Before, there were an order in which timers triggered and people relied on this order. But with multiple timer wheels, reorderings are possible and their code broke.

My advice is to code for the rules of physics (causality). It tends to produce the best results in the long run.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Message send guarantees

Per Hedeland
In reply to this post by Fred Hebert-2
On 2017-01-26 14:09, Fred Hebert wrote:

> On 01/25, Dmitry Kakurin wrote:
>>
>> The reason I'm asking is because I need a guarantee of proper message ordering in the following scenario when all processes involved are local to a node:
>> 1. A --msg1--> C (process A sends msg1 to process C)
>> 2. A --msg2--> B
>> 3. B --msg2--> C (process B simply proxies msg2 from A to C)
>>
>> I need a guarantee that msg2 will always appear in C's queue after msg1.
>>
>> [...]
>>
>> Unfortunately I have a limitation. In my case A talks to C thru proxy B and would like to short-circuit the communication by getting rid of B and start talking to C directly. Process A needs to do
>> that while messages thru B could be in flight. The limitation is that C is not aware of this proxy business, and cannot be changed (it's a regular GenServer that is not written by me).
>> With lack of cooperation from C I cannot think of a communication protocol to accomplish what I want.
>>
>> Is there a way for processes A and B to observe C's queue in any way and derive some information about it to help me with my problem?
>>
>
> As others have mentioned, this is not really possible. You *could* observe the queue, but it is not practical.
>
> The interesting thing with Erlang there is that a lot of these design decisions work the same whether the processes are local or remote. The problem you see here of message ordering is very similar to
> the one you get of packet ordering when delivering data over a network.
>
> Generally, the way to solve the problem is to label every message you send with a sequence number that increments. On every message received, the final receiver can then look at the sequence numbers
> and see if any message is missing. It can then block, ask for a retransmission, or disregard duplicated messages.
>
> Because Erlang's guarantees for message passing only involve any given pair of processes (A -> B, or A -> C) but not more, we get a case where the following race conditions are very possible:
>
>  A            B           C
>  | --- 1 ---> |           |
>  | --- 2 ---------------> |
>  |            | --- 1 --> |
>
> The stop at B means that through scheduling, the message can be delayed for unkown periods of time and the delays can mess up the ordering.
>
> The only surefire way of getting the messages to C in order is to either
> a) send them directly from a single sender every time, or b) to let C know if the messages are not in order so it can do the reordering itself.

Well, there is also (already hinted at I think) c) have A receive an ack
from C for message 1 before it sends message 2. Doing this for every
message may be an unacceptable throughput limitation in some cases, in
some other cases it is what happens "anyway" due to the nature of the
communication. In this particular case, "A talks to C thru proxy B and
would like to short-circuit the communication by getting rid of B and
start talking to C directly", it would be sufficient that A got an ack
for only the *last* via-B-message (could be a special "let's skip the
proxy" message) before it started sending directly to C. Seems like a
clean solution to me.

--Per
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Message send guarantees

Per Hedeland
On 2017-01-26 18:05, Per Hedeland wrote:

> On 2017-01-26 14:09, Fred Hebert wrote:
>> On 01/25, Dmitry Kakurin wrote:
>>>
>>> The reason I'm asking is because I need a guarantee of proper message ordering in the following scenario when all processes involved are local to a node:
>>> 1. A --msg1--> C (process A sends msg1 to process C)
>>> 2. A --msg2--> B
>>> 3. B --msg2--> C (process B simply proxies msg2 from A to C)
>>>
>>> I need a guarantee that msg2 will always appear in C's queue after msg1.
>>>
>>> [...]
>>>
>>> Unfortunately I have a limitation. In my case A talks to C thru proxy B and would like to short-circuit the communication by getting rid of B and start talking to C directly. Process A needs to do
>>> that while messages thru B could be in flight. The limitation is that C is not aware of this proxy business, and cannot be changed (it's a regular GenServer that is not written by me).
>>> With lack of cooperation from C I cannot think of a communication protocol to accomplish what I want.

Sorry, I missed that last part in my comment below - indeed it prevents
the "ack solution" if C won't cooperate.

--Per

>>> Is there a way for processes A and B to observe C's queue in any way and derive some information about it to help me with my problem?
>>>
>>
>> As others have mentioned, this is not really possible. You *could* observe the queue, but it is not practical.
>>
>> The interesting thing with Erlang there is that a lot of these design decisions work the same whether the processes are local or remote. The problem you see here of message ordering is very similar to
>> the one you get of packet ordering when delivering data over a network.
>>
>> Generally, the way to solve the problem is to label every message you send with a sequence number that increments. On every message received, the final receiver can then look at the sequence numbers
>> and see if any message is missing. It can then block, ask for a retransmission, or disregard duplicated messages.
>>
>> Because Erlang's guarantees for message passing only involve any given pair of processes (A -> B, or A -> C) but not more, we get a case where the following race conditions are very possible:
>>
>>  A            B           C
>>  | --- 1 ---> |           |
>>  | --- 2 ---------------> |
>>  |            | --- 1 --> |
>>
>> The stop at B means that through scheduling, the message can be delayed for unkown periods of time and the delays can mess up the ordering.
>>
>> The only surefire way of getting the messages to C in order is to either
>> a) send them directly from a single sender every time, or b) to let C know if the messages are not in order so it can do the reordering itself.
>
> Well, there is also (already hinted at I think) c) have A receive an ack
> from C for message 1 before it sends message 2. Doing this for every
> message may be an unacceptable throughput limitation in some cases, in
> some other cases it is what happens "anyway" due to the nature of the
> communication. In this particular case, "A talks to C thru proxy B and
> would like to short-circuit the communication by getting rid of B and
> start talking to C directly", it would be sufficient that A got an ack
> for only the *last* via-B-message (could be a special "let's skip the
> proxy" message) before it started sending directly to C. Seems like a
> clean solution to me.
>
> --Per
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Message send guarantees

Dmitry Kakurin-2
In reply to this post by Joe Armstrong-2

From: Joe Armstrong <[hidden email]>

Sent: Thursday, January 26, 2017 5:29 AM
 
On Wed, Jan 25, 2017 at 12:38 AM, Dmitry Kakurin
> When I execute "pid ! msg" and it returns, what are the guarantees if pid
> and self are on the same node?
> Can I assume that by the time it returns, msg is in the pid's queue? And as
> a result all messages sent to pid afterwards will be queued after msg?
>
> The reason I'm asking is because I need a guarantee of proper message
> ordering in the following scenario when all processes involved are local to
> a node:
> 1. A --msg1--> C (process A sends msg1 to process C)
> 2. A --msg2--> B
> 3. B --msg2--> C (process B simply proxies msg2 from A to C)
>
> I need a guarantee that msg2 will always appear in C's queue after msg1.

This cannot be guaranteed

Thanks Joe and others. I understand now that it's fruitless to pursue a direction where I'd try to somehow artificially establish this guarantee.

I have to re-evaluate my constraints then. All along my main constraint was that C can be any gen_server, it's not owned by me and I cannot change it.
If I could change C then the following would work:
1. A --msg1--> C
2. A --ping--> C
3. A <--pong-- C
4. A --msg2--> B
5. B --msg2--> C

All we need from C is to respond "pong" to "ping" message. In this case A knows msg2 is enqueued after msg1 because A has observed "pong" in response to "ping" that was sent after msg1. The ordering is as desired here.
Now the only problem I have to solve is: what can be used as a benign message that I can send to any gen_server such that it would reply (to emulate ping/pong exchange above)?
Looking thru gen_server source code I see that it's processing sys messages. So I can use get_state message as "ping" and response with state as "pong". Except it may not be really benign perf-wise if amount of state kept by C is significant.
There is also get_status, but I don't understand how it's processed and if I can use it for ping/pong purposes. Can I? Is there a better message?

Do you have a better idea for ping/pong messages on C? Or how to solve this in general?

Thank you, Dmitry.

P.S. I know that in this simplified example I could piggyback on msg1 if it returns something but in real code I cannot, so let's assume it's a one-way message (a cast, not a call). So I'm OK with ping/pong.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Message send guarantees

Raimo Niskanen-2
On Fri, Jan 27, 2017 at 02:25:49AM +0000, Dmitry Kakurin wrote:

> From: Joe Armstrong <[hidden email]>
>
> Sent: Thursday, January 26, 2017 5:29 AM
>
> On Wed, Jan 25, 2017 at 12:38 AM, Dmitry Kakurin
> <[hidden email]> wrote:
> > When I execute "pid ! msg" and it returns, what are the guarantees if pid
> > and self are on the same node?
> > Can I assume that by the time it returns, msg is in the pid's queue? And as
> > a result all messages sent to pid afterwards will be queued after msg?
> >
> > The reason I'm asking is because I need a guarantee of proper message
> > ordering in the following scenario when all processes involved are local to
> > a node:
> > 1. A --msg1--> C (process A sends msg1 to process C)
> > 2. A --msg2--> B
> > 3. B --msg2--> C (process B simply proxies msg2 from A to C)
> >
> > I need a guarantee that msg2 will always appear in C's queue after msg1.
>
> This cannot be guaranteed
>
> Thanks Joe and others. I understand now that it's fruitless to pursue a direction where I'd try to somehow artificially establish this guarantee.
>
> I have to re-evaluate my constraints then. All along my main constraint was that C can be any gen_server, it's not owned by me and I cannot change it.
> If I could change C then the following would work:
> 1. A --msg1--> C
> 2. A --ping--> C
> 3. A <--pong-- C
> 4. A --msg2--> B
> 5. B --msg2--> C
>

Have you changed your question?  Previously the problem was to send
non-proxied messages after proxied messages.

But if B is a message relay (receive Msg -> C ! Msg, loop() end),
and provided the ping/pong is a gen_server call (since the request
contains who should have the reply, then this should also work:
1. A --msg1---> B
2. A --req----> B
3. B --msg1---> C
4. B --req----> C
5. A <--reply-- C
6. A --msg2---> C

> All we need from C is to respond "pong" to "ping" message. In this case A knows msg2 is enqueued after msg1 because A has observed "pong" in response to "ping" that was sent after msg1. The ordering is as desired here.
> Now the only problem I have to solve is: what can be used as a benign message that I can send to any gen_server such that it would reply (to emulate ping/pong exchange above)?
> Looking thru gen_server source code I see that it's processing sys messages. So I can use get_state message as "ping" and response with state as "pong". Except it may not be really benign perf-wise if amount of state kept by C is significant.
> There is also get_status, but I don't understand how it's processed and if I can use it for ping/pong purposes. Can I? Is there a better message?

sys:get_status/1,2 returns much more data including what sys:get_state/1,2
returns, so it is less suitable.

If you know anything more about your gen_server C, it may have some status
quering handle_call that can be used.

Maybe sys:statistics/2 can be good enough.  Mostly no statistics collection is
activated for a given gen_server so you will get just {ok,no_statistics} back
as pong, and if statistics collection should be activated the gen_server
does not much more than process_info(self(), reductions) and erlang:localtime(),
which may and may not be regarded as heavy, but compared to returning the whole
arbitrary gen_server state it may be better.

>
> Do you have a better idea for ping/pong messages on C? Or how to solve this in general?
>
> Thank you, Dmitry.
>
> P.S. I know that in this simplified example I could piggyback on msg1 if it returns something but in real code I cannot, so let's assume it's a one-way message (a cast, not a call). So I'm OK with ping/pong.
>


--

/ Raimo Niskanen, Erlang/OTP, Ericsson AB
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Message send guarantees

Andrey Tsirulev-2
In reply to this post by Raimo Niskanen-2
Hello,

26.01.2017 10:14, Raimo Niskanen wrote:
> On Wed, Jan 25, 2017 at 11:06:05PM +0000, Dmitry Kakurin wrote:
>> No you cannot assume that. The message is considered "in transit" until the receiving process inspects it somehow.
>>
>> What aspect of implementation can cause such reordering? Could you give an example how this could happen? Maybe I'll be able to artificially create conditions where it would not happen. Or work around it in some other way. Please see below why.
> On a multi-threade system it is very hard to know in which order things
> happen, so there is no "reordering" since there is no temporal order
> to begin with.
What about the following example?

A -- msg1 ---> A
A -- msg2 ---> B
B -- msg2 ---> A

In this case, is it guaranteed that if both msg2 and msg1 get to A's msg
queue, msg1 gets there earlier?

If this is guaranteed, what makes this example *really* different from
the original example?

I use this pattern for delayed gen_server's initialization if it's time
consuming, like that:
init(Args) ->
     self() ! {init, Args},
    ...
handle_info({init, Args}, State) ->
   %% time consuming initialization here

to return from init ASAP so that start/start_link do not block for a
long time. Then I expect that after

P = gen_server:start_link(...),
P ! test

when test is received, gen_server is fully initialized.  It never caused
problems, but it might be luck.

Best Regards,
Andrey

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Message send guarantees

Richard A. O'Keefe-2


On 27/01/17 11:46 PM, Andrey Tsirulev wrote:
> What about the following example?
>
> A -- msg1 ---> A
> A -- msg2 ---> B
> B -- msg2 ---> A
>
> In this case, is it guaranteed that if both msg2 and msg1 get to A's msg
> queue, msg1 gets there earlier?

Let's start by changing the destination from A to C.

A -> msg1 -> C
A -> msg2 -> B
B -> f(msg2) -> C

In this case, it is clearly NOT guaranteed that msg1 will arrive
before f(msg2).

Why would changing C to A make a difference?
Answer: because A -> A is happening in the same place.

Let E1 and E2 be two events.
E1 happens-before E2 if and only if
  (1) E1 and E2 happen in the same process
      E1 happens before E2 in that process's world-line
or
  (2) E1 is the transmission of a message
      and E2 is the receipt of the same message
or
  (3) There is an event Em such that
      E1 happens-before Em and Em happens-before E2

If we change the example to

A -> msg2 -> B
A -> msg1 -> A
B -> f(msg2) -> A

the transmission of msg1 now happens after the transmission
of msg2, so there's no reason to expect it to arrive earlier
than f(msg2).  Imagine a single-CPU implementation where
A -> msg2 -> B puts A to sleep and wakes B, then
B -> f(msg2) -> A puts B to sleep and wakes A,
then A -> msg1 -> A puts msg1 in A's queue after f(msg2).

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Message send guarantees

Raimo Niskanen-2
In reply to this post by Andrey Tsirulev-2
On Fri, Jan 27, 2017 at 01:46:53PM +0300, Andrey Tsirulev wrote:

> Hello,
>
> 26.01.2017 10:14, Raimo Niskanen wrote:
> > On Wed, Jan 25, 2017 at 11:06:05PM +0000, Dmitry Kakurin wrote:
> >> No you cannot assume that. The message is considered "in transit" until the receiving process inspects it somehow.
> >>
> >> What aspect of implementation can cause such reordering? Could you give an example how this could happen? Maybe I'll be able to artificially create conditions where it would not happen. Or work around it in some other way. Please see below why.
> > On a multi-threade system it is very hard to know in which order things
> > happen, so there is no "reordering" since there is no temporal order
> > to begin with.
> What about the following example?
>
> A -- msg1 ---> A
> A -- msg2 ---> B
> B -- msg2 ---> A
>
> In this case, is it guaranteed that if both msg2 and msg1 get to A's msg
> queue, msg1 gets there earlier?

It may be that in the current and all previous implementations msg1 is read
before msg2, but it is not _guaranteed_?

We had a little discussion and the VM guys after 2 minutes started suggesting
possible implementations that could cause msg2 to be read before msg1...

http://erlang.org/doc/apps/erts/communication.html 2.1, second paragraph:

    The only signal ordering guarantee given is the following: if an entity
    sends multiple signals to the same destination entity, the order is
    preserved; that is, if A sends a signal S1 to B, and later sends signal
    S2 to B, S1 is guaranteed not to arrive after S2.

>
> If this is guaranteed, what makes this example *really* different from
> the original example?
>
> I use this pattern for delayed gen_server's initialization if it's time
> consuming, like that:
> init(Args) ->
>      self() ! {init, Args},
>     ...
> handle_info({init, Args}, State) ->
>    %% time consuming initialization here
>
> to return from init ASAP so that start/start_link do not block for a
> long time. Then I expect that after
>
> P = gen_server:start_link(...),
> P ! test
>
> when test is received, gen_server is fully initialized.  It never caused
> problems, but it might be luck.

I'd say that is luck ;-)  In this case you might get a message from any
client that arrives before your {init,Args} message.

This sounds like a use case for proc_lib:spawn..., proc_lib:init_ack/1,2,
and gen_server:enter_loop/3-5.  Using these you can send an early init_ack
to the supervisor, do your time consuming initialization, and then enter
the gen_server receive loop when you are ready for the first message.

Another way could be to use gen_statem and start in an init state where you
postpone all messages until you get your {init,Args} message (there could be
a reference() in that message just for fun), do your time consuming
initialization and change states to running.  Or, of course, design your
own event queueing before handling the {init,Args} message.

>
> Best Regards,
> Andrey
>

Best Regards
--

/ Raimo Niskanen, Erlang/OTP, Ericsson AB
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Message send guarantees

Dmitry Kakurin-2
In reply to this post by Raimo Niskanen-2

On Thu, Jan 26, 2017 at 11:41 PM, Raimo Niskanen <[hidden email]> wrote:
Have you changed your question?  Previously the problem was to send
non-proxied messages after proxied messages.
 
I've used inserting proxy in my original email (as it was the smallest setup I can distill my question to), and removing proxy in followup. Sorry about confusion. The problems are actually isomorphic and have the same solution :-).


But if B is a message relay (receive Msg -> C ! Msg, loop() end),
and provided the ping/pong is a gen_server call (since the request
contains who should have the reply, then this should also work:
1. A --msg1---> B
2. A --req----> B
3. B --msg1---> C
4. B --req----> C
5. A <--reply-- C
6. A --msg2---> C

I had the same solution in mind. Also it's a good idea and optimization to reply directly from C to A in step 5 instead of going thru B. I didn't think of that, thanks.

> Now the only problem I have to solve is: what can be used as a benign message that I can send to any gen_server such that it would reply (to emulate ping/pong exchange above)?

sys:get_status/1,2 returns much more data including what sys:get_state/1,2
returns, so it is less suitable.

If you know anything more about your gen_server C, it may have some status
quering handle_call that can be used.

Maybe sys:statistics/2 can be good enough.  Mostly no statistics collection is
activated for a given gen_server so you will get just {ok,no_statistics} back
as pong, and if statistics collection should be activated the gen_server
does not much more than process_info(self(), reductions) and erlang:localtime(),
which may and may not be regarded as heavy, but compared to returning the whole
arbitrary gen_server state it may be better.

After digging thru sys module code a little more I've settled on using sys:remove(C,nil).
It's asking to remove debug function 'nil' that is of course bogus, but returns ok quickly and is a single list.remove operation from a list that's usually empty.

Thank you, Dmitry



From: [hidden email] <[hidden email]> on behalf of Raimo Niskanen <raimo+[hidden email]>
Sent: Thursday, January 26, 2017 11:41 PM
To: [hidden email]
Subject: Re: [erlang-questions] Message send guarantees
 
On Fri, Jan 27, 2017 at 02:25:49AM +0000, Dmitry Kakurin wrote:
> From: Joe Armstrong <[hidden email]>
>
> Sent: Thursday, January 26, 2017 5:29 AM
>
> On Wed, Jan 25, 2017 at 12:38 AM, Dmitry Kakurin
> <[hidden email]> wrote:
> > When I execute "pid ! msg" and it returns, what are the guarantees if pid
> > and self are on the same node?
> > Can I assume that by the time it returns, msg is in the pid's queue? And as
> > a result all messages sent to pid afterwards will be queued after msg?
> >
> > The reason I'm asking is because I need a guarantee of proper message
> > ordering in the following scenario when all processes involved are local to
> > a node:
> > 1. A --msg1--> C (process A sends msg1 to process C)
> > 2. A --msg2--> B
> > 3. B --msg2--> C (process B simply proxies msg2 from A to C)
> >
> > I need a guarantee that msg2 will always appear in C's queue after msg1.
>
> This cannot be guaranteed
>
> Thanks Joe and others. I understand now that it's fruitless to pursue a direction where I'd try to somehow artificially establish this guarantee.
>
> I have to re-evaluate my constraints then. All along my main constraint was that C can be any gen_server, it's not owned by me and I cannot change it.
> If I could change C then the following would work:
> 1. A --msg1--> C
> 2. A --ping--> C
> 3. A <--pong-- C
> 4. A --msg2--> B
> 5. B --msg2--> C
>

Have you changed your question?  Previously the problem was to send
non-proxied messages after proxied messages.

But if B is a message relay (receive Msg -> C ! Msg, loop() end),
and provided the ping/pong is a gen_server call (since the request
contains who should have the reply, then this should also work:
1. A --msg1---> B
2. A --req----> B
3. B --msg1---> C
4. B --req----> C
5. A <--reply-- C
6. A --msg2---> C

> All we need from C is to respond "pong" to "ping" message. In this case A knows msg2 is enqueued after msg1 because A has observed "pong" in response to "ping" that was sent after msg1. The ordering is as desired here.
> Now the only problem I have to solve is: what can be used as a benign message that I can send to any gen_server such that it would reply (to emulate ping/pong exchange above)?
> Looking thru gen_server source code I see that it's processing sys messages. So I can use get_state message as "ping" and response with state as "pong". Except it may not be really benign perf-wise if amount of state kept by C is significant.
> There is also get_status, but I don't understand how it's processed and if I can use it for ping/pong purposes. Can I? Is there a better message?

sys:get_status/1,2 returns much more data including what sys:get_state/1,2
returns, so it is less suitable.

If you know anything more about your gen_server C, it may have some status
quering handle_call that can be used.

Maybe sys:statistics/2 can be good enough.  Mostly no statistics collection is
activated for a given gen_server so you will get just {ok,no_statistics} back
as pong, and if statistics collection should be activated the gen_server
does not much more than process_info(self(), reductions) and erlang:localtime(),
which may and may not be regarded as heavy, but compared to returning the whole
arbitrary gen_server state it may be better.

>
> Do you have a better idea for ping/pong messages on C? Or how to solve this in general?
>
> Thank you, Dmitry.
>
> P.S. I know that in this simplified example I could piggyback on msg1 if it returns something but in real code I cannot, so let's assume it's a one-way message (a cast, not a call). So I'm OK with ping/pong.
>


--

/ Raimo Niskanen, Erlang/OTP, Ericsson AB
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Loading...