Abusing OTP-8623 for high-priority messages

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Abusing OTP-8623 for high-priority messages

Dmitry Demeshchuk-2
Hi list,

Today (yeah, just a couple years have passed since R14A) I learned
about OTP-8623, which allows to do selective receive for messages that
contain a known reference with O(1) complexity.

Which made me think: what if we make a gen_server'ish process that:
1. Creates a reference that is being passed along with the pid, meaning,
our start_link returns {ok, {Pid, ManageRef}} or something like that.
ManageRef is actually created within the new process inside init/1 or such.
2. Introduce an operation named priority_call, which, basically, sends
something like {ManageRef, {Pid, Ref}, Msg}.
3. The process itself (on its own side) does a selective receive like that:

receive
    {ManageRef, From, Msg} -> handle_call(From, Msg, State)
after 0 ->
    receive
        Msg -> gen_server_like_dispatch_msg(Msg)
    end
end

This could potentially help with sending managing messages to the processes
that may have overloaded inbox with thousands of messages. Say, we see that
the process gets overloaded and we send an operational message of some kind
("discard the entire inbox", for example).

Thoughts?

--
Best regards,
Dmitry Demeshchuk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140410/d04a093d/attachment.html>

Reply | Threaded
Open this post in threaded view
|

Abusing OTP-8623 for high-priority messages

Michael Truog-2
On 04/10/2014 05:02 PM, Dmitry Demeshchuk wrote:

> Hi list,
>
> Today (yeah, just a couple years have passed since R14A) I learned about OTP-8623, which allows to do selective receive for messages that contain a known reference with O(1) complexity.
>
> Which made me think: what if we make a gen_server'ish process that:
> 1. Creates a reference that is being passed along with the pid, meaning, our start_link returns {ok, {Pid, ManageRef}} or something like that. ManageRef is actually created within the new process inside init/1 or such.
> 2. Introduce an operation named priority_call, which, basically, sends something like {ManageRef, {Pid, Ref}, Msg}.
> 3. The process itself (on its own side) does a selective receive like that:
>
> receive
>     {ManageRef, From, Msg} -> handle_call(From, Msg, State)
> after 0 ->
>     receive
>         Msg -> gen_server_like_dispatch_msg(Msg)
>     end
> end
>
> This could potentially help with sending managing messages to the processes that may have overloaded inbox with thousands of messages. Say, we see that the process gets overloaded and we send an operational message of some kind ("discard the entire inbox", for example).
>
> Thoughts?
You could get this interaction by creating an internal service with CloudI (http://cloudi.org).  That means you just use the cloudi_service behaviour for the service request receiver.  You can use the cloudi module to send from normal Erlang pids, or use the cloudi_service behaviour to send from a second internal service. Every CloudI service requests provides a unique v1 UUID to identify the service request, a priority (defaulting to 0, but between -128 (high) and 127 (low), inclusive), and the source Erlang pid, in addition to the two payloads RequestInfo and Request (where RequestInfo is generally key/value message meta-data and Request is the main data, both are any type when used with internal services). The priority of the service request impacts the queuing of the received service request, to determine the order service requests get processed by a service.  There is a queue_limit service configuration option which can be provided for any CloudI service to limit the
number of pending service requests (i.e. unhandled) that may be stored (http://cloudi.org/api.html#2_services_add).  So, the same use-case you have described with added features that includes details for high-availability and partition tolerance.

>
> --
> Best regards,
> Dmitry Demeshchuk
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions
> http://erlang.org/mailman/listinfo/erlang-questions

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140410/0c118188/attachment.html>

Reply | Threaded
Open this post in threaded view
|

Abusing OTP-8623 for high-priority messages

zxq9
On Thursday 10 April 2014 17:24:54 Michael Truog wrote:
> On 04/10/2014 05:02 PM, Dmitry Demeshchuk wrote:
> > Hi list,
> >
> > Today (yeah, just a couple years have passed since R14A) I learned about
> > OTP-8623, which allows to do selective receive for messages that contain
> > a known reference with O(1) complexity.
...
> > This could potentially help with sending managing messages to the
> > processes that may have overloaded inbox with thousands of messages. Say,
> > we see that the process gets overloaded and we send an operational
> > message of some kind ("discard the entire inbox", for example).
> >
> > Thoughts?
>
> You could get this interaction by creating an internal service with CloudI
> (http://cloudi.org).

Not so sure this is what the OP was getting at.

"Is OTP8623 a lightweight way to overcome a clogged queue?" /= "Base
everything you do on some new service stack"

Reply | Threaded
Open this post in threaded view
|

Abusing OTP-8623 for high-priority messages

Dmitry Demeshchuk-2
Well, actually, if Michael pointed me at a module that handles that and it
was something similar, that could have been helpful :)

But, judging by the priorities from -128 to 127, it's probably a manual
filter of some kind, probably with an intermediate mailbox-reducer process.

Which is fine, actually, but my question wasn't "how to handle huge
queues?", but "is using OTP-8623 for that an anti-pattern of some sort?".

And yeah, definitely not "what platform should I use?".


On Thu, Apr 10, 2014 at 7:28 PM, zxq9 <zxq9> wrote:

> On Thursday 10 April 2014 17:24:54 Michael Truog wrote:
> > On 04/10/2014 05:02 PM, Dmitry Demeshchuk wrote:
> > > Hi list,
> > >
> > > Today (yeah, just a couple years have passed since R14A) I learned
> about
> > > OTP-8623, which allows to do selective receive for messages that
> contain
> > > a known reference with O(1) complexity.
> ...
> > > This could potentially help with sending managing messages to the
> > > processes that may have overloaded inbox with thousands of messages.
> Say,
> > > we see that the process gets overloaded and we send an operational
> > > message of some kind ("discard the entire inbox", for example).
> > >
> > > Thoughts?
> >
> > You could get this interaction by creating an internal service with
> CloudI
> > (http://cloudi.org).
>
> Not so sure this is what the OP was getting at.
>
> "Is OTP8623 a lightweight way to overcome a clogged queue?" /= "Base
> everything you do on some new service stack"
> _______________________________________________
> erlang-questions mailing list
> erlang-questions
> http://erlang.org/mailman/listinfo/erlang-questions
>



--
Best regards,
Dmitry Demeshchuk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140410/f82af0bb/attachment.html>

Reply | Threaded
Open this post in threaded view
|

Abusing OTP-8623 for high-priority messages

Michael Truog-2
On 04/10/2014 08:56 PM, Dmitry Demeshchuk wrote:
> Well, actually, if Michael pointed me at a module that handles that and it was something similar, that could have been helpful :)
>
> But, judging by the priorities from -128 to 127, it's probably a manual filter of some kind, probably with an intermediate mailbox-reducer process.
The v1 UUID acts as the known reference with O(1) complexity here. The priority is being handled by an Erlang priority queue data structure called pqueue4 since there has to be some queue structure for queuing service requests so that atomic transactions can occur for service requests.  This might not have been the priority you had in mind if you were thinking about various pattern matching cases on different priority values, but I have found the approach with a data structure more flexible and useful for the dynamic nature of a service, due to the need for a queue (which enforces the incoming limit, since it is easier than trying to do something error-prone with an Erlang pid directly, i.e., something that would modify the Erlang pid state or try to check it repeatedly).

>
> Which is fine, actually, but my question wasn't "how to handle huge queues?", but "is using OTP-8623 for that an anti-pattern of some sort?".
>
> And yeah, definitely not "what platform should I use?".
>
>
> On Thu, Apr 10, 2014 at 7:28 PM, zxq9 <zxq9 <mailto:zxq9>> wrote:
>
>     On Thursday 10 April 2014 17:24:54 Michael Truog wrote:
>     > On 04/10/2014 05:02 PM, Dmitry Demeshchuk wrote:
>     > > Hi list,
>     > >
>     > > Today (yeah, just a couple years have passed since R14A) I learned about
>     > > OTP-8623, which allows to do selective receive for messages that contain
>     > > a known reference with O(1) complexity.
>     ...
>     > > This could potentially help with sending managing messages to the
>     > > processes that may have overloaded inbox with thousands of messages. Say,
>     > > we see that the process gets overloaded and we send an operational
>     > > message of some kind ("discard the entire inbox", for example).
>     > >
>     > > Thoughts?
>     >
>     > You could get this interaction by creating an internal service with CloudI
>     > (http://cloudi.org).
>
>     Not so sure this is what the OP was getting at.
>
>     "Is OTP8623 a lightweight way to overcome a clogged queue?" /= "Base
>     everything you do on some new service stack"
>     _______________________________________________
>     erlang-questions mailing list
>     erlang-questions <mailto:erlang-questions>
>     http://erlang.org/mailman/listinfo/erlang-questions
>
>
>
>
> --
> Best regards,
> Dmitry Demeshchuk
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions
> http://erlang.org/mailman/listinfo/erlang-questions

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140410/db4c3ace/attachment.html>

Reply | Threaded
Open this post in threaded view
|

Abusing OTP-8623 for high-priority messages

Peti Gömöri
Hi

As far as I understand OTP-8623 how it works is it puts a marker in the
message queue when a reference is created (and other special conditions
apply) and in case of a selective receive which contains that reference
only the messages received after that marker are scanned. This makes sense
as messages that were sent before the reference was created cannot contain
that reference.

Considering this it makes no sense to use a reference that was created in
init since all (or almost all) messages will be sent after init.

The readme stating "constant time" is at least a bit misleading. It
probably only refers to the fact that no matter how long your message queue
already is if you create a reference now and you only expect to receive one
or few new messages after this it will be cheap to find that message in the
queue. It will still scan the new messages in sequence, no hashing or
anything.

I might be wrong of course

br
Peter


On Fri, Apr 11, 2014 at 7:50 AM, Michael Truog <mjtruog> wrote:

>  On 04/10/2014 08:56 PM, Dmitry Demeshchuk wrote:
>
> Well, actually, if Michael pointed me at a module that handles that and it
> was something similar, that could have been helpful :)
>
>  But, judging by the priorities from -128 to 127, it's probably a manual
> filter of some kind, probably with an intermediate mailbox-reducer process.
>
> The v1 UUID acts as the known reference with O(1) complexity here.  The
> priority is being handled by an Erlang priority queue data structure called
> pqueue4 since there has to be some queue structure for queuing service
> requests so that atomic transactions can occur for service requests.  This
> might not have been the priority you had in mind if you were thinking about
> various pattern matching cases on different priority values, but I have
> found the approach with a data structure more flexible and useful for the
> dynamic nature of a service, due to the need for a queue (which enforces
> the incoming limit, since it is easier than trying to do something
> error-prone with an Erlang pid directly, i.e., something that would modify
> the Erlang pid state or try to check it repeatedly).
>
>
>
>  Which is fine, actually, but my question wasn't "how to handle huge
> queues?", but "is using OTP-8623 for that an anti-pattern of some sort?".
>
>  And yeah, definitely not "what platform should I use?".
>
>
> On Thu, Apr 10, 2014 at 7:28 PM, zxq9 <zxq9> wrote:
>
>> On Thursday 10 April 2014 17:24:54 Michael Truog wrote:
>> > On 04/10/2014 05:02 PM, Dmitry Demeshchuk wrote:
>> > > Hi list,
>> > >
>> > > Today (yeah, just a couple years have passed since R14A) I learned
>> about
>> > > OTP-8623, which allows to do selective receive for messages that
>> contain
>> > > a known reference with O(1) complexity.
>>  ...
>> > > This could potentially help with sending managing messages to the
>> > > processes that may have overloaded inbox with thousands of messages.
>> Say,
>> > > we see that the process gets overloaded and we send an operational
>> > > message of some kind ("discard the entire inbox", for example).
>> > >
>> > > Thoughts?
>> >
>> > You could get this interaction by creating an internal service with
>> CloudI
>> > (http://cloudi.org).
>>
>>  Not so sure this is what the OP was getting at.
>>
>> "Is OTP8623 a lightweight way to overcome a clogged queue?" /= "Base
>> everything you do on some new service stack"
>>  _______________________________________________
>> erlang-questions mailing list
>> erlang-questions
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>
>
>
>  --
> Best regards,
> Dmitry Demeshchuk
>
>
> _______________________________________________
> erlang-questions mailing listerlang-questions://erlang.org/mailman/listinfo/erlang-questions
>
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140411/68fc6808/attachment.html>

Reply | Threaded
Open this post in threaded view
|

Abusing OTP-8623 for high-priority messages

Björn Gustavsson-3
In reply to this post by Dmitry Demeshchuk-2
On Fri, Apr 11, 2014 at 2:02 AM, Dmitry Demeshchuk <demeshchuk> wrote:
> Hi list,
>
> Today (yeah, just a couple years have passed since R14A) I learned about
> OTP-8623, which allows to do selective receive for messages that contain a
> known reference with O(1) complexity.

Yes, that is basically how it was described in
the release note and how it works in practice.

It is implemented by not looking at the first
part of the message queue that contain
messages received before the reference
was created. In practice, there should only
be a small number of new messages
received after the creation of the reference,
so in practice we have O(1) complexity for
reading out the message.

> Which made me think: what if we make a gen_server'ish process that:
> 1. Creates a reference that is being passed along with the pid, meaning, our
> start_link returns {ok, {Pid, ManageRef}} or something like that. ManageRef
> is actually created within the new process inside init/1 or such.
> 2. Introduce an operation named priority_call, which, basically, sends
> something like {ManageRef, {Pid, Ref}, Msg}.

Apart from the fact that the optimization cannot
at all be applied in this context because it is a
compiler optimization, it would not help at all
because the reference is created when the
process queue is empty, so there are no messages
that can be skipped.

/Bjorn
--
Bj?rn Gustavsson, Erlang/OTP, Ericsson AB

Reply | Threaded
Open this post in threaded view
|

Abusing OTP-8623 for high-priority messages

Tony Rogvall-3
In reply to this post by Dmitry Demeshchuk-2
Reacting (as always) when the word abuse is tagged in the subject line :-)

Reading the comments in beam_receive.erl (compiler) the layout of the code
using this "selective" feature must have the following layout:

1) Ref= make_ref(),    %% or monitor(process, Pid) / spawn_monitor(..)
...
receive
   2) _Mesg = {Ref, Data} -> do_some_stuff()
end.

The compiler will only put out the code you want (recv_mark) if it follows this pattern.
The actual reference created in 1) must be matched in a syntactic pattern
that contain the variable 'Ref'. The beam_receive optimization is done pretty
late in the compiler process so it is a bit more complicated that that.

Example

-module(beam_prio_msg).
-compile(export_all).

test1() ->
    Ref = make_ref(),
    receive
        {Ref, x} -> x
    after
        100 -> timeout
    end.

test2() ->
    Ref = make_ref(),
    Ref1 = Ref,
    receive
        {Ref1, x} -> x
    after
        100 -> timeout
    end.

test3() ->
    Ref = make_ref(),
    Ref1 = {Ref},
    receive
        {Ref1, x} -> x
    after
        100 -> timeout
    end.

Both test1 and test2 put out the (recv_mark) instruction even though test2 is matching on an
other variable Ref1. But test3 will not put out the (recv_mark) instruction, since the reference is put
into an other term and therefor is potentially not in a register anymore.

Anyway. If your plan is to work at all the code fragment must somehow send a message somewhere
saying that "now it is time to send a priority message to me" and then wait for that message.

PrioRef = make_ref(),
ManagerPid ! {send_me_prio_request_with_this_ref, PrioRef},

receive
    {PrioRef, From, Msg} -> handle_call(From, Msg, State)
after 0 ->
    receive
        Msg -> gen_server_like_dispatch_msg(Msg)
    end
end

This plan could almost work (theoretically), but would spam ManagerPid for every call to probe for
high prio message. And you surly add latency since you need to wait more than 0
millis for a high prio response.

In other words, with my current mindset, the scheme is not really working.
But with some new and fresh input this may inspire to even more abuse :-)

/Tony


On 11 apr 2014, at 02:02, Dmitry Demeshchuk <demeshchuk> wrote:

> Hi list,
>
> Today (yeah, just a couple years have passed since R14A) I learned about OTP-8623, which allows to do selective receive for messages that contain a known reference with O(1) complexity.
>
> Which made me think: what if we make a gen_server'ish process that:
> 1. Creates a reference that is being passed along with the pid, meaning, our start_link returns {ok, {Pid, ManageRef}} or something like that. ManageRef is actually created within the new process inside init/1 or such.
> 2. Introduce an operation named priority_call, which, basically, sends something like {ManageRef, {Pid, Ref}, Msg}.
> 3. The process itself (on its own side) does a selective receive like that:
>
> receive
>     {ManageRef, From, Msg} -> handle_call(From, Msg, State)
> after 0 ->
>     receive
>         Msg -> gen_server_like_dispatch_msg(Msg)
>     end
> end
>
> This could potentially help with sending managing messages to the processes that may have overloaded inbox with thousands of messages. Say, we see that the process gets overloaded and we send an operational message of some kind ("discard the entire inbox", for example).
>
> Thoughts?
>
> --
> Best regards,
> Dmitry Demeshchuk
> _______________________________________________
> erlang-questions mailing list
> erlang-questions
> http://erlang.org/mailman/listinfo/erlang-questions

"Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix"



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140411/cd4624c8/attachment.html>

Reply | Threaded
Open this post in threaded view
|

Abusing OTP-8623 for high-priority messages

Ulf Wiger-5

On 11 Apr 2014, at 11:04, Tony Rogvall <tony> wrote:

> This plan could almost work (theoretically), but would spam ManagerPid for every call to probe for
> high prio message. And you surly add latency since you need to wait more than 0
> millis for a high prio response.
>
> In other words, with my current mindset, the scheme is not really working.
> But with some new and fresh input this may inspire to even more abuse :-)

This is how to do it, of course:   ;-)

PrioRef = make_ref(),
ManagerPid ! {send_me_prio_request_with_this_ref, PrioRef},
erlang:hibernate(
    fun() ->
        receive
            {PrioRef, From, Msg} -> handle_call(From, Msg, State)
        after 0 ->
            receive
                Msg -> gen_server_like_dispatch_msg(Msg)
        end
    end).

Of course, hibernate/1 doesn?t exist, and the compiler would have
to make the fun ?inherit? the receive mark.

BR,
Ulf

Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc.
http://feuerlabs.com



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140411/0d491ec9/attachment.html>

Reply | Threaded
Open this post in threaded view
|

Abusing OTP-8623 for high-priority messages

Tony Rogvall-3
Now we are getting somewhere !
Good thinking ulf.

:-)

/Tony

On 11 apr 2014, at 12:52, Ulf Wiger <ulf> wrote:

>
> On 11 Apr 2014, at 11:04, Tony Rogvall <tony> wrote:
>
>> This plan could almost work (theoretically), but would spam ManagerPid for every call to probe for
>> high prio message. And you surly add latency since you need to wait more than 0
>> millis for a high prio response.
>>
>> In other words, with my current mindset, the scheme is not really working.
>> But with some new and fresh input this may inspire to even more abuse :-)
>
> This is how to do it, of course:   ;-)
>
> PrioRef = make_ref(),
> ManagerPid ! {send_me_prio_request_with_this_ref, PrioRef},
> erlang:hibernate(
>     fun() ->
>         receive
>             {PrioRef, From, Msg} -> handle_call(From, Msg, State)
>         after 0 ->
>             receive
>                 Msg -> gen_server_like_dispatch_msg(Msg)
>         end
>     end).
>
> Of course, hibernate/1 doesn?t exist, and the compiler would have
> to make the fun ?inherit? the receive mark.
>
> BR,
> Ulf
>
> Ulf Wiger, Co-founder & Developer Advocate, Feuerlabs Inc.
> http://feuerlabs.com
>
>
>

"Installing applications can lead to corruption over time. Applications gradually write over each other's libraries, partial upgrades occur, user and system errors happen, and minute changes may be unnoticeable and difficult to fix"



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140411/3aed0ffd/attachment.html>