Semantic Monitors: a proposal

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Semantic Monitors: a proposal

Guilherme Andrade
Hello list,

I've spent years writing boilerplate for associating process monitors with other identifiers and react appropriately (and performantly) when one of those processes terminates.

But: what if we could attach data to a monitor and get it back in its DOWN message?

You'd optionally associate data to a monitor when creating it:
    * Ref = monitor(process, Pid, [{data, {user_id, <<"12345">>}}])

And you'd get it back upon the monitor being triggered:
    * {'DOWN', Ref, process, Pid, Reason, {user_id, <<"12345">>}}

Regular monitors would still work the same way and keep the current DOWN message format - nothing would break.

I've explored this concept using NIFs and published the result to GitHub:
    * https://github.com/g-andrade/vaktari

Is this a good idea? If so, would it be feasible to implement it in OTP? Should there be an EEP first? I'd be willing to contribute to either.

Any input is welcome.

Cheers!

--
Guilherme
Reply | Threaded
Open this post in threaded view
|

Re: Semantic Monitors: a proposal

Michael Truog
Hi Guilherme,

I have mainly wanted to get the native monotonic time added to monitor messages because of a monitor creation option, to avoid any variation due to Erlang process message queue delays (so the monotonic time value would be stored when the DOWN message is created, equivalent to a erlang:monotonic_time/0 function call).

If it was possible to provide the data as a string, like "
erlang:monotonic_time()" to get evaluated when the DOWN message is created, that would make me happy though it may seem like an odd approach that adds extra latency (it could be a separate option).  The evaluation could be similar to the expand function below:

expand(L) ->
    {ok, S, _} = erl_scan:string(L ++ "."),
    case erl_parse:parse_term(S) of
        {ok, Term} ->
            Term;
        {error, _} ->
            badarg
    end.

I would want to avoid using a NIF outside of Erlang/OTP for this functionality, to keep all the core logic as Erlang source code.  Adding the monotonic time to a monitor DOWN message helps to ensure you have the time of the process death in a consistent way where the time isn't being influenced by the Erlang process receiving the DOWN message (which remains completely separate and unrelated to the process that died).

Best Regards,
Michael

On 2/16/21 9:32 AM, Guilherme Andrade wrote:
Hello list,

I've spent years writing boilerplate for associating process monitors with other identifiers and react appropriately (and performantly) when one of those processes terminates.

But: what if we could attach data to a monitor and get it back in its DOWN message?

You'd optionally associate data to a monitor when creating it:
    * Ref = monitor(process, Pid, [{data, {user_id, <<"12345">>}}])

And you'd get it back upon the monitor being triggered:
    * {'DOWN', Ref, process, Pid, Reason, {user_id, <<"12345">>}}

Regular monitors would still work the same way and keep the current DOWN message format - nothing would break.

I've explored this concept using NIFs and published the result to GitHub:
    * https://github.com/g-andrade/vaktari

Is this a good idea? If so, would it be feasible to implement it in OTP? Should there be an EEP first? I'd be willing to contribute to either.

Any input is welcome.

Cheers!

--
Guilherme

Reply | Threaded
Open this post in threaded view
|

Re: Semantic Monitors: a proposal

Michael Truog
Using a string this way wouldn't be a good approach due to a lack of visibility and quick evaluation (the expand function only works for data, not evaluating functions).  Providing an anonymous function for evaluation would be the proper approach:  fun erlang:monotonic_time/0.  So, I am suggesting allowing function evaluation when the DOWN message is created unless the monotonic time could be added to the DOWN message due to a separate option.


On 2/16/21 1:14 PM, Michael Truog wrote:
Hi Guilherme,

I have mainly wanted to get the native monotonic time added to monitor messages because of a monitor creation option, to avoid any variation due to Erlang process message queue delays (so the monotonic time value would be stored when the DOWN message is created, equivalent to a erlang:monotonic_time/0 function call).

If it was possible to provide the data as a string, like "
erlang:monotonic_time()" to get evaluated when the DOWN message is created, that would make me happy though it may seem like an odd approach that adds extra latency (it could be a separate option).  The evaluation could be similar to the expand function below:

expand(L) ->
    {ok, S, _} = erl_scan:string(L ++ "."),
    case erl_parse:parse_term(S) of
        {ok, Term} ->
            Term;
        {error, _} ->
            badarg
    end.

I would want to avoid using a NIF outside of Erlang/OTP for this functionality, to keep all the core logic as Erlang source code.  Adding the monotonic time to a monitor DOWN message helps to ensure you have the time of the process death in a consistent way where the time isn't being influenced by the Erlang process receiving the DOWN message (which remains completely separate and unrelated to the process that died).

Best Regards,
Michael

On 2/16/21 9:32 AM, Guilherme Andrade wrote:
Hello list,

I've spent years writing boilerplate for associating process monitors with other identifiers and react appropriately (and performantly) when one of those processes terminates.

But: what if we could attach data to a monitor and get it back in its DOWN message?

You'd optionally associate data to a monitor when creating it:
    * Ref = monitor(process, Pid, [{data, {user_id, <<"12345">>}}])

And you'd get it back upon the monitor being triggered:
    * {'DOWN', Ref, process, Pid, Reason, {user_id, <<"12345">>}}

Regular monitors would still work the same way and keep the current DOWN message format - nothing would break.

I've explored this concept using NIFs and published the result to GitHub:
    * https://github.com/g-andrade/vaktari

Is this a good idea? If so, would it be feasible to implement it in OTP? Should there be an EEP first? I'd be willing to contribute to either.

Any input is welcome.

Cheers!

--
Guilherme


Reply | Threaded
Open this post in threaded view
|

Re: Semantic Monitors: a proposal

Lukas Larsson-8
In reply to this post by Guilherme Andrade
Hello!


On Tue, Feb 16, 2021 at 7:23 PM Guilherme Andrade <[hidden email]> wrote:
Hello list,

I've spent years writing boilerplate for associating process monitors with other identifiers and react appropriately (and performantly) when one of those processes terminates.

But: what if we could attach data to a monitor and get it back in its DOWN message?

You'd optionally associate data to a monitor when creating it:
    * Ref = monitor(process, Pid, [{data, {user_id, <<"12345">>}}])

And you'd get it back upon the monitor being triggered:
    * {'DOWN', Ref, process, Pid, Reason, {user_id, <<"12345">>}}

Regular monitors would still work the same way and keep the current DOWN message format - nothing would break.

I've explored this concept using NIFs and published the result to GitHub:
    * https://github.com/g-andrade/vaktari

Is this a good idea? If so, would it be feasible to implement it in OTP? Should there be an EEP first? I'd be willing to contribute to either.

You are in luck!

PR-2735[1] aka EEP-53[2] adds erlang:monitor/3[3] which can be used to do what you want. It will be released in Erlang/OTP 24.

 

Any input is welcome.

Cheers!

--
Guilherme
Reply | Threaded
Open this post in threaded view
|

Re: Semantic Monitors: a proposal

Viktor Söderqvist
Hello!

On 2021-02-17 08:07, Lukas Larsson wrote:
> You are in luck!
>
> PR-2735[1] aka EEP-53[2] adds erlang:monitor/3[3] which can be used to
> do what you want. It will be released in Erlang/OTP 24.
>
> [1]: https://github.com/erlang/otp/pull/2735 
> [2]: https://github.com/erlang/eep/blob/master/eeps/eep-0053.md 
> [3]:
> https://erlang.org/~rickard/OTP-16718/erts-11.1.1/doc/html/erlang.html#monitor-3 

Nice! But Michael's use case would require an extra monitor option.

On 2/16/21 1:14 PM, Michael Truog wrote:
 > If it was possible to provide the data as a string, like
 > "erlang:monotonic_time()" to get evaluated when the DOWN message
 > is created, that would make me happy though it may seem like an
 > odd approach that adds extra latency (it could be a separate option).

For example {tag_fun, fun() -> {'DOWN', erlang:monotonic_time()} end} or
so, to create the tag at the time of demotinor time. I suppose there
will still be a short time between the death of the process and the
creation of the tag though.

Viktor
Reply | Threaded
Open this post in threaded view
|

Re: Semantic Monitors: a proposal

Lukas Larsson-8


On Wed, Feb 17, 2021 at 10:34 AM Viktor Söderqvist <[hidden email]> wrote:
Hello!

On 2021-02-17 08:07, Lukas Larsson wrote:
> You are in luck!
>
> PR-2735[1] aka EEP-53[2] adds erlang:monitor/3[3] which can be used to
> do what you want. It will be released in Erlang/OTP 24.
>
> [1]: https://github.com/erlang/otp/pull/2735
> [2]: https://github.com/erlang/eep/blob/master/eeps/eep-0053.md
> [3]:
> https://erlang.org/~rickard/OTP-16718/erts-11.1.1/doc/html/erlang.html#monitor-3

Nice! But Michael's use case would require an extra monitor option.

On 2/16/21 1:14 PM, Michael Truog wrote:
 > If it was possible to provide the data as a string, like
 > "erlang:monotonic_time()" to get evaluated when the DOWN message
 > is created, that would make me happy though it may seem like an
 > odd approach that adds extra latency (it could be a separate option).

For example {tag_fun, fun() -> {'DOWN', erlang:monotonic_time()} end} or
so, to create the tag at the time of demotinor time. I suppose there
will still be a short time between the death of the process and the
creation of the tag though.

I don't think we want to enable executing code in the context of an already dead or dying process. What happens if the code does a gen_server:call? What happens if it does gen_server:enter_loop? Could a process be resurrected?

There are also a lot of assumptions within the VM that a dead process does not execute code, so it would a lot of work to do.


Viktor
Reply | Threaded
Open this post in threaded view
|

Re: Semantic Monitors: a proposal

Michael Truog
On 2/17/21 2:03 AM, Lukas Larsson wrote:


On Wed, Feb 17, 2021 at 10:34 AM Viktor Söderqvist <[hidden email]> wrote:
Hello!

On 2021-02-17 08:07, Lukas Larsson wrote:
> You are in luck!
>
> PR-2735[1] aka EEP-53[2] adds erlang:monitor/3[3] which can be used to
> do what you want. It will be released in Erlang/OTP 24.
>
> [1]: https://github.com/erlang/otp/pull/2735
> [2]: https://github.com/erlang/eep/blob/master/eeps/eep-0053.md
> [3]:
> https://erlang.org/~rickard/OTP-16718/erts-11.1.1/doc/html/erlang.html#monitor-3

Nice! But Michael's use case would require an extra monitor option.

On 2/16/21 1:14 PM, Michael Truog wrote:
 > If it was possible to provide the data as a string, like
 > "erlang:monotonic_time()" to get evaluated when the DOWN message
 > is created, that would make me happy though it may seem like an
 > odd approach that adds extra latency (it could be a separate option).

For example {tag_fun, fun() -> {'DOWN', erlang:monotonic_time()} end} or
so, to create the tag at the time of demotinor time. I suppose there
will still be a short time between the death of the process and the
creation of the tag though.

I don't think we want to enable executing code in the context of an already dead or dying process. What happens if the code does a gen_server:call? What happens if it does gen_server:enter_loop? Could a process be resurrected?

There are also a lot of assumptions within the VM that a dead process does not execute code, so it would a lot of work to do.
Is there interest in adding the native monotonic time to the monitor DOWN message due to a separate option?  It should be beneficial to have a more accurate time-of-process-death value than is currently possible when the DOWN message is received.

Best Regards,
Michael

Reply | Threaded
Open this post in threaded view
|

Re: Semantic Monitors: a proposal

Roger Lipscombe-2
On Wed, 17 Feb 2021 at 10:28, Michael Truog <[hidden email]> wrote:
> It should be beneficial to have a more accurate time-of-process-death value than is currently possible when the DOWN message is received.

Why? What's the use case here?
Reply | Threaded
Open this post in threaded view
|

Re: Semantic Monitors: a proposal

Michael Truog
On 2/17/21 2:40 AM, Roger Lipscombe wrote:
> On Wed, 17 Feb 2021 at 10:28, Michael Truog <[hidden email]> wrote:
>> It should be beneficial to have a more accurate time-of-process-death value than is currently possible when the DOWN message is received.
> Why? What's the use case here?
Anytime you want to understand the lifetime of an Erlang process you
would want an accurate understanding of when it is first created and
when it dies.  When it is created (spawned) isn't difficult, it can be
the monotonic time immediately before the spawn.  However, the process
death time is currently at the mercy of the Erlang process that owns its
monitor, when considering the delay of the DOWN message spent in the
message queue.  So, when I think of a use-case, I think of
https://github.com/CloudI/CloudI/blob/228d09fe64e86f1316221de514482a82486e1034/src/lib/cloudi_core/src/cloudi_core_i_services_monitor.erl#L585 
.

The current restart time is a time after the DOWN message is received,
though I would prefer to know when the death of the process really
occurred to have a better understanding of the Erlang process "uptime". 
By that, I mean, how long did this particular Erlang process really have
doing Erlang process things, not the extra latency related to other
Erlang processes like the one that had a monitor and received the DOWN
message.  So, a more accurate Erlang process time could be used in other
Erlang source code.  The Erlang/OTP supervisor is currently relying on
link/trap_exit for the restart, so that wouldn't benefit unless a
separate trap_exit message was possible, assuming there was motivation
to modify the supervisor restart time (to avoid any potential delay
spent due to the 'EXIT' message in the supervisor's message queue).

In CloudI, the services_status CloudI Service API function provides the
various time related information about the lifetime of CloudI services. 
So, for that source code I would prefer to have the most accurate
monotonic time values possible, to ensure the service lifetime data is
accurate.

I understand an argument against a monitor option would be to create a
separate Erlang process to own each monitor, to avoid having extra
messages in the message queue.  However, I don't think that is a
realistic solution and would only make things more complex than they
need to be.

Best Regards,
Michael
Reply | Threaded
Open this post in threaded view
|

Re: Semantic Monitors: a proposal

Max Lapshin-2
In reply to this post by Lukas Larsson-8
Excellent feature!

On Wed, Feb 17, 2021 at 10:07 AM Lukas Larsson <[hidden email]> wrote:

>
> Hello!
>
>
> On Tue, Feb 16, 2021 at 7:23 PM Guilherme Andrade <[hidden email]> wrote:
>>
>> Hello list,
>>
>> I've spent years writing boilerplate for associating process monitors with other identifiers and react appropriately (and performantly) when one of those processes terminates.
>>
>> But: what if we could attach data to a monitor and get it back in its DOWN message?
>>
>> You'd optionally associate data to a monitor when creating it:
>>     * Ref = monitor(process, Pid, [{data, {user_id, <<"12345">>}}])
>>
>> And you'd get it back upon the monitor being triggered:
>>     * {'DOWN', Ref, process, Pid, Reason, {user_id, <<"12345">>}}
>>
>> Regular monitors would still work the same way and keep the current DOWN message format - nothing would break.
>>
>> I've explored this concept using NIFs and published the result to GitHub:
>>     * https://github.com/g-andrade/vaktari
>>
>> Is this a good idea? If so, would it be feasible to implement it in OTP? Should there be an EEP first? I'd be willing to contribute to either.
>
>
> You are in luck!
>
> PR-2735[1] aka EEP-53[2] adds erlang:monitor/3[3] which can be used to do what you want. It will be released in Erlang/OTP 24.
>
> [1]: https://github.com/erlang/otp/pull/2735
> [2]: https://github.com/erlang/eep/blob/master/eeps/eep-0053.md
> [3]: https://erlang.org/~rickard/OTP-16718/erts-11.1.1/doc/html/erlang.html#monitor-3
>
>>
>>
>> Any input is welcome.
>>
>> Cheers!
>>
>> --
>> Guilherme
Reply | Threaded
Open this post in threaded view
|

Re: Semantic Monitors: a proposal

Guilherme Andrade
In reply to this post by Lukas Larsson-8


On Wed, 17 Feb 2021 at 07:07, Lukas Larsson <[hidden email]> wrote:
Hello!


On Tue, Feb 16, 2021 at 7:23 PM Guilherme Andrade <[hidden email]> wrote:
Hello list,

I've spent years writing boilerplate for associating process monitors with other identifiers and react appropriately (and performantly) when one of those processes terminates.

But: what if we could attach data to a monitor and get it back in its DOWN message?

You'd optionally associate data to a monitor when creating it:
    * Ref = monitor(process, Pid, [{data, {user_id, <<"12345">>}}])

And you'd get it back upon the monitor being triggered:
    * {'DOWN', Ref, process, Pid, Reason, {user_id, <<"12345">>}}

Regular monitors would still work the same way and keep the current DOWN message format - nothing would break.

I've explored this concept using NIFs and published the result to GitHub:
    * https://github.com/g-andrade/vaktari

Is this a good idea? If so, would it be feasible to implement it in OTP? Should there be an EEP first? I'd be willing to contribute to either.

You are in luck!

PR-2735[1] aka EEP-53[2] adds erlang:monitor/3[3] which can be used to do what you want. It will be released in Erlang/OTP 24.

This is great news! Can't wait to use it :-)

 


--
Guilherme