Obtain message content of Erlang messages in dtrace

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Obtain message content of Erlang messages in dtrace

Duncan Paul Attard
Hi,

I was experimenting with Erlang and dtrace, and am interested to know whether the message content exchanged between two Erlang processes can be obtained. In particular, I am interested in the `message-send` and `message-receive` probes.

I looked at erlang_dtrace.d (https://github.com/erlang/otp/blob/master/erts/emulator/beam/erlang_dtrace.d) as well as messages.d (https://github.com/erlang/otp/blob/master/lib/runtime_tools/examples/messages.d) to see whether this is possible, but I was not able to make any progress.

Is there a way in which this can be achieved. And if not, are there any alternatives?

Thanks,
Duncan
Reply | Threaded
Open this post in threaded view
|

Re: Obtain message content of Erlang messages in dtrace

Duncan Paul Attard
Thanks a lot!

Yes, currently I’m using erlang:trace/3, but was curious whether the same thing can be achieved `externally` via dtrace (or even LTTng).

Does `Dtrace is meant more to observe VM itself, not to trace you application.` mean that the message content cannot be retrieved though?

Duncan


> On 16 Dec 2019, at 13:10, Łukasz Niemier <[hidden email]> wrote:
>
>> I was experimenting with Erlang and dtrace, and am interested to know whether the message content exchanged between two Erlang processes can be obtained. In particular, I am interested in the `message-send` and `message-receive` probes.
>
> Dtrace is meant more to observe VM itself, not to trace you application.
>
>> Is there a way in which this can be achieved. And if not, are there any alternatives?
>
> For tracing your own application you have erlang:trace/3, seq_trace, dbg, dyntrace, etc. If you are more interested in tracing and debugging applications in Erlang then you should check https://www.erlang-in-anger.com
>
> --
>
> Łukasz Niemier
> [hidden email]
>
>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Obtain message content of Erlang messages in dtrace

Duncan Paul Attard
Hello,

Just wanted to check on this.


> On 16 Dec 2019, at 13:16, Duncan Paul Attard <[hidden email]> wrote:
>
> Thanks a lot!
>
> Yes, currently I’m using erlang:trace/3, but was curious whether the same thing can be achieved `externally` via dtrace (or even LTTng).
>
> Does `Dtrace is meant more to observe VM itself, not to trace you application.` mean that the message content cannot be retrieved though?
>
> Duncan
>
>
>> On 16 Dec 2019, at 13:10, Łukasz Niemier <[hidden email]> wrote:
>>
>>> I was experimenting with Erlang and dtrace, and am interested to know whether the message content exchanged between two Erlang processes can be obtained. In particular, I am interested in the `message-send` and `message-receive` probes.
>>
>> Dtrace is meant more to observe VM itself, not to trace you application.
>>
>>> Is there a way in which this can be achieved. And if not, are there any alternatives?
>>
>> For tracing your own application you have erlang:trace/3, seq_trace, dbg, dyntrace, etc. If you are more interested in tracing and debugging applications in Erlang then you should check https://www.erlang-in-anger.com
>>
>> --
>>
>> Łukasz Niemier
>> [hidden email]
>>
>>
>>
>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Obtain message content of Erlang messages in dtrace

Harris, Robert
In reply to this post by Duncan Paul Attard


> On 16 Dec 2019, at 12:02, Duncan Paul Attard <[hidden email]> wrote:
>
> I was experimenting with Erlang and dtrace, and am interested to know whether the message content exchanged between two Erlang processes can be obtained. In particular, I am interested in the `message-send` and `message-receive` probes.

TL;DR you can use DTrace but you'll have to modify the Erlang VM.

DTrace was intended to provide observation at the very lowest level.  In
the user land case, a process is viewed as a collection of load objects
whose symbol tables describe functions and global variables.  DTrace is
organised as a framework of "providers" that expose certain events ---
locations in executable code or moments in time --- and its "pid"
provider permits the instrumentation of functions at their entry and
return and at any offset between.  One may inspect a process's
environment in terms of its registers and its address space;  also
exposed are integer arguments on function entry and a return value on
exit, but these are simply special cases of register use.

This abstraction is particularly well suited to simple C applications:
tracing function arguments is trivial and often sufficient in its own
right.  With some work to find their locations, local variables are
visible.  If some quantity is a pointer then dereferencing it is
reasonably straight-forward.  Composite data types are fully supported
and may be navigated with varying degrees of ease, depending on the
DTrace implementation.

For more complex programs, the pid provider alone falls short and your
question provides a perfect example.  It may happen that some
significant event, e.g. the receipt of a message, does not correspond to
a well known location such as the entry to or the return from a
function.  Instead, it might lie somewhere within a function at an
unstable offset determined only at compilation.  How may one describe
the location of this event?  Worse, DTrace will reveal only the contents
of a register or a memory location.  Suppose that we obtain the value of
the variable used in the Erlang VM to identify a message;  perhaps it is
a pointer.  To what does it point?  Likely something unwieldy to the
developer and opaque to the user.

DTrace provides a solution in the form of a static tracing point, in
essence a C macro with a stable name that is visible to DTrace.  Thus
the developer can associate a named probe with some logical event (e.g.
message receipt) and expose useful diagnostic data (e.g. a textual
representation of the message itself).

Turning at last to your actual problem, the message-receive probe is
used at line 242 here:

https://github.com/erlang/otp/blob/4d0c23bd19f138e4fcfedd11283636e96d6bbc4f/erts/emulator/beam/msg_instrs.tab#L227

(Note that variables exposed by the probe are initialised only if the
probe itself is enabled, which is determined by the earlier macro at
227.)

Interestingly, the message itself isn't exposed, which is why you have
been unable to trace it.  However, I imagine it would be feasible to
patch the VM to do what you want.  You would need to render the message
as a string into a buffer and expose that buffer's address;  an existing
example of this idiom is

https://github.com/erlang/otp/blob/4d0c23bd19f138e4fcfedd11283636e96d6bbc4f/erts/emulator/beam/global.h#L1603

The name for such static DTrace probes is USDT;  you can read more about
it at

https://docs.oracle.com/cd/E37838_01/html/E61035/gkycs.html#scrolltoc

but there may be differences on your OS (e.g. see DTrace(1) on macos).

Robert
Confidentiality Notice | This email and any included attachments may be privileged, confidential and/or otherwise protected from disclosure. Access to this email by anyone other than the intended recipient is unauthorized. If you believe you have received this email in error, please contact the sender immediately and delete all copies. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
Reply | Threaded
Open this post in threaded view
|

Re: Obtain message content of Erlang messages in dtrace

Duncan Paul Attard
Thanks a lot for your detailed answer!


> On 05 Jan 2020, at 01:39, Harris, Robert <[hidden email]> wrote:
>
>
>
>> On 16 Dec 2019, at 12:02, Duncan Paul Attard <[hidden email]> wrote:
>>
>> I was experimenting with Erlang and dtrace, and am interested to know whether the message content exchanged between two Erlang processes can be obtained. In particular, I am interested in the `message-send` and `message-receive` probes.
>
> TL;DR you can use DTrace but you'll have to modify the Erlang VM.
>
> DTrace was intended to provide observation at the very lowest level.  In
> the user land case, a process is viewed as a collection of load objects
> whose symbol tables describe functions and global variables.  DTrace is
> organised as a framework of "providers" that expose certain events ---
> locations in executable code or moments in time --- and its "pid"
> provider permits the instrumentation of functions at their entry and
> return and at any offset between.  One may inspect a process's
> environment in terms of its registers and its address space;  also
> exposed are integer arguments on function entry and a return value on
> exit, but these are simply special cases of register use.
>
> This abstraction is particularly well suited to simple C applications:
> tracing function arguments is trivial and often sufficient in its own
> right.  With some work to find their locations, local variables are
> visible.  If some quantity is a pointer then dereferencing it is
> reasonably straight-forward.  Composite data types are fully supported
> and may be navigated with varying degrees of ease, depending on the
> DTrace implementation.
>
> For more complex programs, the pid provider alone falls short and your
> question provides a perfect example.  It may happen that some
> significant event, e.g. the receipt of a message, does not correspond to
> a well known location such as the entry to or the return from a
> function.  Instead, it might lie somewhere within a function at an
> unstable offset determined only at compilation.  How may one describe
> the location of this event?  Worse, DTrace will reveal only the contents
> of a register or a memory location.  Suppose that we obtain the value of
> the variable used in the Erlang VM to identify a message;  perhaps it is
> a pointer.  To what does it point?  Likely something unwieldy to the
> developer and opaque to the user.
>
> DTrace provides a solution in the form of a static tracing point, in
> essence a C macro with a stable name that is visible to DTrace.  Thus
> the developer can associate a named probe with some logical event (e.g.
> message receipt) and expose useful diagnostic data (e.g. a textual
> representation of the message itself).
>
> Turning at last to your actual problem, the message-receive probe is
> used at line 242 here:
>
> https://github.com/erlang/otp/blob/4d0c23bd19f138e4fcfedd11283636e96d6bbc4f/erts/emulator/beam/msg_instrs.tab#L227
>
> (Note that variables exposed by the probe are initialised only if the
> probe itself is enabled, which is determined by the earlier macro at
> 227.)
>
> Interestingly, the message itself isn't exposed, which is why you have
> been unable to trace it.  However, I imagine it would be feasible to
> patch the VM to do what you want.  You would need to render the message
> as a string into a buffer and expose that buffer's address;  an existing
> example of this idiom is
>
> https://github.com/erlang/otp/blob/4d0c23bd19f138e4fcfedd11283636e96d6bbc4f/erts/emulator/beam/global.h#L1603
>
> The name for such static DTrace probes is USDT;  you can read more about
> it at
>
> https://docs.oracle.com/cd/E37838_01/html/E61035/gkycs.html#scrolltoc
>
> but there may be differences on your OS (e.g. see DTrace(1) on macos).
>
> Robert
> Confidentiality Notice | This email and any included attachments may be privileged, confidential and/or otherwise protected from disclosure. Access to this email by anyone other than the intended recipient is unauthorized. If you believe you have received this email in error, please contact the sender immediately and delete all copies. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.