Instrumenting Erlang code

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Instrumenting Erlang code

Frank Muller
Hi everyone

I would like to implement a custom instrumentation module for my Erlang code.
Let assume this function:

f1() ->
   … code here …
   ok.

When instrumenting this function, I would like it to look like:

f1() ->
   instrument:start(),

   … code here …

   instrument:end(), <— just before retuning from f1 
   ok.

Other problem I can think of is when we have multiple return paths and/or recursive loop:

f2() ->
  instrument:start(),

  receive ->
    {msg1, M1} -> instrument:end(), M1;
    {msg2, M2} -> instrument:end(), M2;
    _ -> instrument:end(), f2()
  end.

Is that doable?
If yes, can I apply like this logic to all modules running in my Erlang node?

@Kostis: does Concuerror (https://github.com/parapluu/Concuerror) use this technique?

Thanks for you help.
/Frank

Reply | Threaded
Open this post in threaded view
|

Re: Instrumenting Erlang code

Jesper Louis Andersen-2
On Sun, Jan 26, 2020 at 1:58 PM Frank Muller <[hidden email]> wrote:
Hi everyone

I would like to implement a custom instrumentation module for my Erlang code.
Let assume this function:


If you have

f1() ->
  Exprs,
  ok.

You can write

f1() ->
  S = instrument:start(),
  try
    Exprs,
    ok
  finally
    instrument:end(S)
  end.

However, some times you want to track an error apart from a success value. Positive and negative exit paths often require different instrumentation. The success path is often interested in processing time (latency) in addition to processing count. The error path often centers around the idea of an error count, especially if the exit is fast. If you blend them, the problem is that the fast errors muddles your view of the successes, as your latency distribution becomes multi-modal. In turn, your mean and median doesn't work like you think they do anymore.


Other problem I can think of is when we have multiple return paths and/or recursive loop:


If you loop, you want to wrap:

f1(0) -> ok;
f1(N) ->
  do(),
  f(N-1).

f1_(N) ->
  S = instrument:start(),
  try
    f1(N)
  finally
    instrument:end(S)
  end.

This only pushes a single exception handler to the stack. An important caveat is when Exprs or do() hibernates. Beware of that situation, since it removes the stack and invokes a continuation.


Is that doable?
If yes, can I apply like this logic to all modules running in my Erlang node?


Yes, parse transforms.

But in my experience, applying something to all modules is usually the wrong way to go about instrumentation. Maybe with a 

-instrument([F, ...]).

to help the parsetransform along the way.

(

This is considerably easier in something like Elixir with its macro system, or OCaml, because you have PPX extensions.

In OCaml, you would have

let f1() ->
    Expr.

And you could just annotate it

let%instrument f1() ->
  Expr.

And write a PPX rewriter for that. 

)


--
J.
Reply | Threaded
Open this post in threaded view
|

Re: Instrumenting Erlang code

Michael Truog
In reply to this post by Frank Muller
On 1/26/20 4:58 AM, Frank Muller wrote:
Hi everyone

I would like to implement a custom instrumentation module for my Erlang code.
Let assume this function:

You are describing that you need aspects from aspect oriented programming (that should be the best concept for what you want, when thinking about the problem).  I implemented this in CloudI (https://cloudi.org) as a list of functions that are executed in order with separate lists for before and after a callback function.  My approach used the following to represent a function [1]:

{Module :: atom(), Function :: atom()}

to represent a function of the expected arity (call it arity N), and:

{{Module :: atom(), Function :: atom()}}

to represent a function of arity 0 that returns a function of arity N, or a normal Erlang function.  Representing the function as a tuple with atoms allows specifying the function before it is loaded.  Using the arity 0 function indirection allows execution of source code before the function is created, normally when a function retains variables created before the function is created (a closure).  Using this approach made it easy to add monitoring to CloudI services and other things like batch execution of CloudI services.

Using functions as data (configuration for the source code [2]) avoids source code that lacks transparency, like macros or parse transforms.

Best Regards,
Michael

[1] Example type specification https://github.com/CloudI/CloudI/blob/7cac9108827ce58008eac7aa24e192c44e2fbfd6/src/lib/cloudi_core/src/cloudi_service_api.erl#L203-L205
[2] Example configuration of aspects https://github.com/CloudI/CloudI/blob/7cac9108827ce58008eac7aa24e192c44e2fbfd6/src/lib/cloudi_core/src/cloudi_service_api.erl#L429-L434
Reply | Threaded
Open this post in threaded view
|

Re: Instrumenting Erlang code

Vans S
On this topic, is there a way to attach instrumentation on the fly in a running system?  That can give you a callback onentry and onexit? To profile wall time and perhap do conditional matching / stack trace inspection. To see which calls trigger the slow path for example.

On Sunday, January 26, 2020, 12:46:57 p.m. EST, Michael Truog <[hidden email]> wrote:


On 1/26/20 4:58 AM, Frank Muller wrote:
Hi everyone

I would like to implement a custom instrumentation module for my Erlang code.
Let assume this function:


You are describing that you need aspects from aspect oriented programming (that should be the best concept for what you want, when thinking about the problem).  I implemented this in CloudI (https://cloudi.org) as a list of functions that are executed in order with separate lists for before and after a callback function.  My approach used the following to represent a function [1]:

{Module :: atom(), Function :: atom()}

to represent a function of the expected arity (call it arity N), and:

{{Module :: atom(), Function :: atom()}}

to represent a function of arity 0 that returns a function of arity N, or a normal Erlang function.  Representing the function as a tuple with atoms allows specifying the function before it is loaded.  Using the arity 0 function indirection allows execution of source code before the function is created, normally when a function retains variables created before the function is created (a closure).  Using this approach made it easy to add monitoring to CloudI services and other things like batch execution of CloudI services.

Using functions as data (configuration for the source code [2]) avoids source code that lacks transparency, like macros or parse transforms.

Best Regards,
Michael

[1] Example type specification https://github.com/CloudI/CloudI/blob/7cac9108827ce58008eac7aa24e192c44e2fbfd6/src/lib/cloudi_core/src/cloudi_service_api.erl#L203-L205
[2] Example configuration of aspects https://github.com/CloudI/CloudI/blob/7cac9108827ce58008eac7aa24e192c44e2fbfd6/src/lib/cloudi_core/src/cloudi_service_api.erl#L429-L434

Reply | Threaded
Open this post in threaded view
|

Re: Instrumenting Erlang code

Tristan Sloughter-4
In reply to this post by Frank Muller
On Sun, Jan 26, 2020, at 05:58, Frank Muller wrote:
Is that doable?
If yes, can I apply like this logic to all modules running in my Erlang node?

The cases you mention are exactly why it is best to not attempt to apply the logic throughout the code automatically but to write it manually where it makes sense to do so. Except in the case of live tracing, where you can use matchspecs to setup traces on functions you care about.

I'd also suggest checking out existing instrumentation libraries for Erlang, so maybe you don't need to write your own:



As well as the Foundation's Observability working group, https://github.com/erlef/eef-observability-wg/

Tristan
Reply | Threaded
Open this post in threaded view
|

Re: Instrumenting Erlang code

Michael Truog
In reply to this post by Vans S
On 1/26/20 10:41 AM, Vans S wrote:
On this topic, is there a way to attach instrumentation on the fly in a running system?  That can give you a callback onentry and onexit? To profile wall time and perhap do conditional matching / stack trace inspection. To see which calls trigger the slow path for example.

You could take a look at eprof to see how it works ( https://github.com/erlang/otp/blob/c15eb5fdf721afed280afdbe0fff37706eef979c/lib/tools/src/eprof.erl#L373 ).  The eprof approach relies on tracing, which is mutating global state, so it lacks referential transparency and seems best to avoid in production source code (not saying tracing shouldn't be used in production).

Best Regards,
Michael
Reply | Threaded
Open this post in threaded view
|

Re: Instrumenting Erlang code

Attila Rajmund Nohl
In reply to this post by Frank Muller
Frank Muller <[hidden email]> ezt írta (időpont: 2020.
jan. 26., V, 13:58):
>
> Hi everyone
>
> I would like to implement a custom instrumentation module for my Erlang code.
[...]

> f2() ->
>   instrument:start(),
>
>   receive ->
>     {msg1, M1} -> instrument:end(), M1;
>     {msg2, M2} -> instrument:end(), M2;
>     _ -> instrument:end(), f2()
>   end.
>
> Is that doable?

My (not really thought out) idea is that you could use dbg to start a
process that receives a messages when f2 is called and is returning.
If instrument:start or instrument:end does not need to execute in the
same process as f2, then it might work.
Reply | Threaded
Open this post in threaded view
|

Re: Instrumenting Erlang code

Kostis Sagonas-2
In reply to this post by Frank Muller
On 1/26/20 1:58 PM, Frank Muller wrote:
> @Kostis: does Concuerror (https://github.com/parapluu/Concuerror) use
> this technique?

Concuerror is an open-source project and its source files have, for the
most part, pretty descriptive names:

 
https://github.com/parapluu/Concuerror/blob/master/src/concuerror_instrumenter.erl

As you can see, Concuerror does not do its transformation at the source
level but at the level of Core Erlang instead.  The instrumentation code
is not so involved either.

Hope this helps,
Kostis
Reply | Threaded
Open this post in threaded view
|

Re: Instrumenting Erlang code

Stavros Aronis
In reply to this post by Frank Muller
@Kostis: does Concuerror (https://github.com/parapluu/Concuerror) use this technique?

Concuerror's instrumenter does not really work in the way you need. It is instead doing the following things:
* wraps every function call / apply into a call to an inspector function concuerror_inspect:inspect (this function does a simple runtime check using the process dictionary to decide whether a process is under Concuerror or not)
* adds a similar inspector function to every receive statement

The process dictionary trick used to decide which modules should have the instrumentation on and which not might be useful to you, if you want to instrument all modules but not affect all processes: if the process has a specific atom in its dictionary then instrumentation is on, otherwise not.

Without thinking too much about it, if I wanted to implement your variant I'd write a parse transform or custom instrumenter that would just introduce a helper function for every function like this:

Original:

f1() ->
  Body1.

Becomes:

f1() ->
  instrument:start(),
  R = f1_orig(),
  instrument:stop(),
  R

fi_orig() ->
  Body1.

Hope this helps!
Stavros

Reply | Threaded
Open this post in threaded view
|

Re: Instrumenting Erlang code

Edmond Begumisa
In reply to this post by Frank Muller
Hi,

Years ago, Tim Watson wrote a neat AOP-style library which uses function  
attributes to do just that...

https://github.com/hyperthunk/annotations
       
I haven't used it in a while, but I see the README was more recently  
updated so I presume it still works.

- Edmond -

On Sun, 26 Jan 2020 22:58:04 +1000, Frank Muller  
<[hidden email]> wrote:

> Hi everyone
>
> I would like to implement a custom instrumentation module for my Erlang  
> code.
> Let assume this function:
>
> f1() ->
>   … code here …
>   ok.
>
> When instrumenting this function, I would like it to look like:
>
> f1() ->
>   instrument:start(),
>
>   … code here …
>
>   instrument:end(), <— just before retuning from f1  ok.
>
>> Other problem I can think of is when we have multiple return paths  
>> and/or recursive loop:
>
> f2() ->
>  instrument:start(),
>
>  receive ->
>    {msg1, M1} -> instrument:end(), M1;
>    {msg2, M2} -> instrument:end(), M2;
>    _ -> instrument:end(), f2()
>  end.
>
> Is that doable?
> If yes, can I apply like this logic to all modules running in my Erlang  
> node?
>
> @Kostis: does Concuerror (https://github.com/parapluu/Concuerror) use  
> this technique?
>
> Thanks for you help.
> /Frank
>
>



--
Using Opera's mail client: http://www.opera.com/mail/