seq_trace with bigint labels

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

seq_trace with bigint labels

Tristan Sloughter-4
I've gone back and forth for a while now on how seq_trace, http://erlang.org/doc/man/seq_trace.html, could best be used, if at all, to benefit distributed tracing libraries.

In particular I've been working on OpenCensus (http://opencensus.io) -- a recent announcement on the Google open source blog about it: https://opensource.googleblog.com/2018/01/opencensus.html -- implementation in Erlang, https://github.com/census-instrumentation/opencensus-erlang. But this is not unique to Census, the OpenTracing (http://opentracing.io) implementation Otter, https://github.com/Bluehouse-Technology/otter/, works in a similar manner.

A trace must track a trace id and span id and propagate them to children. Both opencensus and Otter provide functionality to track the context in a variable or in the process dictionary. But when it comes to message passing the only option is  adding a variable.

Libraries that are instrumented internally can make this not have to modify the user's api, for example in Erleans we have 'call' extract the trace context behind the scenes and pass it through the gen_statem:call, https://github.com/SpaceTime-IoT/erleans/blob/master/src/erleans_grain.erl#L142

If it were possible to carry the trace context through the message pass in the background in general this would open up possibility for easier instrumentation with few changes to a user's code.

This is essentially what seq_trace already does. The first issue hit when looking at using it to carry a census trace id (128 bit integer) is that while the docs say:

> The label component is an integer which identifies all events belonging to the same sequential trace.

'integer' here does not mean any Erlang integer:

seq_trace:set_token(label, 38995684955506843782500595084643303673).
** exception error: bad argument

Looking in the seq trace code I found it does a `is_small` check on the integer, resulting in the badarg for integers the size of a census trace id.

So finally.. the main point of this email is really just to raise the idea of increasing the size of allowable labels in seq_trace.

Otherwise we must keep track of a label -> trace mapping in a central process or ets table. The first issue that solution would have to solve is, what happens when a message goes to another node where that label mapping doesn't exist.

I'm interested to hear if this is a possibility, what other blockers might arise from using seq_trace like this, if someone has an alternative idea or if someone just thinks this is a bad idea in general :)

--
  Tristan Sloughter
  "I am not a crackpot" - Abe Simpson
  [hidden email]
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: seq_trace with bigint labels

Tristan Sloughter-4
> Ideally we'd have a solution where the individual applications would not
> have to be modified/recompiled but the whole node could be configured to
> send trace data out to a trace collector.

Mostly. You'd still need to include the app that is capable of receiving the seq_trace messages and formatting spans to be reported from them, but you'd be able to trace any message path without code modifications.

But yea, I don't know either if it is as simple as letting it use bigints, or what the consequences of doing so would be.

Hopefully someone on the OTP team sees this and can answer :)
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: seq_trace with bigint labels

Kenneth Lundin-5
I will take it up to discussion within the OTP team and come back with an answer as soon as possible.
The idea with the small_int solution is that we wanted an immediate data here to avoid dynamic memory allocation. But I think it would be possible to allow a bigger
value which still is of fixed size (big int is dynamic).

I think it would be very good if seqtrace could support use cases like this.

/Kenneth, Erlang/OTP team at Ericsson

On Tue, Jan 23, 2018 at 6:12 PM, Tristan Sloughter <[hidden email]> wrote:
> Ideally we'd have a solution where the individual applications would not
> have to be modified/recompiled but the whole node could be configured to
> send trace data out to a trace collector.

Mostly. You'd still need to include the app that is capable of receiving the seq_trace messages and formatting spans to be reported from them, but you'd be able to trace any message path without code modifications.

But yea, I don't know either if it is as simple as letting it use bigints, or what the consequences of doing so would be.

Hopefully someone on the OTP team sees this and can answer :)
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: seq_trace with bigint labels

Tristan Sloughter-4
Great! And yea, it doesn't have to be dynamic. For census it needs to be 128bits for the trace id and span ids are 64bits.

--
  Tristan Sloughter
  "I am not a crackpot" - Abe Simpson


On Wed, Jan 24, 2018, at 1:34 AM, Kenneth Lundin wrote:
I will take it up to discussion within the OTP team and come back with an answer as soon as possible.
The idea with the small_int solution is that we wanted an immediate data here to avoid dynamic memory allocation. But I think it would be possible to allow a bigger
value which still is of fixed size (big int is dynamic).

I think it would be very good if seqtrace could support use cases like this.

/Kenneth, Erlang/OTP team at Ericsson

On Tue, Jan 23, 2018 at 6:12 PM, Tristan Sloughter <[hidden email]> wrote:
> Ideally we'd have a solution where the individual applications would not
> have to be modified/recompiled but the whole node could be configured to
> send trace data out to a trace collector.

Mostly. You'd still need to include the app that is capable of receiving the seq_trace messages and formatting spans to be reported from them, but you'd be able to trace any message path without code modifications.

But yea, I don't know either if it is as simple as letting it use bigints, or what the consequences of doing so would be.

Hopefully someone on the OTP team sees this and can answer :)

_______________________________________________
erlang-questions mailing list


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions