Opentracing for Erlang Processes

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Opentracing for Erlang Processes

Ricardo Oliveira
I'm trying to design a distributed tracing system for Erlang processes following the opentracing standard. For the purpose, I'm supporting myself in otter for generating and emitting traces, and jaeger for collecting, querying and storing them.. The question for which I can't figure out a clear answer is how should I pass the trace payload between processes. The first idea that comes to me is to put the trace information in the all the messages passed between processes, and with that define a sort of message protocol that all messages would have to `extend`. This was the approach Uber used to solve this problem, with the implementation of TChannel. This would be a bit awkward to implement in Erlang, as we would have to `force` all messages exchanged between processes to include this payload, and I don't know how I could do this properly. Do you have a way of doing this ?

Also , would you see a better (simpler) way of doing this in Erlang, by taking advantage of the ERTS and OTP, for example, like seq_trace is passing `implicitly` the trace_token in the messages sent between the processes.

Thank you,
Ricardo Oliveira

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Opentracing for Erlang Processes

Ferenc Holzhauser
Hi Ricardo,

Otter has different APIs, but indeed you do need to pass something on the process boundaries. 
I think this discussion will be very otter specific, so I'd suggest to open an issue on github. We can brainstorm there and see if we can come up with something nice for this.

Regards,
Ferenc

On 28 September 2017 at 17:54, Ricardo Oliveira <[hidden email]> wrote:
I'm trying to design a distributed tracing system for Erlang processes following the opentracing standard. For the purpose, I'm supporting myself in otter for generating and emitting traces, and jaeger for collecting, querying and storing them.. The question for which I can't figure out a clear answer is how should I pass the trace payload between processes. The first idea that comes to me is to put the trace information in the all the messages passed between processes, and with that define a sort of message protocol that all messages would have to `extend`. This was the approach Uber used to solve this problem, with the implementation of TChannel. This would be a bit awkward to implement in Erlang, as we would have to `force` all messages exchanged between processes to include this payload, and I don't know how I could do this properly. Do you have a way of doing this ?

Also , would you see a better (simpler) way of doing this in Erlang, by taking advantage of the ERTS and OTP, for example, like seq_trace is passing `implicitly` the trace_token in the messages sent between the processes.

Thank you,
Ricardo Oliveira

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Opentracing for Erlang Processes

Ricardo Oliveira
Hi Ferenc, 

Thank you for your answer.

I don't believe this is really otter specific. The way I see it, otter is just acting as trace emitter, following the opentracing standard. If we would replace it by other lib, we would still have the same problem.
For example, I can have a task that in order to be completed needs to go through multiple Erlang processes that do different things, like access a db, interact with external api, etc. 
Typically an application contains many tasks or flows, that can run concurrently, and each task can also have many instantiations.
So when one of the processes involved in a task gets a message from another process, how does that process knows to which specific task (context) will it be realizing work for ? Why ? Because with that we could track the flow of execution of a task between all the involved processes. So my question is about how we can propagate this tracing context between processes? Explicitly/forced in all the messages exchanged between the processes? Or if there is another implicit way (like seq_trace does) of doing this ?

Thanks

On Thu, Sep 28, 2017 at 6:06 PM Ferenc Holzhauser <[hidden email]> wrote:
Hi Ricardo,

Otter has different APIs, but indeed you do need to pass something on the process boundaries. 
I think this discussion will be very otter specific, so I'd suggest to open an issue on github. We can brainstorm there and see if we can come up with something nice for this.

Regards,
Ferenc

On 28 September 2017 at 17:54, Ricardo Oliveira <[hidden email]> wrote:
I'm trying to design a distributed tracing system for Erlang processes following the opentracing standard. For the purpose, I'm supporting myself in otter for generating and emitting traces, and jaeger for collecting, querying and storing them.. The question for which I can't figure out a clear answer is how should I pass the trace payload between processes. The first idea that comes to me is to put the trace information in the all the messages passed between processes, and with that define a sort of message protocol that all messages would have to `extend`. This was the approach Uber used to solve this problem, with the implementation of TChannel. This would be a bit awkward to implement in Erlang, as we would have to `force` all messages exchanged between processes to include this payload, and I don't know how I could do this properly. Do you have a way of doing this ?

Also , would you see a better (simpler) way of doing this in Erlang, by taking advantage of the ERTS and OTP, for example, like seq_trace is passing `implicitly` the trace_token in the messages sent between the processes.

Thank you,
Ricardo Oliveira

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Opentracing for Erlang Processes

Tristan Sloughter-4
In reply to this post by Ricardo Oliveira
Also , would you see a better (simpler) way of doing this in Erlang, by taking advantage of the ERTS and OTP, for example, like seq_trace is passing `implicitly` the trace_token in the messages sent between the processes.

I've been meaning to investigate the seq_trace implementation further. It would be awesome to allow for the user to set a 128 and 64 bit integer (trace and parent span id) to use as the id's passed along with the message instead of them being generated.

Currently you set the label but then serial is a 2-tuple that gets automatically incremented, so can't be used for the parent span ids, unless there was a seprate tracking of {Label, Serial} -> SpanId state kept in some process.

Quickest option is to just wrap the behaviours. For example with a custom behaviour we use in a project Erleans it will check for trace context when you make a `call` and pass it along to create a span around the callback.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions