Unidirectional architectures in Erlang

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Unidirectional architectures in Erlang

Rodrigo Stevaux
Hi,

If I may ask, I have a question about actor-based systems design that arose while writing a simple system for integrating GitHub and Slack.

GitHub events are sent to a server using their Webhook API. We do some application logic on the events, and forward them to Slack.

So our architecture is unidirectional: messages are followed from the HTTP server to a sequence of processes until they finally get to a message dispatcher process that sends messages to Slack.

Message comes in, logic applies, message goes out.
The generic servers are very easy to test in a pure fashion: just test the return values of handle_call for a given state.

But:
- unit testing should test that the unit under test forwards a message to a given address (and that is not reflected on the handle_call return value)
- doing the integration testing of a "pipeline" of process involves testing that ultimately a message gets to the last process.

Given the maturity of Erlang, I'd expect to promptly find solutions to this case.

I haven't found a nice solution yet, so I suppose I am using OTP in a wrong way (maybe it's not a good fit for my project), or I am choosing the wrong architecture, or that I am missing key knowledge about how to test things in Erlang.

Can anyone shed some light on this questions? Thanks in advance!

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Unidirectional architectures in Erlang

Fred Hebert-2
On 08/30, Rodrigo Stevaux wrote:
>Message comes in, logic applies, message goes out.
>The generic servers are very easy to test in a pure fashion: just test the
>return values of handle_call for a given state.
>

That can be done as a regular unit test, yes.

>But:
>- unit testing should test that the unit under test forwards a message to a
>given address (and that is not reflected on the handle_call return value)

Aside from using mocks to know that the proper function calls have taken
place, you can't easily test the forwarding of messages since that would
imply side-effects turning your unit tests into integration tests.

>- doing the integration testing of a "pipeline" of process involves testing
>that ultimately a message gets to the last process.
>

Yes and no. You can test that each element of the chain properly
forwards data to the right destination. You can do that if you make the
endpoint/address/destination for each forwarded message parametrizable
or configurable. Forward the data to the test process instead to confirm
that the right side-effect has taken place.

This lets you test each section of the chain in isolation; then if all
sections are well-behaved, you can infer that the whole chain should
behave well. You can confirm that with a more naive end-to-end test,
knowing that individual components are tested more thoroughly.

>Given the maturity of Erlang, I'd expect to promptly find solutions to this
>case.

On top of the techniques above, you can try using the `sys' module to
introspect all OTP behaviours and look at their transitions as a neutral
observer:

    1> sys:get_status(disk_log_server).
    {status,<0.75.0>,
            {module,gen_server},
            [[{<0.76.0>,'$#group_history'},
              {'$initial_call',{disk_log_server,init,1}},
              {'$ancestors',[kernel_safe_sup,kernel_sup,<0.45.0>]}],
             running,<0.66.0>,[],
             [{header,"Status for generic server disk_log_server"},
              {data,[{"Status",running},
                     {"Parent",<0.66.0>},
                     {"Logged events",[]}]},
              {data,[{"State",{state,[]}}]}]]}
    2> P = self(),
    2> sys:install(
    2>   disk_log_server,
    2>   {fun(_Acc, Event, State) -> P ! {trace, Event, State} end,
    2>    nodata}
    2> ).
    ok
    3> disk_log_server ! fake_data.
    fake_data
    4> flush().
    Shell got {trace,{in,fake_data},disk_log_server}
    Shell got {trace,{noreply,{state,[]}},disk_log_server}
    ok

Those won't let you see arbitrary calls being made, but if you have a
process chain A -> B -> C, then you can, by looking at the traces of all
three processes, track the history of a call and the consequences they
have.

The sys module is full of really cool functionality few people know
about, and that can make live debugging, but also testing much simpler.

>
>I haven't found a nice solution yet, so I suppose I am using OTP in a wrong
>way (maybe it's not a good fit for my project), or I am choosing the wrong
>architecture, or that I am missing key knowledge about how to test things
>in Erlang.

There aren't enough details to know at this point. I frequently just use
`meck' (https://github.com/eproxus/meck) as a library, which lets you
replace arbitrary function calls through hot code loadig, get traces and
counts of all calls and values returned of unmodified code, and so on.

If what you're looking for is a way to track side-effects in a very
whitebox manner, meck is hard to beat.

>
>Can anyone shed some light on this questions? Thanks in advance!

please provide more details about the nature of the side-effects you are
trying to observe in a unit test. My default reflex would be to say that
the moment you're testing side-effects, you are doing integration
testing.

Regards,
Fred.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Unidirectional architectures in Erlang

Rodrigo Stevaux
yes, we fundamentally are trying to do integration testing.

unit testing process communication pairwise by sending a message to the parent process is something we tried to do and it worked reasonably well. but we want to know that the whole composition of processes works well.

the challenge we found on that approach is that the processes have names and we would have to find a way to inject the pid of the "next" process in the chain and make it the test process during testing and other process during production (which can be done with a little bit of instrumentation and conditional compiling -- we are on Elixir actually).

is using the "sys" module a regular practice?

thanks for the reply.

Em qui, 30 de ago de 2018 às 20:33, Fred Hebert <[hidden email]> escreveu:
On 08/30, Rodrigo Stevaux wrote:
>Message comes in, logic applies, message goes out.
>The generic servers are very easy to test in a pure fashion: just test the
>return values of handle_call for a given state.
>

That can be done as a regular unit test, yes.

>But:
>- unit testing should test that the unit under test forwards a message to a
>given address (and that is not reflected on the handle_call return value)

Aside from using mocks to know that the proper function calls have taken
place, you can't easily test the forwarding of messages since that would
imply side-effects turning your unit tests into integration tests.

>- doing the integration testing of a "pipeline" of process involves testing
>that ultimately a message gets to the last process.
>

Yes and no. You can test that each element of the chain properly
forwards data to the right destination. You can do that if you make the
endpoint/address/destination for each forwarded message parametrizable
or configurable. Forward the data to the test process instead to confirm
that the right side-effect has taken place.

This lets you test each section of the chain in isolation; then if all
sections are well-behaved, you can infer that the whole chain should
behave well. You can confirm that with a more naive end-to-end test,
knowing that individual components are tested more thoroughly.

>Given the maturity of Erlang, I'd expect to promptly find solutions to this
>case.

On top of the techniques above, you can try using the `sys' module to
introspect all OTP behaviours and look at their transitions as a neutral
observer:

    1> sys:get_status(disk_log_server).
    {status,<0.75.0>,
            {module,gen_server},
            [[{<0.76.0>,'$#group_history'},
              {'$initial_call',{disk_log_server,init,1}},
              {'$ancestors',[kernel_safe_sup,kernel_sup,<0.45.0>]}],
             running,<0.66.0>,[],
             [{header,"Status for generic server disk_log_server"},
              {data,[{"Status",running},
                     {"Parent",<0.66.0>},
                     {"Logged events",[]}]},
              {data,[{"State",{state,[]}}]}]]}
    2> P = self(),
    2> sys:install(
    2>   disk_log_server,
    2>   {fun(_Acc, Event, State) -> P ! {trace, Event, State} end,
    2>    nodata}
    2> ).
    ok
    3> disk_log_server ! fake_data.
    fake_data
    4> flush().
    Shell got {trace,{in,fake_data},disk_log_server}
    Shell got {trace,{noreply,{state,[]}},disk_log_server}
    ok

Those won't let you see arbitrary calls being made, but if you have a
process chain A -> B -> C, then you can, by looking at the traces of all
three processes, track the history of a call and the consequences they
have.

The sys module is full of really cool functionality few people know
about, and that can make live debugging, but also testing much simpler.

>
>I haven't found a nice solution yet, so I suppose I am using OTP in a wrong
>way (maybe it's not a good fit for my project), or I am choosing the wrong
>architecture, or that I am missing key knowledge about how to test things
>in Erlang.

There aren't enough details to know at this point. I frequently just use
`meck' (https://github.com/eproxus/meck) as a library, which lets you
replace arbitrary function calls through hot code loadig, get traces and
counts of all calls and values returned of unmodified code, and so on.

If what you're looking for is a way to track side-effects in a very
whitebox manner, meck is hard to beat.

>
>Can anyone shed some light on this questions? Thanks in advance!

please provide more details about the nature of the side-effects you are
trying to observe in a unit test. My default reflex would be to say that
the moment you're testing side-effects, you are doing integration
testing.

Regards,
Fred.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Unidirectional architectures in Erlang

Jachym Holecek
In reply to this post by Rodrigo Stevaux
Hi,

# Rodrigo Stevaux 2018-08-30:

> If I may ask, I have a question about actor-based systems design that arose
> while writing a simple system for integrating GitHub and Slack.
>
> GitHub events are sent to a server using their Webhook API. We do some
> application logic on the events, and forward them to Slack.
>
> So our architecture is unidirectional: messages are followed from the HTTP
> server to a sequence of processes until they finally get to a message
> dispatcher process that sends messages to Slack.
>
> Message comes in, logic applies, message goes out.

Erlang comes by default with the brilliant Common Test application, I would
very much recommend giving it a try even if it may take a bit of effort to
learn.

> The generic servers are very easy to test in a pure fashion: just test the
> return values of handle_call for a given state.

Not really -- doing so violates abstraction boundaries and processes of such
variety typically do most of their useful work by performing side effects. So
either you have to introduce artificial complexity to satisfy requirements of
the test machinery (highly undesirable) or your test won't really capture
much (but you're still paying the maintenance cost for them).

> But:
> - unit testing should test that the unit under test forwards a message to a
> given address (and that is not reflected on the handle_call return value)
> - doing the integration testing of a "pipeline" of process involves testing
> that ultimately a message gets to the last process.

Good questions to be asking -- what assurances do you want to obtain from the
tests and consequently what granularity should you being testing at and what
methods and tools are available to assist. Also: what is the maintenance cost
of these tests looking forward, and does the utility of these justify them?

Black-box tests, whereby you start the whole thing, tickle its public
interfaces and see the right reactions come out the other end, tend to
always be desirable. You can do this against full cluster, individual
node, individual application, individual process, individual functions.

Common Test, with project-specific scaffolding around it, will be helpful
at all these levels. With it being ordinary Erlang code running as ordinary
Erlang node all the native communication and diagnostic facilities are also
available to you should you need to inspect or tweak something behind the
system's back. It is also not entirely out of the question for the production
code to be somewhat tweaked to cater for testing needs specifically, where
you can't get your way by more standard means.

Perhaps too broad an answer... my two cents anyways. :)

BR,
        -- Jachym
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Unidirectional architectures in Erlang

Jachym Holecek
In reply to this post by Fred Hebert-2
# Fred Hebert 2018-08-30:
> This lets you test each section of the chain in isolation; then if all
> sections are well-behaved, you can infer that the whole chain should behave
> well.

That sounds like a very dangerous conclusion to be making, personally.

It's not unheard of that thousands of unit-tests "pass" and everything is
"ready" and then it all falls like a house of cards with a single naive
end-to-end flow. ;-)

BR,
        -- Jachym
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Unidirectional architectures in Erlang

Rodrigo Stevaux
It's not unheard of that thousands of unit-tests "pass" and everything is
"ready" and then it all falls like a house of cards with a single naive
end-to-end flow. ;-)

My exact experience. Even though everything passes unit tests, the system might be mis-configured for example -- moreover if everything is unit testable, dependency injection is probably being used so configuration matters.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Unidirectional architectures in Erlang

Rodrigo Stevaux
In reply to this post by Jachym Holecek
Not really -- doing so violates abstraction boundaries and processes of such
variety typically do most of their useful work by performing side effects. So
either you have to introduce artificial complexity to satisfy requirements of
the test machinery (highly undesirable) or your test won't really capture
much (but you're still paying the maintenance cost for them).

This is something I am curious about. I am not deeply familiar with any erlang code base and I ask: is it common for Genservers to do side-effects (for example, sending a message to a process which is not the sender)? If not, should there be an OTP behavior for processes that forward messages to others (like returning a list of messages instead of a single reply)?

tks!

Em sex, 31 de ago de 2018 às 15:24, Jachym Holecek <[hidden email]> escreveu:
Hi,

# Rodrigo Stevaux 2018-08-30:
> If I may ask, I have a question about actor-based systems design that arose
> while writing a simple system for integrating GitHub and Slack.
>
> GitHub events are sent to a server using their Webhook API. We do some
> application logic on the events, and forward them to Slack.
>
> So our architecture is unidirectional: messages are followed from the HTTP
> server to a sequence of processes until they finally get to a message
> dispatcher process that sends messages to Slack.
>
> Message comes in, logic applies, message goes out.

Erlang comes by default with the brilliant Common Test application, I would
very much recommend giving it a try even if it may take a bit of effort to
learn.

> The generic servers are very easy to test in a pure fashion: just test the
> return values of handle_call for a given state.

Not really -- doing so violates abstraction boundaries and processes of such
variety typically do most of their useful work by performing side effects. So
either you have to introduce artificial complexity to satisfy requirements of
the test machinery (highly undesirable) or your test won't really capture
much (but you're still paying the maintenance cost for them).

> But:
> - unit testing should test that the unit under test forwards a message to a
> given address (and that is not reflected on the handle_call return value)
> - doing the integration testing of a "pipeline" of process involves testing
> that ultimately a message gets to the last process.

Good questions to be asking -- what assurances do you want to obtain from the
tests and consequently what granularity should you being testing at and what
methods and tools are available to assist. Also: what is the maintenance cost
of these tests looking forward, and does the utility of these justify them?

Black-box tests, whereby you start the whole thing, tickle its public
interfaces and see the right reactions come out the other end, tend to
always be desirable. You can do this against full cluster, individual
node, individual application, individual process, individual functions.

Common Test, with project-specific scaffolding around it, will be helpful
at all these levels. With it being ordinary Erlang code running as ordinary
Erlang node all the native communication and diagnostic facilities are also
available to you should you need to inspect or tweak something behind the
system's back. It is also not entirely out of the question for the production
code to be somewhat tweaked to cater for testing needs specifically, where
you can't get your way by more standard means.

Perhaps too broad an answer... my two cents anyways. :)

BR,
        -- Jachym

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Unidirectional architectures in Erlang

Jesper Louis Andersen-2
In reply to this post by Rodrigo Stevaux
Hi,

I've used the sys module more than once, in both debugging and testing. It is useful, also because you can use sys:log/2,3 to have a trace log of a running process and inspect it later.

Another approach I've used is to add trace messages to a system. A module defines this function:

report_event(_DetailLevel, _From, _To, _Label, _Contents) -> hopefully_traced.

And you call it in your system. Now, you can attach a tracer to this call in tests and verify the trace you get is what you expect. This allows you to "see inside" your system and verify its behavior. More advanced solutions involves rewriting traces into the notion of a temporal monad and then asking questions on the trace for correctness.

Yet another approach is to rewrite your code into a computational (pure) part which sets up a set of actions but has no effect, and an effectful part which runs said actions. This allows you to scrutinize the actions without having to bind your system up. Look up, e.g., the Elm architecture for this approach. The basic idea is that of a free monad: rather than running the program, return a command set which can be interpreted either as a test (where the effects are logged) or as a real system (where the effects are run). We do this at $WORK, which means we can test all of our stack without ever having to run the system as a whole in unit tests. In fact, we don't even need a database, or supervisor tree, since we just simulate the universe around a process. Integration tests are run later via a QuickCheck like tool written by Martin Gausby - treating the system as a black box and poking it from the outside to verify we can glue things together as expected.


On Fri, Aug 31, 2018 at 7:07 PM Rodrigo Stevaux <[hidden email]> wrote:
yes, we fundamentally are trying to do integration testing.

unit testing process communication pairwise by sending a message to the parent process is something we tried to do and it worked reasonably well. but we want to know that the whole composition of processes works well.

the challenge we found on that approach is that the processes have names and we would have to find a way to inject the pid of the "next" process in the chain and make it the test process during testing and other process during production (which can be done with a little bit of instrumentation and conditional compiling -- we are on Elixir actually).

is using the "sys" module a regular practice?

thanks for the reply.

Em qui, 30 de ago de 2018 às 20:33, Fred Hebert <[hidden email]> escreveu:
On 08/30, Rodrigo Stevaux wrote:
>Message comes in, logic applies, message goes out.
>The generic servers are very easy to test in a pure fashion: just test the
>return values of handle_call for a given state.
>

That can be done as a regular unit test, yes.

>But:
>- unit testing should test that the unit under test forwards a message to a
>given address (and that is not reflected on the handle_call return value)

Aside from using mocks to know that the proper function calls have taken
place, you can't easily test the forwarding of messages since that would
imply side-effects turning your unit tests into integration tests.

>- doing the integration testing of a "pipeline" of process involves testing
>that ultimately a message gets to the last process.
>

Yes and no. You can test that each element of the chain properly
forwards data to the right destination. You can do that if you make the
endpoint/address/destination for each forwarded message parametrizable
or configurable. Forward the data to the test process instead to confirm
that the right side-effect has taken place.

This lets you test each section of the chain in isolation; then if all
sections are well-behaved, you can infer that the whole chain should
behave well. You can confirm that with a more naive end-to-end test,
knowing that individual components are tested more thoroughly.

>Given the maturity of Erlang, I'd expect to promptly find solutions to this
>case.

On top of the techniques above, you can try using the `sys' module to
introspect all OTP behaviours and look at their transitions as a neutral
observer:

    1> sys:get_status(disk_log_server).
    {status,<0.75.0>,
            {module,gen_server},
            [[{<0.76.0>,'$#group_history'},
              {'$initial_call',{disk_log_server,init,1}},
              {'$ancestors',[kernel_safe_sup,kernel_sup,<0.45.0>]}],
             running,<0.66.0>,[],
             [{header,"Status for generic server disk_log_server"},
              {data,[{"Status",running},
                     {"Parent",<0.66.0>},
                     {"Logged events",[]}]},
              {data,[{"State",{state,[]}}]}]]}
    2> P = self(),
    2> sys:install(
    2>   disk_log_server,
    2>   {fun(_Acc, Event, State) -> P ! {trace, Event, State} end,
    2>    nodata}
    2> ).
    ok
    3> disk_log_server ! fake_data.
    fake_data
    4> flush().
    Shell got {trace,{in,fake_data},disk_log_server}
    Shell got {trace,{noreply,{state,[]}},disk_log_server}
    ok

Those won't let you see arbitrary calls being made, but if you have a
process chain A -> B -> C, then you can, by looking at the traces of all
three processes, track the history of a call and the consequences they
have.

The sys module is full of really cool functionality few people know
about, and that can make live debugging, but also testing much simpler.

>
>I haven't found a nice solution yet, so I suppose I am using OTP in a wrong
>way (maybe it's not a good fit for my project), or I am choosing the wrong
>architecture, or that I am missing key knowledge about how to test things
>in Erlang.

There aren't enough details to know at this point. I frequently just use
`meck' (https://github.com/eproxus/meck) as a library, which lets you
replace arbitrary function calls through hot code loadig, get traces and
counts of all calls and values returned of unmodified code, and so on.

If what you're looking for is a way to track side-effects in a very
whitebox manner, meck is hard to beat.

>
>Can anyone shed some light on this questions? Thanks in advance!

please provide more details about the nature of the side-effects you are
trying to observe in a unit test. My default reflex would be to say that
the moment you're testing side-effects, you are doing integration
testing.

Regards,
Fred.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


--
J.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Unidirectional architectures in Erlang

Rodrigo Stevaux
This brings the question, is there a library for writing temporal properties for testing with quickcheck or proper? I mean properties that say something about the sequence of trace messages and not about the contents of the list of trace mssages for example.

Em sex, 31 de ago de 2018 às 16:24, Jesper Louis Andersen <[hidden email]> escreveu:
Hi,

I've used the sys module more than once, in both debugging and testing. It is useful, also because you can use sys:log/2,3 to have a trace log of a running process and inspect it later.

Another approach I've used is to add trace messages to a system. A module defines this function:

report_event(_DetailLevel, _From, _To, _Label, _Contents) -> hopefully_traced.

And you call it in your system. Now, you can attach a tracer to this call in tests and verify the trace you get is what you expect. This allows you to "see inside" your system and verify its behavior. More advanced solutions involves rewriting traces into the notion of a temporal monad and then asking questions on the trace for correctness.

Yet another approach is to rewrite your code into a computational (pure) part which sets up a set of actions but has no effect, and an effectful part which runs said actions. This allows you to scrutinize the actions without having to bind your system up. Look up, e.g., the Elm architecture for this approach. The basic idea is that of a free monad: rather than running the program, return a command set which can be interpreted either as a test (where the effects are logged) or as a real system (where the effects are run). We do this at $WORK, which means we can test all of our stack without ever having to run the system as a whole in unit tests. In fact, we don't even need a database, or supervisor tree, since we just simulate the universe around a process. Integration tests are run later via a QuickCheck like tool written by Martin Gausby - treating the system as a black box and poking it from the outside to verify we can glue things together as expected.


On Fri, Aug 31, 2018 at 7:07 PM Rodrigo Stevaux <[hidden email]> wrote:
yes, we fundamentally are trying to do integration testing.

unit testing process communication pairwise by sending a message to the parent process is something we tried to do and it worked reasonably well. but we want to know that the whole composition of processes works well.

the challenge we found on that approach is that the processes have names and we would have to find a way to inject the pid of the "next" process in the chain and make it the test process during testing and other process during production (which can be done with a little bit of instrumentation and conditional compiling -- we are on Elixir actually).

is using the "sys" module a regular practice?

thanks for the reply.

Em qui, 30 de ago de 2018 às 20:33, Fred Hebert <[hidden email]> escreveu:
On 08/30, Rodrigo Stevaux wrote:
>Message comes in, logic applies, message goes out.
>The generic servers are very easy to test in a pure fashion: just test the
>return values of handle_call for a given state.
>

That can be done as a regular unit test, yes.

>But:
>- unit testing should test that the unit under test forwards a message to a
>given address (and that is not reflected on the handle_call return value)

Aside from using mocks to know that the proper function calls have taken
place, you can't easily test the forwarding of messages since that would
imply side-effects turning your unit tests into integration tests.

>- doing the integration testing of a "pipeline" of process involves testing
>that ultimately a message gets to the last process.
>

Yes and no. You can test that each element of the chain properly
forwards data to the right destination. You can do that if you make the
endpoint/address/destination for each forwarded message parametrizable
or configurable. Forward the data to the test process instead to confirm
that the right side-effect has taken place.

This lets you test each section of the chain in isolation; then if all
sections are well-behaved, you can infer that the whole chain should
behave well. You can confirm that with a more naive end-to-end test,
knowing that individual components are tested more thoroughly.

>Given the maturity of Erlang, I'd expect to promptly find solutions to this
>case.

On top of the techniques above, you can try using the `sys' module to
introspect all OTP behaviours and look at their transitions as a neutral
observer:

    1> sys:get_status(disk_log_server).
    {status,<0.75.0>,
            {module,gen_server},
            [[{<0.76.0>,'$#group_history'},
              {'$initial_call',{disk_log_server,init,1}},
              {'$ancestors',[kernel_safe_sup,kernel_sup,<0.45.0>]}],
             running,<0.66.0>,[],
             [{header,"Status for generic server disk_log_server"},
              {data,[{"Status",running},
                     {"Parent",<0.66.0>},
                     {"Logged events",[]}]},
              {data,[{"State",{state,[]}}]}]]}
    2> P = self(),
    2> sys:install(
    2>   disk_log_server,
    2>   {fun(_Acc, Event, State) -> P ! {trace, Event, State} end,
    2>    nodata}
    2> ).
    ok
    3> disk_log_server ! fake_data.
    fake_data
    4> flush().
    Shell got {trace,{in,fake_data},disk_log_server}
    Shell got {trace,{noreply,{state,[]}},disk_log_server}
    ok

Those won't let you see arbitrary calls being made, but if you have a
process chain A -> B -> C, then you can, by looking at the traces of all
three processes, track the history of a call and the consequences they
have.

The sys module is full of really cool functionality few people know
about, and that can make live debugging, but also testing much simpler.

>
>I haven't found a nice solution yet, so I suppose I am using OTP in a wrong
>way (maybe it's not a good fit for my project), or I am choosing the wrong
>architecture, or that I am missing key knowledge about how to test things
>in Erlang.

There aren't enough details to know at this point. I frequently just use
`meck' (https://github.com/eproxus/meck) as a library, which lets you
replace arbitrary function calls through hot code loadig, get traces and
counts of all calls and values returned of unmodified code, and so on.

If what you're looking for is a way to track side-effects in a very
whitebox manner, meck is hard to beat.

>
>Can anyone shed some light on this questions? Thanks in advance!

please provide more details about the nature of the side-effects you are
trying to observe in a unit test. My default reflex would be to say that
the moment you're testing side-effects, you are doing integration
testing.

Regards,
Fred.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


--
J.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Unidirectional architectures in Erlang

Jesper Louis Andersen-2
QuickCheck has eqc_temporal (which is a temporal monad). I don't think proper currently implements the interface, but it isn't too hard to write it yourself if you want to attack the problem.

You generate random commands like in a statem model.
The system-under-test (SUT) generates trace events.
You stuff those trace events into eqc_temporal or equivalent.
You ask properties of the eqc_temporal trace representation, failing the test if the rules are broken.


On Fri, Aug 31, 2018 at 9:29 PM Rodrigo Stevaux <[hidden email]> wrote:
This brings the question, is there a library for writing temporal properties for testing with quickcheck or proper? I mean properties that say something about the sequence of trace messages and not about the contents of the list of trace mssages for example.

Em sex, 31 de ago de 2018 às 16:24, Jesper Louis Andersen <[hidden email]> escreveu:
Hi,

I've used the sys module more than once, in both debugging and testing. It is useful, also because you can use sys:log/2,3 to have a trace log of a running process and inspect it later.

Another approach I've used is to add trace messages to a system. A module defines this function:

report_event(_DetailLevel, _From, _To, _Label, _Contents) -> hopefully_traced.

And you call it in your system. Now, you can attach a tracer to this call in tests and verify the trace you get is what you expect. This allows you to "see inside" your system and verify its behavior. More advanced solutions involves rewriting traces into the notion of a temporal monad and then asking questions on the trace for correctness.

Yet another approach is to rewrite your code into a computational (pure) part which sets up a set of actions but has no effect, and an effectful part which runs said actions. This allows you to scrutinize the actions without having to bind your system up. Look up, e.g., the Elm architecture for this approach. The basic idea is that of a free monad: rather than running the program, return a command set which can be interpreted either as a test (where the effects are logged) or as a real system (where the effects are run). We do this at $WORK, which means we can test all of our stack without ever having to run the system as a whole in unit tests. In fact, we don't even need a database, or supervisor tree, since we just simulate the universe around a process. Integration tests are run later via a QuickCheck like tool written by Martin Gausby - treating the system as a black box and poking it from the outside to verify we can glue things together as expected.


On Fri, Aug 31, 2018 at 7:07 PM Rodrigo Stevaux <[hidden email]> wrote:
yes, we fundamentally are trying to do integration testing.

unit testing process communication pairwise by sending a message to the parent process is something we tried to do and it worked reasonably well. but we want to know that the whole composition of processes works well.

the challenge we found on that approach is that the processes have names and we would have to find a way to inject the pid of the "next" process in the chain and make it the test process during testing and other process during production (which can be done with a little bit of instrumentation and conditional compiling -- we are on Elixir actually).

is using the "sys" module a regular practice?

thanks for the reply.

Em qui, 30 de ago de 2018 às 20:33, Fred Hebert <[hidden email]> escreveu:
On 08/30, Rodrigo Stevaux wrote:
>Message comes in, logic applies, message goes out.
>The generic servers are very easy to test in a pure fashion: just test the
>return values of handle_call for a given state.
>

That can be done as a regular unit test, yes.

>But:
>- unit testing should test that the unit under test forwards a message to a
>given address (and that is not reflected on the handle_call return value)

Aside from using mocks to know that the proper function calls have taken
place, you can't easily test the forwarding of messages since that would
imply side-effects turning your unit tests into integration tests.

>- doing the integration testing of a "pipeline" of process involves testing
>that ultimately a message gets to the last process.
>

Yes and no. You can test that each element of the chain properly
forwards data to the right destination. You can do that if you make the
endpoint/address/destination for each forwarded message parametrizable
or configurable. Forward the data to the test process instead to confirm
that the right side-effect has taken place.

This lets you test each section of the chain in isolation; then if all
sections are well-behaved, you can infer that the whole chain should
behave well. You can confirm that with a more naive end-to-end test,
knowing that individual components are tested more thoroughly.

>Given the maturity of Erlang, I'd expect to promptly find solutions to this
>case.

On top of the techniques above, you can try using the `sys' module to
introspect all OTP behaviours and look at their transitions as a neutral
observer:

    1> sys:get_status(disk_log_server).
    {status,<0.75.0>,
            {module,gen_server},
            [[{<0.76.0>,'$#group_history'},
              {'$initial_call',{disk_log_server,init,1}},
              {'$ancestors',[kernel_safe_sup,kernel_sup,<0.45.0>]}],
             running,<0.66.0>,[],
             [{header,"Status for generic server disk_log_server"},
              {data,[{"Status",running},
                     {"Parent",<0.66.0>},
                     {"Logged events",[]}]},
              {data,[{"State",{state,[]}}]}]]}
    2> P = self(),
    2> sys:install(
    2>   disk_log_server,
    2>   {fun(_Acc, Event, State) -> P ! {trace, Event, State} end,
    2>    nodata}
    2> ).
    ok
    3> disk_log_server ! fake_data.
    fake_data
    4> flush().
    Shell got {trace,{in,fake_data},disk_log_server}
    Shell got {trace,{noreply,{state,[]}},disk_log_server}
    ok

Those won't let you see arbitrary calls being made, but if you have a
process chain A -> B -> C, then you can, by looking at the traces of all
three processes, track the history of a call and the consequences they
have.

The sys module is full of really cool functionality few people know
about, and that can make live debugging, but also testing much simpler.

>
>I haven't found a nice solution yet, so I suppose I am using OTP in a wrong
>way (maybe it's not a good fit for my project), or I am choosing the wrong
>architecture, or that I am missing key knowledge about how to test things
>in Erlang.

There aren't enough details to know at this point. I frequently just use
`meck' (https://github.com/eproxus/meck) as a library, which lets you
replace arbitrary function calls through hot code loadig, get traces and
counts of all calls and values returned of unmodified code, and so on.

If what you're looking for is a way to track side-effects in a very
whitebox manner, meck is hard to beat.

>
>Can anyone shed some light on this questions? Thanks in advance!

please provide more details about the nature of the side-effects you are
trying to observe in a unit test. My default reflex would be to say that
the moment you're testing side-effects, you are doing integration
testing.

Regards,
Fred.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


--
J.


--
J.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Unidirectional architectures in Erlang

Rodrigo Stevaux
Great man, I think this is what I was looking after. Exactly this!

Em sex, 31 de ago de 2018 às 16:33, Jesper Louis Andersen <[hidden email]> escreveu:
QuickCheck has eqc_temporal (which is a temporal monad). I don't think proper currently implements the interface, but it isn't too hard to write it yourself if you want to attack the problem.

You generate random commands like in a statem model.
The system-under-test (SUT) generates trace events.
You stuff those trace events into eqc_temporal or equivalent.
You ask properties of the eqc_temporal trace representation, failing the test if the rules are broken.


On Fri, Aug 31, 2018 at 9:29 PM Rodrigo Stevaux <[hidden email]> wrote:
This brings the question, is there a library for writing temporal properties for testing with quickcheck or proper? I mean properties that say something about the sequence of trace messages and not about the contents of the list of trace mssages for example.

Em sex, 31 de ago de 2018 às 16:24, Jesper Louis Andersen <[hidden email]> escreveu:
Hi,

I've used the sys module more than once, in both debugging and testing. It is useful, also because you can use sys:log/2,3 to have a trace log of a running process and inspect it later.

Another approach I've used is to add trace messages to a system. A module defines this function:

report_event(_DetailLevel, _From, _To, _Label, _Contents) -> hopefully_traced.

And you call it in your system. Now, you can attach a tracer to this call in tests and verify the trace you get is what you expect. This allows you to "see inside" your system and verify its behavior. More advanced solutions involves rewriting traces into the notion of a temporal monad and then asking questions on the trace for correctness.

Yet another approach is to rewrite your code into a computational (pure) part which sets up a set of actions but has no effect, and an effectful part which runs said actions. This allows you to scrutinize the actions without having to bind your system up. Look up, e.g., the Elm architecture for this approach. The basic idea is that of a free monad: rather than running the program, return a command set which can be interpreted either as a test (where the effects are logged) or as a real system (where the effects are run). We do this at $WORK, which means we can test all of our stack without ever having to run the system as a whole in unit tests. In fact, we don't even need a database, or supervisor tree, since we just simulate the universe around a process. Integration tests are run later via a QuickCheck like tool written by Martin Gausby - treating the system as a black box and poking it from the outside to verify we can glue things together as expected.


On Fri, Aug 31, 2018 at 7:07 PM Rodrigo Stevaux <[hidden email]> wrote:
yes, we fundamentally are trying to do integration testing.

unit testing process communication pairwise by sending a message to the parent process is something we tried to do and it worked reasonably well. but we want to know that the whole composition of processes works well.

the challenge we found on that approach is that the processes have names and we would have to find a way to inject the pid of the "next" process in the chain and make it the test process during testing and other process during production (which can be done with a little bit of instrumentation and conditional compiling -- we are on Elixir actually).

is using the "sys" module a regular practice?

thanks for the reply.

Em qui, 30 de ago de 2018 às 20:33, Fred Hebert <[hidden email]> escreveu:
On 08/30, Rodrigo Stevaux wrote:
>Message comes in, logic applies, message goes out.
>The generic servers are very easy to test in a pure fashion: just test the
>return values of handle_call for a given state.
>

That can be done as a regular unit test, yes.

>But:
>- unit testing should test that the unit under test forwards a message to a
>given address (and that is not reflected on the handle_call return value)

Aside from using mocks to know that the proper function calls have taken
place, you can't easily test the forwarding of messages since that would
imply side-effects turning your unit tests into integration tests.

>- doing the integration testing of a "pipeline" of process involves testing
>that ultimately a message gets to the last process.
>

Yes and no. You can test that each element of the chain properly
forwards data to the right destination. You can do that if you make the
endpoint/address/destination for each forwarded message parametrizable
or configurable. Forward the data to the test process instead to confirm
that the right side-effect has taken place.

This lets you test each section of the chain in isolation; then if all
sections are well-behaved, you can infer that the whole chain should
behave well. You can confirm that with a more naive end-to-end test,
knowing that individual components are tested more thoroughly.

>Given the maturity of Erlang, I'd expect to promptly find solutions to this
>case.

On top of the techniques above, you can try using the `sys' module to
introspect all OTP behaviours and look at their transitions as a neutral
observer:

    1> sys:get_status(disk_log_server).
    {status,<0.75.0>,
            {module,gen_server},
            [[{<0.76.0>,'$#group_history'},
              {'$initial_call',{disk_log_server,init,1}},
              {'$ancestors',[kernel_safe_sup,kernel_sup,<0.45.0>]}],
             running,<0.66.0>,[],
             [{header,"Status for generic server disk_log_server"},
              {data,[{"Status",running},
                     {"Parent",<0.66.0>},
                     {"Logged events",[]}]},
              {data,[{"State",{state,[]}}]}]]}
    2> P = self(),
    2> sys:install(
    2>   disk_log_server,
    2>   {fun(_Acc, Event, State) -> P ! {trace, Event, State} end,
    2>    nodata}
    2> ).
    ok
    3> disk_log_server ! fake_data.
    fake_data
    4> flush().
    Shell got {trace,{in,fake_data},disk_log_server}
    Shell got {trace,{noreply,{state,[]}},disk_log_server}
    ok

Those won't let you see arbitrary calls being made, but if you have a
process chain A -> B -> C, then you can, by looking at the traces of all
three processes, track the history of a call and the consequences they
have.

The sys module is full of really cool functionality few people know
about, and that can make live debugging, but also testing much simpler.

>
>I haven't found a nice solution yet, so I suppose I am using OTP in a wrong
>way (maybe it's not a good fit for my project), or I am choosing the wrong
>architecture, or that I am missing key knowledge about how to test things
>in Erlang.

There aren't enough details to know at this point. I frequently just use
`meck' (https://github.com/eproxus/meck) as a library, which lets you
replace arbitrary function calls through hot code loadig, get traces and
counts of all calls and values returned of unmodified code, and so on.

If what you're looking for is a way to track side-effects in a very
whitebox manner, meck is hard to beat.

>
>Can anyone shed some light on this questions? Thanks in advance!

please provide more details about the nature of the side-effects you are
trying to observe in a unit test. My default reflex would be to say that
the moment you're testing side-effects, you are doing integration
testing.

Regards,
Fred.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


--
J.


--
J.

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Unidirectional architectures in Erlang

Fred Hebert-2
In reply to this post by Jachym Holecek
On 08/31, Jachym Holecek wrote:

># Fred Hebert 2018-08-30:
>> This lets you test each section of the chain in isolation; then if all
>> sections are well-behaved, you can infer that the whole chain should behave
>> well.
>
>That sounds like a very dangerous conclusion to be making, personally.
>
>It's not unheard of that thousands of unit-tests "pass" and everything is
>"ready" and then it all falls like a house of cards with a single naive
>end-to-end flow. ;-)
>

Yes, this is why the original quote also strongly suggested having at
least one end-to-end pipeline on top of it. It's just that you may have
fewer details to worry about in the end-to-end pipeline when what you
have to focus on is "is it all plumbed together right" rather than "are
all elements of the whole chain correct and also plumbed together
right".

This is about scoping your tests, and defining which behaviours and
properties of your code you want to validate. You'd want the unit tests
to validate the minutiae of each step, but the integration suite to
check that they're wired up correctly, for example.

I've worked and seen lots of integration and system tests that, through
obtuse combination of inputs, exercised very narrow behaviour of
deeply-entrenched bits of code. Those tests are generally slow to set
up, very fragile, and rarely deterministic.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions