Looking for practical testing advice

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Looking for practical testing advice

Sölvi Páll Ásgeirsson
Hi all

I'm wondering about various different approaches to writing testable
Erlang and I'd be super interested in hearing about how you approach
this.

As a concrete example, let's assume that we're building software that
talks to financial exchanges.
When submitting orders, you might need to map symbols to/from order
book ids, so you run a named gen_server that:

- Knows how to map an order book id to symbol (i.e., 1234 -> ERIC)
- Knows how to map symbol to id (ERIC -> 1234)
- Knows how to refresh it's map periodically from an external service,
such as a database or whatever

Then you want to test your protocol handler, which depends on this id
mapping server, but you don't want to make an expensive call to an
external service for every test you run, so you want to provide fake
data.

I see a few different possible approaches to orchestrate testing of
this, but I'm having a hard time figuring out which of these I should
follow in general:

1. You can start the protocol handler with a parameter, IdMapMod ::
module(), and implement a stub_idmap module that knows how to map
to/from certain ids to use in your tests.
The downside of this is that all calls from the protocol handler to
the id map must extract a dynamic module name from the state, and then
you lose dialyzer, xref, ...
Another downside(or upside?) is that you won't actually perform any
actual calls to your id mapping server.

2. You can start the id mapping server with a parameter,
ExternalServiceMod :: module(), and fake out the external service call
only.
Then your tests of the protocol handler would actually exercise the id
mapping server; but it's calls to an external service would be
answered with a fake stub.
This way you get the benefit of dialyzer + xref as your protocol
handler simply calls id_map:symbol_to_id/1 or whatever.

3. You can use a mocking library such as meck to do something very
similar to either of the two above things.
In past lives using other languages, I've generally developed a
distaste for mocking, as I've felt that it can create brittle tests.
Generally, I think I'd prefer to be able to test things in terms of
interfaces, but I might be swayed otherwise.

Is there anything else I'm missing?  How do you write testable Erlang?
 Is there a general approach you follow?

Thanks & regards
Sölvi Páll Á
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Looking for practical testing advice

Brujo Benavides-3
Not so long ago I would’ve gone with #3 immediately, but lately my co-worker Marcelo Gornstein is slowly persuading me of using something like #2.
I think he’s about to write a blog post exactly about that.

On 28 Mar 2018, at 10:18, Sölvi Páll Ásgeirsson <[hidden email]> wrote:

Hi all

I'm wondering about various different approaches to writing testable
Erlang and I'd be super interested in hearing about how you approach
this.

As a concrete example, let's assume that we're building software that
talks to financial exchanges.
When submitting orders, you might need to map symbols to/from order
book ids, so you run a named gen_server that:

- Knows how to map an order book id to symbol (i.e., 1234 -> ERIC)
- Knows how to map symbol to id (ERIC -> 1234)
- Knows how to refresh it's map periodically from an external service,
such as a database or whatever

Then you want to test your protocol handler, which depends on this id
mapping server, but you don't want to make an expensive call to an
external service for every test you run, so you want to provide fake
data.

I see a few different possible approaches to orchestrate testing of
this, but I'm having a hard time figuring out which of these I should
follow in general:

1. You can start the protocol handler with a parameter, IdMapMod ::
module(), and implement a stub_idmap module that knows how to map
to/from certain ids to use in your tests.
The downside of this is that all calls from the protocol handler to
the id map must extract a dynamic module name from the state, and then
you lose dialyzer, xref, ...
Another downside(or upside?) is that you won't actually perform any
actual calls to your id mapping server.

2. You can start the id mapping server with a parameter,
ExternalServiceMod :: module(), and fake out the external service call
only.
Then your tests of the protocol handler would actually exercise the id
mapping server; but it's calls to an external service would be
answered with a fake stub.
This way you get the benefit of dialyzer + xref as your protocol
handler simply calls id_map:symbol_to_id/1 or whatever.

3. You can use a mocking library such as meck to do something very
similar to either of the two above things.
In past lives using other languages, I've generally developed a
distaste for mocking, as I've felt that it can create brittle tests.
Generally, I think I'd prefer to be able to test things in terms of
interfaces, but I might be swayed otherwise.

Is there anything else I'm missing?  How do you write testable Erlang?
Is there a general approach you follow?

Thanks & regards
Sölvi Páll Á
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Looking for practical testing advice

Jesper Louis Andersen-2
Hi,

My solution is to think functionally: decouple computation from effect. You don't have to go total Monadic style, but split the code which computes results from the code which has some kind of side-effect (messaging, file I/O, network I/O, etc). This lets you test your code by injecting the data into the computational chain in a unit test, and run the effectful tests separately from that. In your case, we might run 3 operations:

1. Compute what data we need
2. Message an external party for that data
3. Compute a result based on that data

This suggests we need 3 functions, where the middle function has a side effect. But functions 1 and 3 can be unit-tested under the assumption of a result from 2. And 2 can be tested independently of 1 and 3. Thinking a bit more like an Erlang programmer would, this is a finite state machine(!). The function 1 returns a *Command* to a command runner, which then executes 2 and feeds the result back into the state machine. The runner then executes 3. If you decouple the generation of what-to-do from actually doing it, testing is easy.

The aside here is that it is exactly what you would do in a monadic style implementation of the same. You define the program as a Free monad (i.e., as a program which you can interpret in several different ways). Then you run said program in a test-interpreter when you want to test the program. But you run it in a "real system" when you want it to execute the program in the real world.

In practice, I like to build the above model because it enables you to test your program without ever having to run the system as a whole. Once you start having many different data sources and dependent processes, you have to think about isolation of them in test settings. Otherwise it becomes prohibitively hard to work with the system as you add more complexity.

The astute reader will recognize that the above is similar to the Elm Architecture, but that is a subject for another post :)


On Wed, Mar 28, 2018 at 4:10 PM Brujo Benavides <[hidden email]> wrote:
Not so long ago I would’ve gone with #3 immediately, but lately my co-worker Marcelo Gornstein is slowly persuading me of using something like #2.
I think he’s about to write a blog post exactly about that.



On 28 Mar 2018, at 10:18, Sölvi Páll Ásgeirsson <[hidden email]> wrote:

Hi all

I'm wondering about various different approaches to writing testable
Erlang and I'd be super interested in hearing about how you approach
this.

As a concrete example, let's assume that we're building software that
talks to financial exchanges.
When submitting orders, you might need to map symbols to/from order
book ids, so you run a named gen_server that:

- Knows how to map an order book id to symbol (i.e., 1234 -> ERIC)
- Knows how to map symbol to id (ERIC -> 1234)
- Knows how to refresh it's map periodically from an external service,
such as a database or whatever

Then you want to test your protocol handler, which depends on this id
mapping server, but you don't want to make an expensive call to an
external service for every test you run, so you want to provide fake
data.

I see a few different possible approaches to orchestrate testing of
this, but I'm having a hard time figuring out which of these I should
follow in general:

1. You can start the protocol handler with a parameter, IdMapMod ::
module(), and implement a stub_idmap module that knows how to map
to/from certain ids to use in your tests.
The downside of this is that all calls from the protocol handler to
the id map must extract a dynamic module name from the state, and then
you lose dialyzer, xref, ...
Another downside(or upside?) is that you won't actually perform any
actual calls to your id mapping server.

2. You can start the id mapping server with a parameter,
ExternalServiceMod :: module(), and fake out the external service call
only.
Then your tests of the protocol handler would actually exercise the id
mapping server; but it's calls to an external service would be
answered with a fake stub.
This way you get the benefit of dialyzer + xref as your protocol
handler simply calls id_map:symbol_to_id/1 or whatever.

3. You can use a mocking library such as meck to do something very
similar to either of the two above things.
In past lives using other languages, I've generally developed a
distaste for mocking, as I've felt that it can create brittle tests.
Generally, I think I'd prefer to be able to test things in terms of
interfaces, but I might be swayed otherwise.

Is there anything else I'm missing?  How do you write testable Erlang?
Is there a general approach you follow?

Thanks & regards
Sölvi Páll Á
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Looking for practical testing advice

Sölvi Páll Ásgeirsson
Thanks Jesper.

I think that we've written some of our things in the manner you
describe, although not intentionally or explicitly.
Do you know of any code I can read which is written using your approach?

Sölvi

On Wed, Mar 28, 2018 at 2:43 PM, Jesper Louis Andersen
<[hidden email]> wrote:

> Hi,
>
> My solution is to think functionally: decouple computation from effect. You
> don't have to go total Monadic style, but split the code which computes
> results from the code which has some kind of side-effect (messaging, file
> I/O, network I/O, etc). This lets you test your code by injecting the data
> into the computational chain in a unit test, and run the effectful tests
> separately from that. In your case, we might run 3 operations:
>
> 1. Compute what data we need
> 2. Message an external party for that data
> 3. Compute a result based on that data
>
> This suggests we need 3 functions, where the middle function has a side
> effect. But functions 1 and 3 can be unit-tested under the assumption of a
> result from 2. And 2 can be tested independently of 1 and 3. Thinking a bit
> more like an Erlang programmer would, this is a finite state machine(!). The
> function 1 returns a *Command* to a command runner, which then executes 2
> and feeds the result back into the state machine. The runner then executes
> 3. If you decouple the generation of what-to-do from actually doing it,
> testing is easy.
>
> The aside here is that it is exactly what you would do in a monadic style
> implementation of the same. You define the program as a Free monad (i.e., as
> a program which you can interpret in several different ways). Then you run
> said program in a test-interpreter when you want to test the program. But
> you run it in a "real system" when you want it to execute the program in the
> real world.
>
> In practice, I like to build the above model because it enables you to test
> your program without ever having to run the system as a whole. Once you
> start having many different data sources and dependent processes, you have
> to think about isolation of them in test settings. Otherwise it becomes
> prohibitively hard to work with the system as you add more complexity.
>
> The astute reader will recognize that the above is similar to the Elm
> Architecture, but that is a subject for another post :)
>
>
> On Wed, Mar 28, 2018 at 4:10 PM Brujo Benavides <[hidden email]>
> wrote:
>>
>> Not so long ago I would’ve gone with #3 immediately, but lately my
>> co-worker Marcelo Gornstein is slowly persuading me of using something like
>> #2.
>> I think he’s about to write a blog post exactly about that.
>>
>> ________________________________
>> Brujo Benavides
>>
>> On 28 Mar 2018, at 10:18, Sölvi Páll Ásgeirsson <[hidden email]> wrote:
>>
>> Hi all
>>
>> I'm wondering about various different approaches to writing testable
>> Erlang and I'd be super interested in hearing about how you approach
>> this.
>>
>> As a concrete example, let's assume that we're building software that
>> talks to financial exchanges.
>> When submitting orders, you might need to map symbols to/from order
>> book ids, so you run a named gen_server that:
>>
>> - Knows how to map an order book id to symbol (i.e., 1234 -> ERIC)
>> - Knows how to map symbol to id (ERIC -> 1234)
>> - Knows how to refresh it's map periodically from an external service,
>> such as a database or whatever
>>
>> Then you want to test your protocol handler, which depends on this id
>> mapping server, but you don't want to make an expensive call to an
>> external service for every test you run, so you want to provide fake
>> data.
>>
>> I see a few different possible approaches to orchestrate testing of
>> this, but I'm having a hard time figuring out which of these I should
>> follow in general:
>>
>> 1. You can start the protocol handler with a parameter, IdMapMod ::
>> module(), and implement a stub_idmap module that knows how to map
>> to/from certain ids to use in your tests.
>> The downside of this is that all calls from the protocol handler to
>> the id map must extract a dynamic module name from the state, and then
>> you lose dialyzer, xref, ...
>> Another downside(or upside?) is that you won't actually perform any
>> actual calls to your id mapping server.
>>
>> 2. You can start the id mapping server with a parameter,
>> ExternalServiceMod :: module(), and fake out the external service call
>> only.
>> Then your tests of the protocol handler would actually exercise the id
>> mapping server; but it's calls to an external service would be
>> answered with a fake stub.
>> This way you get the benefit of dialyzer + xref as your protocol
>> handler simply calls id_map:symbol_to_id/1 or whatever.
>>
>> 3. You can use a mocking library such as meck to do something very
>> similar to either of the two above things.
>> In past lives using other languages, I've generally developed a
>> distaste for mocking, as I've felt that it can create brittle tests.
>> Generally, I think I'd prefer to be able to test things in terms of
>> interfaces, but I might be swayed otherwise.
>>
>> Is there anything else I'm missing?  How do you write testable Erlang?
>> Is there a general approach you follow?
>>
>> Thanks & regards
>> Sölvi Páll Á
>> _______________________________________________
>> erlang-questions mailing list
>> [hidden email]
>> http://erlang.org/mailman/listinfo/erlang-questions
>>
>>
>> _______________________________________________
>> erlang-questions mailing list
>> [hidden email]
>> http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Looking for practical testing advice

Sölvi Páll Ásgeirsson
In reply to this post by Brujo Benavides-3
i'm looking forward to reading that blog post! :)

Sölvi

On Wed, Mar 28, 2018 at 2:10 PM, Brujo Benavides
<[hidden email]> wrote:

> Not so long ago I would’ve gone with #3 immediately, but lately my co-worker
> Marcelo Gornstein is slowly persuading me of using something like #2.
> I think he’s about to write a blog post exactly about that.
>
> ________________________________
> Brujo Benavides
>
>
>
> On 28 Mar 2018, at 10:18, Sölvi Páll Ásgeirsson <[hidden email]> wrote:
>
> Hi all
>
> I'm wondering about various different approaches to writing testable
> Erlang and I'd be super interested in hearing about how you approach
> this.
>
> As a concrete example, let's assume that we're building software that
> talks to financial exchanges.
> When submitting orders, you might need to map symbols to/from order
> book ids, so you run a named gen_server that:
>
> - Knows how to map an order book id to symbol (i.e., 1234 -> ERIC)
> - Knows how to map symbol to id (ERIC -> 1234)
> - Knows how to refresh it's map periodically from an external service,
> such as a database or whatever
>
> Then you want to test your protocol handler, which depends on this id
> mapping server, but you don't want to make an expensive call to an
> external service for every test you run, so you want to provide fake
> data.
>
> I see a few different possible approaches to orchestrate testing of
> this, but I'm having a hard time figuring out which of these I should
> follow in general:
>
> 1. You can start the protocol handler with a parameter, IdMapMod ::
> module(), and implement a stub_idmap module that knows how to map
> to/from certain ids to use in your tests.
> The downside of this is that all calls from the protocol handler to
> the id map must extract a dynamic module name from the state, and then
> you lose dialyzer, xref, ...
> Another downside(or upside?) is that you won't actually perform any
> actual calls to your id mapping server.
>
> 2. You can start the id mapping server with a parameter,
> ExternalServiceMod :: module(), and fake out the external service call
> only.
> Then your tests of the protocol handler would actually exercise the id
> mapping server; but it's calls to an external service would be
> answered with a fake stub.
> This way you get the benefit of dialyzer + xref as your protocol
> handler simply calls id_map:symbol_to_id/1 or whatever.
>
> 3. You can use a mocking library such as meck to do something very
> similar to either of the two above things.
> In past lives using other languages, I've generally developed a
> distaste for mocking, as I've felt that it can create brittle tests.
> Generally, I think I'd prefer to be able to test things in terms of
> interfaces, but I might be swayed otherwise.
>
> Is there anything else I'm missing?  How do you write testable Erlang?
> Is there a general approach you follow?
>
> Thanks & regards
> Sölvi Páll Á
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
>
>
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions