EUnit quirks

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

EUnit quirks

Gareth Adams
Hi,

I've been writing some eunit tests using fixtures (http://erlang.org/doc/apps/eunit/chapter.html#Fixtures) and have noticed a couple of quirks.

1. If the setup fails for some reason cleanup is not called. This can be an issue if you're mecking things as the don't get unmecked so the next test setup fails etc. I find myself lost in a cascade of failures, not knowing which tests are really failing.

2. If something fails in the test itself that's not part of an assert, eunit cancels the tests and they are not run. Although this is reported in the console, if you have a lot of tests with debug statements this can be lost. The only thing that indicates that they've not been run is the number of tests decreases, which  au not be noticed.

Has anyone else had these issues? If so, how did you handle them?

Thanks

Gareth

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: EUnit quirks

Dániel Szoboszlay
Hi,

1:
This specific problem with meck could be worked around by wrapping your fixture into a spawn: meck (by default) unloads mocked modules when the process creating the mocks dies.

There's also a more generic approach to this problem. I've seen two different strategies for writing test setup/cleanup code:
  1. Setup assumes you start from a known state of the system, and it only has to change things that your test needs in addition to that known initial state. It's the cleanup's responsibility to return to the known state by undoing whatever the setup and the actual test could have changed.
  2. Setup doesn't assume anything about the initial state of the system: it has the responsibility of getting it from an arbitrary state to a known state that is suitable for running the test. Cleanup isn't really needed in this model: the next test's setup has to deal with whatever state you leave behind your test.
The second strategy is usually more robust, because it is very easy to accidentally start testing when your system is not in it's assumed initial state (at least on your own machine during development). On the other hand, writing setup code that can take you from literally any (possibly broken) state of the system to a known good start state is very hard. But in case of this particular problem it's easy: just call meck:unload() from the setup function as the first thing.

2:
Do you have a minimal example showing this behaviour? I tried to reproduce, but at least for me eunit:test/1 does correctly report a test failed outside of an assert as a failure too, and the return value reveals that there was a problem as well.

Regards,
Daniel

On Fri, 14 Jul 2017 at 19:16 Gareth Adams <[hidden email]> wrote:
Hi,

I've been writing some eunit tests using fixtures (http://erlang.org/doc/apps/eunit/chapter.html#Fixtures) and have noticed a couple of quirks.

1. If the setup fails for some reason cleanup is not called. This can be an issue if you're mecking things as the don't get unmecked so the next test setup fails etc. I find myself lost in a cascade of failures, not knowing which tests are really failing.

2. If something fails in the test itself that's not part of an assert, eunit cancels the tests and they are not run. Although this is reported in the console, if you have a lot of tests with debug statements this can be lost. The only thing that indicates that they've not been run is the number of tests decreases, which  au not be noticed.

Has anyone else had these issues? If so, how did you handle them?

Thanks

Gareth
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: EUnit quirks

Richard Carlsson-3
In reply to this post by Gareth Adams
1: A setup function is assumed to either succeed, with a possibly modified world as result, or fail, with an unchanged world as result. If you need to make multiple changes, the idea is that you either write separate setup/cleanup functions for each change and nest them to create a complete setup, or you write a single setup function that does its own internal catching of errors and partial cleanup.

2. All exceptions in a test case should be handled the same by eunit. Asserts are just one way of throwing an exception. Do you have an example to demonstrate what you are seeing?

        /Richard

2017-07-14 18:45 GMT+02:00 Gareth Adams <[hidden email]>:
Hi,

I've been writing some eunit tests using fixtures (http://erlang.org/doc/apps/eunit/chapter.html#Fixturesand have noticed a couple of quirks.

1. If the setup fails for some reason cleanup is not called. This can be an issue if you're mecking things as the don't get unmecked so the next test setup fails etc. I find myself lost in a cascade of failures, not knowing which tests are really failing.

2. If something fails in the test itself that's not part of an assert, eunit cancels the tests and they are not run. Although this is reported in the console, if you have a lot of tests with debug statements this can be lost. The only thing that indicates that they've not been run is the number of tests decreases, which  au not be noticed.

Has anyone else had these issues? If so, how did you handle them?

Thanks

Gareth

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: EUnit quirks

Gareth Adams

Hi,

 

This reply is to both you, Richard, and Daniel (didn’t want to pollute the list with two answers).

 

Regarding 1:

 

I’d worked out that I could wrap up the setup so it doesn’t blow and cleans up after itself if there’s an error, I’d just hoped that eunit could have done that for me by always running the cleanup function.

I like the idea of spawning the tests – I’ll try that in work tomorrow and see what happens.

 

I suppose the setup making sure that the system is in a known state before it starts is not a bad idea either but it may lead to lazy tests where no one cleans up after themselves.

 

Regarding 2:

 

I’ve written a quick test where the first two tests are not run because I’ve called an unknown function but the third test is run. As I say, the console does report the failing test but when we’ve got ~1000 tests that often produce debug printouts this gets lost. I’ve noticed the “One or more tests were cancelled” – I don’t recall seeing that in work, I’ll check. However, we use intellij in work which doesn’t display the console results in the test runner by default.

 

-module(aborted).

 

-include_lib("eunit/include/eunit.hrl").

 

aborted_test_() ->

    {

        foreach,

        fun setup/0,

        fun cleanup/1,

        [

            fun test1/1,

            fun test2/1

        ]

    }.

 

setup() ->

    state.

 

cleanup(_State) ->

    ok.

 

test1(State) ->

    call:unknown(State),

    [

        {"Passing test", ?_assertEqual(state, State)},

        {"First Failing test", ?_assertEqual(not_state, State)}

    ].

 

test2(State) ->

    [

        {"Second Failing test", ?_assertEqual(really_not_state, State)}

    ].

 

 

Sent from Mail for Windows 10

 

From: [hidden email]
Sent: 16 July 2017 10:51
To: [hidden email]
Cc: [hidden email]
Subject: Re: [erlang-questions] EUnit quirks

 

1: A setup function is assumed to either succeed, with a possibly modified world as result, or fail, with an unchanged world as result. If you need to make multiple changes, the idea is that you either write separate setup/cleanup functions for each change and nest them to create a complete setup, or you write a single setup function that does its own internal catching of errors and partial cleanup.

 

2. All exceptions in a test case should be handled the same by eunit. Asserts are just one way of throwing an exception. Do you have an example to demonstrate what you are seeing?

 

        /Richard

 

2017-07-14 18:45 GMT+02:00 Gareth Adams <[hidden email]>:

Hi,

 

I've been writing some eunit tests using fixtures (http://erlang.org/doc/apps/eunit/chapter.html#Fixtures) and have noticed a couple of quirks.

 

1. If the setup fails for some reason cleanup is not called. This can be an issue if you're mecking things as the don't get unmecked so the next test setup fails etc. I find myself lost in a cascade of failures, not knowing which tests are really failing.

 

2. If something fails in the test itself that's not part of an assert, eunit cancels the tests and they are not run. Although this is reported in the console, if you have a lot of tests with debug statements this can be lost. The only thing that indicates that they've not been run is the number of tests decreases, which  au not be noticed.

 

Has anyone else had these issues? If so, how did you handle them?

 

Thanks

Gareth


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

 

 


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions