Common Test repeat_until and exit status

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Common Test repeat_until and exit status

Loïc Hoguin-3
Hello,

When I use the test group property repeat_until_all_ok I still get an
error status from ct_run (exit status 1) because the tests that failed
and were repeated are still counted as errors.

For example:



Testing ninenines.gun.gun_SUITE: Starting test (with repeated test cases)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - -
gun_SUITE:shutdown_reason failed on line 399
Reason: timeout
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - -
gun_SUITE:retry_1 failed on line 324
Reason: timeout
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - -

Testing ninenines.gun.gun_SUITE: *** FAILED test case 17 ***

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - -
gun_SUITE:retry_fun failed on line 343
Reason: shutdown_too_late
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - -

Testing ninenines.gun.gun_SUITE: *** FAILED test case 18 ***
Testing ninenines.gun.gun_SUITE: *** FAILED test case 21 ***

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - -
gun_SUITE:retry_1 failed on line 324
Reason: timeout
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - -

Testing ninenines.gun.gun_SUITE: *** FAILED test case 46 ***
Testing ninenines.gun.gun_SUITE: Stopping repeat operation
{repeat_until_all_ok,98}
Testing ninenines.gun.gun_SUITE: TEST COMPLETE, 83 ok, 4 failed of 87
test cases
[...]
make[1]: *** [erlang.mk:6452: ct] Error 1



Is there a way to repeat flaky tests without Common Test counting the
test failures as errors? In the example above I want Common Test
(ct_run) to return the exit status 0 because the tests were OK after a
few tries.

This would be very useful to quickly get tests green on environments
that are more likely to have timing issues (in cases where it's the test
that is the problem, not the application). For example on slower
environments, or on Windows/FreeBSD where the network stack doesn't
always behave the same as Linux. I spend far too much time on fixing
timing issues in tests.

Cheers,

--
Loïc Hoguin
https://ninenines.eu
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions