Testing gen_statem timers

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Testing gen_statem timers

more_than_3_characters
Hi, first time poster (but I've read many threads in here).

How does one usually test gen_statem internal timer events?
Usually you test the effects of the behaviour somehow, and I can test that. But in this particular case I had a situation where timers with uniq references are created, later on they usually get removed and all is fine - they're created for each extra worker started and at some point the worker is removed by the timer event that trigger checking some conditions.

I noticed after that, the way I had written it, if an extra worker was started then died for some other reason than the timer event taking it out, I could theoretically accumulate timers that wouldn't be removed, I've since then corrected that but still I have no tests ensuring me that it actually works. 
So my question is if there's any even way to test it (even if with tracing)?
Thanks

Micael Nussbaumer

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Testing gen_statem timers

Raimo Niskanen-11
On Thu, Aug 15, 2019 at 12:33:03PM +0100, Micael Nussbaumer wrote:

> Hi, first time poster (but I've read many threads in here).
>
> How does one usually test gen_statem internal timer events?
> Usually you test the effects of the behaviour somehow, and I can test that.
> But in this particular case I had a situation where timers with uniq
> references are created, later on they usually get removed and all is fine -
> they're created for each extra worker started and at some point the worker
> is removed by the timer event that trigger checking some conditions.
>
> I noticed after that, the way I had written it, if an extra worker was
> started then died for some other reason than the timer event taking it out,
> I could theoretically accumulate timers that wouldn't be removed, I've
> since then corrected that but still I have no tests ensuring me that it
> actually works.
> So my question is if there's any even way to test it (even if with tracing)?
> Thanks

For the current code on maint and master (I think):

If you trace gen_statem:loop/3; the third argument is a #state{} record
whose 3:rd field (3:rd record field - 4:th tuple field) are the timers.
The timers is a 2-tuple with two maps; one Ref => Name and one Name => Ref.
You might observe these maps and verify that they do not grow.

It should be possible to write a trace match spec that filters out and only
delivers this field to the tracer.

>
> *M*icael *N*ussbaumer
> artistic portfolio <http://cargocollective.com/micaelnussbaumer>


--

/ Raimo Niskanen, Erlang/OTP, Ericsson AB
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Testing gen_statem timers

Torben Hoffmann
In reply to this post by more_than_3_characters
Hi Micael,

I'm very biassed here, but apart from my creator fondness many of my colleagues are very happy about the approach that my Chronos library takes when it comes to timers.


Basically, don't use the internal timers, but do read the full README from Chronos.

Very happy to assist you and improve anything in Chronos, if you find it confusing.

Cheers,
Torben

On Thu, 15 Aug 2019 at 13:35, Micael Nussbaumer <[hidden email]> wrote:
Hi, first time poster (but I've read many threads in here).

How does one usually test gen_statem internal timer events?
Usually you test the effects of the behaviour somehow, and I can test that. But in this particular case I had a situation where timers with uniq references are created, later on they usually get removed and all is fine - they're created for each extra worker started and at some point the worker is removed by the timer event that trigger checking some conditions.

I noticed after that, the way I had written it, if an extra worker was started then died for some other reason than the timer event taking it out, I could theoretically accumulate timers that wouldn't be removed, I've since then corrected that but still I have no tests ensuring me that it actually works. 
So my question is if there's any even way to test it (even if with tracing)?
Thanks

Micael Nussbaumer
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


--

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Testing gen_statem timers

Roger Lipscombe-2
In reply to this post by more_than_3_characters
I'm not convinced that this is how I'd do it these days, but I dug
through some of our existing unit tests and found some code that uses
dbg:tpl(erlang, send_after, ...), etc. to track whether a process has
pending timers.

I yoinked it out of our source tree and stuck it in a gist:
https://gist.github.com/rlipscombe/a0970b147b35cfea6330533431e3de4c

On Thu, 15 Aug 2019 at 12:35, Micael Nussbaumer
<[hidden email]> wrote:

>
> Hi, first time poster (but I've read many threads in here).
>
> How does one usually test gen_statem internal timer events?
> Usually you test the effects of the behaviour somehow, and I can test that. But in this particular case I had a situation where timers with uniq references are created, later on they usually get removed and all is fine - they're created for each extra worker started and at some point the worker is removed by the timer event that trigger checking some conditions.
>
> I noticed after that, the way I had written it, if an extra worker was started then died for some other reason than the timer event taking it out, I could theoretically accumulate timers that wouldn't be removed, I've since then corrected that but still I have no tests ensuring me that it actually works.
> So my question is if there's any even way to test it (even if with tracing)?
> Thanks
>
> Micael Nussbaumer
> artistic portfolio
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Testing gen_statem timers

Raimo Niskanen-11
In reply to this post by Torben Hoffmann
On Thu, Aug 15, 2019 at 06:44:43PM +0200, Torben Hoffmann wrote:
> Hi Micael,
>
> I'm very biassed here, but apart from my creator fondness many of my
> colleagues are very happy about the approach that my Chronos library takes
> when it comes to timers.
>
> https://github.com/lehoff/chronos

I notice that in that README.md you compare Chronos with the timeout feature
in gen_server and gen_fsm.  The timers in gen_statem were specifically
designed to remedy most of their shortcomings.

So, from a usability point of view, can you elaborate what Cronos
can do better than gen_statem timers?

>
> Basically, don't use the internal timers, but do read the full README from
> Chronos.
>
> Very happy to assist you and improve anything in Chronos, if you find it
> confusing.
>
> Cheers,
> Torben
>
> On Thu, 15 Aug 2019 at 13:35, Micael Nussbaumer <[hidden email]>
> wrote:
>
> > Hi, first time poster (but I've read many threads in here).
> >
> > How does one usually test gen_statem internal timer events?
> > Usually you test the effects of the behaviour somehow, and I can test
> > that. But in this particular case I had a situation where timers with uniq
> > references are created, later on they usually get removed and all is fine -
> > they're created for each extra worker started and at some point the worker
> > is removed by the timer event that trigger checking some conditions.
> >
> > I noticed after that, the way I had written it, if an extra worker was
> > started then died for some other reason than the timer event taking it out,
> > I could theoretically accumulate timers that wouldn't be removed, I've
> > since then corrected that but still I have no tests ensuring me that it
> > actually works.
> > So my question is if there's any even way to test it (even if with
> > tracing)?
> > Thanks
> >
> > *M*icael *N*ussbaumer
> > artistic portfolio <http://cargocollective.com/micaelnussbaumer>
> >
>
>
> --
> http://www.linkedin.com/in/torbenhoffmann
> @LeHoff


--

/ Raimo Niskanen, Erlang/OTP, Ericsson AB
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Testing gen_statem timers

Raimo Niskanen-11
In reply to this post by more_than_3_characters
Might timer logging / status be a desirable missing debug feature
for gen_statem?
/ Raimo


On Thu, Aug 15, 2019 at 12:33:03PM +0100, Micael Nussbaumer wrote:

> Hi, first time poster (but I've read many threads in here).
>
> How does one usually test gen_statem internal timer events?
> Usually you test the effects of the behaviour somehow, and I can test that.
> But in this particular case I had a situation where timers with uniq
> references are created, later on they usually get removed and all is fine -
> they're created for each extra worker started and at some point the worker
> is removed by the timer event that trigger checking some conditions.
>
> I noticed after that, the way I had written it, if an extra worker was
> started then died for some other reason than the timer event taking it out,
> I could theoretically accumulate timers that wouldn't be removed, I've
> since then corrected that but still I have no tests ensuring me that it
> actually works.
> So my question is if there's any even way to test it (even if with tracing)?
> Thanks
>
> *M*icael *N*ussbaumer
> artistic portfolio <http://cargocollective.com/micaelnussbaumer>

> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions


--

/ Raimo Niskanen, Erlang/OTP, Ericsson AB
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Testing gen_statem timers

Torben Hoffmann
In reply to this post by Raimo Niskanen-11


On Fri, 16 Aug 2019 at 10:43, Raimo Niskanen <[hidden email]> wrote:
On Thu, Aug 15, 2019 at 06:44:43PM +0200, Torben Hoffmann wrote:
> Hi Micael,
>
> I'm very biassed here, but apart from my creator fondness many of my
> colleagues are very happy about the approach that my Chronos library takes
> when it comes to timers.
>
> https://github.com/lehoff/chronos

I notice that in that README.md you compare Chronos with the timeout feature
in gen_server and gen_fsm.  The timers in gen_statem were specifically
designed to remedy most of their shortcomings.

So, from a usability point of view, can you elaborate what Cronos
can do better than gen_statem timers?

I have a todo to update the README.md to cover gen_statem as well - thanks for reminding me!

I have worked a lot with gen_statem - absolutely a very good component and easy to work with.
But the problem of mocking still remains the same.
When testing I prefer to mock time so that I don't have to wait for the timeout to occur.
Furthermore, I'm so heavily influenced by my ten years at Motorola that I want the timers to stand out as in all the nice SDL diagrams in the standards shows.
That means that I want an easy way to check that a timer has been started (mock the call to chronos:start_timer()) and then an easy way to induce the expiry of said timer without having to wait for whatever the delay is.
When you have timers in the minutes range you just don't want your automated tests to wait it out.

If there is a way to avoid this with gen_statem I'd be most interested!

Cheers,
Torben
 
>
> Basically, don't use the internal timers, but do read the full README from
> Chronos.
>
> Very happy to assist you and improve anything in Chronos, if you find it
> confusing.
>
> Cheers,
> Torben
>
> On Thu, 15 Aug 2019 at 13:35, Micael Nussbaumer <[hidden email]>
> wrote:
>
> > Hi, first time poster (but I've read many threads in here).
> >
> > How does one usually test gen_statem internal timer events?
> > Usually you test the effects of the behaviour somehow, and I can test
> > that. But in this particular case I had a situation where timers with uniq
> > references are created, later on they usually get removed and all is fine -
> > they're created for each extra worker started and at some point the worker
> > is removed by the timer event that trigger checking some conditions.
> >
> > I noticed after that, the way I had written it, if an extra worker was
> > started then died for some other reason than the timer event taking it out,
> > I could theoretically accumulate timers that wouldn't be removed, I've
> > since then corrected that but still I have no tests ensuring me that it
> > actually works.
> > So my question is if there's any even way to test it (even if with
> > tracing)?
> > Thanks
> >
> > *M*icael *N*ussbaumer
> > artistic portfolio <http://cargocollective.com/micaelnussbaumer>
> >
>
>
> --
> http://www.linkedin.com/in/torbenhoffmann
> @LeHoff


--

/ Raimo Niskanen, Erlang/OTP, Ericsson AB
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


--

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Testing gen_statem timers

Jesper Louis Andersen-2
On Fri, Aug 16, 2019 at 12:36 PM Torben Hoffmann <[hidden email]> wrote:
When testing I prefer to mock time so that I don't have to wait for the timeout to occur.

The key observation is this:

You want time to be an injected parameter.

First, note that there is no concept of "now". Any call to e.g., `erlang:monotonic_time()` is behind as soon as your code has that value, by a small epsilon. In many cases, it is late by a lot; several milliseconds. So it is better to have your code take a parameter which is a point in time. When this is received, you roll time forward to that point in your internal state.

Second, timers are the scourge of testing. They are side-effects which means they are usually handled by mocking. And mocking is never the answer if there is a decent way around it. However, if a timer turns into a message, then we can choose to have timers handled outside the component and then send messages into the component. This helps testing, as we can run with a time-compression component, removing typical long waits in our test cases. In fact, separation of effects from computation is almost always a win for code in general.

Third, in systems you want a clock that is stationary and only moves when you call `advance_time(Ms)` or something such. A clock which moves behind your back is just going to create a lot of havoc in your code and computation. So it is better to ignore the real physical world, and then use messaging to handle the advance of the clock.

When I did some QuickCheck work, I built up a submodel which plays the role as the internal erlang timers. This means the model can simulate timers in the system and make choices as to when a timer fires, when time advances, what the interesting timestamps are, and so on. In particular, the order in which timers fire is nondeterministic. This is a mocking solution to the problem, and it required the code to be structured around being easy to hook. In QuickCheck tests, you must know in advance what is going to happen to the system-under-test. Perhaps a way to ask a component for its current timers, a way to disable the firing of timers and a way to inject a given timer back would give a tester the tooling for doing dynamic interaction, like most unit tests are doing.

Also, I've been toying a bit with Janes St. Incremental implementation (In ocaml). They also have a stationary clock which can be advanced from the outside[0].

[0] Getting self-adjusting computations into Erlang is actually a hard problem. The implementation relies on being able to control the firing schedule of incremental nodes, something you cannot easily do in Erlang processes say.




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Testing gen_statem timers

Fred Hebert-2

On Fri, Aug 16, 2019 at 7:07 AM Jesper Louis Andersen <[hidden email]> wrote:

Second, timers are the scourge of testing. They are side-effects which means they are usually handled by mocking. And mocking is never the answer if there is a decent way around it. However, if a timer turns into a message, then we can choose to have timers handled outside the component and then send messages into the component. This helps testing, as we can run with a time-compression component, removing typical long waits in our test cases. In fact, separation of effects from computation is almost always a win for code in general.

This has been my general go-to strategy, but note that this isn't really doable with a gen_statem since the timer events have their own type of message, which you cannot fake in a generic manner (the same way you would just do it with a timeout message in other behaviours). You at the very least need to add a special injection-creating callback that uses the next_event action to inject a timeout of the type you want, and the way -ifdefs work in Erlang, that clause can't optionally be compiled out for production.

Other types of mocking or time handling otherwise need to be aware of implementation details in gen_statem to know which clocks to mock and advance.


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Testing gen_statem timers

Raimo Niskanen-11
In reply to this post by Torben Hoffmann
On Fri, Aug 16, 2019 at 12:36:29PM +0200, Torben Hoffmann wrote:

> On Fri, 16 Aug 2019 at 10:43, Raimo Niskanen <
> [hidden email]> wrote:
>
> > On Thu, Aug 15, 2019 at 06:44:43PM +0200, Torben Hoffmann wrote:
> > > Hi Micael,
> > >
> > > I'm very biassed here, but apart from my creator fondness many of my
> > > colleagues are very happy about the approach that my Chronos library
> > takes
> > > when it comes to timers.
> > >
> > > https://github.com/lehoff/chronos
> >
> > I notice that in that README.md you compare Chronos with the timeout
> > feature
> > in gen_server and gen_fsm.  The timers in gen_statem were specifically
> > designed to remedy most of their shortcomings.
> >
> > So, from a usability point of view, can you elaborate what Cronos
> > can do better than gen_statem timers?
> >
> > I have a todo to update the README.md to cover gen_statem as well - thanks
> for reminding me!

Great!  No problem ;-)

>
> I have worked a lot with gen_statem - absolutely a very good component and
> easy to work with.

Always nice to hear! :-)

> But the problem of mocking still remains the same.
> When testing I prefer to mock time so that I don't have to wait for the
> timeout to occur.
> Furthermore, I'm so heavily influenced by my ten years at Motorola that I
> want the timers to stand out as in all the nice SDL diagrams in the
> standards shows.
> That means that I want an easy way to check that a timer has been started
> (mock the call to chronos:start_timer()) and then an easy way to induce the
> expiry of said timer without having to wait for whatever the delay is.
> When you have timers in the minutes range you just don't want your
> automated tests to wait it out.
>
> If there is a way to avoid this with gen_statem I'd be most interested!

Not today.

For debugging only, not mocking, it would be possible to log timer actions
in the sys log, and to include timer statistics i.e the number of running
timers or a list of them in sys:get_status/1.

Would that be interesting fetures?

For mocking, a possible new feature would be to make the calls to
erlang:start_timer/4 and erlang:cancel_timer/1 pluggable, i.e call them
through fun:s.  Then the plugged in functions may for example communicate
with a registered timer server that can do as it pleases about when to
fire timeouts, log timer starts and cancels, etc...

How dynamic would such a feature need to be?
* An option to gen_statem:start{,_link}/3,4?
* Read application variable at server start?
* Return an action from the callback module (so the callback module
  can implement a function to activate mocking)?
* Other suggestions?

Interesting topic - I did not forsee this need...

BRs
/ Raimo


>
> Cheers,
> Torben
>
>
> > >
> > > Basically, don't use the internal timers, but do read the full README
> > from
> > > Chronos.
> > >
> > > Very happy to assist you and improve anything in Chronos, if you find it
> > > confusing.
> > >
> > > Cheers,
> > > Torben
> > >
> > > On Thu, 15 Aug 2019 at 13:35, Micael Nussbaumer <
> > [hidden email]>
> > > wrote:
> > >
> > > > Hi, first time poster (but I've read many threads in here).
> > > >
> > > > How does one usually test gen_statem internal timer events?
> > > > Usually you test the effects of the behaviour somehow, and I can test
> > > > that. But in this particular case I had a situation where timers with
> > uniq
> > > > references are created, later on they usually get removed and all is
> > fine -
> > > > they're created for each extra worker started and at some point the
> > worker
> > > > is removed by the timer event that trigger checking some conditions.
> > > >
> > > > I noticed after that, the way I had written it, if an extra worker was
> > > > started then died for some other reason than the timer event taking it
> > out,
> > > > I could theoretically accumulate timers that wouldn't be removed, I've
> > > > since then corrected that but still I have no tests ensuring me that it
> > > > actually works.
> > > > So my question is if there's any even way to test it (even if with
> > > > tracing)?
> > > > Thanks
> > > >
> > > > *M*icael *N*ussbaumer
> > > > artistic portfolio <http://cargocollective.com/micaelnussbaumer>
> > > >
> > >
> > >
> > > --
> > > http://www.linkedin.com/in/torbenhoffmann
> > > @LeHoff
> >
> >
> > --
> >
> > / Raimo Niskanen, Erlang/OTP, Ericsson AB
> > _______________________________________________
> > erlang-questions mailing list
> > [hidden email]
> > http://erlang.org/mailman/listinfo/erlang-questions
> >
>
>
> --
> http://www.linkedin.com/in/torbenhoffmann
> @LeHoff

> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions


--

/ Raimo Niskanen, Erlang/OTP, Ericsson AB
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Testing gen_statem timers

more_than_3_characters
In reply to this post by more_than_3_characters
Hi, thank you all for your inputs. I ended up doing a trace as Raimo suggested and it works fine

I think I could also have implemented `format_status` for the gen_statem and extract the timers tuple from it? Not sure. Regarding having an API for it I think it could be useful since that implementation detail is opaque.

Torben, thank you sharing your lib - it looks like it can be quite handy but in this particular case it's for a library as well and I would rather not use a dependency, and since the timer options in the gen_statem work great by default I would like to keep using those instead of incurring another message to set the timer & executing a callback. I can definitively understand the appeal of having it instead be mockable though. 

For the timeouts when I wrote the module that uses these timers I added the option of setting their value on start, so that indeed I would be able to test the behaviour easily in terms of ms and for that part it works great. It was just this detail that was bugging me.
On the other hand, mocking also introduces a layer of "this passes the tests but in reality it's not being tested" correct? I'm not very familiar with mocking in tests.

Roger, thank you so much for taking the time to extract that! So I ended up doing the state inspection instead, I haven't used dbg yet but it looks like it can be useful. What is the difference between using dbg instead of the bare trace? 
I guess now by looking at your example that indeed I could have done the same approach by tracing the timer sets/cancels instead of tracing the call functions and extracting the state, that is probably cleaner. 

Raimo, I replied to you by accident privately when using the list web gui - I thought it was going to the "thread" as a reply to you. And yes, perhaps on the sys:get_status it could include by default/on option also an element for the timers? I did in fact try calling that at some point after your first reply:

{data,[{"Status",running},

                 {"Parent",<0.86.0>},

                 {"Logged Events",[]},

                 {"Postponed",[]}]},


Anyway, thank you all, great reading your replies,

Micael

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Testing gen_statem timers

Adam Lindberg-7
In reply to this post by Raimo Niskanen-11
I concur with Jesper and Fred. Timer data should be injected into the process, with a sensible production default.

Personally I would keep all the different timer values in a map in the process state, seeded by a default argument in my own MyProc:star_link/X function. This map can then be merged with an optional map supplied by the user. When a timer is started in gen_statem, you would do a lookup in this map based on e.g. the timer name. For tests, you could then inject a custom map where you override some timer values to match what is desired for the test. In my opinion, no extra support in gen_statem is needed for this.

Relying on shorter but still not exactly precise timeouts in tests is still tricky. What could be useful to have support for in gen_statem is firing the timeout events manually. I.e. in a test where you set up the timer in gen_statem, there would then be a way to manually fire a timer event in a gen_statem process. That way, the test becomes very readable and deterministic. Not sure how such an API would look like though.

Cheers,
Adam

> On 16. Aug 2019, at 15:47, Raimo Niskanen <[hidden email]> wrote:
>
> On Fri, Aug 16, 2019 at 12:36:29PM +0200, Torben Hoffmann wrote:
>> On Fri, 16 Aug 2019 at 10:43, Raimo Niskanen <
>> [hidden email]> wrote:
>>
>>> On Thu, Aug 15, 2019 at 06:44:43PM +0200, Torben Hoffmann wrote:
>>>> Hi Micael,
>>>>
>>>> I'm very biassed here, but apart from my creator fondness many of my
>>>> colleagues are very happy about the approach that my Chronos library
>>> takes
>>>> when it comes to timers.
>>>>
>>>> https://github.com/lehoff/chronos
>>>
>>> I notice that in that README.md you compare Chronos with the timeout
>>> feature
>>> in gen_server and gen_fsm.  The timers in gen_statem were specifically
>>> designed to remedy most of their shortcomings.
>>>
>>> So, from a usability point of view, can you elaborate what Cronos
>>> can do better than gen_statem timers?
>>>
>>> I have a todo to update the README.md to cover gen_statem as well - thanks
>> for reminding me!
>
> Great!  No problem ;-)
>
>>
>> I have worked a lot with gen_statem - absolutely a very good component and
>> easy to work with.
>
> Always nice to hear! :-)
>
>> But the problem of mocking still remains the same.
>> When testing I prefer to mock time so that I don't have to wait for the
>> timeout to occur.
>> Furthermore, I'm so heavily influenced by my ten years at Motorola that I
>> want the timers to stand out as in all the nice SDL diagrams in the
>> standards shows.
>> That means that I want an easy way to check that a timer has been started
>> (mock the call to chronos:start_timer()) and then an easy way to induce the
>> expiry of said timer without having to wait for whatever the delay is.
>> When you have timers in the minutes range you just don't want your
>> automated tests to wait it out.
>>
>> If there is a way to avoid this with gen_statem I'd be most interested!
>
> Not today.
>
> For debugging only, not mocking, it would be possible to log timer actions
> in the sys log, and to include timer statistics i.e the number of running
> timers or a list of them in sys:get_status/1.
>
> Would that be interesting fetures?
>
> For mocking, a possible new feature would be to make the calls to
> erlang:start_timer/4 and erlang:cancel_timer/1 pluggable, i.e call them
> through fun:s.  Then the plugged in functions may for example communicate
> with a registered timer server that can do as it pleases about when to
> fire timeouts, log timer starts and cancels, etc...
>
> How dynamic would such a feature need to be?
> * An option to gen_statem:start{,_link}/3,4?
> * Read application variable at server start?
> * Return an action from the callback module (so the callback module
>  can implement a function to activate mocking)?
> * Other suggestions?
>
> Interesting topic - I did not forsee this need...
>
> BRs
> / Raimo
>
>
>>
>> Cheers,
>> Torben
>>
>>
>>>>
>>>> Basically, don't use the internal timers, but do read the full README
>>> from
>>>> Chronos.
>>>>
>>>> Very happy to assist you and improve anything in Chronos, if you find it
>>>> confusing.
>>>>
>>>> Cheers,
>>>> Torben
>>>>
>>>> On Thu, 15 Aug 2019 at 13:35, Micael Nussbaumer <
>>> [hidden email]>
>>>> wrote:
>>>>
>>>>> Hi, first time poster (but I've read many threads in here).
>>>>>
>>>>> How does one usually test gen_statem internal timer events?
>>>>> Usually you test the effects of the behaviour somehow, and I can test
>>>>> that. But in this particular case I had a situation where timers with
>>> uniq
>>>>> references are created, later on they usually get removed and all is
>>> fine -
>>>>> they're created for each extra worker started and at some point the
>>> worker
>>>>> is removed by the timer event that trigger checking some conditions.
>>>>>
>>>>> I noticed after that, the way I had written it, if an extra worker was
>>>>> started then died for some other reason than the timer event taking it
>>> out,
>>>>> I could theoretically accumulate timers that wouldn't be removed, I've
>>>>> since then corrected that but still I have no tests ensuring me that it
>>>>> actually works.
>>>>> So my question is if there's any even way to test it (even if with
>>>>> tracing)?
>>>>> Thanks
>>>>>
>>>>> *M*icael *N*ussbaumer
>>>>> artistic portfolio <http://cargocollective.com/micaelnussbaumer>
>>>>>
>>>>
>>>>
>>>> --
>>>> http://www.linkedin.com/in/torbenhoffmann
>>>> @LeHoff
>>>
>>>
>>> --
>>>
>>> / Raimo Niskanen, Erlang/OTP, Ericsson AB
>>> _______________________________________________
>>> erlang-questions mailing list
>>> [hidden email]
>>> http://erlang.org/mailman/listinfo/erlang-questions
>>>
>>
>>
>> --
>> http://www.linkedin.com/in/torbenhoffmann
>> @LeHoff
>
>> _______________________________________________
>> erlang-questions mailing list
>> [hidden email]
>> http://erlang.org/mailman/listinfo/erlang-questions
>
>
> --
>
> / Raimo Niskanen, Erlang/OTP, Ericsson AB
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Testing gen_statem timers

Torben Hoffmann
In reply to this post by Raimo Niskanen-11


On Fri, 16 Aug 2019 at 15:47, Raimo Niskanen <[hidden email]> wrote:

> But the problem of mocking still remains the same.
> When testing I prefer to mock time so that I don't have to wait for the
> timeout to occur.
> Furthermore, I'm so heavily influenced by my ten years at Motorola that I
> want the timers to stand out as in all the nice SDL diagrams in the
> standards shows.
> That means that I want an easy way to check that a timer has been started
> (mock the call to chronos:start_timer()) and then an easy way to induce the
> expiry of said timer without having to wait for whatever the delay is.
> When you have timers in the minutes range you just don't want your
> automated tests to wait it out.
>
> If there is a way to avoid this with gen_statem I'd be most interested!

Not today.

For debugging only, not mocking, it would be possible to log timer actions
in the sys log, and to include timer statistics i.e the number of running
timers or a list of them in sys:get_status/1.

Would that be interesting fetures?

For mocking, a possible new feature would be to make the calls to
erlang:start_timer/4 and erlang:cancel_timer/1 pluggable, i.e call them
through fun:s.  Then the plugged in functions may for example communicate
with a registered timer server that can do as it pleases about when to
fire timeouts, log timer starts and cancels, etc...

How dynamic would such a feature need to be?
* An option to gen_statem:start{,_link}/3,4?
* Read application variable at server start?
* Return an action from the callback module (so the callback module
  can implement a function to activate mocking)?
* Other suggestions?

Interesting topic - I did not forsee this need...

BRs
/ Raimo

I like the idea of making the calls to start an cancel timers pluggable.

The key requirement is that there is a function call that one can detect and that the start of a timer doesn't actually trigger the creation of a timer since one needs the ability to inject the timer expiry into the gen_statem when the test calls for it.
On that note it might not be good to just expose the erlang:*_timer() functions as they would be un-mockable.
Maybe something like a call to a dummy function when the timers start/cancel. That can be mocked or spotted by a trace.
And then a way to inject by using a function in the gen_statem module that would only work in timers_testing mode.
I would welcome this as an option to gen_statem:start{,_link}/3,4 that one would only use for testing. It should simply turn the erlang:*_timer() calls into calls to those dummy functions.

This seems like the least invasive way of doing it. And then we can all expect a beer from the first person to ship to production with the testing_timers option ;-)

Given that gen_statem has introduced the notion of generic timeouts a mechanism such as the above would render Chronos superfluous when it comes to testing timers with gen_statem. 
I would welcome that despite all the help I have gotten from Chronos over the years.

Cheers,
Torben


--

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Testing gen_statem timers

I Gusti Ngurah Oka Prinarjaya
In reply to this post by more_than_3_characters
Hi,

> On the other hand, mocking also introduces a layer of "this passes the tests but in reality it's not being tested" correct? I'm not very familiar with mocking in tests.

For me, you should not use Unit Testing as an evidence to guarantee your system working well in reality. To do that, you should doing Integration Testing so you will never met a thing called "Mock" so it can guarantee your system working well in reality. 

Unit testing is excellent for locking your rules, locking your logic flow so nobody can change your rules, logic carelessly. 


Cheers!




Pada tanggal Sab, 17 Agu 2019 pukul 23.44 Micael Nussbaumer <[hidden email]> menulis:
Hi, thank you all for your inputs. I ended up doing a trace as Raimo suggested and it works fine

I think I could also have implemented `format_status` for the gen_statem and extract the timers tuple from it? Not sure. Regarding having an API for it I think it could be useful since that implementation detail is opaque.

Torben, thank you sharing your lib - it looks like it can be quite handy but in this particular case it's for a library as well and I would rather not use a dependency, and since the timer options in the gen_statem work great by default I would like to keep using those instead of incurring another message to set the timer & executing a callback. I can definitively understand the appeal of having it instead be mockable though. 

For the timeouts when I wrote the module that uses these timers I added the option of setting their value on start, so that indeed I would be able to test the behaviour easily in terms of ms and for that part it works great. It was just this detail that was bugging me.
On the other hand, mocking also introduces a layer of "this passes the tests but in reality it's not being tested" correct? I'm not very familiar with mocking in tests.

Roger, thank you so much for taking the time to extract that! So I ended up doing the state inspection instead, I haven't used dbg yet but it looks like it can be useful. What is the difference between using dbg instead of the bare trace? 
I guess now by looking at your example that indeed I could have done the same approach by tracing the timer sets/cancels instead of tracing the call functions and extracting the state, that is probably cleaner. 

Raimo, I replied to you by accident privately when using the list web gui - I thought it was going to the "thread" as a reply to you. And yes, perhaps on the sys:get_status it could include by default/on option also an element for the timers? I did in fact try calling that at some point after your first reply:

{data,[{"Status",running},

                 {"Parent",<0.86.0>},

                 {"Logged Events",[]},

                 {"Postponed",[]}]},


Anyway, thank you all, great reading your replies,

Micael
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions