Do you know what your code is doing at night?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Do you know what your code is doing at night?

Fred Hebert-2
Hello everyone,

If you've been curious about property-based testing for a while but
found it kind of intimidating or didn't really feel like digging in the
source or asking bloggers or people who implement their own libraries to
get going, I've got a thing you might like.

I've been assembling my own notes on property-based testing over the
last few months, and have just made http://propertesting.com/ public,
even if it is not fully complete yet.

Hopefully, if you're one of those people who want to figure out what the
properties of their code is or you want to make sure they're actually
enforced properly, this could be of help. Have a read and let me know
what you think.

Regards,
Fred.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Do you know what your code is doing at night?

Manfred Lotz
Hi Fred,
I'm an Erlang beginner so I'm not the best qualified guy to make
comments.

Nevertheless, I took a look and I must say you are an extremely good
writer (ok, no surprise looking at your LYSE book).

Hopefully, in August I will have more time to work thorougly thru your
article. Then I will give you some more feedback.


However, I have a general question:

Are unit testing and property based testing to be seen as both
complementing each other, or is property based testing to be regarded
to fully replace unit testing as it should (?) cover a proper superset
of the tests in unit testing?

Thanks a lot.

--
Best, Manfred


On Mon, 10 Jul 2017 13:38:08 -0400
Fred Hebert <[hidden email]> wrote:

> Hello everyone,
>
> If you've been curious about property-based testing for a while but
> found it kind of intimidating or didn't really feel like digging in
> the source or asking bloggers or people who implement their own
> libraries to get going, I've got a thing you might like.
>
> I've been assembling my own notes on property-based testing over the
> last few months, and have just made http://propertesting.com/ public,
> even if it is not fully complete yet.
>
> Hopefully, if you're one of those people who want to figure out what
> the properties of their code is or you want to make sure they're
> actually enforced properly, this could be of help. Have a read and
> let me know what you think.
>
> Regards,
> Fred.
> _______________________________________________
> erlang-questions mailing list
> [hidden email]
> http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Do you know what your code is doing at night?

Loïc Hoguin-3
On 07/13/2017 09:12 AM, Manfred Lotz wrote:
> However, I have a general question:
>
> Are unit testing and property based testing to be seen as both
> complementing each other, or is property based testing to be regarded
> to fully replace unit testing as it should (?) cover a proper superset
> of the tests in unit testing?
I'm not Fred, but I do have some insight on that.

Property based testing can largely replace the few random unit tests you
would write to test cases that you think about. It will also test cases
that you will not think about, and that's good.

Unit testing is still useful to make sure the textbook cases still work
though. If you have a spec (like an RFC) with some examples, you will
want to write unit tests for those, because properties are code and
might not be perfect.

And similarly, unit testing is still useful for testing against
regressions. Even if you fix your property, you should write a unit test
with the previously failing case to make sure you don't break your code
and your property again.

--
Loïc Hoguin
https://ninenines.eu
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Do you know what your code is doing at night?

Fred Hebert-2
I pretty much agree with Loïc here.

I'd say standard tests will still make sense for:

- working around when finding a good property is really hard and/or
  taking time, especially if you're on a tighter schedule, even more
  likely to happen as a beginner.
- To augment a property. For example, the classic 'decode(encode(X)) ==
  X' case is made stronger if you have ways to say what the encoded
  format should look like as well. Regular tests can work as 'anchors'
  that cover some edges that are hard to do as a property
- To store counter-examples of past failures as regression suites
- To write some explicit examples of your code to ensure it works (as
  Loïc mentioned, RFCs are a good source of that)
- As a way to get started writing your properties (write a few examples,
  and *then* generalize them into a property)

The anchoring bit is a kind of important one, especially since if you're
not careful, it's easy to over-estimate the quality of property test.  
Usual tricks to help there (aside from anchoring with regular tests)
will be to:

- look at coverage
- look at the statistics of the distribution of the generators
- do negative testing of your tests (break the code on purpose in a
  subtle way, make sure one property finds the error)

So the TL:DR; would probably be that a bunch of unit tests are
complementary, but a bunch of them can be made redundant as well.

On 07/13, Loïc Hoguin wrote:

>On 07/13/2017 09:12 AM, Manfred Lotz wrote:
>>However, I have a general question:
>>
>>Are unit testing and property based testing to be seen as both
>>complementing each other, or is property based testing to be regarded
>>to fully replace unit testing as it should (?) cover a proper superset
>>of the tests in unit testing?
>I'm not Fred, but I do have some insight on that.
>
>Property based testing can largely replace the few random unit tests
>you would write to test cases that you think about. It will also test
>cases that you will not think about, and that's good.
>
>Unit testing is still useful to make sure the textbook cases still
>work though. If you have a spec (like an RFC) with some examples, you
>will want to write unit tests for those, because properties are code
>and might not be perfect.
>
>And similarly, unit testing is still useful for testing against
>regressions. Even if you fix your property, you should write a unit
>test with the previously failing case to make sure you don't break
>your code and your property again.
>
>--
>Loïc Hoguin
>https://ninenines.eu
>_______________________________________________
>erlang-questions mailing list
>[hidden email]
>http://erlang.org/mailman/listinfo/erlang-questions
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Do you know what your code is doing at night?

Manfred Lotz
In reply to this post by Loïc Hoguin-3
Hi Loic,
Thanks for your reply.

On Thu, 13 Jul 2017 09:46:09 +0200
Loïc Hoguin <[hidden email]> wrote:

> On 07/13/2017 09:12 AM, Manfred Lotz wrote:
> > However, I have a general question:
> >
> > Are unit testing and property based testing to be seen as both
> > complementing each other, or is property based testing to be
> > regarded to fully replace unit testing as it should (?) cover a
> > proper superset of the tests in unit testing?  
> I'm not Fred, but I do have some insight on that.
>
> Property based testing can largely replace the few random unit tests
> you would write to test cases that you think about. It will also test
> cases that you will not think about, and that's good.
>
> Unit testing is still useful to make sure the textbook cases still
> work though. If you have a spec (like an RFC) with some examples, you
> will want to write unit tests for those, because properties are code
> and might not be perfect.
>
> And similarly, unit testing is still useful for testing against
> regressions. Even if you fix your property, you should write a unit
> test with the previously failing case to make sure you don't break
> your code and your property again.
>

Your reply is absolutely excellent, and makes much sense to me.

I think Fred could add a section "Unit testing versus property based
testing" and add your explanations.


Many thanks, Manfred



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Do you know what your code is doing at night?

Fred Hebert-2
On 07/13, Manfred Lotz wrote:
>I think Fred could add a section "Unit testing versus property based
>testing" and add your explanations.
>

Similar content is already part of the book, though not necessarily in a
dedicated section.
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Do you know what your code is doing at night?

Manfred Lotz
On Thu, 13 Jul 2017 07:16:06 -0400
Fred Hebert <[hidden email]> wrote:

> On 07/13, Manfred Lotz wrote:
> >I think Fred could add a section "Unit testing versus property based
> >testing" and add your explanations.
> >  
>
> Similar content is already part of the book, though not necessarily
> in a dedicated section.

I think it is worth to make your and Loic's answer a dedicated section
or subsection.


--
Manfred
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Do you know what your code is doing at night?

Jesper Louis Andersen-2
The question of linking unit tests and property based testing has been covered by Gerdes, Hughes, Smallbone, and Wang:


The idea here is to ask: Suppose we have a property based test suite and a set of unit tests. Can we check if the unit tests are already covered by the property suite? The paper answers this in the affirmative. You can construct a system which can answer the question, with great probability.

It is highly useful in the case where you have a specification and the specification contains certain examples. By making sure your suite covers those examples, you gain the knowledge that your suite is not totally off w.r.t the unit test examples.

In a system with property based testing, you can often track each counterexample you've encountered in a small suite of things to try first. This allows you to quickly check for regressions rather than generating new traces to try. OTOH, if you don't regularly generate the cases, then chances are your generator is off and not being evil enough >:-)

On Thu, Jul 13, 2017 at 3:11 PM Manfred Lotz <[hidden email]> wrote:
On Thu, 13 Jul 2017 07:16:06 -0400
Fred Hebert <[hidden email]> wrote:

> On 07/13, Manfred Lotz wrote:
> >I think Fred could add a section "Unit testing versus property based
> >testing" and add your explanations.
> >
>
> Similar content is already part of the book, though not necessarily
> in a dedicated section.

I think it is worth to make your and Loic's answer a dedicated section
or subsection.


--
Manfred
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Do you know what your code is doing at night?

Manfred Lotz
On Thu, 13 Jul 2017 21:01:17 +0000
Jesper Louis Andersen <[hidden email]> wrote:

> The question of linking unit tests and property based testing has been
> covered by Gerdes, Hughes, Smallbone, and Wang:
>
> http://www.cse.chalmers.se/~nicsma/papers/unit-tests.pdf
>
> The idea here is to ask: Suppose we have a property based test suite
> and a set of unit tests. Can we check if the unit tests are already
> covered by the property suite? The paper answers this in the
> affirmative. You can construct a system which can answer the
> question, with great probability.
>
> It is highly useful in the case where you have a specification and the
> specification contains certain examples. By making sure your suite
> covers those examples, you gain the knowledge that your suite is not
> totally off w.r.t the unit test examples.
>
> In a system with property based testing, you can often track each
> counterexample you've encountered in a small suite of things to try
> first. This allows you to quickly check for regressions rather than
> generating new traces to try. OTOH, if you don't regularly generate
> the cases, then chances are your generator is off and not being evil
> enough >:-)
>


Thanks for this. Very interesting.

Manfred
_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions