Storing failing testcases for QuickCheck

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Storing failing testcases for QuickCheck

Soenke Hahn
Hi!

I would like to have a library that would allow to use QuickCheck in the
normal manner, but it would save test data for failing properties on the
filesystem (maybe using the shiny new acid-state?). On consecutive test
runs, the saved test data would be used first (before generating new
arbitrary test data). I think, this would come in quite handy when doing
test driven development.

Any comments on this? Does something like this already exist?

Cheers,
Sönke



_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Storing failing testcases for QuickCheck

wren ng thornton
On 4/23/11 8:02 AM, Sönke Hahn wrote:

> Hi!
>
> I would like to have a library that would allow to use QuickCheck in the
> normal manner, but it would save test data for failing properties on the
> filesystem (maybe using the shiny new acid-state?). On consecutive test
> runs, the saved test data would be used first (before generating new
> arbitrary test data). I think, this would come in quite handy when doing
> test driven development.
>
> Any comments on this? Does something like this already exist?

The closest I can think of is using HUnit on the same property functions
you use in QuickCheck. I do this often for regression testing and to
ensure that corner cases are covered[1]. Unfortunately this still
requires a bit of hand-holding to keep your regression suite up to date.

I would *love* there to be a tool which (a) automatically saves failing
QuickCheck values to disk, and (b) automates using HUnit to load those
in and test them. I'm not so sure that QuickCheck should be doing the
second step of that since that'd really mess with the QuickCheck
infrastructure; once you have the code for reading from disk, it'd be
trivial to just use HUnit.


[1] I also do some overloading of (==>) and similar, so that I can use
the same properties in both QuickCheck and lazy SmallCheck.
Unfortunately, I've yet to come up with a good way to ensure that
QuickCheck doesn't explore territory already covered by SmallCheck.

--
Live well,
~wren

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Storing failing testcases for QuickCheck

Roel van Dijk-3
On 24 April 2011 01:49, wren ng thornton <[hidden email]> wrote:
> I would *love* there to be a tool which (a) automatically saves failing
> QuickCheck values to disk, and (b) automates using HUnit to load those in
> and test them. I'm not so sure that QuickCheck should be doing the second
> step of that since that'd really mess with the QuickCheck infrastructure;
> once you have the code for reading from disk, it'd be trivial to just use
> HUnit.

Maybe this is a job for test-framework? I think the API's of
QuickCheck and HUnit expose enough information for this to be
possible.

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Storing failing testcases for QuickCheck

Soenke Hahn
Roel van Dijk wrote:

> On 24 April 2011 01:49, wren ng thornton <[hidden email]> wrote:
>> I would *love* there to be a tool which (a) automatically saves failing
>> QuickCheck values to disk, and (b) automates using HUnit to load those in
>> and test them. I'm not so sure that QuickCheck should be doing the second
>> step of that since that'd really mess with the QuickCheck infrastructure;
>> once you have the code for reading from disk, it'd be trivial to just use
>> HUnit.
>
> Maybe this is a job for test-framework? I think the API's of
> QuickCheck and HUnit expose enough information for this to be
> possible.

I've hacked something together: [1]. It doesn't use neither HUnit nor test-
framework, but I can see, why that would probably be better. It seems to do
the job for me at the moment.

[1] https://patch-tag.com/r/shahn/QuickCheckStore/home



_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe