QuickCheck and State monad

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

QuickCheck and State monad

George Boulougaris
Hello,

I have written a Haskell module that contains functions that operate on some state. Let's say that the code looks like this (the actual functions may return an actual result instead of `()`, but this is irrelevant to the question):

    data StateContext = StateContext {
      -- some records
    }
    
    handleEventA :: EventA -> State StateContext ()
    handleEventB :: EventB -> State StateContext ()
    handleEventC :: EventC -> State StateContext ()

As you can imagine the behavior of each function depends on the current state. For example `handleEventA >> handleEventB` will not produce the same result as `handleEventB >> handleEventA`. So I have several HUnit tests that verify the behavior of each function at several states.

But now I would like to write more tests, that exercise all functions at all possible states (the number of states is finite). Writing them with HUnit would be quite labor-itensive, so I thought that using QuickCheck might be helpful in that case (I have only used it before for trivial functions). 

But I cannot see which properties should I test, or what kind of data should the test generate. I suspect that the test should generate random sequence of events (e.g. `handleEventB >> handleEventC >> handleEventA` etc), but I cannot see what properties should be satisfied.

Thanks


_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: QuickCheck and State monad

Mihai Maruseac
I'm assuming that each state in your example has something to
differentiate it from the others. That can be expanded to some
properties to test.

Another thing is that one event can move you from one state to only a
set of states. This can also be encoded as a property.

Finally, try to see if there are some properties/invariants that hold
when handling an event in a state. Doesn't have to be for all events
and all states, but for those where you can find some it should be
possible to write some tests.

On Sat, Feb 4, 2017 at 12:20 PM, George Boulougaris
<[hidden email]> wrote:

> Hello,
>
> I have written a Haskell module that contains functions that operate on some
> state. Let's say that the code looks like this (the actual functions may
> return an actual result instead of `()`, but this is irrelevant to the
> question):
>
>     data StateContext = StateContext {
>       -- some records
>     }
>
>     handleEventA :: EventA -> State StateContext ()
>     handleEventB :: EventB -> State StateContext ()
>     handleEventC :: EventC -> State StateContext ()
>
> As you can imagine the behavior of each function depends on the current
> state. For example `handleEventA >> handleEventB` will not produce the same
> result as `handleEventB >> handleEventA`. So I have several HUnit tests that
> verify the behavior of each function at several states.
>
> But now I would like to write more tests, that exercise all functions at all
> possible states (the number of states is finite). Writing them with HUnit
> would be quite labor-itensive, so I thought that using QuickCheck might be
> helpful in that case (I have only used it before for trivial functions).
>
> But I cannot see which properties should I test, or what kind of data should
> the test generate. I suspect that the test should generate random sequence
> of events (e.g. `handleEventB >> handleEventC >> handleEventA` etc), but I
> cannot see what properties should be satisfied.
>
> Thanks
>
>
> _______________________________________________
> Haskell-Cafe mailing list
> To (un)subscribe, modify options or view archives go to:
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
> Only members subscribed via the mailman list are allowed to post.



--
Mihai Maruseac (MM)
"If you can't solve a problem, then there's an easier problem you can
solve: find it." -- George Polya
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: QuickCheck and State monad

Robin Palotai
A very powerful idea is described in Testing Monadic Code With QuickCheck[1] (or a pdf version at [2]). I think the ideas apply to non-monadic (or non IO-like monadic) code as well.

The basic idea is to generate a sequence of events as test input, like

    events = [EventA, EventA, EventC, EventB]

and execute the corresponding actions (starting from an initial state). Now you can make assertions on the final state S and events that happened.

Some ideas for your case:
- If EventB happened, S must be foo
- If EventC never happened, S must be bar (note that negative event tests are very powerful)
- S must have baz=0 only if the count of EventAs is same as count of EventBs after the first EventC
- ...

You can also compare final states resulting from different event sequences:
- if E1 is a sequence with result S1, sticking EventD anywhere in E1 must also result in S1
- ...

The paper goes into details about generating a valid event sequence, if there are restrictions about in which state can a given event happen.

Section 3 & 4 are about testing that (the effect of) two given action sequences are equal, independent of previous/subsequent actions. This is useful if you don't only handle events, but can do some custom actions on your state. Like you can test that "foo >> barBaz" is always the same as "foo >> bar >> foo >> unicorn".

Section 10-12 bring an other example. Section 13 shows a way to more easily generate a valid action sequence, but only in case you have a parallel abstract (pure) implementation, which is not often the case (except algos), and maybe this is really specific to IO/ST.

[1]: www.cse.chalmers.se/~rjmh/Papers/QuickCheckST.ps
[2]: https://www.researchgate.net/publication/2831386_Testing_Monadic_Code_with_QuickCheck

Robin

2017-02-04 22:04 GMT+01:00 Mihai Maruseac <[hidden email]>:
I'm assuming that each state in your example has something to
differentiate it from the others. That can be expanded to some
properties to test.

Another thing is that one event can move you from one state to only a
set of states. This can also be encoded as a property.

Finally, try to see if there are some properties/invariants that hold
when handling an event in a state. Doesn't have to be for all events
and all states, but for those where you can find some it should be
possible to write some tests.

On Sat, Feb 4, 2017 at 12:20 PM, George Boulougaris
<[hidden email]> wrote:
> Hello,
>
> I have written a Haskell module that contains functions that operate on some
> state. Let's say that the code looks like this (the actual functions may
> return an actual result instead of `()`, but this is irrelevant to the
> question):
>
>     data StateContext = StateContext {
>       -- some records
>     }
>
>     handleEventA :: EventA -> State StateContext ()
>     handleEventB :: EventB -> State StateContext ()
>     handleEventC :: EventC -> State StateContext ()
>
> As you can imagine the behavior of each function depends on the current
> state. For example `handleEventA >> handleEventB` will not produce the same
> result as `handleEventB >> handleEventA`. So I have several HUnit tests that
> verify the behavior of each function at several states.
>
> But now I would like to write more tests, that exercise all functions at all
> possible states (the number of states is finite). Writing them with HUnit
> would be quite labor-itensive, so I thought that using QuickCheck might be
> helpful in that case (I have only used it before for trivial functions).
>
> But I cannot see which properties should I test, or what kind of data should
> the test generate. I suspect that the test should generate random sequence
> of events (e.g. `handleEventB >> handleEventC >> handleEventA` etc), but I
> cannot see what properties should be satisfied.
>
> Thanks
>
>
> _______________________________________________
> Haskell-Cafe mailing list
> To (un)subscribe, modify options or view archives go to:
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
> Only members subscribed via the mailman list are allowed to post.



--
Mihai Maruseac (MM)
"If you can't solve a problem, then there's an easier problem you can
solve: find it." -- George Polya
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.


_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: QuickCheck and State monad

George Boulougaris
Thanks for the replies,

I ended up generating random sequence of events and then checking some properties/invariants on the final state. I realized that in other languages I usually expressed these properties by writing assertions inside the code (e.g. by using the assert() function in C/C++ or throwing a RuntimeException in Java).

On Sun, Feb 5, 2017 at 5:41 AM, Robin Palotai <[hidden email]> wrote:
A very powerful idea is described in Testing Monadic Code With QuickCheck[1] (or a pdf version at [2]). I think the ideas apply to non-monadic (or non IO-like monadic) code as well.

The basic idea is to generate a sequence of events as test input, like

    events = [EventA, EventA, EventC, EventB]

and execute the corresponding actions (starting from an initial state). Now you can make assertions on the final state S and events that happened.

Some ideas for your case:
- If EventB happened, S must be foo
- If EventC never happened, S must be bar (note that negative event tests are very powerful)
- S must have baz=0 only if the count of EventAs is same as count of EventBs after the first EventC
- ...

You can also compare final states resulting from different event sequences:
- if E1 is a sequence with result S1, sticking EventD anywhere in E1 must also result in S1
- ...

The paper goes into details about generating a valid event sequence, if there are restrictions about in which state can a given event happen.

Section 3 & 4 are about testing that (the effect of) two given action sequences are equal, independent of previous/subsequent actions. This is useful if you don't only handle events, but can do some custom actions on your state. Like you can test that "foo >> barBaz" is always the same as "foo >> bar >> foo >> unicorn".

Section 10-12 bring an other example. Section 13 shows a way to more easily generate a valid action sequence, but only in case you have a parallel abstract (pure) implementation, which is not often the case (except algos), and maybe this is really specific to IO/ST.

[1]: www.cse.chalmers.se/~rjmh/Papers/QuickCheckST.ps
[2]: https://www.researchgate.net/publication/2831386_Testing_Monadic_Code_with_QuickCheck

Robin

2017-02-04 22:04 GMT+01:00 Mihai Maruseac <[hidden email]>:
I'm assuming that each state in your example has something to
differentiate it from the others. That can be expanded to some
properties to test.

Another thing is that one event can move you from one state to only a
set of states. This can also be encoded as a property.

Finally, try to see if there are some properties/invariants that hold
when handling an event in a state. Doesn't have to be for all events
and all states, but for those where you can find some it should be
possible to write some tests.

On Sat, Feb 4, 2017 at 12:20 PM, George Boulougaris
<[hidden email]> wrote:
> Hello,
>
> I have written a Haskell module that contains functions that operate on some
> state. Let's say that the code looks like this (the actual functions may
> return an actual result instead of `()`, but this is irrelevant to the
> question):
>
>     data StateContext = StateContext {
>       -- some records
>     }
>
>     handleEventA :: EventA -> State StateContext ()
>     handleEventB :: EventB -> State StateContext ()
>     handleEventC :: EventC -> State StateContext ()
>
> As you can imagine the behavior of each function depends on the current
> state. For example `handleEventA >> handleEventB` will not produce the same
> result as `handleEventB >> handleEventA`. So I have several HUnit tests that
> verify the behavior of each function at several states.
>
> But now I would like to write more tests, that exercise all functions at all
> possible states (the number of states is finite). Writing them with HUnit
> would be quite labor-itensive, so I thought that using QuickCheck might be
> helpful in that case (I have only used it before for trivial functions).
>
> But I cannot see which properties should I test, or what kind of data should
> the test generate. I suspect that the test should generate random sequence
> of events (e.g. `handleEventB >> handleEventC >> handleEventA` etc), but I
> cannot see what properties should be satisfied.
>
> Thanks
>
>
> _______________________________________________
> Haskell-Cafe mailing list
> To (un)subscribe, modify options or view archives go to:
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
> Only members subscribed via the mailman list are allowed to post.



--
Mihai Maruseac (MM)
"If you can't solve a problem, then there's an easier problem you can
solve: find it." -- George Polya
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.



_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.