How does GHC's testsuite work?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

How does GHC's testsuite work?

Sébastien Hinderer
Dear all,

I am a member of OCaml's developement team. More specifically, I am
working on a test-driver for the OCaml compiler, which will be part of
OCaml's 4.06 release.

I am currently writing an article to describe the tool and its
principles. In this article, I would like to also talk about how other
compilers' testsuites are handled and loking how things are done in GHC
is natural.

In OCaml, our testsuite essentially consist in whole programs that
we compile and run, checking that the compilation and execution results
match the expected ones.

From what I could see from GHC's testsuite, it seemed to me that it uses
Python to drive the tests. I also understood that the testsuite has
tests that are more kind of unit-tests, in the .T file. Am I correct
here? Or do you guys also have whole program tests?
If you do, how do you compile and run them?

Any comment / hint on this aspect of the test harness' design would be
really helpful.

Many thanks in advance,

Sébastien.





_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: How does GHC's testsuite work?

Edward Z. Yang
Actually, it's the reverse of what you said: like OCaml, GHC essentially
has ~no unit tests; it's entirely Haskell programs which we compile
(and sometimes run; a lot of tests are for the typechecker only so
we don't bother running those.)  The .T file is just a way of letting
the Python driver know what tests exist.

Edward

Excerpts from Sébastien Hinderer's message of 2017-10-30 16:17:38 +0100:

> Dear all,
>
> I am a member of OCaml's developement team. More specifically, I am
> working on a test-driver for the OCaml compiler, which will be part of
> OCaml's 4.06 release.
>
> I am currently writing an article to describe the tool and its
> principles. In this article, I would like to also talk about how other
> compilers' testsuites are handled and loking how things are done in GHC
> is natural.
>
> In OCaml, our testsuite essentially consist in whole programs that
> we compile and run, checking that the compilation and execution results
> match the expected ones.
>
> From what I could see from GHC's testsuite, it seemed to me that it uses
> Python to drive the tests. I also understood that the testsuite has
> tests that are more kind of unit-tests, in the .T file. Am I correct
> here? Or do you guys also have whole program tests?
> If you do, how do you compile and run them?
>
> Any comment / hint on this aspect of the test harness' design would be
> really helpful.
>
> Many thanks in advance,
>
> Sébastien.
>
_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: How does GHC's testsuite work?

Sébastien Hinderer
Dear Edward,

Many thanks for your prompt response!

Edward Z. Yang (2017/10/30 11:25 -0400):
> Actually, it's the reverse of what you said: like OCaml, GHC essentially
> has ~no unit tests; it's entirely Haskell programs which we compile
> (and sometimes run; a lot of tests are for the typechecker only so
> we don't bother running those.)  The .T file is just a way of letting
> the Python driver know what tests exist.

Oh okay! Would you be able to point me to just a few tests to get an
idea of a few typical situations, please?

One other question I forgot to ask: how do you deal with conditional
tests? For instance, if a test should be run only on some platforms? Or,
in OCaml we have tests for Fortran bindings that should be run only if a
Fortran compiler is available. How would you deal with such tests?

Thanks!

Sébastien.
_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: How does GHC's testsuite work?

Edward Z. Yang
Excerpts from Sébastien Hinderer's message of 2017-10-30 16:39:24 +0100:

> Dear Edward,
>
> Many thanks for your prompt response!
>
> Edward Z. Yang (2017/10/30 11:25 -0400):
> > Actually, it's the reverse of what you said: like OCaml, GHC essentially
> > has ~no unit tests; it's entirely Haskell programs which we compile
> > (and sometimes run; a lot of tests are for the typechecker only so
> > we don't bother running those.)  The .T file is just a way of letting
> > the Python driver know what tests exist.
>
> Oh okay! Would you be able to point me to just a few tests to get an
> idea of a few typical situations, please?

For example:

    The metadata
    https://github.com/ghc/ghc/blob/master/testsuite/tests/typecheck/should_fail/all.T

    The source file
    https://github.com/ghc/ghc/blob/master/testsuite/tests/typecheck/should_fail/tcfail011.hs

    The expected error output
    https://github.com/ghc/ghc/blob/master/testsuite/tests/typecheck/should_fail/tcfail011.stderr

> One other question I forgot to ask: how do you deal with conditional
> tests? For instance, if a test should be run only on some platforms? Or,
> in OCaml we have tests for Fortran bindings that should be run only if a
> Fortran compiler is available. How would you deal with such tests?

All managed inside the Python driver code.

Example:
https://github.com/ghc/ghc/blob/master/testsuite/tests/rts/all.T#L32

Edward
_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: How does GHC's testsuite work?

Joachim Breitner-2
In reply to this post by Edward Z. Yang
Hi Sebastien,

I’m looking forward to your report, surely there will be some
interesting inspirations for us.

Am Montag, den 30.10.2017, 11:25 -0400 schrieb Edward Z. Yang:
> Actually, it's the reverse of what you said: like OCaml, GHC essentially
> has ~no unit tests; it's entirely Haskell programs which we compile
> (and sometimes run; a lot of tests are for the typechecker only so
> we don't bother running those.)  The .T file is just a way of letting
> the Python driver know what tests exist.

let me add that these tests rarely check the actual output of the
compiler (i.e. the program, or even the simplified code). Often it is
enough to check
 * whether the compile succeeds or fails as expected, or maybe
 * what messages the compiler prints.

In a few cases we do dump the complete intermediate code (-ddump-
simpl), but then the test case specifies a “normalization function”
that checks the output for a certain property, e.g. by grepping for
certain patterns.

The only real unit tests that I know of are these:
http://git.haskell.org/ghc.git/tree/HEAD:/testsuite/tests/callarity/unittest
These are effectively programs using “GHC-the-library”

Joachim


--
Joachim Breitner
  [hidden email]
  http://www.joachim-breitner.de/

_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: How does GHC's testsuite work?

Sébastien Hinderer
Dear Joachim,

Many thanks for your prompt, positive and encouraging response.

Joachim Breitner (2017/10/30 13:44 -0400):
> Hi Sebastien,
>
> I’m looking forward to your report, surely there will be some
> interesting inspirations for us.

Thanks. Unfortunately, the paper has been written in French. i'll see
whether I can find the time to translate it into english soon. If I can
do so I'll definitely post a link.

> Am Montag, den 30.10.2017, 11:25 -0400 schrieb Edward Z. Yang:
> > Actually, it's the reverse of what you said: like OCaml, GHC essentially
> > has ~no unit tests; it's entirely Haskell programs which we compile
> > (and sometimes run; a lot of tests are for the typechecker only so
> > we don't bother running those.)  The .T file is just a way of letting
> > the Python driver know what tests exist.
>
> let me add that these tests rarely check the actual output of the
> compiler (i.e. the program, or even the simplified code). Often it is
> enough to check
>  * whether the compile succeeds or fails as expected, or maybe
>  * what messages the compiler prints.

I see. I think it is quite similar for OCaml.

> In a few cases we do dump the complete intermediate code (-ddump-
> simpl), but then the test case specifies a “normalization function”
> that checks the output for a certain property, e.g. by grepping for
> certain patterns.

Got it, thanks. We also have options to pretty-print the few internal
representations we have, but as far as I know, we don't use any
normalization function in such cases and just make a diff with an
expected reference (yuk!).

> The only real unit tests that I know of are these:
> http://git.haskell.org/ghc.git/tree/HEAD:/testsuite/tests/callarity/unittest
> These are effectively programs using “GHC-the-library”

Okay, thanks!

Will come back with a link ASAP!

Sébastien.
_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: How does GHC's testsuite work?

Sébastien Hinderer
In reply to this post by Edward Z. Yang
Dear Edward,

Many thanks to you, too, for your prompt response.

Edward Z. Yang (2017/10/30 11:51 -0400):

> Excerpts from Sébastien Hinderer's message of 2017-10-30 16:39:24 +0100:
> > Dear Edward,
> >
> > Many thanks for your prompt response!
> >
> > Edward Z. Yang (2017/10/30 11:25 -0400):
> > > Actually, it's the reverse of what you said: like OCaml, GHC essentially
> > > has ~no unit tests; it's entirely Haskell programs which we compile
> > > (and sometimes run; a lot of tests are for the typechecker only so
> > > we don't bother running those.)  The .T file is just a way of letting
> > > the Python driver know what tests exist.
> >
> > Oh okay! Would you be able to point me to just a few tests to get an
> > idea of a few typical situations, please?
>
> For example:
>
>     The metadata
>     https://github.com/ghc/ghc/blob/master/testsuite/tests/typecheck/should_fail/all.T
>
>     The source file
>     https://github.com/ghc/ghc/blob/master/testsuite/tests/typecheck/should_fail/tcfail011.hs
>
>     The expected error output
>     https://github.com/ghc/ghc/blob/master/testsuite/tests/typecheck/should_fail/tcfail011.stderr

Excellent, thanks! With these few hints I really got the understanding I
was looking for so I'm really grateful for that, thanks!

> > One other question I forgot to ask: how do you deal with conditional
> > tests? For instance, if a test should be run only on some platforms? Or,
> > in OCaml we have tests for Fortran bindings that should be run only if a
> > Fortran compiler is available. How would you deal with such tests?
>
> All managed inside the Python driver code.
>
> Example:
> https://github.com/ghc/ghc/blob/master/testsuite/tests/rts/all.T#L32

okay thanks, awesome!

Best wishes,

Sébastien.
_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: How does GHC's testsuite work?

Wolfram Kahl
Cher Sébastien,

> Thanks. Unfortunately, the paper has been written in French.

No need to add ``Unfortunately''...  ;-)


> Will come back with a link ASAP!

I guess that many will agree with my opinion:
Links to French papers are welcome, too!


Amicalement,

Wolfram
_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/glasgow-haskell-users