[commit: testsuite] master: Higher residency in Haddock (15c09a3)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[commit: testsuite] master: Higher residency in Haddock (15c09a3)

Simon Marlow-7
On 22/11/13 16:43, git at git.haskell.org wrote:

> Repository : ssh://git at git.haskell.org/testsuite
>
> On branch  : master
> Link       : http://ghc.haskell.org/trac/ghc/changeset/15c09a332cd24e5d6c4c9a0ede9b151c9a095f66/testsuite
>
>> ---------------------------------------------------------------
>
> commit 15c09a332cd24e5d6c4c9a0ede9b151c9a095f66
> Author: Simon Peyton Jones <simonpj at microsoft.com>
> Date:   Fri Nov 22 16:42:43 2013 +0000
>
>      Higher residency in Haddock
>
>      I think there really is a slight worsening in the situation here, but
>      it needs someone to build a profiled compiler and take a proper look.

Maybe make a ticket for the next milestone then?

I'm getting slightly worried that we keep pushing these performance
bounds up without really investigating why.  The great thing about the
perf tests is that they catch something immediately, rather than us
having to bisect it or do lengthy profiling investigations later.

What should we do about this?  I realise it's a pain when you've got a
working patch and the only thing holding it up is some random perf test
that seems to be completely unrelated.  Let's have a brainstorm on how
we can improve things.  What tools can we build to help?  Maybe this is
a good way that new contributors could get involved.

Here's one idea off the top of my head: let's make a way to take heap
samples at deterministic points during compilation, say between phases.
  Then make a tool to compare profiles made this way, so that we can say
what changed between two compiler versions.  I'm pretty sure this will
spot leaks pretty quickly, and tell you something about what is leaking
(which constructors).

I think we've accrued some bloat recently.  I had to increase the memory
in my VM because it ceased to be able to build HEAD at some point.

Cheers,
Simon


Reply | Threaded
Open this post in threaded view
|

[commit: testsuite] master: Higher residency in Haddock (15c09a3)

Simon Peyton Jones
I agree.  The most direct approach is for someone to build profiled compilers and get some sense of where the space and time is going in a run of Haddock, or GHC.  

I would be thrilled if someone felt able to do some forensic performance comparison between different versions of GHC and/or Haddock. Would anyone like to do that?  There is almost certainly plenty of low-hanging fruit -- you might speed up GHC by 20% and everyone would be Very Happy.

On particular compilation of the Haddock tests is that they embody two distinct sources of variation:
  1.  Cabal (the source code examined by the run of Haddock
      has more/different code
  2.  Haddock itself may have gotten slower/fatter

(1) is a motivation-sapping problem.  Performance tests should have stable input data.

Better tools would certainly help. If we are to retain memory-size numbers, maybe the tests should always be run in such a way that the numbers are stable and predictable.

The other real difficulty is this: if space usage really does go up by 10% in an extremely large and complicated program like Haddock or GHC, it can be jolly hard to nail down what's causing it.  If it goes through the roof, it's (relatively) easy to find.  If bytes-allocated goes up it's easy to find (use -ticky). But a 10% increase in space usage is always going be hard.

Simon



| -----Original Message-----
| From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of Simon
| Marlow
| Sent: 28 November 2013 12:31
| To: ghc-devs at haskell.org
| Subject: Re: [commit: testsuite] master: Higher residency in Haddock
| (15c09a3)
|
| On 22/11/13 16:43, git at git.haskell.org wrote:
| > Repository : ssh://git at git.haskell.org/testsuite
| >
| > On branch  : master
| > Link       :
| http://ghc.haskell.org/trac/ghc/changeset/15c09a332cd24e5d6c4c9a0ede9b1
| 51c9a095f66/testsuite
| >
| >> ---------------------------------------------------------------
| >
| > commit 15c09a332cd24e5d6c4c9a0ede9b151c9a095f66
| > Author: Simon Peyton Jones <simonpj at microsoft.com>
| > Date:   Fri Nov 22 16:42:43 2013 +0000
| >
| >      Higher residency in Haddock
| >
| >      I think there really is a slight worsening in the situation
| here, but
| >      it needs someone to build a profiled compiler and take a proper
| look.
|
| Maybe make a ticket for the next milestone then?
|
| I'm getting slightly worried that we keep pushing these performance
| bounds up without really investigating why.  The great thing about the
| perf tests is that they catch something immediately, rather than us
| having to bisect it or do lengthy profiling investigations later.
|
| What should we do about this?  I realise it's a pain when you've got a
| working patch and the only thing holding it up is some random perf test
| that seems to be completely unrelated.  Let's have a brainstorm on how
| we can improve things.  What tools can we build to help?  Maybe this is
| a good way that new contributors could get involved.
|
| Here's one idea off the top of my head: let's make a way to take heap
| samples at deterministic points during compilation, say between phases.
|   Then make a tool to compare profiles made this way, so that we can
| say
| what changed between two compiler versions.  I'm pretty sure this will
| spot leaks pretty quickly, and tell you something about what is leaking
| (which constructors).
|
| I think we've accrued some bloat recently.  I had to increase the
| memory
| in my VM because it ceased to be able to build HEAD at some point.
|
| Cheers,
| Simon
|
| _______________________________________________
| ghc-devs mailing list
| ghc-devs at haskell.org
| http://www.haskell.org/mailman/listinfo/ghc-devs

Reply | Threaded
Open this post in threaded view
|

[commit: testsuite] master: Higher residency in Haddock (15c09a3)

Simon Marlow-7
On 28/11/13 12:56, Simon Peyton-Jones wrote:
> I agree.  The most direct approach is for someone to build profiled compilers and get some sense of where the space and time is going in a run of Haddock, or GHC.
>
> I would be thrilled if someone felt able to do some forensic performance comparison between different versions of GHC and/or Haddock. Would anyone like to do that?  There is almost certainly plenty of low-hanging fruit -- you might speed up GHC by 20% and everyone would be Very Happy.
>
> On particular compilation of the Haddock tests is that they embody two distinct sources of variation:
>    1.  Cabal (the source code examined by the run of Haddock
>        has more/different code
>    2.  Haddock itself may have gotten slower/fatter

Most of the cost in Haddock is in typechecking all the source code,
which is done by GHC.  I think this is why we have perf tests for
Haddock - they're GHC tests, kind-of.  The annoying thing about the
Haddock perf tests is that they're difficult to re-run though.

> (1) is a motivation-sapping problem.  Performance tests should have stable input data.

Yes, that's not good.  Cabal changing is certainly a legitimate reason
to bump the thresholds for the Haddock tests though.

> Better tools would certainly help. If we are to retain memory-size numbers, maybe the tests should always be run in such a way that the numbers are stable and predictable.

I've been thinking about this.  Just increasing the frequency of samples
would help, but I think taking samples at deterministic points (like
between phases) would help more.  It needs someone to add some support
for this though, in the form of an API for heap sampling that we can
call from Haskell.

Cheers,
Simon


> Simon
>
>
>
> | -----Original Message-----
> | From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of Simon
> | Marlow
> | Sent: 28 November 2013 12:31
> | To: ghc-devs at haskell.org
> | Subject: Re: [commit: testsuite] master: Higher residency in Haddock
> | (15c09a3)
> |
> | On 22/11/13 16:43, git at git.haskell.org wrote:
> | > Repository : ssh://git at git.haskell.org/testsuite
> | >
> | > On branch  : master
> | > Link       :
> | http://ghc.haskell.org/trac/ghc/changeset/15c09a332cd24e5d6c4c9a0ede9b1
> | 51c9a095f66/testsuite
> | >
> | >> ---------------------------------------------------------------
> | >
> | > commit 15c09a332cd24e5d6c4c9a0ede9b151c9a095f66
> | > Author: Simon Peyton Jones <simonpj at microsoft.com>
> | > Date:   Fri Nov 22 16:42:43 2013 +0000
> | >
> | >      Higher residency in Haddock
> | >
> | >      I think there really is a slight worsening in the situation
> | here, but
> | >      it needs someone to build a profiled compiler and take a proper
> | look.
> |
> | Maybe make a ticket for the next milestone then?
> |
> | I'm getting slightly worried that we keep pushing these performance
> | bounds up without really investigating why.  The great thing about the
> | perf tests is that they catch something immediately, rather than us
> | having to bisect it or do lengthy profiling investigations later.
> |
> | What should we do about this?  I realise it's a pain when you've got a
> | working patch and the only thing holding it up is some random perf test
> | that seems to be completely unrelated.  Let's have a brainstorm on how
> | we can improve things.  What tools can we build to help?  Maybe this is
> | a good way that new contributors could get involved.
> |
> | Here's one idea off the top of my head: let's make a way to take heap
> | samples at deterministic points during compilation, say between phases.
> |   Then make a tool to compare profiles made this way, so that we can
> | say
> | what changed between two compiler versions.  I'm pretty sure this will
> | spot leaks pretty quickly, and tell you something about what is leaking
> | (which constructors).
> |
> | I think we've accrued some bloat recently.  I had to increase the
> | memory
> | in my VM because it ceased to be able to build HEAD at some point.
> |
> | Cheers,
> | Simon
> |
> | _______________________________________________
> | ghc-devs mailing list
> | ghc-devs at haskell.org
> | http://www.haskell.org/mailman/listinfo/ghc-devs
>