Measuring compiler performance

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Measuring compiler performance

GHC - devs mailing list
Hi devs!

I've recently started working on a few compiler perf related tickets,
and wondered how to verify that a change actually has a positive
impact on compiler performance.

I first looked at the wiki for information on this, but didn't find much:

https://gitlab.haskell.org/ghc/ghc/-/wikis/performance/compiler
doesn't contain any information on how to measure compiler performance

The best tip I found was the recommendation in
https://gitlab.haskell.org/ghc/ghc/-/wikis/building/running-no-fib to
compile nofib/spectral/simple/Main.hs. With -O0 and -O that takes
respectively about 1.5s and 5s for me, so the effort is manageable.
I've also done full nofib runs, but they take a very long time.

A problem in this context is that reliable performance measurements
require a quiet machine. Closing my browser, and turning off other
programs is – in my perception – rather inconvenient, particularly
when I have to do it for a prolonged time.

Ideally I wouldn't have to perform these measurements on my local
machine at all! Do you usually use a separate machine for this? _Very_
convenient would be some kind of bot whom I could tell e.g.

@perf-bot compiler perf

…or more concretely

@perf-bot compile nofib/spectral/simple/Main.hs

…or just

@nofib-bot run

… or something like that.

I've noticed that CI now includes a perf-nofib job. But since it
appears to run on a different machine each time, I'm not sure whether
it's actually useful for comparing performance. Could it be made more
useful by running it consistently on the same dedicated machine?

Another question regarding performing compiler perf measurements
locally is which build flavour to use: So far I have used the "perf"
flavour. A problem here is that a full build seems to take close to an
hour. A rebuild with --freeze1 takes ~15 minutes on my machine. Is
this the right flavour to use?

BTW what's the purpose of the profiled GHC modules built with this
flavour which just seem to additionally prolong compile time? I don't
see a ghc-prof binary or similar in _build/stage1/bin.

Also, what's the status of gipeda? The most recent commit at
https://perf.haskell.org/ghc/ is from "about a year ago"?

Sorry for this load of questions and complaints! I do believe though
that if work on compiler performance was a bit better documented and
more convenient, we might see even more progress on that front. :)

Cheers,
Simon
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Measuring compiler performance

Richard Eisenberg-5
Because of the fragility of time-based performance tests, I often use allocations as a proxy for time. The number of allocations is very stable. On the down side, it is sometimes possible to reduce allocations while increasing time, so one shouldn't try too hard to blindly optimize allocations.

Separately, in order to get measurements, I build Cabal with ghc --make. Cabal is convenient because it is sizeable, real-world, updated, and needs no dependencies. Compiling the full library definitely takes more than 5 seconds, though.

I agree that there is plenty of room for improvement here!

Richard

> On Apr 5, 2020, at 12:24 AM, Simon Jakobi via ghc-devs <[hidden email]> wrote:
>
> Hi devs!
>
> I've recently started working on a few compiler perf related tickets,
> and wondered how to verify that a change actually has a positive
> impact on compiler performance.
>
> I first looked at the wiki for information on this, but didn't find much:
>
> https://gitlab.haskell.org/ghc/ghc/-/wikis/performance/compiler
> doesn't contain any information on how to measure compiler performance
>
> The best tip I found was the recommendation in
> https://gitlab.haskell.org/ghc/ghc/-/wikis/building/running-no-fib to
> compile nofib/spectral/simple/Main.hs. With -O0 and -O that takes
> respectively about 1.5s and 5s for me, so the effort is manageable.
> I've also done full nofib runs, but they take a very long time.
>
> A problem in this context is that reliable performance measurements
> require a quiet machine. Closing my browser, and turning off other
> programs is – in my perception – rather inconvenient, particularly
> when I have to do it for a prolonged time.
>
> Ideally I wouldn't have to perform these measurements on my local
> machine at all! Do you usually use a separate machine for this? _Very_
> convenient would be some kind of bot whom I could tell e.g.
>
> @perf-bot compiler perf
>
> …or more concretely
>
> @perf-bot compile nofib/spectral/simple/Main.hs
>
> …or just
>
> @nofib-bot run
>
> … or something like that.
>
> I've noticed that CI now includes a perf-nofib job. But since it
> appears to run on a different machine each time, I'm not sure whether
> it's actually useful for comparing performance. Could it be made more
> useful by running it consistently on the same dedicated machine?
>
> Another question regarding performing compiler perf measurements
> locally is which build flavour to use: So far I have used the "perf"
> flavour. A problem here is that a full build seems to take close to an
> hour. A rebuild with --freeze1 takes ~15 minutes on my machine. Is
> this the right flavour to use?
>
> BTW what's the purpose of the profiled GHC modules built with this
> flavour which just seem to additionally prolong compile time? I don't
> see a ghc-prof binary or similar in _build/stage1/bin.
>
> Also, what's the status of gipeda? The most recent commit at
> https://perf.haskell.org/ghc/ is from "about a year ago"?
>
> Sorry for this load of questions and complaints! I do believe though
> that if work on compiler performance was a bit better documented and
> more convenient, we might see even more progress on that front. :)
>
> Cheers,
> Simon
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Measuring compiler performance

Andreas Klebinger
In reply to this post by GHC - devs mailing list
Hi Simon,

things I do to measure performance:

* compile nofib/spectral/simple/Main.hs, look at instructions (perf) and allocations/time (+RTS -s)
* compile nofib as a whole (Use NoFibRuns=0 to avoid running the benchmarks). Look at compile time/allocations.
* compile Cabal the library (cd cabal-head/Cabal && ghc Setup.hs -fforce-recomp). Look at allocations time via +RTS -s or instructions using perf.
* compile a particular files triggering the case I want to optimize

In general:
Adjust depending on flags you want to look at. If you optimize the simplifier -O0 will be useless.
If you optimize type-checking -O2 will be pointless. And so on.

In general I only compile as linking adds overhead which isn't really part of GHC.

Another question regarding performing compiler perf measurements
locally is which build flavour to use: So far I have used the "perf"
flavour. A problem here is that a full build seems to take close to an
hour. A rebuild with --freeze1 takes ~15 minutes on my machine. Is
this the right flavour to use?
Personally I use the quick flavour, freeze stage 1 and configure hadrian to pass -O to stage2
unless I know the thing I'm working on will benefit significantly from -O2.

That is if I optimize an algorithm -O2 won't really make a difference so I use -O.
If I optimize a particular hotspot in the implementation of an algorithm by using
bangs it's worthwhile to look at -O2 as well.

You can also set particular flags for only specific files using OPTIONS_GHC pragmas.
This way you can avoid compiling the whole of GHC with -O/-O2.

Ideally I wouldn't have to perform these measurements on my local
machine at all! Do you usually use a separate machine for this? _Very_
convenient would be some kind of bot whom I could tell e.g.
I use another machine. Others only look at metrics which are less affected by system load like allocations.

Ideally I wouldn't have to perform these measurements on my local
machine at all! Do you usually use a separate machine for this? _Very_
convenient would be some kind of bot whom I could tell e.g.
Various people have come up with scripts to automate the measurements on nofib which get's you
closer to this. I discussed with ben and others a few times in the past having a wider framework for
collecting compiler performance indicators. But it's a lot of work to get right and once the immediate
need is gone those ideas usually get shelved again.
BTW what's the purpose of the profiled GHC modules built with this
flavour which just seem to additionally prolong compile time? I don't
see a ghc-prof binary or similar in _build/stage1/bin.
As far as I know if you compile (non-ghc) code using -prof then you will need the ghc library
available in the prof way. But it would be good to have the option to disable this.

Also, what's the status of gipeda? The most recent commit at
https://perf.haskell.org/ghc/ is from "about a year ago"?
I think the author stopped maintaining it after he switched jobs. So it's currently not useful
for investigating performance. But I'm sure he wouldn't object if anyone were to pick it up.
Cheers
Andreas

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Measuring compiler performance

Joachim Breitner-2
Hi,

Am Montag, den 06.04.2020, 11:03 +0200 schrieb Andreas Klebinger:
> > Also, what's the status of gipeda? The most recent commit at
> > https://perf.haskell.org/ghc/ is from "about a year ago"?
>
>
> I think the author stopped maintaining it after he switched jobs. So
> it's currently not useful
> for investigating performance.

I think a few things happened.
For a while, the dedicated builder (which was a machine on Richard’s
university) was down.
Then there was a phase where data was collected and the page was
generated, but somehow the upload to the web hosting stopped working.
Ben started to work on running these things “more properly” as part of
Circle CI (but that never finished).
And then there is the ways of recording performance data via git notes,
which maybe was meant to be fed into gipeda somehow?

Somewhere in this confusion (and the fact that there wasn’t much
complaining about the lack of this service) it fell off the table.

> But I'm sure he wouldn't object if anyone were to pick it up.

Definitey not! It seems that “the other” solutions didn’t materialize
as quickly as we hoped for, so reactivating perf.haskell.org might be
useful. Anybody willing to drive this? Happy to advise!

Cheers,
Joachim


--
Joachim Breitner
  [hidden email]
  http://www.joachim-breitner.de/


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Measuring compiler performance

Ben Gamari-3
In reply to this post by GHC - devs mailing list
Simon Jakobi via ghc-devs <[hidden email]> writes:

> Hi devs!
>
Hi Simon!

> I've recently started working on a few compiler perf related tickets,
> and wondered how to verify that a change actually has a positive
> impact on compiler performance.
>
> I first looked at the wiki for information on this, but didn't find much:
>
> https://gitlab.haskell.org/ghc/ghc/-/wikis/performance/compiler
> doesn't contain any information on how to measure compiler performance
>
> The best tip I found was the recommendation in
> https://gitlab.haskell.org/ghc/ghc/-/wikis/building/running-no-fib to
> compile nofib/spectral/simple/Main.hs. With -O0 and -O that takes
> respectively about 1.5s and 5s for me, so the effort is manageable.
> I've also done full nofib runs, but they take a very long time.
>
As Joachim points out, progress on this proceeds in fits and starts.
I have a number of pointers to share:

 * When performing compiler measurements myself I generally use a
   combination of the following:

    * nofib
    * the GHC performance testsuite
    * what I call the "Cabal test"; namely:

        $ _build/stage1/bin/ghc -O -ilibraries/Cabal/Cabal \
          libraries/Cabal/Cabal/Setup.hs +RTS -s

 * My WIP nofib branch [1] makes nofib much faster and easier to work
   with and adds the ability to measure perf counters, in addition to
   the usual RTS and cachegrind statistics.

 * My nofib branch produces output in a uniform, easy to consume format
   and provides a tool for comparing sets of measurements in this format.

 * My ghc_perf tool [2] is very useful for extracting runtime and perf
   statistics from Haskell program runs; furthermore, it produces output
   in the same format as expected by the aforementioned nofib-compare
   utility.

 * I have a utility [3] which I use to reproducibly build a set of
   branches, run the testsuite, nofib, and the Cabal test on each of
   them. Admittedly it could use a bit of cleanup but it does its job
   reasonably well, making performance measurement a "set it and forget
   it" sort of task.

 * We collect and record a complete set of testsuite statistics (saved
   to git notes 43]); however, we currently do not import these into
   gipeda.

 * We don't currently have a box which can measure reliable timings
   (since our builders are nearly all virtualised cloud instances). I'm
   going to need to do some shuffling to change this.

 * One potentially useful source of performance information (which sadly
   we currently do not exploit) is the -ddump-timing output produced
   during head.hackage runs.

[1] https://gitlab.haskell.org/ghc/nofib/merge_requests/24
[2] https://gitlab.haskell.org/bgamari/ghc-utils/blob/master/ghc_perf.py
[3] https://gitlab.haskell.org/bgamari/ghc-utils/-/tree/master/build-all
[4] https://gitlab.haskell.org/ghc/ghc/-/wikis/building/running-tests/performance-tests


> A problem in this context is that reliable performance measurements
> require a quiet machine. Closing my browser, and turning off other
> programs is – in my perception – rather inconvenient, particularly
> when I have to do it for a prolonged time.
>
> Ideally I wouldn't have to perform these measurements on my local
> machine at all! Do you usually use a separate machine for this? _Very_
> convenient would be some kind of bot whom I could tell e.g.
>

Indeed it is inconvenient. I am in the lucky situation that I have
another machine locally that can be made reasonably quiet without
interfering with my worflow. However, in general

> @perf-bot compiler perf
>
> …or more concretely
>
> @perf-bot compile nofib/spectral/simple/Main.hs
>
> …or just
>
> @nofib-bot run
>
> … or something like that.
>
> I've noticed that CI now includes a perf-nofib job. But since it
> appears to run on a different machine each time, I'm not sure whether
> it's actually useful for comparing performance. Could it be made more
> useful by running it consistently on the same dedicated machine?
>
Indeed, we currently don't have a dedicated machine for timings.
However, allocations and executable sizes are still useful.

Nevertheless, as noted above I think that we should make more of an
effort to measure time. I need to do some shuffling of our runners so we
have a quiet bare-metal which can be dedicated to performance
measurement. I'll try to get to this in the next day or so.

> Another question regarding performing compiler perf measurements
> locally is which build flavour to use: So far I have used the "perf"
> flavour. A problem here is that a full build seems to take close to an
> hour. A rebuild with --freeze1 takes ~15 minutes on my machine. Is
> this the right flavour to use?
>
I think perf is the best option for performance measurement (afterall,
we want to know what users would see). However, it is indeed a bit
painful.

> BTW what's the purpose of the profiled GHC modules built with this
> flavour which just seem to additionally prolong compile time? I don't
> see a ghc-prof binary or similar in _build/stage1/bin.
>
Indeed; there is little sense in building profiled modules just for
performance measurement. However, I don't believe we currently have a
build flavour which provides comparable optimisation but without the
profiled way. Perhaps we should add one.

> Also, what's the status of gipeda? The most recent commit at
> https://perf.haskell.org/ghc/ is from "about a year ago"?
>
Indeed the machine which was previously providing gipeda builds is sadly
no longer around; consequently it's on ice at the moment. I would like
to get it going again but recently correctness issues have been taking
up more time than I would like to admit.

> Sorry for this load of questions and complaints! I do believe though
> that if work on compiler performance was a bit better documented and
> more convenient, we might see even more progress on that front. :)
>
Quite alright! Typing out the points above made me realize that there is
indeed quite a bit of knowledge that the wiki leaves un-said.

Cheers,

- Ben


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Measuring compiler performance

GHC - devs mailing list
Many thanks, Richard, Andreas, Joachim, and Ben, for your responses! I
have a few things to try now. :)

>     * what I call the "Cabal test"; namely:
>
>         $ _build/stage1/bin/ghc -O -ilibraries/Cabal/Cabal \
>           libraries/Cabal/Cabal/Setup.hs +RTS -s

Thanks for spelling it out like that, Ben! I'm slightly embarrassed to
say that I hadn't been aware that I could use GHC directly in this way
to build a package!

Andreas, you wrote:

> In general I only compile as linking adds overhead which isn't really part of GHC.

How do I tell GHC to build e.g. nofib/spectral/simple/Main.hs or Cabal
without linking?

I'll eventually try to distill a wiki page from all this!

Cheers,
Simon





>
>  * My WIP nofib branch [1] makes nofib much faster and easier to work
>    with and adds the ability to measure perf counters, in addition to
>    the usual RTS and cachegrind statistics.
>
>  * My nofib branch produces output in a uniform, easy to consume format
>    and provides a tool for comparing sets of measurements in this format.
>
>  * My ghc_perf tool [2] is very useful for extracting runtime and perf
>    statistics from Haskell program runs; furthermore, it produces output
>    in the same format as expected by the aforementioned nofib-compare
>    utility.
>
>  * I have a utility [3] which I use to reproducibly build a set of
>    branches, run the testsuite, nofib, and the Cabal test on each of
>    them. Admittedly it could use a bit of cleanup but it does its job
>    reasonably well, making performance measurement a "set it and forget
>    it" sort of task.
>
>  * We collect and record a complete set of testsuite statistics (saved
>    to git notes 43]); however, we currently do not import these into
>    gipeda.
>
>  * We don't currently have a box which can measure reliable timings
>    (since our builders are nearly all virtualised cloud instances). I'm
>    going to need to do some shuffling to change this.
>
>  * One potentially useful source of performance information (which sadly
>    we currently do not exploit) is the -ddump-timing output produced
>    during head.hackage runs.
>
> [1] https://gitlab.haskell.org/ghc/nofib/merge_requests/24
> [2] https://gitlab.haskell.org/bgamari/ghc-utils/blob/master/ghc_perf.py
> [3] https://gitlab.haskell.org/bgamari/ghc-utils/-/tree/master/build-all
> [4] https://gitlab.haskell.org/ghc/ghc/-/wikis/building/running-tests/performance-tests
>
>
> > A problem in this context is that reliable performance measurements
> > require a quiet machine. Closing my browser, and turning off other
> > programs is – in my perception – rather inconvenient, particularly
> > when I have to do it for a prolonged time.
> >
> > Ideally I wouldn't have to perform these measurements on my local
> > machine at all! Do you usually use a separate machine for this? _Very_
> > convenient would be some kind of bot whom I could tell e.g.
> >
>
> Indeed it is inconvenient. I am in the lucky situation that I have
> another machine locally that can be made reasonably quiet without
> interfering with my worflow. However, in general
>
> > @perf-bot compiler perf
> >
> > …or more concretely
> >
> > @perf-bot compile nofib/spectral/simple/Main.hs
> >
> > …or just
> >
> > @nofib-bot run
> >
> > … or something like that.
> >
> > I've noticed that CI now includes a perf-nofib job. But since it
> > appears to run on a different machine each time, I'm not sure whether
> > it's actually useful for comparing performance. Could it be made more
> > useful by running it consistently on the same dedicated machine?
> >
> Indeed, we currently don't have a dedicated machine for timings.
> However, allocations and executable sizes are still useful.
>
> Nevertheless, as noted above I think that we should make more of an
> effort to measure time. I need to do some shuffling of our runners so we
> have a quiet bare-metal which can be dedicated to performance
> measurement. I'll try to get to this in the next day or so.
>
> > Another question regarding performing compiler perf measurements
> > locally is which build flavour to use: So far I have used the "perf"
> > flavour. A problem here is that a full build seems to take close to an
> > hour. A rebuild with --freeze1 takes ~15 minutes on my machine. Is
> > this the right flavour to use?
> >
> I think perf is the best option for performance measurement (afterall,
> we want to know what users would see). However, it is indeed a bit
> painful.
>
> > BTW what's the purpose of the profiled GHC modules built with this
> > flavour which just seem to additionally prolong compile time? I don't
> > see a ghc-prof binary or similar in _build/stage1/bin.
> >
> Indeed; there is little sense in building profiled modules just for
> performance measurement. However, I don't believe we currently have a
> build flavour which provides comparable optimisation but without the
> profiled way. Perhaps we should add one.
>
> > Also, what's the status of gipeda? The most recent commit at
> > https://perf.haskell.org/ghc/ is from "about a year ago"?
> >
> Indeed the machine which was previously providing gipeda builds is sadly
> no longer around; consequently it's on ice at the moment. I would like
> to get it going again but recently correctness issues have been taking
> up more time than I would like to admit.
>
> > Sorry for this load of questions and complaints! I do believe though
> > that if work on compiler performance was a bit better documented and
> > more convenient, we might see even more progress on that front. :)
> >
> Quite alright! Typing out the points above made me realize that there is
> indeed quite a bit of knowledge that the wiki leaves un-said.
>
> Cheers,
>
> - Ben
>
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Measuring compiler performance

Matthew Pickering
Simon, I assume the `-no-link` flag does this.

```
> ghc --show-options | grep link
-copy-libs-when-linking
-no-link
-no-auto-link-packages
--print-c-compiler-link-flags
```

Cheers,

Matt

On Wed, Apr 8, 2020 at 4:15 PM Simon Jakobi via ghc-devs
<[hidden email]> wrote:

>
> Many thanks, Richard, Andreas, Joachim, and Ben, for your responses! I
> have a few things to try now. :)
>
> >     * what I call the "Cabal test"; namely:
> >
> >         $ _build/stage1/bin/ghc -O -ilibraries/Cabal/Cabal \
> >           libraries/Cabal/Cabal/Setup.hs +RTS -s
>
> Thanks for spelling it out like that, Ben! I'm slightly embarrassed to
> say that I hadn't been aware that I could use GHC directly in this way
> to build a package!
>
> Andreas, you wrote:
>
> > In general I only compile as linking adds overhead which isn't really part of GHC.
>
> How do I tell GHC to build e.g. nofib/spectral/simple/Main.hs or Cabal
> without linking?
>
> I'll eventually try to distill a wiki page from all this!
>
> Cheers,
> Simon
>
>
>
>
>
> >
> >  * My WIP nofib branch [1] makes nofib much faster and easier to work
> >    with and adds the ability to measure perf counters, in addition to
> >    the usual RTS and cachegrind statistics.
> >
> >  * My nofib branch produces output in a uniform, easy to consume format
> >    and provides a tool for comparing sets of measurements in this format.
> >
> >  * My ghc_perf tool [2] is very useful for extracting runtime and perf
> >    statistics from Haskell program runs; furthermore, it produces output
> >    in the same format as expected by the aforementioned nofib-compare
> >    utility.
> >
> >  * I have a utility [3] which I use to reproducibly build a set of
> >    branches, run the testsuite, nofib, and the Cabal test on each of
> >    them. Admittedly it could use a bit of cleanup but it does its job
> >    reasonably well, making performance measurement a "set it and forget
> >    it" sort of task.
> >
> >  * We collect and record a complete set of testsuite statistics (saved
> >    to git notes 43]); however, we currently do not import these into
> >    gipeda.
> >
> >  * We don't currently have a box which can measure reliable timings
> >    (since our builders are nearly all virtualised cloud instances). I'm
> >    going to need to do some shuffling to change this.
> >
> >  * One potentially useful source of performance information (which sadly
> >    we currently do not exploit) is the -ddump-timing output produced
> >    during head.hackage runs.
> >
> > [1] https://gitlab.haskell.org/ghc/nofib/merge_requests/24
> > [2] https://gitlab.haskell.org/bgamari/ghc-utils/blob/master/ghc_perf.py
> > [3] https://gitlab.haskell.org/bgamari/ghc-utils/-/tree/master/build-all
> > [4] https://gitlab.haskell.org/ghc/ghc/-/wikis/building/running-tests/performance-tests
> >
> >
> > > A problem in this context is that reliable performance measurements
> > > require a quiet machine. Closing my browser, and turning off other
> > > programs is – in my perception – rather inconvenient, particularly
> > > when I have to do it for a prolonged time.
> > >
> > > Ideally I wouldn't have to perform these measurements on my local
> > > machine at all! Do you usually use a separate machine for this? _Very_
> > > convenient would be some kind of bot whom I could tell e.g.
> > >
> >
> > Indeed it is inconvenient. I am in the lucky situation that I have
> > another machine locally that can be made reasonably quiet without
> > interfering with my worflow. However, in general
> >
> > > @perf-bot compiler perf
> > >
> > > …or more concretely
> > >
> > > @perf-bot compile nofib/spectral/simple/Main.hs
> > >
> > > …or just
> > >
> > > @nofib-bot run
> > >
> > > … or something like that.
> > >
> > > I've noticed that CI now includes a perf-nofib job. But since it
> > > appears to run on a different machine each time, I'm not sure whether
> > > it's actually useful for comparing performance. Could it be made more
> > > useful by running it consistently on the same dedicated machine?
> > >
> > Indeed, we currently don't have a dedicated machine for timings.
> > However, allocations and executable sizes are still useful.
> >
> > Nevertheless, as noted above I think that we should make more of an
> > effort to measure time. I need to do some shuffling of our runners so we
> > have a quiet bare-metal which can be dedicated to performance
> > measurement. I'll try to get to this in the next day or so.
> >
> > > Another question regarding performing compiler perf measurements
> > > locally is which build flavour to use: So far I have used the "perf"
> > > flavour. A problem here is that a full build seems to take close to an
> > > hour. A rebuild with --freeze1 takes ~15 minutes on my machine. Is
> > > this the right flavour to use?
> > >
> > I think perf is the best option for performance measurement (afterall,
> > we want to know what users would see). However, it is indeed a bit
> > painful.
> >
> > > BTW what's the purpose of the profiled GHC modules built with this
> > > flavour which just seem to additionally prolong compile time? I don't
> > > see a ghc-prof binary or similar in _build/stage1/bin.
> > >
> > Indeed; there is little sense in building profiled modules just for
> > performance measurement. However, I don't believe we currently have a
> > build flavour which provides comparable optimisation but without the
> > profiled way. Perhaps we should add one.
> >
> > > Also, what's the status of gipeda? The most recent commit at
> > > https://perf.haskell.org/ghc/ is from "about a year ago"?
> > >
> > Indeed the machine which was previously providing gipeda builds is sadly
> > no longer around; consequently it's on ice at the moment. I would like
> > to get it going again but recently correctness issues have been taking
> > up more time than I would like to admit.
> >
> > > Sorry for this load of questions and complaints! I do believe though
> > > that if work on compiler performance was a bit better documented and
> > > more convenient, we might see even more progress on that front. :)
> > >
> > Quite alright! Typing out the points above made me realize that there is
> > indeed quite a bit of knowledge that the wiki leaves un-said.
> >
> > Cheers,
> >
> > - Ben
> >
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Measuring compiler performance

GHC - devs mailing list
Thanks, Matt! That works!

Am Mi., 8. Apr. 2020 um 17:21 Uhr schrieb Matthew Pickering
<[hidden email]>:

>
> Simon, I assume the `-no-link` flag does this.
>
> ```
> > ghc --show-options | grep link
> -copy-libs-when-linking
> -no-link
> -no-auto-link-packages
> --print-c-compiler-link-flags
> ```
>
> Cheers,
>
> Matt
>
> On Wed, Apr 8, 2020 at 4:15 PM Simon Jakobi via ghc-devs
> <[hidden email]> wrote:
> >
> > Many thanks, Richard, Andreas, Joachim, and Ben, for your responses! I
> > have a few things to try now. :)
> >
> > >     * what I call the "Cabal test"; namely:
> > >
> > >         $ _build/stage1/bin/ghc -O -ilibraries/Cabal/Cabal \
> > >           libraries/Cabal/Cabal/Setup.hs +RTS -s
> >
> > Thanks for spelling it out like that, Ben! I'm slightly embarrassed to
> > say that I hadn't been aware that I could use GHC directly in this way
> > to build a package!
> >
> > Andreas, you wrote:
> >
> > > In general I only compile as linking adds overhead which isn't really part of GHC.
> >
> > How do I tell GHC to build e.g. nofib/spectral/simple/Main.hs or Cabal
> > without linking?
> >
> > I'll eventually try to distill a wiki page from all this!
> >
> > Cheers,
> > Simon
> >
> >
> >
> >
> >
> > >
> > >  * My WIP nofib branch [1] makes nofib much faster and easier to work
> > >    with and adds the ability to measure perf counters, in addition to
> > >    the usual RTS and cachegrind statistics.
> > >
> > >  * My nofib branch produces output in a uniform, easy to consume format
> > >    and provides a tool for comparing sets of measurements in this format.
> > >
> > >  * My ghc_perf tool [2] is very useful for extracting runtime and perf
> > >    statistics from Haskell program runs; furthermore, it produces output
> > >    in the same format as expected by the aforementioned nofib-compare
> > >    utility.
> > >
> > >  * I have a utility [3] which I use to reproducibly build a set of
> > >    branches, run the testsuite, nofib, and the Cabal test on each of
> > >    them. Admittedly it could use a bit of cleanup but it does its job
> > >    reasonably well, making performance measurement a "set it and forget
> > >    it" sort of task.
> > >
> > >  * We collect and record a complete set of testsuite statistics (saved
> > >    to git notes 43]); however, we currently do not import these into
> > >    gipeda.
> > >
> > >  * We don't currently have a box which can measure reliable timings
> > >    (since our builders are nearly all virtualised cloud instances). I'm
> > >    going to need to do some shuffling to change this.
> > >
> > >  * One potentially useful source of performance information (which sadly
> > >    we currently do not exploit) is the -ddump-timing output produced
> > >    during head.hackage runs.
> > >
> > > [1] https://gitlab.haskell.org/ghc/nofib/merge_requests/24
> > > [2] https://gitlab.haskell.org/bgamari/ghc-utils/blob/master/ghc_perf.py
> > > [3] https://gitlab.haskell.org/bgamari/ghc-utils/-/tree/master/build-all
> > > [4] https://gitlab.haskell.org/ghc/ghc/-/wikis/building/running-tests/performance-tests
> > >
> > >
> > > > A problem in this context is that reliable performance measurements
> > > > require a quiet machine. Closing my browser, and turning off other
> > > > programs is – in my perception – rather inconvenient, particularly
> > > > when I have to do it for a prolonged time.
> > > >
> > > > Ideally I wouldn't have to perform these measurements on my local
> > > > machine at all! Do you usually use a separate machine for this? _Very_
> > > > convenient would be some kind of bot whom I could tell e.g.
> > > >
> > >
> > > Indeed it is inconvenient. I am in the lucky situation that I have
> > > another machine locally that can be made reasonably quiet without
> > > interfering with my worflow. However, in general
> > >
> > > > @perf-bot compiler perf
> > > >
> > > > …or more concretely
> > > >
> > > > @perf-bot compile nofib/spectral/simple/Main.hs
> > > >
> > > > …or just
> > > >
> > > > @nofib-bot run
> > > >
> > > > … or something like that.
> > > >
> > > > I've noticed that CI now includes a perf-nofib job. But since it
> > > > appears to run on a different machine each time, I'm not sure whether
> > > > it's actually useful for comparing performance. Could it be made more
> > > > useful by running it consistently on the same dedicated machine?
> > > >
> > > Indeed, we currently don't have a dedicated machine for timings.
> > > However, allocations and executable sizes are still useful.
> > >
> > > Nevertheless, as noted above I think that we should make more of an
> > > effort to measure time. I need to do some shuffling of our runners so we
> > > have a quiet bare-metal which can be dedicated to performance
> > > measurement. I'll try to get to this in the next day or so.
> > >
> > > > Another question regarding performing compiler perf measurements
> > > > locally is which build flavour to use: So far I have used the "perf"
> > > > flavour. A problem here is that a full build seems to take close to an
> > > > hour. A rebuild with --freeze1 takes ~15 minutes on my machine. Is
> > > > this the right flavour to use?
> > > >
> > > I think perf is the best option for performance measurement (afterall,
> > > we want to know what users would see). However, it is indeed a bit
> > > painful.
> > >
> > > > BTW what's the purpose of the profiled GHC modules built with this
> > > > flavour which just seem to additionally prolong compile time? I don't
> > > > see a ghc-prof binary or similar in _build/stage1/bin.
> > > >
> > > Indeed; there is little sense in building profiled modules just for
> > > performance measurement. However, I don't believe we currently have a
> > > build flavour which provides comparable optimisation but without the
> > > profiled way. Perhaps we should add one.
> > >
> > > > Also, what's the status of gipeda? The most recent commit at
> > > > https://perf.haskell.org/ghc/ is from "about a year ago"?
> > > >
> > > Indeed the machine which was previously providing gipeda builds is sadly
> > > no longer around; consequently it's on ice at the moment. I would like
> > > to get it going again but recently correctness issues have been taking
> > > up more time than I would like to admit.
> > >
> > > > Sorry for this load of questions and complaints! I do believe though
> > > > that if work on compiler performance was a bit better documented and
> > > > more convenient, we might see even more progress on that front. :)
> > > >
> > > Quite alright! Typing out the points above made me realize that there is
> > > indeed quite a bit of knowledge that the wiki leaves un-said.
> > >
> > > Cheers,
> > >
> > > - Ben
> > >
> > _______________________________________________
> > ghc-devs mailing list
> > [hidden email]
> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs