perf.haskell.org functional again

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

perf.haskell.org functional again

Joachim Breitner-2
Hi,

after a system upgrade to avoid weird linker errors, and after some fixes in the nofib submodule, http://perf.haskell.org/ghc is running again.

I am collecting instruction counts instead of runtime, because the latter was just too often varying too wildly. I hope this will yield less false alarms.

I am also running nofib with mode=fast. This way, building GHC, running the testsuite and nofib takes a bit over one hour. I hope this can keep up with y'all's commits (when it took 2h it couldn't).

Now nothing of the setup requires a quiet dedicated machine, so if there is a need, we could move these builds into the cloud or into some CI infrastructure - but no changes are immediately planned.

Enjoy, Joachim
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: perf.haskell.org functional again

Moritz Angermann-2
Hooray! Thanks for doing this!

Sent from my iPhone

> On 24 Oct 2017, at 8:02 AM, Joachim Breitner <[hidden email]> wrote:
>
> Hi,
>
> after a system upgrade to avoid weird linker errors, and after some fixes in the nofib submodule, http://perf.haskell.org/ghc is running again.
>
> I am collecting instruction counts instead of runtime, because the latter was just too often varying too wildly. I hope this will yield less false alarms.
>
> I am also running nofib with mode=fast. This way, building GHC, running the testsuite and nofib takes a bit over one hour. I hope this can keep up with y'all's commits (when it took 2h it couldn't).
>
> Now nothing of the setup requires a quiet dedicated machine, so if there is a need, we could move these builds into the cloud or into some CI infrastructure - but no changes are immediately planned.
>
> Enjoy, Joachim
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: perf.haskell.org functional again

Manuel M T Chakravarty-4
In reply to this post by Joachim Breitner-2
Hi Joachim,

Great! Just because you mention CI infrastructure and our effort around GHC CI at the moment, do you think, it would make sense to move this to CircleCI eventually?

Cheers,
Manuel

> Am 24.10.2017 um 11:02 schrieb Joachim Breitner <[hidden email]>:
>
> Hi,
>
> after a system upgrade to avoid weird linker errors, and after some fixes in the nofib submodule, http://perf.haskell.org/ghc is running again.
>
> I am collecting instruction counts instead of runtime, because the latter was just too often varying too wildly. I hope this will yield less false alarms.
>
> I am also running nofib with mode=fast. This way, building GHC, running the testsuite and nofib takes a bit over one hour. I hope this can keep up with y'all's commits (when it took 2h it couldn't).
>
> Now nothing of the setup requires a quiet dedicated machine, so if there is a need, we could move these builds into the cloud or into some CI infrastructure - but no changes are immediately planned.
>
> Enjoy, Joachim
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: perf.haskell.org functional again

Joachim Breitner-2
Hi,

Is CircleCI the future now?

In general, yes. But it’s running fine for now, so I would not
prematurely throw it over.

My requirements are:

 * needs to run on every commit (not just every push), including
   branches.
 * needs to be able to push to a repository¹, so it needs access to
   some secret token for depolyment.
   Alternatively, the CI could simply keep track of the build log
   and the perf.haskell.org dashboard scrapes it from there.
 * Occasionally, I find that I want to rebuild a fair number of commits
   that are already in the repository. So a good way to trigger
   rebuilds would be nice.

Greetings,
Joachim

¹ https://github.com/nomeata/ghc-speed-logs/commits/master

Am Dienstag, den 24.10.2017, 12:18 +1100 schrieb Manuel M T
Chakravarty:

> Hi Joachim,
>
> Great! Just because you mention CI infrastructure and our effort around GHC CI at the moment, do you think, it would make sense to move this to CircleCI eventually?
>
> Cheers,
> Manuel
>
> > Am 24.10.2017 um 11:02 schrieb Joachim Breitner <[hidden email]>:
> >
> > Hi,
> >
> > after a system upgrade to avoid weird linker errors, and after some fixes in the nofib submodule, http://perf.haskell.org/ghc is running again.
> >
> > I am collecting instruction counts instead of runtime, because the latter was just too often varying too wildly. I hope this will yield less false alarms.
> >
> > I am also running nofib with mode=fast. This way, building GHC, running the testsuite and nofib takes a bit over one hour. I hope this can keep up with y'all's commits (when it took 2h it couldn't).
> >
> > Now nothing of the setup requires a quiet dedicated machine, so if there is a need, we could move these builds into the cloud or into some CI infrastructure - but no changes are immediately planned.
> >
> > Enjoy, Joachim
--
Joachim Breitner
  [hidden email]
  http://www.joachim-breitner.de/

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: perf.haskell.org functional again

Boespflug, Mathieu
Hi Joachim,

what kind of machine is this running on at the moment? A dedicated
instance or some colocated VM in the cloud? I ask because for perf
results something CircleCI might *not* be suitable here because
execution time on CircleCI varies quite substantially depending on
load from other users of the same physical machine. This is not a
problem for most types of jobs, but perf measurement is one of the few
for which it is, depending on which metrics are tracked.

Of course, we could always have TravisCI/CircleCI drive a dedicated
machine, but it's not clear to me that these services would be the
best fit for this particular use case.

Best,
--
Mathieu Boespflug
Founder at http://tweag.io.


On 24 October 2017 at 05:46, Joachim Breitner <[hidden email]> wrote:

> Hi,
>
> Is CircleCI the future now?
>
> In general, yes. But it’s running fine for now, so I would not
> prematurely throw it over.
>
> My requirements are:
>
>  * needs to run on every commit (not just every push), including
>    branches.
>  * needs to be able to push to a repository¹, so it needs access to
>    some secret token for depolyment.
>    Alternatively, the CI could simply keep track of the build log
>    and the perf.haskell.org dashboard scrapes it from there.
>  * Occasionally, I find that I want to rebuild a fair number of commits
>    that are already in the repository. So a good way to trigger
>    rebuilds would be nice.
>
> Greetings,
> Joachim
>
> ¹ https://github.com/nomeata/ghc-speed-logs/commits/master
>
> Am Dienstag, den 24.10.2017, 12:18 +1100 schrieb Manuel M T
> Chakravarty:
>> Hi Joachim,
>>
>> Great! Just because you mention CI infrastructure and our effort around GHC CI at the moment, do you think, it would make sense to move this to CircleCI eventually?
>>
>> Cheers,
>> Manuel
>>
>> > Am 24.10.2017 um 11:02 schrieb Joachim Breitner <[hidden email]>:
>> >
>> > Hi,
>> >
>> > after a system upgrade to avoid weird linker errors, and after some fixes in the nofib submodule, http://perf.haskell.org/ghc is running again.
>> >
>> > I am collecting instruction counts instead of runtime, because the latter was just too often varying too wildly. I hope this will yield less false alarms.
>> >
>> > I am also running nofib with mode=fast. This way, building GHC, running the testsuite and nofib takes a bit over one hour. I hope this can keep up with y'all's commits (when it took 2h it couldn't).
>> >
>> > Now nothing of the setup requires a quiet dedicated machine, so if there is a need, we could move these builds into the cloud or into some CI infrastructure - but no changes are immediately planned.
>> >
>> > Enjoy, Joachim
> --
> Joachim Breitner
>   [hidden email]
>   http://www.joachim-breitner.de/
>
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: perf.haskell.org functional again

Joachim Breitner-2
Hi,

Am Dienstag, den 24.10.2017, 09:57 +0200 schrieb Boespflug, Mathieu:
> what kind of machine is this running on at the moment? A dedicated
> instance or some colocated VM in the cloud?

A physical box sponsored by Bryn Mawr.
8GB RAM, 8 Core Intel(R) Core(TM) i7 CPU 930  @ 2.80GHz


> I ask because for perf
> results something CircleCI might *not* be suitable here because
> execution time on CircleCI varies quite substantially depending on
> load from other users of the same physical machine. This is not a
> problem for most types of jobs, but perf measurement is one of the few
> for which it is, depending on which metrics are tracked.

But that’s what I am saying: By measuring dynamic instructions (using
cachegrind), the execution time is no longer relevant.

Joachim
--
Joachim Breitner
  [hidden email]
  http://www.joachim-breitner.de/

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: perf.haskell.org functional again

Boespflug, Mathieu
Hi Joachim,

> But that’s what I am saying: By measuring dynamic instructions (using
> cachegrind), the execution time is no longer relevant.

I'm skeptical that this metric is a reliable enough proxy for actual
runtime of user programs to make tracking it and only it sufficient.
For the reasons Sven Panne gave in a previous email. But I'll be very
interested to hear your experience report over time.
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: perf.haskell.org functional again

Joachim Breitner-2
Hi,

Am Dienstag, den 24.10.2017, 16:59 +0200 schrieb Boespflug, Mathieu:
> Hi Joachim,
>
> > But that’s what I am saying: By measuring dynamic instructions (using
> > cachegrind), the execution time is no longer relevant.
>
> I'm skeptical that this metric is a reliable enough proxy for actual
> runtime of user programs to make tracking it and only it sufficient.
> For the reasons Sven Panne gave in a previous email. But I'll be very
> interested to hear your experience report over time.

Sure, it’s a crutch. But we have been working for decades with
“allocations go down, so this must be a win”, which is even more
distant from runtime.

If someone builds me a system where the runtime measurements are
actually reliable (and not drowned in the noise of modern CPU
architectures with sensitivity to layout and alignment), maybe using a
tool like this http://plasma.cs.umass.edu/emery/stabilizer.html
then I’ll happily ditch instruction counts and use that!

But at the current state, working with nofib results is just too
frustrating: You see a change, but you don't know, is it a measurement
error? Did you push something over a performance cliff? Is it really
related to your change to Core? I find it just not viable.

Joachim


--
Joachim Breitner
  [hidden email]
  http://www.joachim-breitner.de/

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: perf.haskell.org functional again

Manuel M T Chakravarty-4
In reply to this post by Joachim Breitner-2
Am 24.10.2017 um 14:46 schrieb Joachim Breitner <[hidden email]>:

Is CircleCI the future now?

Yes, as per


In general, yes. But it’s running fine for now, so I would not
prematurely throw it over.

My requirements are: 

* needs to run on every commit (not just every push), including 
  branches.
* needs to be able to push to a repository¹, so it needs access to 
  some secret token for depolyment.
  Alternatively, the CI could simply keep track of the build log
  and the perf.haskell.org dashboard scrapes it from there.
* Occasionally, I find that I want to rebuild a fair number of commits
  that are already in the repository. So a good way to trigger
  rebuilds would be nice.

No need to mess with a working system, but as long as performance counters are the basis, I think, these constraints can all be met. In particular, CircleCI has facilities to allow deployments including managing of secrets. (That is a pretty common requirements for CIs.)

Cheers,
Manuel


Greetings,
Joachim

¹ https://github.com/nomeata/ghc-speed-logs/commits/master

Am Dienstag, den 24.10.2017, 12:18 +1100 schrieb Manuel M T
Chakravarty:
Hi Joachim,

Great! Just because you mention CI infrastructure and our effort around GHC CI at the moment, do you think, it would make sense to move this to CircleCI eventually?

Cheers,
Manuel

Am 24.10.2017 um 11:02 schrieb Joachim Breitner <[hidden email]>:

Hi,

after a system upgrade to avoid weird linker errors, and after some fixes in the nofib submodule, http://perf.haskell.org/ghc is running again.

I am collecting instruction counts instead of runtime, because the latter was just too often varying too wildly. I hope this will yield less false alarms.

I am also running nofib with mode=fast. This way, building GHC, running the testsuite and nofib takes a bit over one hour. I hope this can keep up with y'all's commits (when it took 2h it couldn't).

Now nothing of the setup requires a quiet dedicated machine, so if there is a need, we could move these builds into the cloud or into some CI infrastructure - but no changes are immediately planned.

Enjoy, Joachim 
-- 
Joachim Breitner
 [hidden email]
 http://www.joachim-breitner.de/
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: perf.haskell.org functional again

Sebastian Graf
Hi,

just wanted to throw in the idea of parallelising the benchmark suite (hurts to even write that, but cachegrind) to speed up the build, if ever need be.

Cheers,
Sebastian

On Wed, Oct 25, 2017 at 2:13 AM, Manuel M T Chakravarty <[hidden email]> wrote:
Am 24.10.2017 um 14:46 schrieb Joachim Breitner <[hidden email]>:

Is CircleCI the future now?

Yes, as per


In general, yes. But it’s running fine for now, so I would not
prematurely throw it over.

My requirements are: 

* needs to run on every commit (not just every push), including 
  branches.
* needs to be able to push to a repository¹, so it needs access to 
  some secret token for depolyment.
  Alternatively, the CI could simply keep track of the build log
  and the perf.haskell.org dashboard scrapes it from there.
* Occasionally, I find that I want to rebuild a fair number of commits
  that are already in the repository. So a good way to trigger
  rebuilds would be nice.

No need to mess with a working system, but as long as performance counters are the basis, I think, these constraints can all be met. In particular, CircleCI has facilities to allow deployments including managing of secrets. (That is a pretty common requirements for CIs.)

Cheers,
Manuel


Greetings,
Joachim

¹ https://github.com/nomeata/ghc-speed-logs/commits/master

Am Dienstag, den 24.10.2017, 12:18 +1100 schrieb Manuel M T
Chakravarty:
Hi Joachim,

Great! Just because you mention CI infrastructure and our effort around GHC CI at the moment, do you think, it would make sense to move this to CircleCI eventually?

Cheers,
Manuel

Am 24.10.2017 um 11:02 schrieb Joachim Breitner <[hidden email]>:

Hi,

after a system upgrade to avoid weird linker errors, and after some fixes in the nofib submodule, http://perf.haskell.org/ghc is running again.

I am collecting instruction counts instead of runtime, because the latter was just too often varying too wildly. I hope this will yield less false alarms.

I am also running nofib with mode=fast. This way, building GHC, running the testsuite and nofib takes a bit over one hour. I hope this can keep up with y'all's commits (when it took 2h it couldn't).

Now nothing of the setup requires a quiet dedicated machine, so if there is a need, we could move these builds into the cloud or into some CI infrastructure - but no changes are immediately planned.

Enjoy, Joachim 
-- 
Joachim Breitner
 [hidden email]
 http://www.joachim-breitner.de/
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs



_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: perf.haskell.org functional again

Joachim Breitner-2
Hi,

Am Sonntag, den 29.10.2017, 23:58 +0100 schrieb Sebastian Graf:
> Hi,
>
> just wanted to throw in the idea of parallelising the benchmark suite
> (hurts to even write that, but cachegrind) to speed up the build, if
> ever need be.

https://github.com/nomeata/gipeda/blob/master/ghc/run-speed.sh#L143

good idea indeed – I’ve been doing that since I started running
cachegrind ;-)


Joachim

--
Joachim Breitner
  [hidden email]
  http://www.joachim-breitner.de/

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (849 bytes) Download Attachment