[GHC] #8173: GHC uses nub

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[GHC] #8173: GHC uses nub

Kim-Ee Yeoh
Administrator
On Fri, Sep 27, 2013 at 8:14 PM, GHC <ghc-devs at haskell.org> wrote:

> A 5% improvement in compile time is remarkable, if it's true.  Great!  But
>  I'm always worried about the noise in compile times measured in seconds.
>

Does anyone else think the noise in runtimes is alarming considering that
the following is the fib-analysis of /binary-identical/ programs?

>              Min          -0.1%     -0.0%    -25.4%    -32.2%     -1.3%
>             Max          +0.1%     +0.0%    +19.0%    +22.2%    +10.0%

Shouldn't we find an explanation for this before believing the compile time
numbers? What would cause these wide swings on the benchmarking machine?

p.s. For the record: Should do more rigorous statistical testing instead of
naive percentages, yes?

-- Kim-Ee
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20130928/d3f997ed/attachment.htm>

Reply | Threaded
Open this post in threaded view
|

[GHC] #8173: GHC uses nub

Muhaimin Ahsan
Kim-Ee,

The updated fib-analyse report from a few hours ago is posted here: https://gist.github.com/leroux/6725810#file-headvordnub-analysis-L2988
Comment (http://ghc.haskell.org/trac/ghc/ticket/8173#comment:9)

Sorry for the misunderstanding.

Muhaimin

On Sep 27, 2013, at 3:56 PM, Kim-Ee Yeoh <ky3 at atamo.com> wrote:

>
> On Fri, Sep 27, 2013 at 8:14 PM, GHC <ghc-devs at haskell.org> wrote:
> A 5% improvement in compile time is remarkable, if it's true.  Great!  But
>  I'm always worried about the noise in compile times measured in seconds.
>
> Does anyone else think the noise in runtimes is alarming considering that the following is the fib-analysis of /binary-identical/ programs?
>
> >              Min          -0.1%     -0.0%    -25.4%    -32.2%     -1.3%
> >             Max          +0.1%     +0.0%    +19.0%    +22.2%    +10.0%
>
> Shouldn't we find an explanation for this before believing the compile time numbers? What would cause these wide swings on the benchmarking machine?
>
> p.s. For the record: Should do more rigorous statistical testing instead of naive percentages, yes?
>
> -- Kim-Ee
> _______________________________________________
> ghc-tickets mailing list
> ghc-tickets at haskell.org
> http://www.haskell.org/mailman/listinfo/ghc-tickets

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20130927/005f230d/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20130927/005f230d/attachment.pgp>

Reply | Threaded
Open this post in threaded view
|

[GHC] #8173: GHC uses nub

Kim-Ee Yeoh
Administrator
Salam Muhaimin,

So to answer my own question, the current practice seems to be to just
eyeball

>   Geometric Mean          -0.0%     -0.0%     -0.3%     -0.1%     +0.1%

and if the numbers are within historical epsilons of 0, that means no
change.

For a moment, I thought this was some erratically behaving VM.

The compile times are across-the-board lower, nice!

-- Kim-Ee


On Sat, Sep 28, 2013 at 4:39 AM, Muhaimin Ahsan <leroux at fezrev.com> wrote:

> Kim-Ee,
>
> The updated fib-analyse report from a few hours ago is posted here:
> https://gist.github.com/leroux/6725810#file-headvordnub-analysis-L2988
> Comment (http://ghc.haskell.org/trac/ghc/ticket/8173#comment:9)
>
> Sorry for the misunderstanding.
>
> Muhaimin
>
> On Sep 27, 2013, at 3:56 PM, Kim-Ee Yeoh <ky3 at atamo.com> wrote:
>
>
> On Fri, Sep 27, 2013 at 8:14 PM, GHC <ghc-devs at haskell.org> wrote:
>
>> A 5% improvement in compile time is remarkable, if it's true.  Great!  But
>>  I'm always worried about the noise in compile times measured in seconds.
>>
>
> Does anyone else think the noise in runtimes is alarming considering that
> the following is the fib-analysis of /binary-identical/ programs?
>
> >              Min          -0.1%     -0.0%    -25.4%    -32.2%     -1.3%
> >             Max          +0.1%     +0.0%    +19.0%    +22.2%    +10.0%
>
> Shouldn't we find an explanation for this before believing the compile
> time numbers? What would cause these wide swings on the benchmarking
> machine?
>
> p.s. For the record: Should do more rigorous statistical testing instead
> of naive percentages, yes?
>
> -- Kim-Ee
>  _______________________________________________
> ghc-tickets mailing list
> ghc-tickets at haskell.org
> http://www.haskell.org/mailman/listinfo/ghc-tickets
>
>
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://www.haskell.org/mailman/listinfo/ghc-devs
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20130928/ad70d704/attachment.htm>