[ANNOUNCE] GHC 8.2.1 release candidate 1

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[ANNOUNCE] GHC 8.2.1 release candidate 1

Ben Gamari-3

Hello everyone,

The GHC team is very pleased to announce the first candidate of the
8.2.1 release of the Glasgow Haskell Compiler. Source and binary
distributions are available at

    https://downloads.haskell.org/~ghc/8.2.1-rc1/

This is the first of a few release candidates leading up the final 8.2.1
release. This release will feature,

  * A new type-indexed Typeable implementation

  * The long awaited Backpack

  * Deriving strategies for disambiguating DeriveAnyClass,
    GeneralizedNewtypeDeriving, and stock mechanisms

  * Overloaded record fields

  * Improved compiler performance

  * Better code generation through more robust tracking of join points

  * Compact regions for more efficient garbage collection and serialization

  * Better support for machines with non-uniform memory architectures

  * More robust support for levity (e.g. RuntimeRep) polymorphism

  * A simple interface for streaming eventlog data from live processes

  * Further refinement of DWARF support

Unfortunately there are still a few known issues in this release,
including a few compiler panics (#13233, #13509) and a memory leak in
the simplifier (#13426) which may adversely affect heap sizes and
compilation time for some modules. This memory leak unfortunately made
it impossible to provide a 32-bit Windows distribution for this
candidate; this will be resolved in -rc2.

As always, please let us know if you have difficulty. Thanks to everyone
who has contributed to this release!

Happy testing,

- Ben


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [ANNOUNCE] GHC 8.2.1 release candidate 1

Boespflug, Mathieu
Hi Ben,

this is great news! I'm particularly keen on learning more about two points you mentioned in your email:

* Compiler performance: do you have any numbers to quantify what 8.0 vs 8.2 is likely to look like? How much has the work that's been done affect performance across the board? Or are these mostly pathological cases (e.g. ghc --make with high number of cores, large number of definitions in a module, large amount of let nesting in definitions, etc)
* DWARF support: could you clarify at a very high-level what typical uses cases can be expected to work and which ones won't? Would be eager to read any resources you could point me at to help me understand what still needs to be done on this front.

Many thanks,

--
Mathieu Boespflug
Founder at http://tweag.io.

On 10 April 2017 at 05:50, Ben Gamari <[hidden email]> wrote:

Hello everyone,

The GHC team is very pleased to announce the first candidate of the
8.2.1 release of the Glasgow Haskell Compiler. Source and binary
distributions are available at

    https://downloads.haskell.org/~ghc/8.2.1-rc1/

This is the first of a few release candidates leading up the final 8.2.1
release. This release will feature,

  * A new type-indexed Typeable implementation

  * The long awaited Backpack

  * Deriving strategies for disambiguating DeriveAnyClass,
    GeneralizedNewtypeDeriving, and stock mechanisms

  * Overloaded record fields

  * Improved compiler performance

  * Better code generation through more robust tracking of join points

  * Compact regions for more efficient garbage collection and serialization

  * Better support for machines with non-uniform memory architectures

  * More robust support for levity (e.g. RuntimeRep) polymorphism

  * A simple interface for streaming eventlog data from live processes

  * Further refinement of DWARF support

Unfortunately there are still a few known issues in this release,
including a few compiler panics (#13233, #13509) and a memory leak in
the simplifier (#13426) which may adversely affect heap sizes and
compilation time for some modules. This memory leak unfortunately made
it impossible to provide a 32-bit Windows distribution for this
candidate; this will be resolved in -rc2.

As always, please let us know if you have difficulty. Thanks to everyone
who has contributed to this release!

Happy testing,

- Ben


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs



_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [ANNOUNCE] GHC 8.2.1 release candidate 1

lonetiger

Hi Ben,

The windows build is unusable. The settings file has $TopDir expanded and hard-coded to the build path on drydock.

Tamar


On Mon, 10 Apr 2017, 08:14 Boespflug, Mathieu, <[hidden email]> wrote:
Hi Ben,

this is great news! I'm particularly keen on learning more about two points you mentioned in your email:

* Compiler performance: do you have any numbers to quantify what 8.0 vs 8.2 is likely to look like? How much has the work that's been done affect performance across the board? Or are these mostly pathological cases (e.g. ghc --make with high number of cores, large number of definitions in a module, large amount of let nesting in definitions, etc)
* DWARF support: could you clarify at a very high-level what typical uses cases can be expected to work and which ones won't? Would be eager to read any resources you could point me at to help me understand what still needs to be done on this front.

Many thanks,

--
Mathieu Boespflug
Founder at http://tweag.io.

On 10 April 2017 at 05:50, Ben Gamari <[hidden email]> wrote:

Hello everyone,

The GHC team is very pleased to announce the first candidate of the
8.2.1 release of the Glasgow Haskell Compiler. Source and binary
distributions are available at

    https://downloads.haskell.org/~ghc/8.2.1-rc1/

This is the first of a few release candidates leading up the final 8.2.1
release. This release will feature,

  * A new type-indexed Typeable implementation

  * The long awaited Backpack

  * Deriving strategies for disambiguating DeriveAnyClass,
    GeneralizedNewtypeDeriving, and stock mechanisms

  * Overloaded record fields

  * Improved compiler performance

  * Better code generation through more robust tracking of join points

  * Compact regions for more efficient garbage collection and serialization

  * Better support for machines with non-uniform memory architectures

  * More robust support for levity (e.g. RuntimeRep) polymorphism

  * A simple interface for streaming eventlog data from live processes

  * Further refinement of DWARF support

Unfortunately there are still a few known issues in this release,
including a few compiler panics (#13233, #13509) and a memory leak in
the simplifier (#13426) which may adversely affect heap sizes and
compilation time for some modules. This memory leak unfortunately made
it impossible to provide a 32-bit Windows distribution for this
candidate; this will be resolved in -rc2.

As always, please let us know if you have difficulty. Thanks to everyone
who has contributed to this release!

Happy testing,

- Ben


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [ANNOUNCE] GHC 8.2.1 release candidate 1

Ben Gamari-3
Phyx <[hidden email]> writes:

> Hi Ben,
>
> The windows build is unusable. The settings file has $TopDir expanded and
> hard-coded to the build path on drydock.
>
Hmm, this is very odd and sounds like a build system bug since I used
the same scripts as usual to generate this bindist. I'll investigate.

Cheers,

- Ben


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [ANNOUNCE] GHC 8.2.1 release candidate 1

Ben Gamari-3
In reply to this post by Boespflug, Mathieu
"Boespflug, Mathieu" <[hidden email]> writes:

> Hi Ben,
>
> this is great news! I'm particularly keen on learning more about two points
> you mentioned in your email:
>
> * Compiler performance: do you have any numbers to quantify what 8.0 vs 8.2
> is likely to look like?

I'm afraid the best I can provide at the moment is [1]. On closer
inspection of these I'm a bit suspicious of the 8.0 numbers; I'll try to
reproduce them (and characterize the current ghc-8.2 branch, which fixes
a significant memory leak present in -rc1) shortly. That being said,
there were a few major fixes in 8.2.

> How much has the work that's been done affect performance across the
> board? Or are these mostly pathological cases (e.g. ghc --make with
> high number of cores, large number of definitions in a module, large
> amount of let nesting in definitions, etc)

The fixes span the gamut but I generally tried to concentrate on issues
that looked like they might affect a large fraction of programs. I fixed
at least one major regression present in 8.0 which significantly
inflated allocations of typeclass instance matching. I've also done
quite a bit of refactoring around our handling of names. These include
improvements in interface file serialization efficiency and name lookup.
These are just some of the major reworks; there are also numerous
smaller fixes which I don't have time to cover here.

Compilation performance has also generally improved as a result of some
of the work in the simplifier. In particular, GHC now performs an early
inlining phase which, quite surprisingly, tends to result in smaller
programs earlier in simplification, reducing the amount of work that the
compiler needs to carry out. Simon summarized some of his preliminary
numbers here [2].

> * DWARF support: could you clarify at a very high-level what typical uses
> cases can be expected to work and which ones won't? Would be eager to read
> any resources you could point me at to help me understand what still needs
> to be done on this front.
>
At this point if GHC compiles your program with -g you should have a
pretty good assurance that the resulting DWARF information is
reasonable. This means that users can use the ExecutionStack mechanism,
RTS stack-trace functionality, and GDB without fear of the act of
capturing a stacktrace leading to a crash.

After starting to write more here, I realized that you will likely not
be the last person to ask about this and updated the DWARF status page
with additional discussion [3]. Let me know if you have any questions.

Cheers,

- Ben

[1] https://gist.github.com/bgamari/fbd3e55047bd041d8208ebe820c0f493
[2] https://mail.haskell.org/pipermail/ghc-devs/2017-February/013818.html
[3] https://ghc.haskell.org/trac/ghc/wiki/DWARF/Status

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [ANNOUNCE] GHC 8.2.1 release candidate 1

Boespflug, Mathieu
Hi Ben,

thank you for this summary. The DWARF status page is helpful. Something was unclear to me though. There are three main potential use cases for DWARF that I see:

1. debugging, possibly with gdb
2. stack traces on exceptions
3. stack sampling, which is a form of performance profiling.

Forgive me for these naive questions, but... Is (1) possible at all at this point? If I compile GHC with all the right options as explained in the status page, do I magically get backtraces associated to all my exceptions? What should I do to get those? It's my understanding that (3) is possible and works fine, even when FFI code is invoked, but slower than for C programs due to running user-level code to inspect the stack for each sample. Is that right?

Best,

--
Mathieu Boespflug
Founder at http://tweag.io.

On 10 April 2017 at 21:28, Ben Gamari <[hidden email]> wrote:
"Boespflug, Mathieu" <[hidden email]> writes:

> Hi Ben,
>
> this is great news! I'm particularly keen on learning more about two points
> you mentioned in your email:
>
> * Compiler performance: do you have any numbers to quantify what 8.0 vs 8.2
> is likely to look like?

I'm afraid the best I can provide at the moment is [1]. On closer
inspection of these I'm a bit suspicious of the 8.0 numbers; I'll try to
reproduce them (and characterize the current ghc-8.2 branch, which fixes
a significant memory leak present in -rc1) shortly. That being said,
there were a few major fixes in 8.2.

> How much has the work that's been done affect performance across the
> board? Or are these mostly pathological cases (e.g. ghc --make with
> high number of cores, large number of definitions in a module, large
> amount of let nesting in definitions, etc)

The fixes span the gamut but I generally tried to concentrate on issues
that looked like they might affect a large fraction of programs. I fixed
at least one major regression present in 8.0 which significantly
inflated allocations of typeclass instance matching. I've also done
quite a bit of refactoring around our handling of names. These include
improvements in interface file serialization efficiency and name lookup.
These are just some of the major reworks; there are also numerous
smaller fixes which I don't have time to cover here.

Compilation performance has also generally improved as a result of some
of the work in the simplifier. In particular, GHC now performs an early
inlining phase which, quite surprisingly, tends to result in smaller
programs earlier in simplification, reducing the amount of work that the
compiler needs to carry out. Simon summarized some of his preliminary
numbers here [2].

> * DWARF support: could you clarify at a very high-level what typical uses
> cases can be expected to work and which ones won't? Would be eager to read
> any resources you could point me at to help me understand what still needs
> to be done on this front.
>
At this point if GHC compiles your program with -g you should have a
pretty good assurance that the resulting DWARF information is
reasonable. This means that users can use the ExecutionStack mechanism,
RTS stack-trace functionality, and GDB without fear of the act of
capturing a stacktrace leading to a crash.

After starting to write more here, I realized that you will likely not
be the last person to ask about this and updated the DWARF status page
with additional discussion [3]. Let me know if you have any questions.

Cheers,

- Ben

[1] https://gist.github.com/bgamari/fbd3e55047bd041d8208ebe820c0f493
[2] https://mail.haskell.org/pipermail/ghc-devs/2017-February/013818.html
[3] https://ghc.haskell.org/trac/ghc/wiki/DWARF/Status


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [ANNOUNCE] GHC 8.2.1 release candidate 1

Boespflug, Mathieu
In reply to this post by Ben Gamari-3
> > * Compiler performance: do you have any numbers to quantify what 8.0 vs 8.2
> > is likely to look like?
>
> I'm afraid the best I can provide at the moment is [1]. On closer
> inspection of these I'm a bit suspicious of the 8.0 numbers; I'll try to
> reproduce them (and characterize the current ghc-8.2 branch, which fixes
> a significant memory leak present in -rc1) shortly. That being said,
> there were a few major fixes in 8.2.

I'm not sure how to interpret these numbers. Looking at the compile
time section, it seems to report an average 81% *increase* in
compilation time, but compared to what - ghc-8.0 or ghc-7.4?
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [ANNOUNCE] GHC 8.2.1 release candidate 1

Ben Gamari-3
In reply to this post by Boespflug, Mathieu
"Boespflug, Mathieu" <[hidden email]> writes:

> Hi Ben,
>
> thank you for this summary. The DWARF status page is helpful. Something was
> unclear to me though. There are three main potential use cases for DWARF
> that I see:
>
> 1. debugging, possibly with gdb
> 2. stack traces on exceptions
> 3. stack sampling, which is a form of performance profiling.
>
> Forgive me for these naive questions, but... Is (1) possible at all at this
> point?
If you break into a program with gdb you will indeed get a usable
stacktrace. However, you cannot probe in-scope bindings as you can in
other languages.

> If I compile GHC with all the right options as explained in the
> status page, do I magically get backtraces associated to all my
> exceptions? What should I do to get those?

Ahh, thanks for pointing this out; this should be discussed on the
status page. I have a proposal allowing this here [1] and there is a
ticket here [2].

It turns out that this is actually a slightly tricky thing due to two
facts,

 * stack unwinding can be expensive

 * exceptions (especially asynchronous exceptions) are sometimes used
   as a mechanism for normal control flow.

Compatibility is quite tricky as under the current plan existing code
would drop callstack information if it catches and rethrows an
exception. I don't see any way to resolve this in a backwards
compatible manner.

> It's my understanding that (3) is possible and works fine, even when
> FFI code is invoked, but slower than for C programs due to running
> user-level code to inspect the stack for each sample. Is that right?

It's possible with my statistical profiling patches, but these are
currently not merged and the project is on hold due to lack of time. You
can, however, use perf. --callgraph will be broken in Haskell code,
however (e.g. you will only see the first stack frame).

Cheers,

- Ben

[1] https://ghc.haskell.org/trac/ghc/wiki/Exceptions/StackTraces
[2] https://ghc.haskell.org/trac/ghc/ticket/12096


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [ANNOUNCE] GHC 8.2.1 release candidate 1

Boespflug, Mathieu
Hi Ben,

it sounds like some of the remaining limitations around DWARF support
(e.g. finishing the stack sampling work, local bindings in GDB, ...)
could make for a good Haskell Summer of Code project. Have you
considered writing this up as one or two project ideas here:
https://summer.haskell.org/ideas.html? As HSoC sponsors, we'd be more
than happy to see someone pick this up.

Best,
--
Mathieu Boespflug
Founder at http://tweag.io.


On 10 April 2017 at 22:52, Ben Gamari <[hidden email]> wrote:

> "Boespflug, Mathieu" <[hidden email]> writes:
>
>> Hi Ben,
>>
>> thank you for this summary. The DWARF status page is helpful. Something was
>> unclear to me though. There are three main potential use cases for DWARF
>> that I see:
>>
>> 1. debugging, possibly with gdb
>> 2. stack traces on exceptions
>> 3. stack sampling, which is a form of performance profiling.
>>
>> Forgive me for these naive questions, but... Is (1) possible at all at this
>> point?
>
> If you break into a program with gdb you will indeed get a usable
> stacktrace. However, you cannot probe in-scope bindings as you can in
> other languages.
>
>> If I compile GHC with all the right options as explained in the
>> status page, do I magically get backtraces associated to all my
>> exceptions? What should I do to get those?
>
> Ahh, thanks for pointing this out; this should be discussed on the
> status page. I have a proposal allowing this here [1] and there is a
> ticket here [2].
>
> It turns out that this is actually a slightly tricky thing due to two
> facts,
>
>  * stack unwinding can be expensive
>
>  * exceptions (especially asynchronous exceptions) are sometimes used
>    as a mechanism for normal control flow.
>
> Compatibility is quite tricky as under the current plan existing code
> would drop callstack information if it catches and rethrows an
> exception. I don't see any way to resolve this in a backwards
> compatible manner.
>
>> It's my understanding that (3) is possible and works fine, even when
>> FFI code is invoked, but slower than for C programs due to running
>> user-level code to inspect the stack for each sample. Is that right?
>
> It's possible with my statistical profiling patches, but these are
> currently not merged and the project is on hold due to lack of time. You
> can, however, use perf. --callgraph will be broken in Haskell code,
> however (e.g. you will only see the first stack frame).
>
> Cheers,
>
> - Ben
>
> [1] https://ghc.haskell.org/trac/ghc/wiki/Exceptions/StackTraces
> [2] https://ghc.haskell.org/trac/ghc/ticket/12096
>
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [ANNOUNCE] GHC 8.2.1 release candidate 1

Ben Gamari-3
"Boespflug, Mathieu" <[hidden email]> writes:

> Hi Ben,
>
> it sounds like some of the remaining limitations around DWARF support
> (e.g. finishing the stack sampling work, local bindings in GDB, ...)
> could make for a good Haskell Summer of Code project. Have you
> considered writing this up as one or two project ideas here:
> https://summer.haskell.org/ideas.html? As HSoC sponsors, we'd be more
> than happy to see someone pick this up.
>
I haven't until now. To be honest I think the local-bindings-in-GDB
issue is a hard problem and is likely well beyond a summer project.
You would need not just the ability track the in-scope bindings in every
context of the source program, but also the ability to reconstitute
their values after simplification. It's also not clear to me how useful
this feature would be given how much simplified code often differs from
the code the user writes.

However, I do see there might be room for a project on the statistical
profiler itself or its associated tooling. We just need to come to a
conclusion on which direction is most appropriate for GHC.

For this having some concrete use-cases would be quite helpful. How do
you envision using statistical profiling on Haskell projects? What is
the minimal set of features that would make for a useful profiler?

Cheers,

- Ben

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [ANNOUNCE] GHC 8.2.1 release candidate 1

Boespflug, Mathieu
> However, I do see there might be room for a project on the statistical
> profiler itself or its associated tooling. We just need to come to a
> conclusion on which direction is most appropriate for GHC.

You mean - given a choice between somehow reusing perf or going down
the route of fully custom tooling?

> For this having some concrete use-cases would be quite helpful. How do
> you envision using statistical profiling on Haskell projects? What is
> the minimal set of features that would make for a useful profiler?

That sounds like a good way to approach this. Here goes...

I'd really prefer seeing a Haskell program as a black box, that I can
profile using the same tools as C programs or native code generated
from any other language. It shouldn't matter that the source is
Haskell. In my ideal workflow, I have a *vanilla* Haskell program
compiled with debug symbols by a *vanilla* GHC (no special ./configure
options as prereqs), that I can hook up perf to, e.g.

$ perf record -g ./mybinary

Then I should be able to use perf report to analyze the results. Or
indeed use existing pipelines to obtain other visualizations (flame
graphs etc).

I'm not particularly interested in integration with the event log,
though others might have a need for that.

I'm also interested in hotspot analysis, à la perf annotate.

As Brendan Gregg says, "perf isn't some random tool: it's part of the
Linux kernel, and is actively developed and enhanced."

I need accurate and informative stack samples (no STG internal details
in the output that I can't connect back to source locations) for
programs that include all manner of FFI calls. Better still if time
spent in the GC doesn't pollute my stack samples.

The tricky part is that for flame graphs you need to sample stacks,
and for that you need to teach perf how to collect that data somehow,
since the C stack isn't used for haskell activation frames and we have
a call-by-need evaluation strategy anyways. But the slow option you
mention in the status page sounds okayish to me, and using eBPF to
perform stack sampling entirely from the kernel looks like a promising
direction.
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [ANNOUNCE] GHC 8.2.1 release candidate 1

Ben Gamari-3
"Boespflug, Mathieu" <[hidden email]> writes:

>> However, I do see there might be room for a project on the statistical
>> profiler itself or its associated tooling. We just need to come to a
>> conclusion on which direction is most appropriate for GHC.
>
> You mean - given a choice between somehow reusing perf or going down
> the route of fully custom tooling?
>
Right.

>> For this having some concrete use-cases would be quite helpful. How do
>> you envision using statistical profiling on Haskell projects? What is
>> the minimal set of features that would make for a useful profiler?
>
> That sounds like a good way to approach this. Here goes...
>
> I'd really prefer seeing a Haskell program as a black box, that I can
> profile using the same tools as C programs or native code generated
> from any other language. It shouldn't matter that the source is
> Haskell. In my ideal workflow, I have a *vanilla* Haskell program
> compiled with debug symbols by a *vanilla* GHC (no special ./configure
> options as prereqs), that I can hook up perf to, e.g.
>
> $ perf record -g ./mybinary
>
> Then I should be able to use perf report to analyze the results. Or
> indeed use existing pipelines to obtain other visualizations (flame
> graphs etc).
>
> I'm not particularly interested in integration with the event log,
> though others might have a need for that.
>
> I'm also interested in hotspot analysis, à la perf annotate.
>
> As Brendan Gregg says, "perf isn't some random tool: it's part of the
> Linux kernel, and is actively developed and enhanced."
>
> I need accurate and informative stack samples (no STG internal details
> in the output that I can't connect back to source locations) for
> programs that include all manner of FFI calls. Better still if time
> spent in the GC doesn't pollute my stack samples.
>
> The tricky part is that for flame graphs you need to sample stacks,
> and for that you need to teach perf how to collect that data somehow,
> since the C stack isn't used for haskell activation frames and we have
> a call-by-need evaluation strategy anyways. But the slow option you
> mention in the status page sounds okayish to me, and using eBPF to
> perform stack sampling entirely from the kernel looks like a promising
> direction.
Indeed that is the engineering side of the trickiness. However, there is
also a theoretically difficult aspect which Peter articulates nicely in
Chapter 2 of his thesis. I refer you to the thesis for the full
explanation but, in short, reasoning about causality in lazy languages
is quite difficult (which is why this whole endeavour was worthy of a
thesis). This leads to some amount of impedance mismatch with existing
tooling, which is the reason I started down the road of a
Haskell-centric solution.

To elaborate, while in an imperative language it is relatively easy to say
"instruction at address $A arose from line $N in $FILE.c", this sort of
unambiguous statement is not possible in Haskell. Instead, we have to
be satisfied with *sets* of source locations. That is,
"instruction at address $A arose from line $N in $FILE, and
line $N' in $FILE', and ...".

Unfortunately, essentially none of the existing profiling and debugging
infrastructure (DWARF and perf included) was designed with this model in
mind. In particular, DWARF's line information encoding requires that GHC
choose a single location to attribute the cost of each instruction to.
This is, as you might expect, a rather tricky choice to make and while
GHC has a few heuristics, we will inevitably be wrong in some
circumstances. From experience, I can say that attempting to analyse
profiles naively, using only GHC's heuristically-guided guess of the
appropriate source location, can lead to rather perplexing results.

For this reason, GHC uses [1] DWARF's extensibility mechanisms to export
an additional set of line information which can be consumed by
Haskell-aware tooling which captures the full richness of GHC's source
ticks. In section 5.8 of his thesis Peter proposes a scheme of "fuzzing"
for making use of this location information. I haven't tried Peter's
fuzzing approach, but I suspect it will make the profiler output
significantly easier to reason about.


All this being said, I totally agree that being able to use
widely-known, well-maintained native tools is a significant advantage.
This is a large part of why I put the profiling project on hold. I have
wondered whether we might be able to provide a preprocessor which would
take sample data from GHC's statistical profiler and export it to
perf's own format, performing fuzzing and any other necessary
Haskell-specific preprocessing. This would at least allow us to tap in
to the ecosystem that has arisen around perf. Alternatively, we could
try to contribute patches to perf's upstream, although I'm not sure how
likely acceptance would be.

Cheers,

- Ben


[1] https://phabricator.haskell.org/D1279#R?query=D1279

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (497 bytes) Download Attachment
Loading...