Changes in GHC API modularity with the "Encode shape information in PMOracle" MR

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Changes in GHC API modularity with the "Encode shape information in PMOracle" MR

GHC - devs mailing list
Some time back, the `ghc-lib` project split into two targets : `ghc-lib-parser` for those projects that just need to produce syntax trees and `ghc-lib` (re-exporting `ghc-lib-parser` modules) having the remaining modules for projects that need to go on and distill parse trees to Core. The idea of course was to reduce build times for tools like hlint that only need parse trees.

Roughly, `ghc-lib-parser` got about 200 files and `ghc-lib` 300. Today after landing `7915afc6bb9539a4534db99aeb6616a6d145918a`, "Encode shape information in `PmOracle`", `ghc-lib-parser` now needs 543 files and `ghc-lib` gets just 25.

That may be just bad luck for `ghc-lib-parser` and the way it has to be but I thought I should at least mention the knock-on effect of this change on the modularity of the GHC API in case this consequence hasn't been considered?

--
Shayne Fletcher
Language Engineer / +1 917 699 7663
Digital Asset, creators of DAML

This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message.
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Changes in GHC API modularity with the "Encode shape information in PMOracle" MR

Sebastian Graf
Hi Shayne,

Sorry to hear that! We didn't consider modularity at all and I would be happy to try to refactor in a way that would allow `ghc-lib-parser` to be properly separated again.
I'm fairly certain that I didn't directly touch anything parser related, but apparently the new cyclic import of PmOracle within TcRnTypes (which is also exposed from `ghc-lib-parser`) pulled in the other half of GHC.
I'll see if I can fix that tomorrow, if only by extracting a separate `Types`-style module.

Cheers,
Sebastian

Am Mo., 16. Sept. 2019 um 22:04 Uhr schrieb Shayne Fletcher via ghc-devs <[hidden email]>:
Some time back, the `ghc-lib` project split into two targets : `ghc-lib-parser` for those projects that just need to produce syntax trees and `ghc-lib` (re-exporting `ghc-lib-parser` modules) having the remaining modules for projects that need to go on and distill parse trees to Core. The idea of course was to reduce build times for tools like hlint that only need parse trees.

Roughly, `ghc-lib-parser` got about 200 files and `ghc-lib` 300. Today after landing `7915afc6bb9539a4534db99aeb6616a6d145918a`, "Encode shape information in `PmOracle`", `ghc-lib-parser` now needs 543 files and `ghc-lib` gets just 25.

That may be just bad luck for `ghc-lib-parser` and the way it has to be but I thought I should at least mention the knock-on effect of this change on the modularity of the GHC API in case this consequence hasn't been considered?

--
Shayne Fletcher
Language Engineer / +1 917 699 7663
Digital Asset, creators of DAML

This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message._______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Changes in GHC API modularity with the "Encode shape information in PMOracle" MR

GHC - devs mailing list
Hi Sebastian,

On Mon, Sep 16, 2019 at 5:23 PM Sebastian Graf <[hidden email]> wrote:
Hi Shayne,

Sorry to hear that! We didn't consider modularity at all and I would be happy to try to refactor in a way that would allow `ghc-lib-parser` to be properly separated again.
I'm fairly certain that I didn't directly touch anything parser related, but apparently the new cyclic import of PmOracle within TcRnTypes (which is also exposed from `ghc-lib-parser`) pulled in the other half of GHC.
I'll see if I can fix that tomorrow, if only by extracting a separate `Types`-style module.


That sounds awesome. Tremendous. Thank-you! Please feel free to reach out to me if there's anything I can do to help your analysis[*]!

[*] For the record, the procedure for calculating the `ghc-lib-parser` modules is a little complicated by there needing to be some generated equivalents of `.hsc` files present for this to work but the procedure is at the end of the day just `ghc -M` invoked over `Parser.hs`.
 
Cheers,
Sebastian

Fingers crossed and all the best!

--
Shayne Fletcher
Language Engineer / +1 917 699 7663
Digital Asset, creators of DAML

This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message.
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Changes in GHC API modularity with the "Encode shape information in PMOracle" MR

Matthew Pickering
This is precisely the problem I was worried about in April.

https://mail.haskell.org/pipermail/ghc-devs/2019-April/017493.html

Any fix should ensure adding a test to make sure it doesn't happen again.

Cheers,

Matt

On Mon, Sep 16, 2019 at 10:38 PM Shayne Fletcher via ghc-devs
<[hidden email]> wrote:

>
> Hi Sebastian,
>
> On Mon, Sep 16, 2019 at 5:23 PM Sebastian Graf <[hidden email]> wrote:
>>
>> Hi Shayne,
>>
>> Sorry to hear that! We didn't consider modularity at all and I would be happy to try to refactor in a way that would allow `ghc-lib-parser` to be properly separated again.
>> I'm fairly certain that I didn't directly touch anything parser related, but apparently the new cyclic import of PmOracle within TcRnTypes (which is also exposed from `ghc-lib-parser`) pulled in the other half of GHC.
>> I'll see if I can fix that tomorrow, if only by extracting a separate `Types`-style module.
>>
>
> That sounds awesome. Tremendous. Thank-you! Please feel free to reach out to me if there's anything I can do to help your analysis[*]!
>
> [*] For the record, the procedure for calculating the `ghc-lib-parser` modules is a little complicated by there needing to be some generated equivalents of `.hsc` files present for this to work but the procedure is at the end of the day just `ghc -M` invoked over `Parser.hs`.
>
>>
>> Cheers,
>> Sebastian
>
>
> Fingers crossed and all the best!
>
> --
> Shayne Fletcher
> Language Engineer / +1 917 699 7663
> Digital Asset, creators of DAML
>
> This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message._______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Changes in GHC API modularity with the "Encode shape information in PMOracle" MR

Sebastian Graf
Hmm. The issue of build parallelism is kind-of orthogonal to .hs-boot files/circular imports.

The impact of .hs-boot files on build parallelism depends on their announced imports; at least in my semantic model of them (which might be incorrect) they just break cycles.
The degree of parallelism is visible as a property of this DAG where cycles are broken through insertion of .hs-boot nodes.

The issue here is one of transitive dependencies, which is a property on the original cyclic dependency graph.

So while we have the dependency graph

HsExpr <--> TcRnTypes <--> PmOracle

This is of no issue for build parallelism, because the .hs-boot files remove "feedback vertices" from the graph, rendering it acyclic:

TcRnTypes.hs-boot <- HsExpr <- TcRnTypes <- PmOracle
PmOracle.hs-boot <- TcRnTypes

Yet a package including HsExpr must also include PmOracle to compile.

The parallelism issue is best tackled by hadrian based profiles, on which we could compute try to come up with the optimal schedule and compare it across multiple CI runs.
But the dependency issue here is rather specific to the set of modules that should be included in ghc-lib-parser, I'm afraid, and I don't see how to solve it in a general way other than pre-computing the set of transitive dependencies for specific blessed modules. Which might be a thing we want... One set for each major entry point to the compiler pipeline, for example.

Am Di., 17. Sept. 2019 um 10:19 Uhr schrieb Matthew Pickering <[hidden email]>:
This is precisely the problem I was worried about in April.

https://mail.haskell.org/pipermail/ghc-devs/2019-April/017493.html

Any fix should ensure adding a test to make sure it doesn't happen again.

Cheers,

Matt

On Mon, Sep 16, 2019 at 10:38 PM Shayne Fletcher via ghc-devs
<[hidden email]> wrote:
>
> Hi Sebastian,
>
> On Mon, Sep 16, 2019 at 5:23 PM Sebastian Graf <[hidden email]> wrote:
>>
>> Hi Shayne,
>>
>> Sorry to hear that! We didn't consider modularity at all and I would be happy to try to refactor in a way that would allow `ghc-lib-parser` to be properly separated again.
>> I'm fairly certain that I didn't directly touch anything parser related, but apparently the new cyclic import of PmOracle within TcRnTypes (which is also exposed from `ghc-lib-parser`) pulled in the other half of GHC.
>> I'll see if I can fix that tomorrow, if only by extracting a separate `Types`-style module.
>>
>
> That sounds awesome. Tremendous. Thank-you! Please feel free to reach out to me if there's anything I can do to help your analysis[*]!
>
> [*] For the record, the procedure for calculating the `ghc-lib-parser` modules is a little complicated by there needing to be some generated equivalents of `.hsc` files present for this to work but the procedure is at the end of the day just `ghc -M` invoked over `Parser.hs`.
>
>>
>> Cheers,
>> Sebastian
>
>
> Fingers crossed and all the best!
>
> --
> Shayne Fletcher
> Language Engineer / +1 917 699 7663
> Digital Asset, creators of DAML
>
> This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message._______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Changes in GHC API modularity with the "Encode shape information in PMOracle" MR

Sebastian Graf
In reply to this post by GHC - devs mailing list
Shayne,

out of curiosity, could you find out which of the three modules DsMonad, FamInst and TcSimplify lead to the blowup? If it's not too much of a hassle, that is.
These are the only imports of PmOracle that aren't already exported from ghc-lib-parser.

Cheers,
Sebastian

Am Mo., 16. Sept. 2019 um 22:38 Uhr schrieb Shayne Fletcher <[hidden email]>:
Hi Sebastian,

On Mon, Sep 16, 2019 at 5:23 PM Sebastian Graf <[hidden email]> wrote:
Hi Shayne,

Sorry to hear that! We didn't consider modularity at all and I would be happy to try to refactor in a way that would allow `ghc-lib-parser` to be properly separated again.
I'm fairly certain that I didn't directly touch anything parser related, but apparently the new cyclic import of PmOracle within TcRnTypes (which is also exposed from `ghc-lib-parser`) pulled in the other half of GHC.
I'll see if I can fix that tomorrow, if only by extracting a separate `Types`-style module.


That sounds awesome. Tremendous. Thank-you! Please feel free to reach out to me if there's anything I can do to help your analysis[*]!

[*] For the record, the procedure for calculating the `ghc-lib-parser` modules is a little complicated by there needing to be some generated equivalents of `.hsc` files present for this to work but the procedure is at the end of the day just `ghc -M` invoked over `Parser.hs`.
 
Cheers,
Sebastian

Fingers crossed and all the best!

--
Shayne Fletcher
Language Engineer / +1 917 699 7663
Digital Asset, creators of DAML

This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message.

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Changes in GHC API modularity with the "Encode shape information in PMOracle" MR

GHC - devs mailing list
Seeing what I can do.

On Tue, Sep 17, 2019 at 10:31 AM Sebastian Graf <[hidden email]> wrote:
Shayne,

out of curiosity, could you find out which of the three modules DsMonad, FamInst and TcSimplify lead to the blowup? If it's not too much of a hassle, that is.
These are the only imports of PmOracle that aren't already exported from ghc-lib-parser.

Cheers,
Sebastian

Am Mo., 16. Sept. 2019 um 22:38 Uhr schrieb Shayne Fletcher <[hidden email]>:
Hi Sebastian,

On Mon, Sep 16, 2019 at 5:23 PM Sebastian Graf <[hidden email]> wrote:
Hi Shayne,

Sorry to hear that! We didn't consider modularity at all and I would be happy to try to refactor in a way that would allow `ghc-lib-parser` to be properly separated again.
I'm fairly certain that I didn't directly touch anything parser related, but apparently the new cyclic import of PmOracle within TcRnTypes (which is also exposed from `ghc-lib-parser`) pulled in the other half of GHC.
I'll see if I can fix that tomorrow, if only by extracting a separate `Types`-style module.


That sounds awesome. Tremendous. Thank-you! Please feel free to reach out to me if there's anything I can do to help your analysis[*]!

[*] For the record, the procedure for calculating the `ghc-lib-parser` modules is a little complicated by there needing to be some generated equivalents of `.hsc` files present for this to work but the procedure is at the end of the day just `ghc -M` invoked over `Parser.hs`.
 
Cheers,
Sebastian

Fingers crossed and all the best!

--
Shayne Fletcher
Language Engineer / +1 917 699 7663
Digital Asset, creators of DAML

This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message.


--
Shayne Fletcher
Language Engineer / +1 917 699 7663
Digital Asset, creators of DAML

This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message.
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Changes in GHC API modularity with the "Encode shape information in PMOracle" MR

GHC - devs mailing list
Getting you some raw data to look at but first results indicate at least that `DsMonad` is pulling in `nativeGen`, `llvmGen`, `stgToCmm`, `simplCore`, `hieFile` and so on.

On Tue, Sep 17, 2019 at 12:05 PM Shayne Fletcher <[hidden email]> wrote:
Seeing what I can do.

On Tue, Sep 17, 2019 at 10:31 AM Sebastian Graf <[hidden email]> wrote:
Shayne,

out of curiosity, could you find out which of the three modules DsMonad, FamInst and TcSimplify lead to the blowup? If it's not too much of a hassle, that is.
These are the only imports of PmOracle that aren't already exported from ghc-lib-parser.

Cheers,
Sebastian

Am Mo., 16. Sept. 2019 um 22:38 Uhr schrieb Shayne Fletcher <[hidden email]>:
Hi Sebastian,

On Mon, Sep 16, 2019 at 5:23 PM Sebastian Graf <[hidden email]> wrote:
Hi Shayne,

Sorry to hear that! We didn't consider modularity at all and I would be happy to try to refactor in a way that would allow `ghc-lib-parser` to be properly separated again.
I'm fairly certain that I didn't directly touch anything parser related, but apparently the new cyclic import of PmOracle within TcRnTypes (which is also exposed from `ghc-lib-parser`) pulled in the other half of GHC.
I'll see if I can fix that tomorrow, if only by extracting a separate `Types`-style module.


That sounds awesome. Tremendous. Thank-you! Please feel free to reach out to me if there's anything I can do to help your analysis[*]!

[*] For the record, the procedure for calculating the `ghc-lib-parser` modules is a little complicated by there needing to be some generated equivalents of `.hsc` files present for this to work but the procedure is at the end of the day just `ghc -M` invoked over `Parser.hs`.
 
Cheers,
Sebastian

Fingers crossed and all the best!

--
Shayne Fletcher
Language Engineer / +1 917 699 7663
Digital Asset, creators of DAML

This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message.


--
Shayne Fletcher
Language Engineer / +1 917 699 7663
Digital Asset, creators of DAML


--
Shayne Fletcher
Language Engineer / +1 917 699 7663
Digital Asset, creators of DAML

This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message.
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Changes in GHC API modularity with the "Encode shape information in PMOracle" MR

GHC - devs mailing list
In reply to this post by Sebastian Graf
Excellent work Sebastian. I confirm that the situation is resolved by https://gitlab.haskell.org/ghc/ghc/merge_requests/1743.

On Mon, Sep 16, 2019 at 5:23 PM Sebastian Graf <[hidden email]> wrote:
Hi Shayne,

Sorry to hear that! We didn't consider modularity at all and I would be happy to try to refactor in a way that would allow `ghc-lib-parser` to be properly separated again.
I'm fairly certain that I didn't directly touch anything parser related, but apparently the new cyclic import of PmOracle within TcRnTypes (which is also exposed from `ghc-lib-parser`) pulled in the other half of GHC.
I'll see if I can fix that tomorrow, if only by extracting a separate `Types`-style module.

Cheers,
Sebastian

Am Mo., 16. Sept. 2019 um 22:04 Uhr schrieb Shayne Fletcher via ghc-devs <[hidden email]>:
Some time back, the `ghc-lib` project split into two targets : `ghc-lib-parser` for those projects that just need to produce syntax trees and `ghc-lib` (re-exporting `ghc-lib-parser` modules) having the remaining modules for projects that need to go on and distill parse trees to Core. The idea of course was to reduce build times for tools like hlint that only need parse trees.

Roughly, `ghc-lib-parser` got about 200 files and `ghc-lib` 300. Today after landing `7915afc6bb9539a4534db99aeb6616a6d145918a`, "Encode shape information in `PmOracle`", `ghc-lib-parser` now needs 543 files and `ghc-lib` gets just 25.

That may be just bad luck for `ghc-lib-parser` and the way it has to be but I thought I should at least mention the knock-on effect of this change on the modularity of the GHC API in case this consequence hasn't been considered?

--
Shayne Fletcher
Language Engineer / +1 917 699 7663
Digital Asset, creators of DAML

This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message._______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


--
Shayne Fletcher
Language Engineer / +1 917 699 7663
Digital Asset, creators of DAML

This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message.
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs