Does it sound a good idea to implement "backend plugins"?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Does it sound a good idea to implement "backend plugins"?

Shao, Cheng
Hi all,

I'm thinking of adding "backend plugins" in the current Plugins
mechanism which allows one to inspect/modify the IRs post simplifier
pass (STG/Cmm), similar to the recently added source plugins for HsSyn
IRs. This can be useful for anyone creating a custom GHC backend to
target an experimental platform (e.g. the Asterius compiler which
targets WebAssembly), and previously in order to retrieve those IRs
from the regular pipeline, we need to use Hooks which is somewhat
hacky.

Does this sound a good idea to you? If so, I can open a trac ticket
and a Phab diff for this feature.

Best,
Shao Cheng
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Does it sound a good idea to implement "backend plugins"?

Matthew Pickering
Sounds like a reasonable idea to me. However, you should take some
time to propose a concrete interface as this part was not obvious in
the design of source plugins.

Matt
On Thu, Oct 4, 2018 at 2:56 PM Shao, Cheng <[hidden email]> wrote:

>
> Hi all,
>
> I'm thinking of adding "backend plugins" in the current Plugins
> mechanism which allows one to inspect/modify the IRs post simplifier
> pass (STG/Cmm), similar to the recently added source plugins for HsSyn
> IRs. This can be useful for anyone creating a custom GHC backend to
> target an experimental platform (e.g. the Asterius compiler which
> targets WebAssembly), and previously in order to retrieve those IRs
> from the regular pipeline, we need to use Hooks which is somewhat
> hacky.
>
> Does this sound a good idea to you? If so, I can open a trac ticket
> and a Phab diff for this feature.
>
> Best,
> Shao Cheng
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Does it sound a good idea to implement "backend plugins"?

Ben Gamari-2
In reply to this post by Shao, Cheng
"Shao, Cheng" <[hidden email]> writes:

> Hi all,
>
> I'm thinking of adding "backend plugins" in the current Plugins
> mechanism which allows one to inspect/modify the IRs post simplifier
> pass (STG/Cmm), similar to the recently added source plugins for HsSyn
> IRs. This can be useful for anyone creating a custom GHC backend to
> target an experimental platform (e.g. the Asterius compiler which
> targets WebAssembly), and previously in order to retrieve those IRs
> from the regular pipeline, we need to use Hooks which is somewhat
> hacky.
>
> Does this sound a good idea to you? If so, I can open a trac ticket
> and a Phab diff for this feature.
>
Yes, during the Implementors' Workshop this year it seemed like there
was considerable interest in such a mechanism. However, as Matthew said,
the devil is in the details; before starting an implementation I would
recommend that you open a ticket describing the specifics of the
proposed interface. It also wouldn't hurt to motivate the proposal with
a discussion of the concrete use-cases that the interface is meant to
address.

Cheers,

- Ben


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Does it sound a good idea to implement "backend plugins"?

Luite Stegeman
In reply to this post by Shao, Cheng
I think it sounds like a potentially good idea in general, but I agree with Ben and Matthew here that a more concrete plan of intended use is needed.

Adding "pluggable backends" to spin up new targets seems to require quite a bit of additional infrastructure for initialising a library directory and package database. But there are probably more specific use cases that need inspecting/modifying STG or Cmm where plugins would already be useful in practice.

Hooks (or rather their locations in the pipeline) are rather ad hoc by nature, but for Asterius a hook that takes Cmm and takes over from there seems like a reasonable approach given the current state of things. I think the Cmm hook you implemented (or something similar) would be perfectly acceptable to use for now.

I don't think it's a problem if a hook exists for some time, and at some point it's superseded by a more general plugin mechanism. Especially with the GHC 6 month release cycle there's not much need for future proofing.

Luite

On Thu, Oct 4, 2018 at 3:56 PM Shao, Cheng <[hidden email]> wrote:
Hi all,

I'm thinking of adding "backend plugins" in the current Plugins
mechanism which allows one to inspect/modify the IRs post simplifier
pass (STG/Cmm), similar to the recently added source plugins for HsSyn
IRs. This can be useful for anyone creating a custom GHC backend to
target an experimental platform (e.g. the Asterius compiler which
targets WebAssembly), and previously in order to retrieve those IRs
from the regular pipeline, we need to use Hooks which is somewhat
hacky.

Does this sound a good idea to you? If so, I can open a trac ticket
and a Phab diff for this feature.

Best,
Shao Cheng
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Does it sound a good idea to implement "backend plugins"?

Shao, Cheng
> Adding "pluggable backends" to spin up new targets seems to require quite a bit of additional infrastructure for initialising a library directory and package database. But there are probably more specific use cases that need inspecting/modifying STG or Cmm where plugins would already be useful in practice.

I think setting up a new global libdir/pkgdb is beyond the scope of
backend plugins. The user shall implement his/her own boot script to
configure for the new architecture, generate relevant headers, run
Cabal's Setup program to launch GHC with the plugin loaded.

> Hooks (or rather their locations in the pipeline) are rather ad hoc by nature, but for Asterius a hook that takes Cmm and takes over from there seems like a reasonable approach given the current state of things. I think the Cmm hook you implemented (or something similar) would be perfectly acceptable to use for now.

For the use case of asterius itself, indeed Hooks already fit the use
case for now. But since we seek to upstream our newly added features
in our ghc fork back to ghc hq, we should upstream those changes early
and make them more principled. Compared to Hooks, I prefer to move to
Plugins entirely since:

* Plugins are more composable, you can load multiple plugins in one
ghc invocation. Hooks are not.
* If I implement the same mechanisms in Plugins, this can be
beneficial to other projects. Currently, in asterius, everything works
via a pile of hacks upon hacks in ghc-toolkit, and it's not good for
reuse.
* The newly added backend plugins shouldn't have visible
correctness/performance impact if they're not used, and it's just a
few local modifications in the ghc codebase.

> On Thu, Oct 4, 2018 at 3:56 PM Shao, Cheng <[hidden email]> wrote:
>>
>> Hi all,
>>
>> I'm thinking of adding "backend plugins" in the current Plugins
>> mechanism which allows one to inspect/modify the IRs post simplifier
>> pass (STG/Cmm), similar to the recently added source plugins for HsSyn
>> IRs. This can be useful for anyone creating a custom GHC backend to
>> target an experimental platform (e.g. the Asterius compiler which
>> targets WebAssembly), and previously in order to retrieve those IRs
>> from the regular pipeline, we need to use Hooks which is somewhat
>> hacky.
>>
>> Does this sound a good idea to you? If so, I can open a trac ticket
>> and a Phab diff for this feature.
>>
>> Best,
>> Shao Cheng
>> _______________________________________________
>> ghc-devs mailing list
>> [hidden email]
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Does it sound a good idea to implement "backend plugins"?

Moritz Angermann-2
A long time ago, I’ve tried to inject plugin logic to allows some control over the driver pipeline (phase ordering) and hooking various code gen related functions.


At that time I ran into issues that might simply not exist with plugins anymore today, but I haven’t looked.

The whole design wasn’t quite right and injects everything into the dynflags.  Also ghc wanted to be able to compile the plugin on the fly, but I needed the plugin to be loaded very early during the startup phase to exert enough control of the rest of the pipeline through the plugin.

Cheers,
 Moritz

Sent from my iPhone

On 5 Oct 2018, at 1:52 AM, Shao, Cheng <[hidden email]> wrote:

Adding "pluggable backends" to spin up new targets seems to require quite a bit of additional infrastructure for initialising a library directory and package database. But there are probably more specific use cases that need inspecting/modifying STG or Cmm where plugins would already be useful in practice.

I think setting up a new global libdir/pkgdb is beyond the scope of
backend plugins. The user shall implement his/her own boot script to
configure for the new architecture, generate relevant headers, run
Cabal's Setup program to launch GHC with the plugin loaded.

Hooks (or rather their locations in the pipeline) are rather ad hoc by nature, but for Asterius a hook that takes Cmm and takes over from there seems like a reasonable approach given the current state of things. I think the Cmm hook you implemented (or something similar) would be perfectly acceptable to use for now.

For the use case of asterius itself, indeed Hooks already fit the use
case for now. But since we seek to upstream our newly added features
in our ghc fork back to ghc hq, we should upstream those changes early
and make them more principled. Compared to Hooks, I prefer to move to
Plugins entirely since:

* Plugins are more composable, you can load multiple plugins in one
ghc invocation. Hooks are not.
* If I implement the same mechanisms in Plugins, this can be
beneficial to other projects. Currently, in asterius, everything works
via a pile of hacks upon hacks in ghc-toolkit, and it's not good for
reuse.
* The newly added backend plugins shouldn't have visible
correctness/performance impact if they're not used, and it's just a
few local modifications in the ghc codebase.

On Thu, Oct 4, 2018 at 3:56 PM Shao, Cheng <[hidden email]> wrote:

Hi all,

I'm thinking of adding "backend plugins" in the current Plugins
mechanism which allows one to inspect/modify the IRs post simplifier
pass (STG/Cmm), similar to the recently added source plugins for HsSyn
IRs. This can be useful for anyone creating a custom GHC backend to
target an experimental platform (e.g. the Asterius compiler which
targets WebAssembly), and previously in order to retrieve those IRs
from the regular pipeline, we need to use Hooks which is somewhat
hacky.

Does this sound a good idea to you? If so, I can open a trac ticket
and a Phab diff for this feature.

Best,
Shao Cheng
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Does it sound a good idea to implement "backend plugins"?

Shao, Cheng
> A long time ago, I’ve tried to inject plugin logic to allows some control over the driver pipeline (phase ordering) and hooking various code gen related functions.
>
> See https://phabricator.haskell.org/D535

Cool! I haven't thoroughly read the history of that diff, but allowing
manipulation of a Hooks via a Plugin seems overkill in this case, and
even if one can do so, it still doesn't lead to the backend IR types;
one would need to use runPhaseHook and modify the behavior after a
CgGuts is generated, which unfortunately leads to quite some
boilerplate code.

> At that time I ran into issues that might simply not exist with plugins anymore today, but I haven’t looked.

Interesting. I'll make sure to consult you in case I'm bitten by some
hidden issues when I actually implement it :)

> The whole design wasn’t quite right and injects everything into the dynflags.  Also ghc wanted to be able to compile the plugin on the fly, but I needed the plugin to be loaded very early during the startup phase to exert enough control of the rest of the pipeline through the plugin.

Well, in the case of backend plugins, it isn't supposed to be a home
plugin to be compiled and used on the fly. A typical use case would be
compiling/installing the plugin to a standalone pkgdb, then used to
compile other packages.

>
> On 5 Oct 2018, at 1:52 AM, Shao, Cheng <[hidden email]> wrote:
>
> Adding "pluggable backends" to spin up new targets seems to require quite a bit of additional infrastructure for initialising a library directory and package database. But there are probably more specific use cases that need inspecting/modifying STG or Cmm where plugins would already be useful in practice.
>
>
> I think setting up a new global libdir/pkgdb is beyond the scope of
> backend plugins. The user shall implement his/her own boot script to
> configure for the new architecture, generate relevant headers, run
> Cabal's Setup program to launch GHC with the plugin loaded.
>
> Hooks (or rather their locations in the pipeline) are rather ad hoc by nature, but for Asterius a hook that takes Cmm and takes over from there seems like a reasonable approach given the current state of things. I think the Cmm hook you implemented (or something similar) would be perfectly acceptable to use for now.
>
>
> For the use case of asterius itself, indeed Hooks already fit the use
> case for now. But since we seek to upstream our newly added features
> in our ghc fork back to ghc hq, we should upstream those changes early
> and make them more principled. Compared to Hooks, I prefer to move to
> Plugins entirely since:
>
> * Plugins are more composable, you can load multiple plugins in one
> ghc invocation. Hooks are not.
> * If I implement the same mechanisms in Plugins, this can be
> beneficial to other projects. Currently, in asterius, everything works
> via a pile of hacks upon hacks in ghc-toolkit, and it's not good for
> reuse.
> * The newly added backend plugins shouldn't have visible
> correctness/performance impact if they're not used, and it's just a
> few local modifications in the ghc codebase.
>
> On Thu, Oct 4, 2018 at 3:56 PM Shao, Cheng <[hidden email]> wrote:
>
>
> Hi all,
>
>
> I'm thinking of adding "backend plugins" in the current Plugins
>
> mechanism which allows one to inspect/modify the IRs post simplifier
>
> pass (STG/Cmm), similar to the recently added source plugins for HsSyn
>
> IRs. This can be useful for anyone creating a custom GHC backend to
>
> target an experimental platform (e.g. the Asterius compiler which
>
> targets WebAssembly), and previously in order to retrieve those IRs
>
> from the regular pipeline, we need to use Hooks which is somewhat
>
> hacky.
>
>
> Does this sound a good idea to you? If so, I can open a trac ticket
>
> and a Phab diff for this feature.
>
>
> Best,
>
> Shao Cheng
>
> _______________________________________________
>
> ghc-devs mailing list
>
> [hidden email]
>
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs