Patch/feature proposal: "Source plugins"

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Patch/feature proposal: "Source plugins"

Edsko de Vries-4
Sorry for the earlier mishap, here's the full email.

Hi all,

The plugin mechanism gives access to the program in Core; this suffices for
many but not quite all purposes. Tools that need access to the original AST
can call typecheckModule directly, but of course this requires using the
GHC API directly. Moreover, even when using the GHC API directly anyway (as
in my case), it means that tools cannot take advantage of ghc's
infrastructure for dependency tracking, recompiling only changed modules,
etc.

Hence it would be useful to have "source plugins", which can be used both
externally and when using ghc API (in the latter case I guess "hooks" would
be the more appropriate terminology). Currently "core plugins" are recorded
as part of DynFlags as

    pluginModNames        :: [ModuleName],
    pluginModNameOpts     :: [(ModuleName,String)],

This makes sense when thinking of plugins only as an external mechanism,
but is less convenient when using them as internal hooks, too. In my draft
patch I introduce a new type "HscPlugin" (described shortly) and added

    sourcePlugins         :: [HscPlugin],

to DynFlags. HscPlugin is a record of a pair of functions; having the
actual record here rather than  a module name means that these functions
can have a non-empty closure, which is obviously convenient when using this
as a hook rather than an external plugin.

In my current version HscPlugin looks like

    data HscPlugin = HscPlugin {
        runHscPlugin :: forall m. MonadIO m
                     => DynFlags
                     -> TcGblEnv
                     -> m TcGblEnv

      , runHscQQ     :: forall m. MonadIO m
                     => Env TcGblEnv TcLclEnv
                     -> HsQuasiQuote Name
                     -> m (HsQuasiQuote Name)
      }

runHscPlugin is the main function; it gets passed the TcGblEnv (which
contains the type checked AST as its tcd_binds field) and gets a change to
return it modified (we don't currently take advantage of that; I did that
only to be in line with "core plugins").

Unfortunately, the typechecked AST is only a subset of the renamed AST (see
http://www.haskell.org/pipermail/ghc-devs/2013-February/000540.html). The
TcGblEnv contains a  tcg_rn_decls field, which is a reference to the full
renamed (as opposed to typechecked) AST, but by default this is not
initialized: the typechecker only optionally retains the renamed AST, and
this is hardcoded to by False. In my current patch I have changed this so
that it's hard coded to be True; ideally this should become an option in
DynFlags (more ideal still would be if the type checked AST would not lose
any information).

Unfortunately, even the renamer loses information: quasi-quotes get
expanded during renaming and all evidence of that there was ever a
quasi-quote there has disappeared when the renamer returns. For this
reason, the HscPlugin type that I'm using at the moment also has a hook for
quasi-quotes.

So what I have currently done is:

   1. Introduced the above HscPlugin type and added a corresponding field
   to DynFlags
   2. Call runHscQQ in the renamer whenever a quasi-quote gets expanded.
   3. Make sure that the typechecker passes the result of the renamer
   through.
   4. Call runHscPlugin on the result of the typechecker.

In my client code I then combine the information obtained from these three
sources (2, 3, 4).

The path of least resistance for me currently to submit this as a patch for
ghc therefore be to submit a patch that does precisely what I described
above, mutatis mutandis based on your feedback, except that I would add an
option to add to DynFlags that would tell the type checker whether or not
to pass the result of the renamer through, rather than hardcoding it.

It is a little bit messy mostly because parts of the AST get lost along the
way: quasi-quotes in the renamer, data type declarations and other things
during type checking. A more ideal way, but also more time consuming, would
be to change this so that the renamer leaves evidence of the quasi-quotes
in the tree, and the type checker returns the entire tree type checked,
rather than just a subset. I think that ultimately this is the better
approach, at least for our purposes -- I'm not sure about other tools, but
since this would be a larger change that affects larger parts of the ghc
pipeline I'm not sure that I'll be able to do it.

Any and all feedback on the above would be appreciated!

Edsko
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20130605/55b5bd0a/attachment-0001.htm>

Reply | Threaded
Open this post in threaded view
|

Patch/feature proposal: "Source plugins"

Simon Marlow-7
+1.  I don't have any comments on the technical issues I'm afraid, but
the more ways we can extend GHC without having to change it directly,
the better.

Cheers,
        Simon

On 05/06/13 12:51, Edsko de Vries wrote:

> Sorry for the earlier mishap, here's the full email.
>
> Hi all,
>
> The plugin mechanism gives access to the program in Core; this suffices
> for many but not quite all purposes. Tools that need access to the
> original AST can call typecheckModule directly, but of course this
> requires using the GHC API directly. Moreover, even when using the GHC
> API directly anyway (as in my case), it means that tools cannot take
> advantage of ghc's infrastructure for dependency tracking, recompiling
> only changed modules, etc.
>
> Hence it would be useful to have "source plugins", which can be used
> both externally and when using ghc API (in the latter case I guess
> "hooks" would be the more appropriate terminology). Currently "core
> plugins" are recorded as part of DynFlags as
>
>      pluginModNames        :: [ModuleName],
>      pluginModNameOpts     :: [(ModuleName,String)],
>
> This makes sense when thinking of plugins only as an external mechanism,
> but is less convenient when using them as internal hooks, too. In my
> draft patch I introduce a new type "HscPlugin" (described shortly) and added
>
>      sourcePlugins         :: [HscPlugin],
>
> to DynFlags. HscPlugin is a record of a pair of functions; having the
> actual record here rather than  a module name means that these functions
> can have a non-empty closure, which is obviously convenient when using
> this as a hook rather than an external plugin.
>
> In my current version HscPlugin looks like
>
>      data HscPlugin = HscPlugin {
>          runHscPlugin :: forall m. MonadIO m
>                       => DynFlags
>                       -> TcGblEnv
>                       -> m TcGblEnv
>        , runHscQQ     :: forall m. MonadIO m
>                       => Env TcGblEnv TcLclEnv
>                       -> HsQuasiQuote Name
>                       -> m (HsQuasiQuote Name)
>        }
>
> runHscPlugin is the main function; it gets passed the TcGblEnv (which
> contains the type checked AST as its tcd_binds field) and gets a change
> to return it modified (we don't currently take advantage of that; I did
> that only to be in line with "core plugins").
>
> Unfortunately, the typechecked AST is only a subset of the renamed AST
> (see
> http://www.haskell.org/pipermail/ghc-devs/2013-February/000540.html).
> The TcGblEnv contains a  tcg_rn_decls field, which is a reference to the
> full renamed (as opposed to typechecked) AST, but by default this is not
> initialized: the typechecker only optionally retains the renamed AST,
> and this is hardcoded to by False. In my current patch I have changed
> this so that it's hard coded to be True; ideally this should become an
> option in DynFlags (more ideal still would be if the type checked AST
> would not lose any information).
>
> Unfortunately, even the renamer loses information: quasi-quotes get
> expanded during renaming and all evidence of that there was ever a
> quasi-quote there has disappeared when the renamer returns. For this
> reason, the HscPlugin type that I'm using at the moment also has a hook
> for quasi-quotes.
>
> So what I have currently done is:
>
>  1. Introduced the above HscPlugin type and added a corresponding field
>     to DynFlags
>  2. Call runHscQQ in the renamer whenever a quasi-quote gets expanded.
>  3. Make sure that the typechecker passes the result of the renamer through.
>  4. Call runHscPlugin on the result of the typechecker.
>
> In my client code I then combine the information obtained from these
> three sources (2, 3, 4).
>
> The path of least resistance for me currently to submit this as a patch
> for ghc therefore be to submit a patch that does precisely what I
> described above, mutatis mutandis based on your feedback, except that I
> would add an option to add to DynFlags that would tell the type checker
> whether or not to pass the result of the renamer through, rather than
> hardcoding it.
>
> It is a little bit messy mostly because parts of the AST get lost along
> the way: quasi-quotes in the renamer, data type declarations and other
> things during type checking. A more ideal way, but also more time
> consuming, would be to change this so that the renamer leaves evidence
> of the quasi-quotes in the tree, and the type checker returns the entire
> tree type checked, rather than just a subset. I think that ultimately
> this is the better approach, at least for our purposes -- I'm not sure
> about other tools, but since this would be a larger change that affects
> larger parts of the ghc pipeline I'm not sure that I'll be able to do it.
>
> Any and all feedback on the above would be appreciated!
>
> Edsko
>
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://www.haskell.org/mailman/listinfo/ghc-devs
>



Reply | Threaded
Open this post in threaded view
|

Patch/feature proposal: "Source plugins"

Luite Stegeman
I'd also be very happy with some plugin interface that allows us to use
more GhcMake functionality from the GHC API. For GHCJS we would want to run
our own (STG -> JavaScript) code generator on sources that need to be
recompiled instead of letting GHC run its pipeline. Do you think that's
possible with your proposal?

luite


On Thu, Jun 6, 2013 at 10:13 PM, Simon Marlow <marlowsd at gmail.com> wrote:

> +1.  I don't have any comments on the technical issues I'm afraid, but the
> more ways we can extend GHC without having to change it directly, the
> better.
>
> Cheers,
>         Simon
>
>
> On 05/06/13 12:51, Edsko de Vries wrote:
>
>> Sorry for the earlier mishap, here's the full email.
>>
>> Hi all,
>>
>> The plugin mechanism gives access to the program in Core; this suffices
>> for many but not quite all purposes. Tools that need access to the
>> original AST can call typecheckModule directly, but of course this
>> requires using the GHC API directly. Moreover, even when using the GHC
>> API directly anyway (as in my case), it means that tools cannot take
>> advantage of ghc's infrastructure for dependency tracking, recompiling
>> only changed modules, etc.
>>
>> Hence it would be useful to have "source plugins", which can be used
>> both externally and when using ghc API (in the latter case I guess
>> "hooks" would be the more appropriate terminology). Currently "core
>> plugins" are recorded as part of DynFlags as
>>
>>      pluginModNames        :: [ModuleName],
>>      pluginModNameOpts     :: [(ModuleName,String)],
>>
>> This makes sense when thinking of plugins only as an external mechanism,
>> but is less convenient when using them as internal hooks, too. In my
>> draft patch I introduce a new type "HscPlugin" (described shortly) and
>> added
>>
>>      sourcePlugins         :: [HscPlugin],
>>
>> to DynFlags. HscPlugin is a record of a pair of functions; having the
>> actual record here rather than  a module name means that these functions
>> can have a non-empty closure, which is obviously convenient when using
>> this as a hook rather than an external plugin.
>>
>> In my current version HscPlugin looks like
>>
>>      data HscPlugin = HscPlugin {
>>          runHscPlugin :: forall m. MonadIO m
>>                       => DynFlags
>>                       -> TcGblEnv
>>                       -> m TcGblEnv
>>        , runHscQQ     :: forall m. MonadIO m
>>                       => Env TcGblEnv TcLclEnv
>>                       -> HsQuasiQuote Name
>>                       -> m (HsQuasiQuote Name)
>>        }
>>
>> runHscPlugin is the main function; it gets passed the TcGblEnv (which
>> contains the type checked AST as its tcd_binds field) and gets a change
>> to return it modified (we don't currently take advantage of that; I did
>> that only to be in line with "core plugins").
>>
>> Unfortunately, the typechecked AST is only a subset of the renamed AST
>> (see
>> http://www.haskell.org/**pipermail/ghc-devs/2013-**February/000540.html<http://www.haskell.org/pipermail/ghc-devs/2013-February/000540.html>
>> ).
>> The TcGblEnv contains a  tcg_rn_decls field, which is a reference to the
>> full renamed (as opposed to typechecked) AST, but by default this is not
>> initialized: the typechecker only optionally retains the renamed AST,
>> and this is hardcoded to by False. In my current patch I have changed
>> this so that it's hard coded to be True; ideally this should become an
>> option in DynFlags (more ideal still would be if the type checked AST
>> would not lose any information).
>>
>> Unfortunately, even the renamer loses information: quasi-quotes get
>> expanded during renaming and all evidence of that there was ever a
>> quasi-quote there has disappeared when the renamer returns. For this
>> reason, the HscPlugin type that I'm using at the moment also has a hook
>> for quasi-quotes.
>>
>> So what I have currently done is:
>>
>>  1. Introduced the above HscPlugin type and added a corresponding field
>>     to DynFlags
>>  2. Call runHscQQ in the renamer whenever a quasi-quote gets expanded.
>>  3. Make sure that the typechecker passes the result of the renamer
>> through.
>>  4. Call runHscPlugin on the result of the typechecker.
>>
>>
>> In my client code I then combine the information obtained from these
>> three sources (2, 3, 4).
>>
>> The path of least resistance for me currently to submit this as a patch
>> for ghc therefore be to submit a patch that does precisely what I
>> described above, mutatis mutandis based on your feedback, except that I
>> would add an option to add to DynFlags that would tell the type checker
>> whether or not to pass the result of the renamer through, rather than
>> hardcoding it.
>>
>> It is a little bit messy mostly because parts of the AST get lost along
>> the way: quasi-quotes in the renamer, data type declarations and other
>> things during type checking. A more ideal way, but also more time
>> consuming, would be to change this so that the renamer leaves evidence
>> of the quasi-quotes in the tree, and the type checker returns the entire
>> tree type checked, rather than just a subset. I think that ultimately
>> this is the better approach, at least for our purposes -- I'm not sure
>> about other tools, but since this would be a larger change that affects
>> larger parts of the ghc pipeline I'm not sure that I'll be able to do it.
>>
>> Any and all feedback on the above would be appreciated!
>>
>> Edsko
>>
>>
>> ______________________________**_________________
>> ghc-devs mailing list
>> ghc-devs at haskell.org
>> http://www.haskell.org/**mailman/listinfo/ghc-devs<http://www.haskell.org/mailman/listinfo/ghc-devs>
>>
>>
>
> ______________________________**_________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://www.haskell.org/**mailman/listinfo/ghc-devs<http://www.haskell.org/mailman/listinfo/ghc-devs>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20130607/1108754c/attachment.htm>

Reply | Threaded
Open this post in threaded view
|

Patch/feature proposal: "Source plugins"

Edsko de Vries-4
Luite,

On Fri, Jun 7, 2013 at 2:08 PM, Luite Stegeman <stegeman at gmail.com> wrote:

> I'd also be very happy with some plugin interface that allows us to use
> more GhcMake functionality from the GHC API. For GHCJS we would want to run
> our own (STG -> JavaScript) code generator on sources that need to be
> recompiled instead of letting GHC run its pipeline. Do you think that's
> possible with your proposal?
>

Probably; if you disable ghc's code generator (set hscTarget to HscNothing,
ghcLink to NoLink) ghc will still do the dependency analysis, still process
modules in the right order, and a source plugin still gets called on every
module that gets typechecked correctly -- so you could potentially generate
code at that point?

Edsko
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20130611/8b2fa8c2/attachment.htm>

Reply | Threaded
Open this post in threaded view
|

Patch/feature proposal: "Source plugins"

Luite Stegeman
Sounds good, but for some reason we have HscAsm as a target at the moment
in GHCJS, and set it to HscNothing only after typechecking. I forgot why,
might have something to do with TH. Do you have an implementation of this
patch that i can test it with (even if it's not the final API or really
ugly)?

luite



On Tue, Jun 11, 2013 at 10:43 AM, Edsko de Vries <edskodevries at gmail.com>wrote:

> Luite,
>
> On Fri, Jun 7, 2013 at 2:08 PM, Luite Stegeman <stegeman at gmail.com> wrote:
>
>> I'd also be very happy with some plugin interface that allows us to use
>> more GhcMake functionality from the GHC API. For GHCJS we would want to run
>> our own (STG -> JavaScript) code generator on sources that need to be
>> recompiled instead of letting GHC run its pipeline. Do you think that's
>> possible with your proposal?
>>
>
> Probably; if you disable ghc's code generator (set hscTarget to
> HscNothing, ghcLink to NoLink) ghc will still do the dependency analysis,
> still process modules in the right order, and a source plugin still gets
> called on every module that gets typechecked correctly -- so you could
> potentially generate code at that point?
>
> Edsko
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20130611/02474f2b/attachment-0001.htm>

Reply | Threaded
Open this post in threaded view
|

Patch/feature proposal: "Source plugins"

Thomas Schilling-2
In reply to this post by Edsko de Vries-4
On 5 June 2013 13:51, Edsko de Vries <edskodevries at gmail.com> wrote:
> It is a little bit messy mostly because parts of the AST get lost along the
> way: quasi-quotes in the renamer, data type declarations and other things
> during type checking. A more ideal way, but also more time consuming, would
> be to change this so that the renamer leaves evidence of the quasi-quotes in
> the tree, and the type checker returns the entire tree type checked, rather
> than just a subset. I think that ultimately this is the better approach, at
> least for our purposes -- I'm not sure about other tools, but since this
> would be a larger change that affects larger parts of the ghc pipeline I'm
> not sure that I'll be able to do it.

I needed something similar.  In particular, I built a custom code
generator, but now I need a similar feature for extracting information
from a Haskell file (for IDE features).

Since I needed to modify one-shot compilation mode I couldn't use the
GHC API.  For the IDE stuff I'm using Shake as the build manager, so
that also needs a customized one-shot mode.  For my current
implementation I just copied and adapted the necessary parts of
HscMain, DriverPipeline, etc.  That's very messy, fragile and breaks
on every GHC release so I'd really like to see the necessary features
put into GHC.

Do you have a working patch somewhere?


Reply | Threaded
Open this post in threaded view
|

Patch/feature proposal: "Source plugins"

Simon Peyton Jones
Guys,

I'm not following the details here, but I'm open to suggestions (patches, even) that improve the GHC API.  

Simon

| -----Original Message-----
| From: ghc-devs-bounces at haskell.org [mailto:ghc-devs-bounces at haskell.org]
| On Behalf Of Thomas Schilling
| Sent: 11 June 2013 12:53
| To: Edsko de Vries
| Cc: ghc-devs at haskell.org
| Subject: Re: Patch/feature proposal: "Source plugins"
|
| On 5 June 2013 13:51, Edsko de Vries <edskodevries at gmail.com> wrote:
| > It is a little bit messy mostly because parts of the AST get lost
| along the
| > way: quasi-quotes in the renamer, data type declarations and other
| things
| > during type checking. A more ideal way, but also more time consuming,
| would
| > be to change this so that the renamer leaves evidence of the quasi-
| quotes in
| > the tree, and the type checker returns the entire tree type checked,
| rather
| > than just a subset. I think that ultimately this is the better
| approach, at
| > least for our purposes -- I'm not sure about other tools, but since
| this
| > would be a larger change that affects larger parts of the ghc pipeline
| I'm
| > not sure that I'll be able to do it.
|
| I needed something similar.  In particular, I built a custom code
| generator, but now I need a similar feature for extracting information
| from a Haskell file (for IDE features).
|
| Since I needed to modify one-shot compilation mode I couldn't use the
| GHC API.  For the IDE stuff I'm using Shake as the build manager, so
| that also needs a customized one-shot mode.  For my current
| implementation I just copied and adapted the necessary parts of
| HscMain, DriverPipeline, etc.  That's very messy, fragile and breaks
| on every GHC release so I'd really like to see the necessary features
| put into GHC.
|
| Do you have a working patch somewhere?
|
| _______________________________________________
| ghc-devs mailing list
| ghc-devs at haskell.org
| http://www.haskell.org/mailman/listinfo/ghc-devs


Reply | Threaded
Open this post in threaded view
|

Patch/feature proposal: "Source plugins"

Luite Stegeman
Any news on this? I'd really like to have this in GHC 7.8.1 so that we can
release a fully working GHCJS with GhcMake functionality based on it. I'd
be happy to help write the patch.

luite



On Tue, Jun 11, 2013 at 3:21 PM, Simon Peyton-Jones
<simonpj at microsoft.com>wrote:

> Guys,
>
> I'm not following the details here, but I'm open to suggestions (patches,
> even) that improve the GHC API.
>
> Simon
>
> | -----Original Message-----
> | From: ghc-devs-bounces at haskell.org [mailto:ghc-devs-bounces at haskell.org]
> | On Behalf Of Thomas Schilling
> | Sent: 11 June 2013 12:53
> | To: Edsko de Vries
> | Cc: ghc-devs at haskell.org
> | Subject: Re: Patch/feature proposal: "Source plugins"
> |
> | On 5 June 2013 13:51, Edsko de Vries <edskodevries at gmail.com> wrote:
> | > It is a little bit messy mostly because parts of the AST get lost
> | along the
> | > way: quasi-quotes in the renamer, data type declarations and other
> | things
> | > during type checking. A more ideal way, but also more time consuming,
> | would
> | > be to change this so that the renamer leaves evidence of the quasi-
> | quotes in
> | > the tree, and the type checker returns the entire tree type checked,
> | rather
> | > than just a subset. I think that ultimately this is the better
> | approach, at
> | > least for our purposes -- I'm not sure about other tools, but since
> | this
> | > would be a larger change that affects larger parts of the ghc pipeline
> | I'm
> | > not sure that I'll be able to do it.
> |
> | I needed something similar.  In particular, I built a custom code
> | generator, but now I need a similar feature for extracting information
> | from a Haskell file (for IDE features).
> |
> | Since I needed to modify one-shot compilation mode I couldn't use the
> | GHC API.  For the IDE stuff I'm using Shake as the build manager, so
> | that also needs a customized one-shot mode.  For my current
> | implementation I just copied and adapted the necessary parts of
> | HscMain, DriverPipeline, etc.  That's very messy, fragile and breaks
> | on every GHC release so I'd really like to see the necessary features
> | put into GHC.
> |
> | Do you have a working patch somewhere?
> |
> | _______________________________________________
> | ghc-devs mailing list
> | ghc-devs at haskell.org
> | http://www.haskell.org/mailman/listinfo/ghc-devs
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://www.haskell.org/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20130626/58393710/attachment.htm>

Reply | Threaded
Open this post in threaded view
|

Patch/feature proposal: "Source plugins"

Edsko de Vries-4
Hi Luite,

I was fully planning on a first version of the patch yesterday, but so far
my efforts were thwarted by annoying problems with dynamic libraries (not
-- directly -- related to the patch at all). I will try again today :)

Edsko


On Wed, Jun 26, 2013 at 8:51 AM, Luite Stegeman <stegeman at gmail.com> wrote:

> Any news on this? I'd really like to have this in GHC 7.8.1 so that we can
> release a fully working GHCJS with GhcMake functionality based on it. I'd
> be happy to help write the patch.
>
> luite
>
>
>
> On Tue, Jun 11, 2013 at 3:21 PM, Simon Peyton-Jones <simonpj at microsoft.com
> > wrote:
>
>> Guys,
>>
>> I'm not following the details here, but I'm open to suggestions (patches,
>> even) that improve the GHC API.
>>
>> Simon
>>
>> | -----Original Message-----
>> | From: ghc-devs-bounces at haskell.org [mailto:ghc-devs-bounces at haskell.org
>> ]
>> | On Behalf Of Thomas Schilling
>> | Sent: 11 June 2013 12:53
>> | To: Edsko de Vries
>> | Cc: ghc-devs at haskell.org
>> | Subject: Re: Patch/feature proposal: "Source plugins"
>> |
>> | On 5 June 2013 13:51, Edsko de Vries <edskodevries at gmail.com> wrote:
>> | > It is a little bit messy mostly because parts of the AST get lost
>> | along the
>> | > way: quasi-quotes in the renamer, data type declarations and other
>> | things
>> | > during type checking. A more ideal way, but also more time consuming,
>> | would
>> | > be to change this so that the renamer leaves evidence of the quasi-
>> | quotes in
>> | > the tree, and the type checker returns the entire tree type checked,
>> | rather
>> | > than just a subset. I think that ultimately this is the better
>> | approach, at
>> | > least for our purposes -- I'm not sure about other tools, but since
>> | this
>> | > would be a larger change that affects larger parts of the ghc pipeline
>> | I'm
>> | > not sure that I'll be able to do it.
>> |
>> | I needed something similar.  In particular, I built a custom code
>> | generator, but now I need a similar feature for extracting information
>> | from a Haskell file (for IDE features).
>> |
>> | Since I needed to modify one-shot compilation mode I couldn't use the
>> | GHC API.  For the IDE stuff I'm using Shake as the build manager, so
>> | that also needs a customized one-shot mode.  For my current
>> | implementation I just copied and adapted the necessary parts of
>> | HscMain, DriverPipeline, etc.  That's very messy, fragile and breaks
>> | on every GHC release so I'd really like to see the necessary features
>> | put into GHC.
>> |
>> | Do you have a working patch somewhere?
>> |
>> | _______________________________________________
>> | ghc-devs mailing list
>> | ghc-devs at haskell.org
>> | http://www.haskell.org/mailman/listinfo/ghc-devs
>>
>> _______________________________________________
>> ghc-devs mailing list
>> ghc-devs at haskell.org
>> http://www.haskell.org/mailman/listinfo/ghc-devs
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20130626/eedc164d/attachment.htm>

Reply | Threaded
Open this post in threaded view
|

Patch/feature proposal: "Source plugins"

Thomas Schilling-2
In reply to this post by Luite Stegeman
I'm building a "wrapper" that is implemented by pulling out
DriverPipeline and HscMain.  Once that API seems OK we can turn it
into a GHC patch.

On 26 June 2013 09:51, Luite Stegeman <stegeman at gmail.com> wrote:

> Any news on this? I'd really like to have this in GHC 7.8.1 so that we can
> release a fully working GHCJS with GhcMake functionality based on it. I'd be
> happy to help write the patch.
>
> luite
>
>
>
> On Tue, Jun 11, 2013 at 3:21 PM, Simon Peyton-Jones <simonpj at microsoft.com>
> wrote:
>>
>> Guys,
>>
>> I'm not following the details here, but I'm open to suggestions (patches,
>> even) that improve the GHC API.
>>
>> Simon
>>
>> | -----Original Message-----
>> | From: ghc-devs-bounces at haskell.org [mailto:ghc-devs-bounces at haskell.org]
>> | On Behalf Of Thomas Schilling
>> | Sent: 11 June 2013 12:53
>> | To: Edsko de Vries
>> | Cc: ghc-devs at haskell.org
>> | Subject: Re: Patch/feature proposal: "Source plugins"
>> |
>> | On 5 June 2013 13:51, Edsko de Vries <edskodevries at gmail.com> wrote:
>> | > It is a little bit messy mostly because parts of the AST get lost
>> | along the
>> | > way: quasi-quotes in the renamer, data type declarations and other
>> | things
>> | > during type checking. A more ideal way, but also more time consuming,
>> | would
>> | > be to change this so that the renamer leaves evidence of the quasi-
>> | quotes in
>> | > the tree, and the type checker returns the entire tree type checked,
>> | rather
>> | > than just a subset. I think that ultimately this is the better
>> | approach, at
>> | > least for our purposes -- I'm not sure about other tools, but since
>> | this
>> | > would be a larger change that affects larger parts of the ghc pipeline
>> | I'm
>> | > not sure that I'll be able to do it.
>> |
>> | I needed something similar.  In particular, I built a custom code
>> | generator, but now I need a similar feature for extracting information
>> | from a Haskell file (for IDE features).
>> |
>> | Since I needed to modify one-shot compilation mode I couldn't use the
>> | GHC API.  For the IDE stuff I'm using Shake as the build manager, so
>> | that also needs a customized one-shot mode.  For my current
>> | implementation I just copied and adapted the necessary parts of
>> | HscMain, DriverPipeline, etc.  That's very messy, fragile and breaks
>> | on every GHC release so I'd really like to see the necessary features
>> | put into GHC.
>> |
>> | Do you have a working patch somewhere?
>> |
>> | _______________________________________________
>> | ghc-devs mailing list
>> | ghc-devs at haskell.org
>> | http://www.haskell.org/mailman/listinfo/ghc-devs
>>
>> _______________________________________________
>> ghc-devs mailing list
>> ghc-devs at haskell.org
>> http://www.haskell.org/mailman/listinfo/ghc-devs
>
>


Reply | Threaded
Open this post in threaded view
|

Patch/feature proposal: "Source plugins"

Simon Peyton Jones
In reply to this post by Edsko de Vries-4
I'm confused as to details here.

*         Edsko is doing something; Nick is doing something else (attached for completeness).

*         I can't locate a Trac Wiki page that describes the design

I'm more than happy to adopt patches that improve the plugin API, but you'll have to lead me through it!

No hurry, just when you are ready.

Simon

From: Edsko de Vries [mailto:edskodevries at gmail.com]
Sent: 26 June 2013 09:21
To: Luite Stegeman
Cc: Simon Peyton-Jones; Thomas Schilling; ghc-devs at haskell.org
Subject: Re: Patch/feature proposal: "Source plugins"

Hi Luite,

I was fully planning on a first version of the patch yesterday, but so far my efforts were thwarted by annoying problems with dynamic libraries (not -- directly -- related to the patch at all). I will try again today :)

Edsko

On Wed, Jun 26, 2013 at 8:51 AM, Luite Stegeman <stegeman at gmail.com<mailto:stegeman at gmail.com>> wrote:
Any news on this? I'd really like to have this in GHC 7.8.1 so that we can release a fully working GHCJS with GhcMake functionality based on it. I'd be happy to help write the patch.

luite


On Tue, Jun 11, 2013 at 3:21 PM, Simon Peyton-Jones <simonpj at microsoft.com<mailto:simonpj at microsoft.com>> wrote:
Guys,

I'm not following the details here, but I'm open to suggestions (patches, even) that improve the GHC API.

Simon

| -----Original Message-----
| From: ghc-devs-bounces at haskell.org<mailto:ghc-devs-bounces at haskell.org> [mailto:ghc-devs-bounces at haskell.org<mailto:ghc-devs-bounces at haskell.org>]
| On Behalf Of Thomas Schilling
| Sent: 11 June 2013 12:53
| To: Edsko de Vries
| Cc: ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>
| Subject: Re: Patch/feature proposal: "Source plugins"
|
| On 5 June 2013 13:51, Edsko de Vries <edskodevries at gmail.com<mailto:edskodevries at gmail.com>> wrote:
| > It is a little bit messy mostly because parts of the AST get lost
| along the
| > way: quasi-quotes in the renamer, data type declarations and other
| things
| > during type checking. A more ideal way, but also more time consuming,
| would
| > be to change this so that the renamer leaves evidence of the quasi-
| quotes in
| > the tree, and the type checker returns the entire tree type checked,
| rather
| > than just a subset. I think that ultimately this is the better
| approach, at
| > least for our purposes -- I'm not sure about other tools, but since
| this
| > would be a larger change that affects larger parts of the ghc pipeline
| I'm
| > not sure that I'll be able to do it.
|
| I needed something similar.  In particular, I built a custom code
| generator, but now I need a similar feature for extracting information
| from a Haskell file (for IDE features).
|
| Since I needed to modify one-shot compilation mode I couldn't use the
| GHC API.  For the IDE stuff I'm using Shake as the build manager, so
| that also needs a customized one-shot mode.  For my current
| implementation I just copied and adapted the necessary parts of
| HscMain, DriverPipeline, etc.  That's very messy, fragile and breaks
| on every GHC release so I'd really like to see the necessary features
| put into GHC.
|
| Do you have a working patch somewhere?
|
| _______________________________________________
| ghc-devs mailing list
| ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>
| http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>
http://www.haskell.org/mailman/listinfo/ghc-devs


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20130628/8ddb6cbb/attachment-0001.htm>
-------------- next part --------------
An embedded message was scrubbed...
From: Nicolas Frisby <nicolas.frisby at gmail.com>
Subject: Re: invoking front-end phases from within a GHC plugin?
Date: Tue, 25 Jun 2013 02:56:39 +0000
Size: 11759
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20130628/8ddb6cbb/attachment-0001.eml>

Reply | Threaded
Open this post in threaded view
|

Patch/feature proposal: "Source plugins"

Edsko de Vries-4
Yes, I fully intend to create a ticket with a detailed description and a
first patch, but I've been struggling with the latest HEAD, and
specifically the fact that it now uses dynamic libraries for TH. I (think I
am) stuck at a Cabal bug and I cannot currently build my code at all :-/
Duncan is looking into this though.

Edsko


On Fri, Jun 28, 2013 at 1:43 PM, Simon Peyton-Jones
<simonpj at microsoft.com>wrote:

>  I?m confused as to details here.  ****
>
> **?         **Edsko is doing something; Nick is doing something else
> (attached for completeness).****
>
> **?         **I can?t locate a Trac Wiki page that describes the design***
> *
>
> ** **
>
> I?m more than happy to adopt patches that improve the plugin API, but
> you?ll have to lead me through it!****
>
> ** **
>
> No hurry, just when you are ready.****
>
> ** **
>
> Simon****
>
> ** **
>
> *From:* Edsko de Vries [mailto:edskodevries at gmail.com]
> *Sent:* 26 June 2013 09:21
> *To:* Luite Stegeman
> *Cc:* Simon Peyton-Jones; Thomas Schilling; ghc-devs at haskell.org
>
> *Subject:* Re: Patch/feature proposal: "Source plugins"****
>
>  ** **
>
> Hi Luite,****
>
> ** **
>
> I was fully planning on a first version of the patch yesterday, but so far
> my efforts were thwarted by annoying problems with dynamic libraries (not
> -- directly -- related to the patch at all). I will try again today :)****
>
> ** **
>
> Edsko****
>
> ** **
>
> On Wed, Jun 26, 2013 at 8:51 AM, Luite Stegeman <stegeman at gmail.com>
> wrote:****
>
>  Any news on this? I'd really like to have this in GHC 7.8.1 so that we
> can release a fully working GHCJS with GhcMake functionality based on it.
> I'd be happy to help write the patch.****
>
> ** **
>
> luite****
>
> ** **
>
> ** **
>
> On Tue, Jun 11, 2013 at 3:21 PM, Simon Peyton-Jones <simonpj at microsoft.com>
> wrote:****
>
>  Guys,
>
> I'm not following the details here, but I'm open to suggestions (patches,
> even) that improve the GHC API.
>
> Simon****
>
>
> | -----Original Message-----
> | From: ghc-devs-bounces at haskell.org [mailto:ghc-devs-bounces at haskell.org]
> | On Behalf Of Thomas Schilling
> | Sent: 11 June 2013 12:53
> | To: Edsko de Vries
> | Cc: ghc-devs at haskell.org
> | Subject: Re: Patch/feature proposal: "Source plugins"
> |
> | On 5 June 2013 13:51, Edsko de Vries <edskodevries at gmail.com> wrote:
> | > It is a little bit messy mostly because parts of the AST get lost
> | along the
> | > way: quasi-quotes in the renamer, data type declarations and other
> | things
> | > during type checking. A more ideal way, but also more time consuming,
> | would
> | > be to change this so that the renamer leaves evidence of the quasi-
> | quotes in
> | > the tree, and the type checker returns the entire tree type checked,
> | rather
> | > than just a subset. I think that ultimately this is the better
> | approach, at
> | > least for our purposes -- I'm not sure about other tools, but since
> | this
> | > would be a larger change that affects larger parts of the ghc pipeline
> | I'm
> | > not sure that I'll be able to do it.
> |
> | I needed something similar.  In particular, I built a custom code
> | generator, but now I need a similar feature for extracting information
> | from a Haskell file (for IDE features).
> |
> | Since I needed to modify one-shot compilation mode I couldn't use the
> | GHC API.  For the IDE stuff I'm using Shake as the build manager, so
> | that also needs a customized one-shot mode.  For my current
> | implementation I just copied and adapted the necessary parts of
> | HscMain, DriverPipeline, etc.  That's very messy, fragile and breaks
> | on every GHC release so I'd really like to see the necessary features
> | put into GHC.
> |
> | Do you have a working patch somewhere?
> |
> | _______________________________________________
> | ghc-devs mailing list
> | ghc-devs at haskell.org
> | http://www.haskell.org/mailman/listinfo/ghc-devs
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://www.haskell.org/mailman/listinfo/ghc-devs****
>
>  ** **
>
>  ** **
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20130701/d55f6534/attachment.htm>

Reply | Threaded
Open this post in threaded view
|

Patch/feature proposal: "Source plugins"

Edsko de Vries-4
Ok, sorry all for the delay. Attached is a "frontend plugins" patch for ghc
7.4.2. It will not work for ghc HEAD because the structure of the compiler
has changed in various places, but I cannot currently compile my code
against HEAD. Making this available now because it's been far too long
already :) Comments/suggestions/feedback welcome. Hopefully the patch is
pretty self-explanatory; and it's only small.

-E


On Mon, Jul 1, 2013 at 9:32 AM, Edsko de Vries <edskodevries at gmail.com>wrote:

> Yes, I fully intend to create a ticket with a detailed description and a
> first patch, but I've been struggling with the latest HEAD, and
> specifically the fact that it now uses dynamic libraries for TH. I (think I
> am) stuck at a Cabal bug and I cannot currently build my code at all :-/
> Duncan is looking into this though.
>
> Edsko
>
>
> On Fri, Jun 28, 2013 at 1:43 PM, Simon Peyton-Jones <simonpj at microsoft.com
> > wrote:
>
>>  I?m confused as to details here.  ****
>>
>> **?         **Edsko is doing something; Nick is doing something else
>> (attached for completeness).****
>>
>> **?         **I can?t locate a Trac Wiki page that describes the design**
>> **
>>
>> ** **
>>
>> I?m more than happy to adopt patches that improve the plugin API, but
>> you?ll have to lead me through it!****
>>
>> ** **
>>
>> No hurry, just when you are ready.****
>>
>> ** **
>>
>> Simon****
>>
>> ** **
>>
>> *From:* Edsko de Vries [mailto:edskodevries at gmail.com]
>> *Sent:* 26 June 2013 09:21
>> *To:* Luite Stegeman
>> *Cc:* Simon Peyton-Jones; Thomas Schilling; ghc-devs at haskell.org
>>
>> *Subject:* Re: Patch/feature proposal: "Source plugins"****
>>
>>  ** **
>>
>> Hi Luite,****
>>
>> ** **
>>
>> I was fully planning on a first version of the patch yesterday, but so
>> far my efforts were thwarted by annoying problems with dynamic libraries
>> (not -- directly -- related to the patch at all). I will try again today :)
>> ****
>>
>> ** **
>>
>> Edsko****
>>
>> ** **
>>
>> On Wed, Jun 26, 2013 at 8:51 AM, Luite Stegeman <stegeman at gmail.com>
>> wrote:****
>>
>>  Any news on this? I'd really like to have this in GHC 7.8.1 so that we
>> can release a fully working GHCJS with GhcMake functionality based on it.
>> I'd be happy to help write the patch.****
>>
>> ** **
>>
>> luite****
>>
>> ** **
>>
>> ** **
>>
>> On Tue, Jun 11, 2013 at 3:21 PM, Simon Peyton-Jones <
>> simonpj at microsoft.com> wrote:****
>>
>>  Guys,
>>
>> I'm not following the details here, but I'm open to suggestions (patches,
>> even) that improve the GHC API.
>>
>> Simon****
>>
>>
>> | -----Original Message-----
>> | From: ghc-devs-bounces at haskell.org [mailto:ghc-devs-bounces at haskell.org
>> ]
>> | On Behalf Of Thomas Schilling
>> | Sent: 11 June 2013 12:53
>> | To: Edsko de Vries
>> | Cc: ghc-devs at haskell.org
>> | Subject: Re: Patch/feature proposal: "Source plugins"
>> |
>> | On 5 June 2013 13:51, Edsko de Vries <edskodevries at gmail.com> wrote:
>> | > It is a little bit messy mostly because parts of the AST get lost
>> | along the
>> | > way: quasi-quotes in the renamer, data type declarations and other
>> | things
>> | > during type checking. A more ideal way, but also more time consuming,
>> | would
>> | > be to change this so that the renamer leaves evidence of the quasi-
>> | quotes in
>> | > the tree, and the type checker returns the entire tree type checked,
>> | rather
>> | > than just a subset. I think that ultimately this is the better
>> | approach, at
>> | > least for our purposes -- I'm not sure about other tools, but since
>> | this
>> | > would be a larger change that affects larger parts of the ghc pipeline
>> | I'm
>> | > not sure that I'll be able to do it.
>> |
>> | I needed something similar.  In particular, I built a custom code
>> | generator, but now I need a similar feature for extracting information
>> | from a Haskell file (for IDE features).
>> |
>> | Since I needed to modify one-shot compilation mode I couldn't use the
>> | GHC API.  For the IDE stuff I'm using Shake as the build manager, so
>> | that also needs a customized one-shot mode.  For my current
>> | implementation I just copied and adapted the necessary parts of
>> | HscMain, DriverPipeline, etc.  That's very messy, fragile and breaks
>> | on every GHC release so I'd really like to see the necessary features
>> | put into GHC.
>> |
>> | Do you have a working patch somewhere?
>> |
>> | _______________________________________________
>> | ghc-devs mailing list
>> | ghc-devs at haskell.org
>> | http://www.haskell.org/mailman/listinfo/ghc-devs
>>
>> _______________________________________________
>> ghc-devs mailing list
>> ghc-devs at haskell.org
>> http://www.haskell.org/mailman/listinfo/ghc-devs****
>>
>>  ** **
>>
>>  ** **
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20130708/3e428234/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ghc-7.4.2-frontend-plugins.diff
Type: application/octet-stream
Size: 5450 bytes
Desc: not available
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20130708/3e428234/attachment-0001.obj>

Reply | Threaded
Open this post in threaded view
|

Patch/feature proposal: "Source plugins"

Austin Seipp-4
Looks simple enough in principle.

This patch only implements hooking, although I agree with your original
email that this would also be useful externally (with -fplugin.) However,
the core plugin mechanism is the opposite: it only has an external
interface, but I'm unsure if there's a similarly easy way to fiddle with
the core pipeline, using only the GHC API. Adding it to dynflags like you
did would be OK (barring other peoples' objections.)

So with that in mind, should we maybe take this time to consolidate the two
APIs a bit for coherency? Just a thought.

On Monday, July 8, 2013, Edsko de Vries wrote:

> Ok, sorry all for the delay. Attached is a "frontend plugins" patch for
> ghc 7.4.2. It will not work for ghc HEAD because the structure of the
> compiler has changed in various places, but I cannot currently compile my
> code against HEAD. Making this available now because it's been far too long
> already :) Comments/suggestions/feedback welcome. Hopefully the patch is
> pretty self-explanatory; and it's only small.
>
> -E
>
>
> On Mon, Jul 1, 2013 at 9:32 AM, Edsko de Vries <edskodevries at gmail.com>wrote:
>
> Yes, I fully intend to create a ticket with a detailed description and a
> first patch, but I've been struggling with the latest HEAD, and
> specifically the fact that it now uses dynamic libraries for TH. I (think I
> am) stuck at a Cabal bug and I cannot currently build my code at all :-/
> Duncan is looking into this though.
>
> Edsko
>
>
> On Fri, Jun 28, 2013 at 1:43 PM, Simon Peyton-Jones <simonpj at microsoft.com
> > wrote:
>
>  I?m confused as to details here.  ****
>
> **?         **Edsko is doing something; Nick is doing something else
> (attached for completeness).****
>
> **?         **I can?t locate a Trac Wiki page that describes the design***
> *
>
> ** **
>
> I?m more than happy to adopt patches that improve the plugin API, but
> you?ll have to lead me through it!****
>
> ** **
>
> No hurry, just when you are ready.****
>
> ** **
>
> Simon****
>
> ** **
>
> *From:* Edsko de Vries [mailto:edskodevries at gmail.com]
> *Sent:* 26 June 2013 09:21
> *To:* Luite Stegeman
> *Cc:* Simon Peyton-Jones; Thomas Schilling; ghc-devs at haskell.org
>
> *Subject:* Re: Patch/feature proposal: "Source plugins"****
>
>  ** **
>
> Hi Luite,****
>
> ** **
>
> I was fully planning on a first version of the patch yesterday, but so far
> my efforts were thwarted by annoying problems with dynamic libraries (not
> -- directly -- related to the patch at all). I will try again today :)****
>
> ** **
>
> Edsko****
>
>

--
Sent from my portable turning machine
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20130708/dc6accf0/attachment.htm>

Reply | Threaded
Open this post in threaded view
|

Patch/feature proposal: "Source plugins"

Simon Peyton Jones
In reply to this post by Edsko de Vries-4
Edsko, Luite, Thomas, and others

Thanks.  I wonder whether you might write a Wiki page saying

*        exactly what you want to achieve, with examples, and

*        sketching how you hope to achieve it

You may want to do a few rounds on that among yourselves; perhaps you have various different goals in mind that we can address at once.

Selfishly, I don't really want to review code until I have a very clear picture of what it is trying to achieve.   (And I'm rushing towards the POPL deadline anyway!)

Simon

From: Edsko de Vries [mailto:edskodevries at gmail.com]
Sent: 08 July 2013 18:02
To: Simon Peyton-Jones
Cc: Luite Stegeman; Thomas Schilling; ghc-devs at haskell.org; Nicolas Frisby
Subject: Re: Patch/feature proposal: "Source plugins"

Ok, sorry all for the delay. Attached is a "frontend plugins" patch for ghc 7.4.2. It will not work for ghc HEAD because the structure of the compiler has changed in various places, but I cannot currently compile my code against HEAD. Making this available now because it's been far too long already :) Comments/suggestions/feedback welcome. Hopefully the patch is pretty self-explanatory; and it's only small.

-E

On Mon, Jul 1, 2013 at 9:32 AM, Edsko de Vries <edskodevries at gmail.com<mailto:edskodevries at gmail.com>> wrote:
Yes, I fully intend to create a ticket with a detailed description and a first patch, but I've been struggling with the latest HEAD, and specifically the fact that it now uses dynamic libraries for TH. I (think I am) stuck at a Cabal bug and I cannot currently build my code at all :-/ Duncan is looking into this though.

Edsko

On Fri, Jun 28, 2013 at 1:43 PM, Simon Peyton-Jones <simonpj at microsoft.com<mailto:simonpj at microsoft.com>> wrote:
I'm confused as to details here.

*         Edsko is doing something; Nick is doing something else (attached for completeness).

*         I can't locate a Trac Wiki page that describes the design

I'm more than happy to adopt patches that improve the plugin API, but you'll have to lead me through it!

No hurry, just when you are ready.

Simon

From: Edsko de Vries [mailto:edskodevries at gmail.com<mailto:edskodevries at gmail.com>]
Sent: 26 June 2013 09:21
To: Luite Stegeman
Cc: Simon Peyton-Jones; Thomas Schilling; ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>

Subject: Re: Patch/feature proposal: "Source plugins"

Hi Luite,

I was fully planning on a first version of the patch yesterday, but so far my efforts were thwarted by annoying problems with dynamic libraries (not -- directly -- related to the patch at all). I will try again today :)

Edsko

On Wed, Jun 26, 2013 at 8:51 AM, Luite Stegeman <stegeman at gmail.com<mailto:stegeman at gmail.com>> wrote:
Any news on this? I'd really like to have this in GHC 7.8.1 so that we can release a fully working GHCJS with GhcMake functionality based on it. I'd be happy to help write the patch.

luite


On Tue, Jun 11, 2013 at 3:21 PM, Simon Peyton-Jones <simonpj at microsoft.com<mailto:simonpj at microsoft.com>> wrote:
Guys,

I'm not following the details here, but I'm open to suggestions (patches, even) that improve the GHC API.

Simon

| -----Original Message-----
| From: ghc-devs-bounces at haskell.org<mailto:ghc-devs-bounces at haskell.org> [mailto:ghc-devs-bounces at haskell.org<mailto:ghc-devs-bounces at haskell.org>]
| On Behalf Of Thomas Schilling
| Sent: 11 June 2013 12:53
| To: Edsko de Vries
| Cc: ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>
| Subject: Re: Patch/feature proposal: "Source plugins"
|
| On 5 June 2013 13:51, Edsko de Vries <edskodevries at gmail.com<mailto:edskodevries at gmail.com>> wrote:
| > It is a little bit messy mostly because parts of the AST get lost
| along the
| > way: quasi-quotes in the renamer, data type declarations and other
| things
| > during type checking. A more ideal way, but also more time consuming,
| would
| > be to change this so that the renamer leaves evidence of the quasi-
| quotes in
| > the tree, and the type checker returns the entire tree type checked,
| rather
| > than just a subset. I think that ultimately this is the better
| approach, at
| > least for our purposes -- I'm not sure about other tools, but since
| this
| > would be a larger change that affects larger parts of the ghc pipeline
| I'm
| > not sure that I'll be able to do it.
|
| I needed something similar.  In particular, I built a custom code
| generator, but now I need a similar feature for extracting information
| from a Haskell file (for IDE features).
|
| Since I needed to modify one-shot compilation mode I couldn't use the
| GHC API.  For the IDE stuff I'm using Shake as the build manager, so
| that also needs a customized one-shot mode.  For my current
| implementation I just copied and adapted the necessary parts of
| HscMain, DriverPipeline, etc.  That's very messy, fragile and breaks
| on every GHC release so I'd really like to see the necessary features
| put into GHC.
|
| Do you have a working patch somewhere?
|
| _______________________________________________
| ghc-devs mailing list
| ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>
| http://www.haskell.org/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>
http://www.haskell.org/mailman/listinfo/ghc-devs




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20130708/6b87ccdd/attachment-0001.htm>

Reply | Threaded
Open this post in threaded view
|

Patch/feature proposal: "Source plugins"

Edsko de Vries-4
Ok, I have now finally got managed to compile and test my code against
ghc head. I have created a wiki page for the proposal

http://ghc.haskell.org/trac/ghc/wiki/FrontendPluginsProposal

and have attached patches against both 7.4.2 and against HEAD.
Hopefully this is now a good starting point for a design that works
for everybody.

-E

On Mon, Jul 8, 2013 at 10:46 PM, Simon Peyton-Jones
<simonpj at microsoft.com> wrote:

> Edsko, Luite, Thomas, and others
>
>
>
> Thanks.  I wonder whether you might write a Wiki page saying
>
> ?        exactly what you want to achieve, with examples, and
>
> ?        sketching how you hope to achieve it
>
>
>
> You may want to do a few rounds on that among yourselves; perhaps you have
> various different goals in mind that we can address at once.
>
>
>
> Selfishly, I don?t really want to review code until I have a very clear
> picture of what it is trying to achieve.   (And I?m rushing towards the POPL
> deadline anyway!)
>
>
>
> Simon
>
>
>
> From: Edsko de Vries [mailto:edskodevries at gmail.com]
> Sent: 08 July 2013 18:02
> To: Simon Peyton-Jones
> Cc: Luite Stegeman; Thomas Schilling; ghc-devs at haskell.org; Nicolas Frisby
>
>
> Subject: Re: Patch/feature proposal: "Source plugins"
>
>
>
> Ok, sorry all for the delay. Attached is a "frontend plugins" patch for ghc
> 7.4.2. It will not work for ghc HEAD because the structure of the compiler
> has changed in various places, but I cannot currently compile my code
> against HEAD. Making this available now because it's been far too long
> already :) Comments/suggestions/feedback welcome. Hopefully the patch is
> pretty self-explanatory; and it's only small.
>
>
>
> -E
>
>
>
> On Mon, Jul 1, 2013 at 9:32 AM, Edsko de Vries <edskodevries at gmail.com>
> wrote:
>
> Yes, I fully intend to create a ticket with a detailed description and a
> first patch, but I've been struggling with the latest HEAD, and specifically
> the fact that it now uses dynamic libraries for TH. I (think I am) stuck at
> a Cabal bug and I cannot currently build my code at all :-/ Duncan is
> looking into this though.
>
>
>
> Edsko
>
>
>
> On Fri, Jun 28, 2013 at 1:43 PM, Simon Peyton-Jones <simonpj at microsoft.com>
> wrote:
>
> I?m confused as to details here.
>
> ?         Edsko is doing something; Nick is doing something else (attached
> for completeness).
>
> ?         I can?t locate a Trac Wiki page that describes the design
>
>
>
> I?m more than happy to adopt patches that improve the plugin API, but you?ll
> have to lead me through it!
>
>
>
> No hurry, just when you are ready.
>
>
>
> Simon
>
>
>
> From: Edsko de Vries [mailto:edskodevries at gmail.com]
> Sent: 26 June 2013 09:21
> To: Luite Stegeman
> Cc: Simon Peyton-Jones; Thomas Schilling; ghc-devs at haskell.org
>
>
> Subject: Re: Patch/feature proposal: "Source plugins"
>
>
>
> Hi Luite,
>
>
>
> I was fully planning on a first version of the patch yesterday, but so far
> my efforts were thwarted by annoying problems with dynamic libraries (not --
> directly -- related to the patch at all). I will try again today :)
>
>
>
> Edsko
>
>
>
> On Wed, Jun 26, 2013 at 8:51 AM, Luite Stegeman <stegeman at gmail.com> wrote:
>
> Any news on this? I'd really like to have this in GHC 7.8.1 so that we can
> release a fully working GHCJS with GhcMake functionality based on it. I'd be
> happy to help write the patch.
>
>
>
> luite
>
>
>
>
>
> On Tue, Jun 11, 2013 at 3:21 PM, Simon Peyton-Jones <simonpj at microsoft.com>
> wrote:
>
> Guys,
>
> I'm not following the details here, but I'm open to suggestions (patches,
> even) that improve the GHC API.
>
> Simon
>
>
> | -----Original Message-----
> | From: ghc-devs-bounces at haskell.org [mailto:ghc-devs-bounces at haskell.org]
> | On Behalf Of Thomas Schilling
> | Sent: 11 June 2013 12:53
> | To: Edsko de Vries
> | Cc: ghc-devs at haskell.org
> | Subject: Re: Patch/feature proposal: "Source plugins"
> |
> | On 5 June 2013 13:51, Edsko de Vries <edskodevries at gmail.com> wrote:
> | > It is a little bit messy mostly because parts of the AST get lost
> | along the
> | > way: quasi-quotes in the renamer, data type declarations and other
> | things
> | > during type checking. A more ideal way, but also more time consuming,
> | would
> | > be to change this so that the renamer leaves evidence of the quasi-
> | quotes in
> | > the tree, and the type checker returns the entire tree type checked,
> | rather
> | > than just a subset. I think that ultimately this is the better
> | approach, at
> | > least for our purposes -- I'm not sure about other tools, but since
> | this
> | > would be a larger change that affects larger parts of the ghc pipeline
> | I'm
> | > not sure that I'll be able to do it.
> |
> | I needed something similar.  In particular, I built a custom code
> | generator, but now I need a similar feature for extracting information
> | from a Haskell file (for IDE features).
> |
> | Since I needed to modify one-shot compilation mode I couldn't use the
> | GHC API.  For the IDE stuff I'm using Shake as the build manager, so
> | that also needs a customized one-shot mode.  For my current
> | implementation I just copied and adapted the necessary parts of
> | HscMain, DriverPipeline, etc.  That's very messy, fragile and breaks
> | on every GHC release so I'd really like to see the necessary features
> | put into GHC.
> |
> | Do you have a working patch somewhere?
> |
> | _______________________________________________
> | ghc-devs mailing list
> | ghc-devs at haskell.org
> | http://www.haskell.org/mailman/listinfo/ghc-devs
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://www.haskell.org/mailman/listinfo/ghc-devs
>
>
>
>
>
>
>
>



Reply | Threaded
Open this post in threaded view
|

Patch/feature proposal: "Source plugins"

Thomas Schilling-2
I added my (very rough) take on this issue at: https://github.com/nominolo/ghc-phase-plugins

I wanted to implement this outside of GHC, so I had to copy HscMain and DriverPipeline.

The key part is the FileHooks datatype:

  type Hook a = a -> a

  data FileHooks = FileHooks
    { hookParse :: Hook (ModSummary -> HHsc HsParsedModule)
    , hookTypecheckRename :: Hook (HscEnv -> ModSummary -> HsParsedModule
                                          -> HHsc TcGblEnv)
    , hookDesugar :: Hook (HscEnv -> ModSummary -> TcGblEnv -> HHsc ModGuts)
    , hookOptimize :: Hook (ModGuts -> HHsc ModGuts)
    , hookWriteIface :: Hook (ModIface -> Bool -> ModSummary -> HHsc ())
    , hookCodeGen :: Hook (ModIface -> ModDetails -> CgGuts -> ModSummary
                                    -> HHsc (Maybe FilePath))
    , hookPostBackendPhase :: Hook (DynFlags -> HscSource -> HscTarget -> Phase)
    -- there are probably more things we want to add here
    }

The basic idea is simple: each significant component of the compiler can be customised by choosing the right hook implementation.  Each hook gets the default implementation as its first argument and can choose whatever it wants to do with it.  Where the current code would call, say, "hscOptimize guts", it now calls "hookOptimize hooks hscOptimize guts".

The messy bit is that it's up to the hook to establish the necessary (often undocumented) invariants for the result.  A cleaner API would probably require larger refactorings and a more careful design of the high-level pipeline API, though.

The HHsc type is currently just "IO (Messages, Maybe a)", but perhaps it's better to just export the Hsc monad currently used internally in HscMain.

Edsko's design, which seems to focus on type checker/quasi-quoting phases, could be included in this.  In fact it would be nice if we could actually untangle the renamed and type checker again. A few years back I discussed this with SPJ, and we concluded that it should be possible to do that.

There are two immediate use cases I have in mind for this design:

 1. IDE tools which run custom analyses in between compilation.  A means to omit certain phases is also useful.  (Right now, specifying -fno-code also skips desugaring which means you lose warnings regarding overlapping and missing pattern matches.)

 2. Custom backends.  There are now a number of projects that implement custom backends for GHC (ghcjs, Lambdachine).  Being able to reuse all of the front-end and plug in a custom code-generator is very useful.

 / Thomas


On 25 Jul 2013, at 17:12, Edsko de Vries <edskodevries at gmail.com> wrote:

> Ok, I have now finally got managed to compile and test my code against
> ghc head. I have created a wiki page for the proposal
>
> http://ghc.haskell.org/trac/ghc/wiki/FrontendPluginsProposal
>
> and have attached patches against both 7.4.2 and against HEAD.
> Hopefully this is now a good starting point for a design that works
> for everybody.
>
> -E
>
> On Mon, Jul 8, 2013 at 10:46 PM, Simon Peyton-Jones
> <simonpj at microsoft.com> wrote:
>> Edsko, Luite, Thomas, and others
>>
>>
>>
>> Thanks.  I wonder whether you might write a Wiki page saying
>>
>> ?        exactly what you want to achieve, with examples, and
>>
>> ?        sketching how you hope to achieve it
>>
>>
>>
>> You may want to do a few rounds on that among yourselves; perhaps you have
>> various different goals in mind that we can address at once.
>>
>>
>>
>> Selfishly, I don?t really want to review code until I have a very clear
>> picture of what it is trying to achieve.   (And I?m rushing towards the POPL
>> deadline anyway!)
>>
>>
>>
>> Simon
>>
>>
>>
>> From: Edsko de Vries [mailto:edskodevries at gmail.com]
>> Sent: 08 July 2013 18:02
>> To: Simon Peyton-Jones
>> Cc: Luite Stegeman; Thomas Schilling; ghc-devs at haskell.org; Nicolas Frisby
>>
>>
>> Subject: Re: Patch/feature proposal: "Source plugins"
>>
>>
>>
>> Ok, sorry all for the delay. Attached is a "frontend plugins" patch for ghc
>> 7.4.2. It will not work for ghc HEAD because the structure of the compiler
>> has changed in various places, but I cannot currently compile my code
>> against HEAD. Making this available now because it's been far too long
>> already :) Comments/suggestions/feedback welcome. Hopefully the patch is
>> pretty self-explanatory; and it's only small.
>>
>>
>>
>> -E
>>
>>
>>
>> On Mon, Jul 1, 2013 at 9:32 AM, Edsko de Vries <edskodevries at gmail.com>
>> wrote:
>>
>> Yes, I fully intend to create a ticket with a detailed description and a
>> first patch, but I've been struggling with the latest HEAD, and specifically
>> the fact that it now uses dynamic libraries for TH. I (think I am) stuck at
>> a Cabal bug and I cannot currently build my code at all :-/ Duncan is
>> looking into this though.
>>
>>
>>
>> Edsko
>>
>>
>>
>> On Fri, Jun 28, 2013 at 1:43 PM, Simon Peyton-Jones <simonpj at microsoft.com>
>> wrote:
>>
>> I?m confused as to details here.
>>
>> ?         Edsko is doing something; Nick is doing something else (attached
>> for completeness).
>>
>> ?         I can?t locate a Trac Wiki page that describes the design
>>
>>
>>
>> I?m more than happy to adopt patches that improve the plugin API, but you?ll
>> have to lead me through it!
>>
>>
>>
>> No hurry, just when you are ready.
>>
>>
>>
>> Simon
>>
>>
>>
>> From: Edsko de Vries [mailto:edskodevries at gmail.com]
>> Sent: 26 June 2013 09:21
>> To: Luite Stegeman
>> Cc: Simon Peyton-Jones; Thomas Schilling; ghc-devs at haskell.org
>>
>>
>> Subject: Re: Patch/feature proposal: "Source plugins"
>>
>>
>>
>> Hi Luite,
>>
>>
>>
>> I was fully planning on a first version of the patch yesterday, but so far
>> my efforts were thwarted by annoying problems with dynamic libraries (not --
>> directly -- related to the patch at all). I will try again today :)
>>
>>
>>
>> Edsko
>>
>>
>>
>> On Wed, Jun 26, 2013 at 8:51 AM, Luite Stegeman <stegeman at gmail.com> wrote:
>>
>> Any news on this? I'd really like to have this in GHC 7.8.1 so that we can
>> release a fully working GHCJS with GhcMake functionality based on it. I'd be
>> happy to help write the patch.
>>
>>
>>
>> luite
>>
>>
>>
>>
>>
>> On Tue, Jun 11, 2013 at 3:21 PM, Simon Peyton-Jones <simonpj at microsoft.com>
>> wrote:
>>
>> Guys,
>>
>> I'm not following the details here, but I'm open to suggestions (patches,
>> even) that improve the GHC API.
>>
>> Simon
>>
>>
>> | -----Original Message-----
>> | From: ghc-devs-bounces at haskell.org [mailto:ghc-devs-bounces at haskell.org]
>> | On Behalf Of Thomas Schilling
>> | Sent: 11 June 2013 12:53
>> | To: Edsko de Vries
>> | Cc: ghc-devs at haskell.org
>> | Subject: Re: Patch/feature proposal: "Source plugins"
>> |
>> | On 5 June 2013 13:51, Edsko de Vries <edskodevries at gmail.com> wrote:
>> | > It is a little bit messy mostly because parts of the AST get lost
>> | along the
>> | > way: quasi-quotes in the renamer, data type declarations and other
>> | things
>> | > during type checking. A more ideal way, but also more time consuming,
>> | would
>> | > be to change this so that the renamer leaves evidence of the quasi-
>> | quotes in
>> | > the tree, and the type checker returns the entire tree type checked,
>> | rather
>> | > than just a subset. I think that ultimately this is the better
>> | approach, at
>> | > least for our purposes -- I'm not sure about other tools, but since
>> | this
>> | > would be a larger change that affects larger parts of the ghc pipeline
>> | I'm
>> | > not sure that I'll be able to do it.
>> |
>> | I needed something similar.  In particular, I built a custom code
>> | generator, but now I need a similar feature for extracting information
>> | from a Haskell file (for IDE features).
>> |
>> | Since I needed to modify one-shot compilation mode I couldn't use the
>> | GHC API.  For the IDE stuff I'm using Shake as the build manager, so
>> | that also needs a customized one-shot mode.  For my current
>> | implementation I just copied and adapted the necessary parts of
>> | HscMain, DriverPipeline, etc.  That's very messy, fragile and breaks
>> | on every GHC release so I'd really like to see the necessary features
>> | put into GHC.
>> |
>> | Do you have a working patch somewhere?
>> |
>> | _______________________________________________
>> | ghc-devs mailing list
>> | ghc-devs at haskell.org
>> | http://www.haskell.org/mailman/listinfo/ghc-devs
>>
>> _______________________________________________
>> ghc-devs mailing list
>> ghc-devs at haskell.org
>> http://www.haskell.org/mailman/listinfo/ghc-devs
>>
>>
>>
>>
>>
>>
>>
>>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 203 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20130731/ae92173e/attachment.pgp>