Integrate GHC-API with Cabal

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Integrate GHC-API with Cabal

Félix Baylac
Dear cafe,

I am currently working on a project [1] which aims to index the code published on
Stackage in order to provide a "code example" database. Basically, for each
library,  I want to generate a real-world example corpus for each exported
symbol.

Before generating this database, I need to retrieve the exported symbol of a
package. So far, I have been gathering the dependencies and integrating them in
the GHC pkg database using an external stack call, I then parse the cabal file
 and load the exposed modules using the GHC API and gather the exported
 symbols*.

It works pretty reliably on the simple packages, however, I end up with some
missing dynamic flags for some more advanced packages (missing c includes, c
libraries, ASM flags, etc.). I then started to parse and load these missing
attributes to GHC until I stepped back for a moment and realized I was
re-implementing cabal build.

I then looked for a way to "hook" my GHC API code in cabal build. This would
allow cabal to both handle the dependencies gathering as well as setting up the
correct GHC dyn-flags. The only resource I found was [2]. It's really clever,
however, looks a bit hacky to me.

Is there a better way to perform this kind of  "hook"?


[1] https://git.alternativebit.fr/NinjaTrappeur/Exhs
[2] http://blog.ezyang.com/2017/02/how-to-integrate-ghc-api-programs-with-cabal/

* I am aware of the hoogle index generated by haddock. However, this index is
  missing when the haddock documentation of a package cannot be generated.
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate GHC-API with Cabal

Imants Cekusins
cabal-install package includes useful modules. With these modules it is possible to read the actual build info. These modules could be added to the Cabal API.

What are the reasons cabal-install can not be added as a project dependency?

Can some modules be moved from cabal-install to Cabal library?



On 25 July 2018 at 11:33, Félix Baylac <[hidden email]> wrote:
Dear cafe,

I am currently working on a project [1] which aims to index the code published on
Stackage in order to provide a "code example" database. Basically, for each
library,  I want to generate a real-world example corpus for each exported
symbol.

Before generating this database, I need to retrieve the exported symbol of a
package. So far, I have been gathering the dependencies and integrating them in
the GHC pkg database using an external stack call, I then parse the cabal file
 and load the exposed modules using the GHC API and gather the exported
 symbols*.

It works pretty reliably on the simple packages, however, I end up with some
missing dynamic flags for some more advanced packages (missing c includes, c
libraries, ASM flags, etc.). I then started to parse and load these missing
attributes to GHC until I stepped back for a moment and realized I was
re-implementing cabal build.

I then looked for a way to "hook" my GHC API code in cabal build. This would
allow cabal to both handle the dependencies gathering as well as setting up the
correct GHC dyn-flags. The only resource I found was [2]. It's really clever,
however, looks a bit hacky to me.

Is there a better way to perform this kind of  "hook"?


[1] https://git.alternativebit.fr/NinjaTrappeur/Exhs
[2] http://blog.ezyang.com/2017/02/how-to-integrate-ghc-api-programs-with-cabal/

* I am aware of the hoogle index generated by haddock. However, this index is
  missing when the haddock documentation of a package cannot be generated.
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.


_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate GHC-API with Cabal

Matthew Pickering
In reply to this post by Félix Baylac
You can implement your tool as a source plugin and then it will work with any method of building. 

On Wed, 25 Jul 2018, 10:34 Félix Baylac, <[hidden email]> wrote:
Dear cafe,

I am currently working on a project [1] which aims to index the code published on
Stackage in order to provide a "code example" database. Basically, for each
library,  I want to generate a real-world example corpus for each exported
symbol.

Before generating this database, I need to retrieve the exported symbol of a
package. So far, I have been gathering the dependencies and integrating them in
the GHC pkg database using an external stack call, I then parse the cabal file
 and load the exposed modules using the GHC API and gather the exported
 symbols*.

It works pretty reliably on the simple packages, however, I end up with some
missing dynamic flags for some more advanced packages (missing c includes, c
libraries, ASM flags, etc.). I then started to parse and load these missing
attributes to GHC until I stepped back for a moment and realized I was
re-implementing cabal build.

I then looked for a way to "hook" my GHC API code in cabal build. This would
allow cabal to both handle the dependencies gathering as well as setting up the
correct GHC dyn-flags. The only resource I found was [2]. It's really clever,
however, looks a bit hacky to me.

Is there a better way to perform this kind of  "hook"?


[1] https://git.alternativebit.fr/NinjaTrappeur/Exhs
[2] http://blog.ezyang.com/2017/02/how-to-integrate-ghc-api-programs-with-cabal/

* I am aware of the hoogle index generated by haddock. However, this index is
  missing when the haddock documentation of a package cannot be generated.
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate GHC-API with Cabal

Daniel Gröber
In reply to this post by Félix Baylac
Hi,

On Wed, Jul 25, 2018 at 10:33:45AM +0200, Félix Baylac wrote:

> Before generating this database, I need to retrieve the exported symbol of a
> package. So far, I have been gathering the dependencies and integrating them in
> the GHC pkg database using an external stack call, I then parse the cabal file
>  and load the exposed modules using the GHC API and gather the exported
>  symbols*.
>
> It works pretty reliably on the simple packages, however, I end up with some
> missing dynamic flags for some more advanced packages (missing c includes, c
> libraries, ASM flags, etc.). I then started to parse and load these missing
> attributes to GHC until I stepped back for a moment and realized I was
> re-implementing cabal build.

I think you might be interrested in one of my packages,
[cabal-helper](https://hackage.haskell.org/package/cabal-helper). It's
geard more towards editor tooling but I think it should work for your
use case too.

Basically you'd have to do `cabal install --only-dependencies && cabal
configure && cabal build` for each package you want to inspect and
then cabal-helpers API boils down to giving you a list of GHC
flags/module names you need to pass into the GHC API to make it work.

(The `cabal build` step is needed to run preprocessors like hsc2hs and
compile C dependencies, but once cabal has taken care of that you can
usually just load the package without problems.)

Here's some example code you can use to get started (from our tests):
https://github.com/DanielG/cabal-helper/blob/7fc3b9d468a4a2997a7fed63e378567f2ef3a401/tests/GhcSession.hs#L96

Quoting from the above, the main part is:

```
let qe = mkQueryEnv dir (dir </> "dist")
cs <- runQuery qe $ components $ (,,) <$> entrypoints <.> ghcOptions
forM cs $ \(ep, opts, cn) -> do
  ...
```

where `dir` is the directory containing the cabal file of the project
you're trying to load into GHC. The do block gets called for each
component in the cabal project with 1) `ep` a datatype containing
(among other things) the list of modules you need to pass to GHC, 2)
the GHC flags `opts` and 3) `ch` the component name.

API docs are over here:
https://hackage.haskell.org/package/cabal-helper-0.8.0.2/docs/Distribution-Helper.html.
They're probably not all that much use if you're not familliar with
Cabal's inner workings though so feel free to ask me questions.

--Daniel
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Integrate GHC-API with Cabal

Daniel Gröber
In reply to this post by Félix Baylac
Hi,

On Wed, Jul 25, 2018 at 08:30:48PM +0200, Félix Baylac wrote:

> Thanks a lot for this really helpful message.
>
> > I think you might be interrested in one of my packages,
> > [cabal-helper](https://hackage.haskell.org/package/cabal-helper). It's
> > geard more towards editor tooling but I think it should work for your
> > use case too.
>
> Indeed, looks like what I'm looking for.
>
> > Basically you'd have to do `cabal install --only-dependencies && cabal
> > configure && cabal build` for each package you want to inspect and
> > then cabal-helpers API boils down to giving you a list of GHC
> > flags/module names you need to pass into the GHC API to make it work.
> >
> > (The `cabal build` step is needed to run preprocessors like hsc2hs and
> > compile C dependencies, but once cabal has taken care of that you can
> > usually just load the package without problems.)
>
> I had to make some modifications and jump in some nix hoops to use
> cabal-install instead of stack to build the packages, but indeed, it does work
> like a charm.
Ah, I forgot you said you were using Stack. You can use it too if you
prefer. You just have to use `stack path --dist-dir` (run in the
project directory) to get the path to the dist-dir after running
`stack build` instead of using the hardcoded `$PWD/dist` value. There
have been some reports of issues with stack using weird combinations
of Cabal-library versions though which so it might need some tweaking.

> This is perfect, I just tried your library following this example on the text
> package and it seems to do the job. Thanks a lot, fixing this problem without
> re-inventing the wheel is a huge relief.
>
> Do you mind if I write a small article on this integration.

No not at all, send me a link when you're done.

> It'll probably prevent some other people from using dodgy hacks.

Well I'm not sure I would call cabal-helper not a dodgy hack but at
least it's a (sort of) well maintained hack :)

--Daniel

PS: Looks like I messed up the list-reply on my original message so I
resent it and I'm CC'ing haskell-cafe on this one. Sorry about that.

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Integrate GHC-API with Cabal

Félix Baylac
In reply to this post by Imants Cekusins
Hi,

Thanks for the reply, I somehow missed your message.

> cabal-install package includes useful modules. With these modules it is
> possible to read the actual build info. These modules could be added to the
> Cabal API.

As mentionned in another reply, there's even a library[1] retrieving this kind of
informations.
 
> What are the reasons cabal-install can not be added as a project dependency?

I can depend on it.
 
> Can some modules be moved from cabal-install to Cabal library?

This is a really good question. Seems like exposing this kind of information
could benefit some other people.

I am not fimiliar with Cabal internals and have no idea whether this is
possible nor the amount of work needed for this. I am trying to avoid digging
yet another rabbit whole for now.

[1]: https://hackage.haskell.org/package/cabal-helper

> On 25 July 2018 at 11:33, Félix Baylac <[hidden email]> wrote:
>
> > Dear cafe,
> >
> > I am currently working on a project [1] which aims to index the code
> > published on
> > Stackage in order to provide a "code example" database. Basically, for each
> > library,  I want to generate a real-world example corpus for each exported
> > symbol.
> >
> > Before generating this database, I need to retrieve the exported symbol of
> > a
> > package. So far, I have been gathering the dependencies and integrating
> > them in
> > the GHC pkg database using an external stack call, I then parse the cabal
> > file
> >  and load the exposed modules using the GHC API and gather the exported
> >  symbols*.
> >
> > It works pretty reliably on the simple packages, however, I end up with
> > some
> > missing dynamic flags for some more advanced packages (missing c includes,
> > c
> > libraries, ASM flags, etc.). I then started to parse and load these
> > missing
> > attributes to GHC until I stepped back for a moment and realized I was
> > re-implementing cabal build.
> >
> > I then looked for a way to "hook" my GHC API code in cabal build. This
> > would
> > allow cabal to both handle the dependencies gathering as well as setting
> > up the
> > correct GHC dyn-flags. The only resource I found was [2]. It's really
> > clever,
> > however, looks a bit hacky to me.
> >
> > Is there a better way to perform this kind of  "hook"?
> >
> >
> > [1] https://git.alternativebit.fr/NinjaTrappeur/Exhs
> > [2] http://blog.ezyang.com/2017/02/how-to-integrate-ghc-api-
> > programs-with-cabal/
> >
> > * I am aware of the hoogle index generated by haddock. However, this index
> > is
> >   missing when the haddock documentation of a package cannot be generated.
> > _______________________________________________
> > Haskell-Cafe mailing list
> > To (un)subscribe, modify options or view archives go to:
> > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
> > Only members subscribed via the mailman list are allowed to post.
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.