Extracting representation from GHC

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Extracting representation from GHC

Németh Boldizsár
Dear GHC Developers,

I would like to ask your opinion on my ideas to make it easier to use
development tools with GHC.

In the past when working on a Haskell refactoring tool I relied on using
the GHC API for parsing and type checking Haskell sources. I extracted
the representation and performed analysis and transformation on it as it
was needed. However using the refactorer would be easier if it could
work with build tools.

To do this, my idea is to instruct GHC with a compilation flag to give
out its internal representation of the source code. Most build tools let
the user to configure the GHC flags so the refactoring tool would be
usable in any build infrastructure. I'm thinking of using the
pre-existing plugin architecture and adding two new fields to the Plugin
datastructure. One would be called with the parsed representation
(HsParsedModule) when parsing succeeds, another with the result of the
type checking (TcGblEnv) when type checking is finished.

What do you think about this solution?

Boldizsár

(ps: My first idea was using frontend plugins, but I could not access
the representation from there and --frontend flag changed GHC
compilation mode.)

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Extracting representation from GHC

Matthew Pickering
I have too wanted this in the past and made a post to a similar effect
on the mailing list 6 months ago.

https://mail.haskell.org/pipermail/ghc-devs/2017-July/014427.html

It references this proposal for a similar feature.

https://ghc.haskell.org/trac/ghc/wiki/FrontendPluginsProposal#no1

If you would be glad to implement it then feel free to add me as a reviewer.

Cheers,

Matt

On Fri, Jan 19, 2018 at 9:35 AM, Németh Boldizsár <[hidden email]> wrote:

> Dear GHC Developers,
>
> I would like to ask your opinion on my ideas to make it easier to use
> development tools with GHC.
>
> In the past when working on a Haskell refactoring tool I relied on using the
> GHC API for parsing and type checking Haskell sources. I extracted the
> representation and performed analysis and transformation on it as it was
> needed. However using the refactorer would be easier if it could work with
> build tools.
>
> To do this, my idea is to instruct GHC with a compilation flag to give out
> its internal representation of the source code. Most build tools let the
> user to configure the GHC flags so the refactoring tool would be usable in
> any build infrastructure. I'm thinking of using the pre-existing plugin
> architecture and adding two new fields to the Plugin datastructure. One
> would be called with the parsed representation (HsParsedModule) when parsing
> succeeds, another with the result of the type checking (TcGblEnv) when type
> checking is finished.
>
> What do you think about this solution?
>
> Boldizsár
>
> (ps: My first idea was using frontend plugins, but I could not access the
> representation from there and --frontend flag changed GHC compilation mode.)
>
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Extracting representation from GHC

Shao Cheng

Hi,

IIRC you can already use hscFrontendHook in the DynFlags hooks to retrieve TcGblEnv, and with a little bit of work, also HsParsedModule.

Regards,
Shao Cheng


On Fri, Jan 19, 2018, 5:41 PM Matthew Pickering <[hidden email]> wrote:
I have too wanted this in the past and made a post to a similar effect
on the mailing list 6 months ago.

https://mail.haskell.org/pipermail/ghc-devs/2017-July/014427.html

It references this proposal for a similar feature.

https://ghc.haskell.org/trac/ghc/wiki/FrontendPluginsProposal#no1

If you would be glad to implement it then feel free to add me as a reviewer.

Cheers,

Matt

On Fri, Jan 19, 2018 at 9:35 AM, Németh Boldizsár <[hidden email]> wrote:
> Dear GHC Developers,
>
> I would like to ask your opinion on my ideas to make it easier to use
> development tools with GHC.
>
> In the past when working on a Haskell refactoring tool I relied on using the
> GHC API for parsing and type checking Haskell sources. I extracted the
> representation and performed analysis and transformation on it as it was
> needed. However using the refactorer would be easier if it could work with
> build tools.
>
> To do this, my idea is to instruct GHC with a compilation flag to give out
> its internal representation of the source code. Most build tools let the
> user to configure the GHC flags so the refactoring tool would be usable in
> any build infrastructure. I'm thinking of using the pre-existing plugin
> architecture and adding two new fields to the Plugin datastructure. One
> would be called with the parsed representation (HsParsedModule) when parsing
> succeeds, another with the result of the type checking (TcGblEnv) when type
> checking is finished.
>
> What do you think about this solution?
>
> Boldizsár
>
> (ps: My first idea was using frontend plugins, but I could not access the
> representation from there and --frontend flag changed GHC compilation mode.)
>
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

RE: Extracting representation from GHC

GHC - devs mailing list
In reply to this post by Németh Boldizsár
|  To do this, my idea is to instruct GHC with a compilation flag to give
|  out its internal representation of the source code.

Why can't you just use GHC as a library, and ask it to parse and typecheck the module and then look at what it gives you.

Others are more used to the GHC API than me, though.

S

|  -----Original Message-----
|  From: ghc-devs [mailto:[hidden email]] On Behalf Of
|  Németh Boldizsár
|  Sent: 19 January 2018 09:35
|  To: [hidden email]
|  Subject: Extracting representation from GHC
|  
|  Dear GHC Developers,
|  
|  I would like to ask your opinion on my ideas to make it easier to use
|  development tools with GHC.
|  
|  In the past when working on a Haskell refactoring tool I relied on
|  using the GHC API for parsing and type checking Haskell sources. I
|  extracted the representation and performed analysis and transformation
|  on it as it was needed. However using the refactorer would be easier
|  if it could work with build tools.
|  
|  To do this, my idea is to instruct GHC with a compilation flag to give
|  out its internal representation of the source code. Most build tools
|  let the user to configure the GHC flags so the refactoring tool would
|  be usable in any build infrastructure. I'm thinking of using the pre-
|  existing plugin architecture and adding two new fields to the Plugin
|  datastructure. One would be called with the parsed representation
|  (HsParsedModule) when parsing succeeds, another with the result of the
|  type checking (TcGblEnv) when type checking is finished.
|  
|  What do you think about this solution?
|  
|  Boldizsár
|  
|  (ps: My first idea was using frontend plugins, but I could not access
|  the representation from there and --frontend flag changed GHC
|  compilation mode.)
|  
|  _______________________________________________
|  ghc-devs mailing list
|  [hidden email]
|  https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.h
|  askell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-
|  devs&data=02%7C01%7Csimonpj%40microsoft.com%7C9d78fd2d16994ade4d9008d5
|  5f2007c3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6365195135360471
|  87&sdata=voUEz%2BKTp0p3CtwP1Hx6xA3cXN0qoYONLPd9T7xRve8%3D&reserved=0
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Extracting representation from GHC

Robin Palotai
See some additions inline.
BR
Robin

2018-01-19 18:27 GMT+01:00 Simon Peyton Jones via ghc-devs <[hidden email]>:
|  To do this, my idea is to instruct GHC with a compilation flag to give
|  out its internal representation of the source code.

Why can't you just use GHC as a library, and ask it to parse and typecheck the module and then look at what it gives you.

Last time I checked (GHC 8.2, for haskell-indexer), using the library is not equivalent to using GHC's Main. GHC's Main does tremendous amount of magic with flag parsing and state setup, and doesn't expose all the functionality for libraries to do the same.

AFAIR we saw two possible ways to get the AST out from a complicated setup (FFI, objects, packages, ...):
1) invoke GHC and use Frontend plugin (but Frontend plugin is/was more limited at the time - the gist in the below trac entry mentions that even the Frontend plugin didn't do everything Main does).
2) Refactor GHC Main and expose all the functionality to GHC API. 

I filed https://ghc.haskell.org/trac/ghc/ticket/14018 a while ago that's slightly related.
 
By the way, you can click around http://stuff.codereview.me/ghc/#ghc/ghc/Main.hs?corpus=ghc-8.2.1-rc2&signature in the 'main' function to see all the magic.

Others are more used to the GHC API than me, though.

S

|  -----Original Message-----
|  From: ghc-devs [mailto:[hidden email]] On Behalf Of
|  Németh Boldizsár
|  Sent: 19 January 2018 09:35
|  To: [hidden email]
|  Subject: Extracting representation from GHC
|
|  Dear GHC Developers,
|
|  I would like to ask your opinion on my ideas to make it easier to use
|  development tools with GHC.
|
|  In the past when working on a Haskell refactoring tool I relied on
|  using the GHC API for parsing and type checking Haskell sources. I
|  extracted the representation and performed analysis and transformation
|  on it as it was needed. However using the refactorer would be easier
|  if it could work with build tools.
|
|  To do this, my idea is to instruct GHC with a compilation flag to give
|  out its internal representation of the source code. Most build tools
|  let the user to configure the GHC flags so the refactoring tool would
|  be usable in any build infrastructure. I'm thinking of using the pre-
|  existing plugin architecture and adding two new fields to the Plugin
|  datastructure. One would be called with the parsed representation
|  (HsParsedModule) when parsing succeeds, another with the result of the
|  type checking (TcGblEnv) when type checking is finished.
|
|  What do you think about this solution?
|
|  Boldizsár
|
|  (ps: My first idea was using frontend plugins, but I could not access
|  the representation from there and --frontend flag changed GHC
|  compilation mode.)
|
|  _______________________________________________
|  ghc-devs mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.h
askell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-
|  devs&data=02%7C01%7Csimonpj%40microsoft.com%7C9d78fd2d16994ade4d9008d5
|  5f2007c3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6365195135360471
|  87&sdata=voUEz%2BKTp0p3CtwP1Hx6xA3cXN0qoYONLPd9T7xRve8%3D&reserved=0
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Extracting representation from GHC

Robin Palotai
See also https://github.com/google/haskell-indexer/blob/master/haskell-indexer-backend-ghc/src/Language/Haskell/Indexer/Backend/GhcApiSupport.hs for an as-complete GHC API based setup as I could get. The comments indicate possible deficiencies.


In practice one can run AST extraction with HscNoLink + HscInterpreted for most targets, but for hairy ones (FFI invoked from TemplateHaskell in certain ways, for examples) that will fail.

2018-01-19 19:46 GMT+01:00 Robin Palotai <[hidden email]>:
See some additions inline.
BR
Robin

2018-01-19 18:27 GMT+01:00 Simon Peyton Jones via ghc-devs <[hidden email]>:
|  To do this, my idea is to instruct GHC with a compilation flag to give
|  out its internal representation of the source code.

Why can't you just use GHC as a library, and ask it to parse and typecheck the module and then look at what it gives you.

Last time I checked (GHC 8.2, for haskell-indexer), using the library is not equivalent to using GHC's Main. GHC's Main does tremendous amount of magic with flag parsing and state setup, and doesn't expose all the functionality for libraries to do the same.

AFAIR we saw two possible ways to get the AST out from a complicated setup (FFI, objects, packages, ...):
1) invoke GHC and use Frontend plugin (but Frontend plugin is/was more limited at the time - the gist in the below trac entry mentions that even the Frontend plugin didn't do everything Main does).
2) Refactor GHC Main and expose all the functionality to GHC API. 

I filed https://ghc.haskell.org/trac/ghc/ticket/14018 a while ago that's slightly related.
 
By the way, you can click around http://stuff.codereview.me/ghc/#ghc/ghc/Main.hs?corpus=ghc-8.2.1-rc2&signature in the 'main' function to see all the magic.

Others are more used to the GHC API than me, though.

S

|  -----Original Message-----
|  From: ghc-devs [mailto:[hidden email]] On Behalf Of
|  Németh Boldizsár
|  Sent: 19 January 2018 09:35
|  To: [hidden email]
|  Subject: Extracting representation from GHC
|
|  Dear GHC Developers,
|
|  I would like to ask your opinion on my ideas to make it easier to use
|  development tools with GHC.
|
|  In the past when working on a Haskell refactoring tool I relied on
|  using the GHC API for parsing and type checking Haskell sources. I
|  extracted the representation and performed analysis and transformation
|  on it as it was needed. However using the refactorer would be easier
|  if it could work with build tools.
|
|  To do this, my idea is to instruct GHC with a compilation flag to give
|  out its internal representation of the source code. Most build tools
|  let the user to configure the GHC flags so the refactoring tool would
|  be usable in any build infrastructure. I'm thinking of using the pre-
|  existing plugin architecture and adding two new fields to the Plugin
|  datastructure. One would be called with the parsed representation
|  (HsParsedModule) when parsing succeeds, another with the result of the
|  type checking (TcGblEnv) when type checking is finished.
|
|  What do you think about this solution?
|
|  Boldizsár
|
|  (ps: My first idea was using frontend plugins, but I could not access
|  the representation from there and --frontend flag changed GHC
|  compilation mode.)
|
|  _______________________________________________
|  ghc-devs mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.h
askell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-
|  devs&data=02%7C01%7Csimonpj%40microsoft.com%7C9d78fd2d16994ade4d9008d5
|  5f2007c3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636519<a href="tel:(513)%20536-0471" value="+15135360471" target="_blank">5135360471
|  87&sdata=voUEz%2BKTp0p3CtwP1Hx6xA3cXN0qoYONLPd9T7xRve8%3D&reserved=0
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs



_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs