HIE Files

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

HIE Files

Zubin Duggal
Hello,

I will be working on a GSOC project that will allow GHC to output a new .hie file to be written next to .hi files. It will contain information about the typechecked Haskell AST, allowing tooling(like haddocks --hyperlinked-source and haskell-ide-engine) to work without having to parse, rename and typecheck files all over again.

I have made a GHC wiki page containing more details here:

https://ghc.haskell.org/trac/ghc/wiki/HIEFiles

Looking forward to any comments and suggestions.

Thanks,
Zubin.

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

RE: HIE Files

GHC - devs mailing list

Interesting.

 

Please do keep the wiki page up to date so that it accurately describes the current design.  For example, I hope you’ll flesh out what a “simplified, source aware, annotated AST derived from the Renamed/Typechecked Source” really is.

 

Why not put the .hie-file info into the .hi file?  (Optionally, of course.)

 

What tools/libraries do you plan to produce to allow clients to read a .hie file and make send of the contents?

 

Simon

 

 

 

From: ghc-devs <[hidden email]> On Behalf Of Zubin Duggal
Sent: 14 May 2018 13:32
To: [hidden email]
Cc: Joachim Breitner <[hidden email]>; Gershom B <[hidden email]>
Subject: HIE Files

 

Hello,

I will be working on a GSOC project that will allow GHC to output a new .hie file to be written next to .hi files. It will contain information about the typechecked Haskell AST, allowing tooling(like haddocks --hyperlinked-source and haskell-ide-engine) to work without having to parse, rename and typecheck files all over again.

I have made a GHC wiki page containing more details here:

https://ghc.haskell.org/trac/ghc/wiki/HIEFiles

 

Looking forward to any comments and suggestions.

Thanks,

Zubin.


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: HIE Files

Peter Podlovics
Hi,

Sometimes, when working with a type-checked AST, it can be useful to know the types of subexpressions as well. A simple use case would be any kind of static analysis of the code. Currently, the type-checker discards all intermediate results after the type checking ends, so the AST only has info about nodes with names. Storing the types of subexpressions somewhere would be a great benefit for many tools. Do you plan on including these intermediate results in the HIE file?

Regards,
Peter

On Mon, May 14, 2018 at 3:30 PM, Simon Peyton Jones via ghc-devs <[hidden email]> wrote:

Interesting.

 

Please do keep the wiki page up to date so that it accurately describes the current design.  For example, I hope you’ll flesh out what a “simplified, source aware, annotated AST derived from the Renamed/Typechecked Source” really is.

 

Why not put the .hie-file info into the .hi file?  (Optionally, of course.)

 

What tools/libraries do you plan to produce to allow clients to read a .hie file and make send of the contents?

 

Simon

 

 

 

From: ghc-devs <[hidden email]> On Behalf Of Zubin Duggal
Sent: 14 May 2018 13:32
To: [hidden email]
Cc: Joachim Breitner <[hidden email]>; Gershom B <[hidden email]>
Subject: HIE Files

 

Hello,

I will be working on a GSOC project that will allow GHC to output a new .hie file to be written next to .hi files. It will contain information about the typechecked Haskell AST, allowing tooling(like haddocks --hyperlinked-source and haskell-ide-engine) to work without having to parse, rename and typecheck files all over again.

I have made a GHC wiki page containing more details here:

https://ghc.haskell.org/trac/ghc/wiki/HIEFiles

 

Looking forward to any comments and suggestions.

Thanks,

Zubin.


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs



_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: HIE Files

Gershom Bazerman
In reply to this post by GHC - devs mailing list
On Mon, May 14, 2018 at 9:30 AM, Simon Peyton Jones
<[hidden email]> wrote:

>
> Why not put the .hie-file info into the .hi file?  (Optionally, of course.)
>

Simon, I'm curious what benefits you think we might get from this?
(I'm one of the mentors on this GSoC project btw).

> What tools/libraries do you plan to produce to allow clients to read a .hie file and make send of the contents?

For GSoC as a proof of concept the idea is to teach haddock's
hyperlinked-source backend to use this information to add
type-annotation-on-hover to the colorized, hyperlinked, html source.

I think what is anticipated more broadly is that other tools like the
Haskell IDE Engine (which Zubin has contributed to in the past) will
also be able to make use of these files to provide ide and tooling
features in a more lightweight way than needing to directly interface
with the GHC API. (This by the way is one of the key benefits of
keeping the file separate from standard hi files -- it should be
parseable and consumable without needing to link in GHC).

-g
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

RE: HIE Files

GHC - devs mailing list
|  > Why not put the .hie-file info into the .hi file?  (Optionally, of
|  > course.)
|  >
|  
|  Simon, I'm curious what benefits you think we might get from this?
|  (I'm one of the mentors on this GSoC project btw).

Well, I've always thought that we should really put the .hi file into the .o file!  Having two files risks getting things out of sync, and three makes that worse.  The file is just a place to keep a blob of info.  What's the motivation for having two .hie as well as .hi?

|  
|  > What tools/libraries do you plan to produce to allow clients to read
|  a .hie file and make send of the contents?
|  
|  For GSoC as a proof of concept the idea is to teach haddock's
|  hyperlinked-source backend to use this information to add type-
|  annotation-on-hover to the colorized, hyperlinked, html source.

That's great.  But would it not be good to offer a library, with a well-defined API, that allows a client (including Haddock) to parse those .hie files into syntax trees or whatever?  You'll need to do that to allow the haddock thing you describe -- and it'd be much better to make the parser (and doubtless lots of utility function like finding things in the tree) available to any client not just haddock.

And that in turn raises the questions of WHAT syntax tree.  HsSyn?  Template Haskell?  Haskell-src-exts?  Or something new?   Shayan and Alan are busy parameterising HsSyn to make it non-GHC-specific, and directly usable for this kind of endeavour ("Trees that grow").  It'd be great to build on their work.

|  with the GHC API. (This by the way is one of the key benefits of
|  keeping the file separate from standard hi files -- it should be
|  parseable and consumable without needing to link in GHC).

Yes, not linking in GHC is a reasonable goal; but having two files and file formats is not a necessary consequence of that goal.  Nothing stops us making a library to parse .hi files -- indeed the entire iface/ directory in GHC is quite well separated for that precise purpose.

None of this is to criticise the plan.  I think it's a great idea to make more info more readily available to more tools.   I'm just poking at it a bit 😊.

Simon
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: HIE Files

Zubin Duggal
And that in turn raises the questions of WHAT syntax tree.  HsSyn?  Template Haskell?  Haskell-src-exts?  Or something new?   Shayan and Alan are busy parameterising HsSyn to make it non-GHC-specific, and directly usable for this kind of endeavour ("Trees that grow").  It'd be great to build on their work.

Mainly, we need information on every Token that appears in the original source. My plan is to further group Tokens into a simple rose-tree based on how they occur in HsSyn. We intentionally want to avoid capturing too much information so the format doesn't change much with changes to the GHC AST.

I've made a file describing roughly what the data structures involved should look like
The plan is to group the Tokens together in a tree in way similar to what structured-haskell-mode does. (The gifs in the following link might provide some idea)

For example, here is what structured-haskell-mode outputs for a small snippet of code: https://gist.github.com/wz1000/db42d4f533ba7d2345934906b312f743

We want something similar for the HIE AST, but grouped into a tree, where each node(roughly corresponding to HsSyn constructors) points to all the subnodes and tokens it spans over.

That's great.  But would it not be good to offer a library, with a well-defined API, that allows a client (including Haddock) to parse those .hie files into syntax trees or whatever?  You'll need to do that to allow the haddock thing you describe -- and it'd be much better to make the parser (and doubtless lots of utility function like finding things in the tree) available to any client not just haddock.

Yes, a library to consume these files is definitely something we need, and I believe it will grow out naturally as we work out the integration with haddock and haskell-ide-engine.

On 15 May 2018 at 14:12, Simon Peyton Jones <[hidden email]> wrote:
|  > Why not put the .hie-file info into the .hi file?  (Optionally, of
|  > course.)
|  >

|  Simon, I'm curious what benefits you think we might get from this?
|  (I'm one of the mentors on this GSoC project btw).

Well, I've always thought that we should really put the .hi file into the .o file!  Having two files risks getting things out of sync, and three makes that worse.  The file is just a place to keep a blob of info.  What's the motivation for having two .hie as well as .hi?


|  > What tools/libraries do you plan to produce to allow clients to read
|  a .hie file and make send of the contents?

|  For GSoC as a proof of concept the idea is to teach haddock's
|  hyperlinked-source backend to use this information to add type-
|  annotation-on-hover to the colorized, hyperlinked, html source.

That's great.  But would it not be good to offer a library, with a well-defined API, that allows a client (including Haddock) to parse those .hie files into syntax trees or whatever?  You'll need to do that to allow the haddock thing you describe -- and it'd be much better to make the parser (and doubtless lots of utility function like finding things in the tree) available to any client not just haddock.

And that in turn raises the questions of WHAT syntax tree.  HsSyn?  Template Haskell?  Haskell-src-exts?  Or something new?   Shayan and Alan are busy parameterising HsSyn to make it non-GHC-specific, and directly usable for this kind of endeavour ("Trees that grow").  It'd be great to build on their work.

|  with the GHC API. (This by the way is one of the key benefits of
|  keeping the file separate from standard hi files -- it should be
|  parseable and consumable without needing to link in GHC).

Yes, not linking in GHC is a reasonable goal; but having two files and file formats is not a necessary consequence of that goal.  Nothing stops us making a library to parse .hi files -- indeed the entire iface/ directory in GHC is quite well separated for that precise purpose.

None of this is to criticise the plan.  I think it's a great idea to make more info more readily available to more tools.   I'm just poking at it a bit 😊.

Simon


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

RE: HIE Files

GHC - devs mailing list

Mainly, we need information on every Token that appears in the original source.

 

Good idea.  Alan Zimmerman’s exact-print stuff has precisely that goal, I believe.   So it’d be worth talking to him; perhaps by working together you can make much more rapid progress.  Or not – but a conversation would be helpful in any case.  I’m very happy to see more attention and effort being devoted to this space.  Thank you!

 

Simon

 

From: Zubin Duggal <[hidden email]>
Sent: 15 May 2018 10:13
To: Simon Peyton Jones <[hidden email]>
Cc: Gershom B <[hidden email]>; [hidden email]; Joachim Breitner <[hidden email]>; Shayan Najd <[hidden email]>; Alan & Kim Zimmerman <[hidden email]>
Subject: Re: HIE Files

 

And that in turn raises the questions of WHAT syntax tree.  HsSyn?  Template Haskell?  Haskell-src-exts?  Or something new?   Shayan and Alan are busy parameterising HsSyn to make it non-GHC-specific, and directly usable for this kind of endeavour ("Trees that grow").  It'd be great to build on their work.

 

Mainly, we need information on every Token that appears in the original source. My plan is to further group Tokens into a simple rose-tree based on how they occur in HsSyn. We intentionally want to avoid capturing too much information so the format doesn't change much with changes to the GHC AST.

I've made a file describing roughly what the data structures involved should look like

The plan is to group the Tokens together in a tree in way similar to what structured-haskell-mode does. (The gifs in the following link might provide some idea)

For example, here is what structured-haskell-mode outputs for a small snippet of code: https://gist.github.com/wz1000/db42d4f533ba7d2345934906b312f743

We want something similar for the HIE AST, but grouped into a tree, where each node(roughly corresponding to HsSyn constructors) points to all the subnodes and tokens it spans over.

That's great.  But would it not be good to offer a library, with a well-defined API, that allows a client (including Haddock) to parse those .hie files into syntax trees or whatever?  You'll need to do that to allow the haddock thing you describe -- and it'd be much better to make the parser (and doubtless lots of utility function like finding things in the tree) available to any client not just haddock.

 

Yes, a library to consume these files is definitely something we need, and I believe it will grow out naturally as we work out the integration with haddock and haskell-ide-engine.


On 15 May 2018 at 14:12, Simon Peyton Jones <[hidden email]> wrote:

|  > Why not put the .hie-file info into the .hi file?  (Optionally, of
|  > course.)
|  >

|  Simon, I'm curious what benefits you think we might get from this?
|  (I'm one of the mentors on this GSoC project btw).

Well, I've always thought that we should really put the .hi file into the .o file!  Having two files risks getting things out of sync, and three makes that worse.  The file is just a place to keep a blob of info.  What's the motivation for having two .hie as well as .hi?


|  > What tools/libraries do you plan to produce to allow clients to read
|  a .hie file and make send of the contents?

|  For GSoC as a proof of concept the idea is to teach haddock's
|  hyperlinked-source backend to use this information to add type-
|  annotation-on-hover to the colorized, hyperlinked, html source.

That's great.  But would it not be good to offer a library, with a well-defined API, that allows a client (including Haddock) to parse those .hie files into syntax trees or whatever?  You'll need to do that to allow the haddock thing you describe -- and it'd be much better to make the parser (and doubtless lots of utility function like finding things in the tree) available to any client not just haddock.

And that in turn raises the questions of WHAT syntax tree.  HsSyn?  Template Haskell?  Haskell-src-exts?  Or something new?   Shayan and Alan are busy parameterising HsSyn to make it non-GHC-specific, and directly usable for this kind of endeavour ("Trees that grow").  It'd be great to build on their work.

|  with the GHC API. (This by the way is one of the key benefits of
|  keeping the file separate from standard hi files -- it should be
|  parseable and consumable without needing to link in GHC).

Yes, not linking in GHC is a reasonable goal; but having two files and file formats is not a necessary consequence of that goal.  Nothing stops us making a library to parse .hi files -- indeed the entire iface/ directory in GHC is quite well separated for that precise purpose.

None of this is to criticise the plan.  I think it's a great idea to make more info more readily available to more tools.   I'm just poking at it a bit 😊.

Simon

 


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs