TTG hsSyn for Batch and Interactive Parsing

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

TTG hsSyn for Batch and Interactive Parsing

Alan & Kim Zimmerman
I want to be able to run the GHC parser in one of two modes, batch which functions as now, and interactive which will (eventually) be incremental.

In addition, the hsSyn AST for each will have different TTG[1] annotations, so that it can better support IDE usage.

I think this can be done by changing the types in HsExtension to introduce a 'Process'  type as follows

data Pass = Parsed Process | Renamed | Typechecked
         deriving (Data)

data Process = Batch | Interactive
  deriving (Show, Data)

We then rename the pass synonyms so that batch is the default

type GhcPs   = GhcPass ('Parsed 'Batch)
type GhcPsI  = GhcPass ('Parsed 'Interactive)

I have attached a simple proof of concept file, which emulates parsing and renaming.

Is this an appropriate approach to take?

Alan


[1] Trees That Grow https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

ParserTypes.hs (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: TTG hsSyn for Batch and Interactive Parsing

GHC - devs mailing list

At first blush, “running the parser in two modes” and “changing the Pass” type don’t match up in my mind.  One seems quite local (how to run the parser).  The other seems more pervasive.

 

Can you say more about your proposed design, perhaps even on a wiki page?

 

Simon

 

From: ghc-devs <[hidden email]> On Behalf Of Alan & Kim Zimmerman
Sent: 07 May 2018 16:17
To: ghc-devs <[hidden email]>
Subject: TTG hsSyn for Batch and Interactive Parsing

 

I want to be able to run the GHC parser in one of two modes, batch which functions as now, and interactive which will (eventually) be incremental.

In addition, the hsSyn AST for each will have different TTG[1] annotations, so that it can better support IDE usage.

I think this can be done by changing the types in HsExtension to introduce a 'Process'  type as follows

data Pass = Parsed Process | Renamed | Typechecked
         deriving (Data)

data Process = Batch | Interactive
  deriving (Show, Data)

We then rename the pass synonyms so that batch is the default

type GhcPs   = GhcPass ('Parsed 'Batch)
type GhcPsI  = GhcPass ('Parsed 'Interactive)

I have attached a simple proof of concept file, which emulates parsing and renaming.

Is this an appropriate approach to take?

Alan

 

[1] Trees That Grow https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: TTG hsSyn for Batch and Interactive Parsing

Alan & Kim Zimmerman

On 8 May 2018 at 10:54, Simon Peyton Jones <[hidden email]> wrote:

At first blush, “running the parser in two modes” and “changing the Pass” type don’t match up in my mind.  One seems quite local (how to run the parser).  The other seems more pervasive.

 

Can you say more about your proposed design, perhaps even on a wiki page?

 

Simon

 

From: ghc-devs <[hidden email]> On Behalf Of Alan & Kim Zimmerman
Sent: 07 May 2018 16:17
To: ghc-devs <[hidden email]>
Subject: TTG hsSyn for Batch and Interactive Parsing

 

I want to be able to run the GHC parser in one of two modes, batch which functions as now, and interactive which will (eventually) be incremental.

In addition, the hsSyn AST for each will have different TTG[1] annotations, so that it can better support IDE usage.

I think this can be done by changing the types in HsExtension to introduce a 'Process'  type as follows

data Pass = Parsed Process | Renamed | Typechecked
         deriving (Data)

data Process = Batch | Interactive
  deriving (Show, Data)

We then rename the pass synonyms so that batch is the default

type GhcPs   = GhcPass ('Parsed 'Batch)
type GhcPsI  = GhcPass ('Parsed 'Interactive)

I have attached a simple proof of concept file, which emulates parsing and renaming.

Is this an appropriate approach to take?

Alan

 

[1] Trees That Grow https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow



_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

RE: TTG hsSyn for Batch and Interactive Parsing

GHC - devs mailing list

Thanks. 

 

I am absolutely behind this objective:

I propose to move the API Annotations to where they belong, inside the AST.

Indeed I thought that was always part of the TTG plan.

 

But I don’t understand what this has to do with interactive vs batch parsing.  Why don’t you unconditionally retain API-annotation info?  How would GhcPs be used differently to GhcPsI?

 

You might want to answer by clarifying on the wiki page, so that it is a persistent record of the design debugged in dialogue by email.

 

Simon

 

From: Alan & Kim Zimmerman <[hidden email]>
Sent: 08 May 2018 21:02
To: Simon Peyton Jones <[hidden email]>
Cc: ghc-devs <[hidden email]>
Subject: Re: TTG hsSyn for Batch and Interactive Parsing

 

 

On 8 May 2018 at 10:54, Simon Peyton Jones <[hidden email]> wrote:

At first blush, “running the parser in two modes” and “changing the Pass” type don’t match up in my mind.  One seems quite local (how to run the parser).  The other seems more pervasive.

 

Can you say more about your proposed design, perhaps even on a wiki page?

 

Simon

 

From: ghc-devs <[hidden email]> On Behalf Of Alan & Kim Zimmerman
Sent: 07 May 2018 16:17
To: ghc-devs <[hidden email]>
Subject: TTG hsSyn for Batch and Interactive Parsing

 

I want to be able to run the GHC parser in one of two modes, batch which functions as now, and interactive which will (eventually) be incremental.

In addition, the hsSyn AST for each will have different TTG[1] annotations, so that it can better support IDE usage.

I think this can be done by changing the types in HsExtension to introduce a 'Process'  type as follows

data Pass = Parsed Process | Renamed | Typechecked
         deriving (Data)

data Process = Batch | Interactive
  deriving (Show, Data)

We then rename the pass synonyms so that batch is the default

type GhcPs   = GhcPass ('Parsed 'Batch)
type GhcPsI  = GhcPass ('Parsed 'Interactive)

I have attached a simple proof of concept file, which emulates parsing and renaming.

Is this an appropriate approach to take?

Alan

 

[1] Trees That Grow https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow

 


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: TTG hsSyn for Batch and Interactive Parsing

Alan & Kim Zimmerman
I have updated the Wiki.

On 9 May 2018 at 10:15, Simon Peyton Jones <[hidden email]> wrote:

Thanks. 

 

I am absolutely behind this objective:

I propose to move the API Annotations to where they belong, inside the AST.

Indeed I thought that was always part of the TTG plan.

 

But I don’t understand what this has to do with interactive vs batch parsing.  Why don’t you unconditionally retain API-annotation info?  How would GhcPs be used differently to GhcPsI?

 

You might want to answer by clarifying on the wiki page, so that it is a persistent record of the design debugged in dialogue by email.

 

Simon

 

From: Alan & Kim Zimmerman <[hidden email]>
Sent: 08 May 2018 21:02
To: Simon Peyton Jones <[hidden email]>
Cc: ghc-devs <[hidden email]>
Subject: Re: TTG hsSyn for Batch and Interactive Parsing

 

 

On 8 May 2018 at 10:54, Simon Peyton Jones <[hidden email]> wrote:

At first blush, “running the parser in two modes” and “changing the Pass” type don’t match up in my mind.  One seems quite local (how to run the parser).  The other seems more pervasive.

 

Can you say more about your proposed design, perhaps even on a wiki page?

 

Simon

 

From: ghc-devs <[hidden email]> On Behalf Of Alan & Kim Zimmerman
Sent: 07 May 2018 16:17
To: ghc-devs <[hidden email]>
Subject: TTG hsSyn for Batch and Interactive Parsing

 

I want to be able to run the GHC parser in one of two modes, batch which functions as now, and interactive which will (eventually) be incremental.

In addition, the hsSyn AST for each will have different TTG[1] annotations, so that it can better support IDE usage.

I think this can be done by changing the types in HsExtension to introduce a 'Process'  type as follows

data Pass = Parsed Process | Renamed | Typechecked
         deriving (Data)

data Process = Batch | Interactive
  deriving (Show, Data)

We then rename the pass synonyms so that batch is the default

type GhcPs   = GhcPass ('Parsed 'Batch)
type GhcPsI  = GhcPass ('Parsed 'Interactive)

I have attached a simple proof of concept file, which emulates parsing and renaming.

Is this an appropriate approach to take?

Alan

 

[1] Trees That Grow https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow

 



_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

RE: TTG hsSyn for Batch and Interactive Parsing

GHC - devs mailing list

Thanks.  But I still don’t see the connection with “interactive”.  Why should maintaining API annotations have anything to do with interactivity?

 

Maybe
  data Process = WithApiAnnotations | WithoutApiAnnotations
 
I understand you want two different variants of the syntax tree, but I don’t understand what functions might produce or consume them.
 
In particular does the parser produce (HsSyn (GhcPs WithApiAnnotations)) or without?
 
Simon
 
 

 

 

From: Alan & Kim Zimmerman <[hidden email]>
Sent: 09 May 2018 21:12
To: Simon Peyton Jones <[hidden email]>
Cc: ghc-devs <[hidden email]>
Subject: Re: TTG hsSyn for Batch and Interactive Parsing

 

I have updated the Wiki.

 

On 9 May 2018 at 10:15, Simon Peyton Jones <[hidden email]> wrote:

Thanks. 

 

I am absolutely behind this objective:

I propose to move the API Annotations to where they belong, inside the AST.

Indeed I thought that was always part of the TTG plan.

 

But I don’t understand what this has to do with interactive vs batch parsing.  Why don’t you unconditionally retain API-annotation info?  How would GhcPs be used differently to GhcPsI?

 

You might want to answer by clarifying on the wiki page, so that it is a persistent record of the design debugged in dialogue by email.

 

Simon

 

From: Alan & Kim Zimmerman <[hidden email]>
Sent: 08 May 2018 21:02
To: Simon Peyton Jones <[hidden email]>
Cc: ghc-devs <[hidden email]>
Subject: Re: TTG hsSyn for Batch and Interactive Parsing

 

 

On 8 May 2018 at 10:54, Simon Peyton Jones <[hidden email]> wrote:

At first blush, “running the parser in two modes” and “changing the Pass” type don’t match up in my mind.  One seems quite local (how to run the parser).  The other seems more pervasive.

 

Can you say more about your proposed design, perhaps even on a wiki page?

 

Simon

 

From: ghc-devs <[hidden email]> On Behalf Of Alan & Kim Zimmerman
Sent: 07 May 2018 16:17
To: ghc-devs <[hidden email]>
Subject: TTG hsSyn for Batch and Interactive Parsing

 

I want to be able to run the GHC parser in one of two modes, batch which functions as now, and interactive which will (eventually) be incremental.

In addition, the hsSyn AST for each will have different TTG[1] annotations, so that it can better support IDE usage.

I think this can be done by changing the types in HsExtension to introduce a 'Process'  type as follows

data Pass = Parsed Process | Renamed | Typechecked
         deriving (Data)

data Process = Batch | Interactive
  deriving (Show, Data)

We then rename the pass synonyms so that batch is the default

type GhcPs   = GhcPass ('Parsed 'Batch)
type GhcPsI  = GhcPass ('Parsed 'Interactive)

I have attached a simple proof of concept file, which emulates parsing and renaming.

Is this an appropriate approach to take?

Alan

 

[1] Trees That Grow https://ghc.haskell.org/trac/ghc/wiki/ImplementingTreesThatGrow

 

 


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: TTG hsSyn for Batch and Interactive Parsing

Alan & Kim Zimmerman
I have updated the Wiki with the clearer names, and noted that a single parser definition would still be used, as at present, but would only keep the extra info if it was requested to.

The naming around interactive and batch is to anticipate a further step I would like to take, to make the parser fully incremental, in the sense that it would process as input the prior parse tree and a list of changes to the source, and then generate a fresh parse tree, with the changed nodes marked.  This mode would be tightly coupled to an external too like haskell-ide-engine, to manage the bookkeeping around this.

My thinking for this is to use the approach presented in the paper "Efficient and Flexible Incremental Parsing" by Wagner and Graham[1].
The plan is to modify `happy`, so that we can reuse the existing GHC Parser.y with minor modifications. This is the same approach as used in the library tree-sitter[2], which is a very active project on github.

WIP is at [3], but it is very early stage.

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

RE: TTG hsSyn for Batch and Interactive Parsing

GHC - devs mailing list

This sounds fine.  But I still don’t understand

 

There would still be a single parser definition in Parser.y, which would make use of functions to add the additional info to the generated source tree, which would be NOPs if the information was not being kept. This is similar to what happens at present with the Api Annotations.

 

What is the type of the parser?  Does it produce

parser :: String -> HsSyn (GhcPass (Parsed WithApiAnnotation)

or

parser :: String -> HsSyn (GhcPass (Parsed WithoutApiAnnotation)

?

 

We can’t make the result type depend on DynFlags! (Yet)(

    parser :: DynFlags -> String

  -> HsSyn (GhcPass (Parsed (if …

                            then WithApiAnnotations

                            else WihoutsApiAnnotations)

 

So I’m puzzled.

 

Nomenclature.  I’d say “NoApiAnnotations” rather than “WithoutApiAnnotations”.

 

Also: do we have data to show that it’s not OK to always keep API annotations.   That would be simpler, wouldn’t it.

 

Incidentally the Haddock stuff, decorations of type (Maybe LHsDocString) somehow belong in this world too, don’t they?

 

SImon

 

 

From: Alan & Kim Zimmerman <[hidden email]>
Sent: 10 May 2018 22:35
To: Simon Peyton Jones <[hidden email]>
Cc: ghc-devs <[hidden email]>
Subject: Re: TTG hsSyn for Batch and Interactive Parsing

 

I have updated the Wiki with the clearer names, and noted that a single parser definition would still be used, as at present, but would only keep the extra info if it was requested to.

 

The naming around interactive and batch is to anticipate a further step I would like to take, to make the parser fully incremental, in the sense that it would process as input the prior parse tree and a list of changes to the source, and then generate a fresh parse tree, with the changed nodes marked.  This mode would be tightly coupled to an external too like haskell-ide-engine, to manage the bookkeeping around this.

 

My thinking for this is to use the approach presented in the paper "Efficient and Flexible Incremental Parsing" by Wagner and Graham[1].

The plan is to modify `happy`, so that we can reuse the existing GHC Parser.y with minor modifications. This is the same approach as used in the library tree-sitter[2], which is a very active project on github.

WIP is at [3], but it is very early stage.


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: TTG hsSyn for Batch and Interactive Parsing

Alan & Kim Zimmerman


On 18 May 2018 at 16:13, Simon Peyton Jones <[hidden email]> wrote:

 

We can’t make the result type depend on DynFlags! (Yet)(

    parser :: DynFlags -> String

  -> HsSyn (GhcPass (Parsed (if …

                            then WithApiAnnotations

                            else WihoutsApiAnnotations)


We could conceptually have

  parser :: DynFlags -> String -> Either (HsSyn (Parsed WithApiAnnotations)) (HsSyn (Parsed NoApiAnnotations))

The main point is that the next phase can make use of either of the variants.

And it may be simplest to just always use the annotations, the ParsedSource is normally discarded after renaming anyway.

Alan

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

RE: TTG hsSyn for Batch and Interactive Parsing

GHC - devs mailing list

If we are always going to generate a parse tree with annotations from the parser, let’s not generate two!

 

I’m fine with always generating the annotations, but we just need to check that it doesn’t have insupportable costs.

 

Simon

 

From: Alan & Kim Zimmerman <[hidden email]>
Sent: 18 May 2018 15:31
To: Simon Peyton Jones <[hidden email]>
Cc: ghc-devs <[hidden email]>
Subject: Re: TTG hsSyn for Batch and Interactive Parsing

 

 

 

On 18 May 2018 at 16:13, Simon Peyton Jones <[hidden email]> wrote:

 

We can’t make the result type depend on DynFlags! (Yet)(

    parser :: DynFlags -> String

  -> HsSyn (GhcPass (Parsed (if …

                            then WithApiAnnotations

                            else WihoutsApiAnnotations)

 

We could conceptually have

 

  parser :: DynFlags -> String -> Either (HsSyn (Parsed WithApiAnnotations)) (HsSyn (Parsed NoApiAnnotations))

 

The main point is that the next phase can make use of either of the variants.

 

And it may be simplest to just always use the annotations, the ParsedSource is normally discarded after renaming anyway.

 

Alan


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs