How should we treat changes to the GHC API?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

How should we treat changes to the GHC API?

GHC - devs mailing list

A recent MR for GHC (adding machinery for plugins to write data to extensible interface files) made me wonder: 

how we should treat significant changes to the GHC API?

Changes to the GHC API, especially to bits used by plugins or by IDEs, are clearly user-visible to an important class of users – they are not just internal to GHC itself.   So, how should we review them?  Should they perhaps be part of the GHC proposals process?  Or some other similar process?   (The collection of experts on the GHC API, plugins, IDEs etc, is rather different to the membership of the GHC steering group.)

I'm asking, not to be obstructive, but because the GHC API deserves to be thought of as a whole; in the past it has grown incrementally, without much discussion, and that has not served us well.  But at the moment there is no process, no group to consult.

Any views?

Simon


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: How should we treat changes to the GHC API?

Mathieu Boespflug-3
I would just point out that decision by committee, and in particular the GHC Proposals process, has a high cost in terms of both total human brain cycles and latency. The cost is entirely justified when it comes to things that are a) hard to revert and b) extremely hard to get right the first time, like new extensions to the language, or c) very sensitive (like governance, say). For things like breaking changes to API's, it's worth writing out what the current problems are. Are users complaining that the API churn is too high? Are they concerned about endemic quality problems with the API?

It may be enough to make sure to know who the main users of the API are and tag them as reviewers on these types of changes in GitLab. Or to avoid extra process but enshrine principles that might be necessary to adopt, like saying that existing API functions should always be kept as-is during some deprecation period and new functionality should be exposed in new additions to the API. Principles to be upheld by reviewers.

On Mon, Jul 27, 2020 at 10:45:50, Simon Peyton Jones <[hidden email]> wrote:

A recent MR for GHC (adding machinery for plugins to write data to extensible interface files) made me wonder: 

how we should treat significant changes to the GHC API?

Changes to the GHC API, especially to bits used by plugins or by IDEs, are clearly user-visible to an important class of users – they are not just internal to GHC itself.   So, how should we review them?  Should they perhaps be part of the GHC proposals process?  Or some other similar process?   (The collection of experts on the GHC API, plugins, IDEs etc, is rather different to the membership of the GHC steering group.)

I'm asking, not to be obstructive, but because the GHC API deserves to be thought of as a whole; in the past it has grown incrementally, without much discussion, and that has not served us well.  But at the moment there is no process, no group to consult.

Any views?

Simon

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs



_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

RE: How should we treat changes to the GHC API?

GHC - devs mailing list

What I’m after is a clear opportunity for informed debate, and a clear yes/no decision.  That need not be high overhead.

 

It means paying some upfront cost for design changes.  But that’s better than the legacy cost of dealing with things that turn out, in retrospect, to be less well designed than they could be.

 

We tend to think of APIs as implementation details.  But they are deeply significant, and express key abstractions, just like language designs do.  I think we should treat them just as seriously.

 

Simon

 

From: Mathieu Boespflug <[hidden email]>
Sent: 27 July 2020 11:11
To: Simon Peyton Jones <[hidden email]>
Cc: [hidden email] Devs <[hidden email]>
Subject: Re: How should we treat changes to the GHC API?

 

I would just point out that decision by committee, and in particular the GHC Proposals process, has a high cost in terms of both total human brain cycles and latency. The cost is entirely justified when it comes to things that are a) hard to revert and b) extremely hard to get right the first time, like new extensions to the language, or c) very sensitive (like governance, say). For things like breaking changes to API's, it's worth writing out what the current problems are. Are users complaining that the API churn is too high? Are they concerned about endemic quality problems with the API?

 

It may be enough to make sure to know who the main users of the API are and tag them as reviewers on these types of changes in GitLab. Or to avoid extra process but enshrine principles that might be necessary to adopt, like saying that existing API functions should always be kept as-is during some deprecation period and new functionality should be exposed in new additions to the API. Principles to be upheld by reviewers.

 

On Mon, Jul 27, 2020 at 10:45:50, Simon Peyton Jones <[hidden email]> wrote:

A recent MR for GHC (adding machinery for plugins to write data to extensible interface files) made me wonder: 

how we should treat significant changes to the GHC API?

Changes to the GHC API, especially to bits used by plugins or by IDEs, are clearly user-visible to an important class of users – they are not just internal to GHC itself.   So, how should we review them?  Should they perhaps be part of the GHC proposals process?  Or some other similar process?   (The collection of experts on the GHC API, plugins, IDEs etc, is rather different to the membership of the GHC steering group.)

I'm asking, not to be obstructive, but because the GHC API deserves to be thought of as a whole; in the past it has grown incrementally, without much discussion, and that has not served us well.  But at the moment there is no process, no group to consult.

Any views?

Simon

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

 


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: How should we treat changes to the GHC API?

Iavor Diatchki
In principle, I think we should treat the GHC API like any other library, and try not to break code unnecessarily.  However, my impression is that the GHC API grew somewhat organically, so we may want to put some additional work before we stabilize things too much.  It's been a while since I used it, so I might be out of date, but last I looked the GHC API was a module exporting some high-level functions from GHC.   I think that a single module is too small of an API for a project as large as GHC.  In fact, it probably makes sense to define more than one API.  For example, each plugin point should probably have its own API, and that's likely different to the GHC API that exposes functionality such as "load and type check this module here", or "parse and evaluate this string".

-Iavor



On Mon, Jul 27, 2020 at 4:05 AM Simon Peyton Jones via ghc-devs <[hidden email]> wrote:

What I’m after is a clear opportunity for informed debate, and a clear yes/no decision.  That need not be high overhead.

 

It means paying some upfront cost for design changes.  But that’s better than the legacy cost of dealing with things that turn out, in retrospect, to be less well designed than they could be.

 

We tend to think of APIs as implementation details.  But they are deeply significant, and express key abstractions, just like language designs do.  I think we should treat them just as seriously.

 

Simon

 

From: Mathieu Boespflug <[hidden email]>
Sent: 27 July 2020 11:11
To: Simon Peyton Jones <[hidden email]>
Cc: [hidden email] Devs <[hidden email]>
Subject: Re: How should we treat changes to the GHC API?

 

I would just point out that decision by committee, and in particular the GHC Proposals process, has a high cost in terms of both total human brain cycles and latency. The cost is entirely justified when it comes to things that are a) hard to revert and b) extremely hard to get right the first time, like new extensions to the language, or c) very sensitive (like governance, say). For things like breaking changes to API's, it's worth writing out what the current problems are. Are users complaining that the API churn is too high? Are they concerned about endemic quality problems with the API?

 

It may be enough to make sure to know who the main users of the API are and tag them as reviewers on these types of changes in GitLab. Or to avoid extra process but enshrine principles that might be necessary to adopt, like saying that existing API functions should always be kept as-is during some deprecation period and new functionality should be exposed in new additions to the API. Principles to be upheld by reviewers.

 

On Mon, Jul 27, 2020 at 10:45:50, Simon Peyton Jones <[hidden email]> wrote:

A recent MR for GHC (adding machinery for plugins to write data to extensible interface files) made me wonder: 

how we should treat significant changes to the GHC API?

Changes to the GHC API, especially to bits used by plugins or by IDEs, are clearly user-visible to an important class of users – they are not just internal to GHC itself.   So, how should we review them?  Should they perhaps be part of the GHC proposals process?  Or some other similar process?   (The collection of experts on the GHC API, plugins, IDEs etc, is rather different to the membership of the GHC steering group.)

I'm asking, not to be obstructive, but because the GHC API deserves to be thought of as a whole; in the past it has grown incrementally, without much discussion, and that has not served us well.  But at the moment there is no process, no group to consult.

Any views?

Simon

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

 

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: How should we treat changes to the GHC API?

Eric Seidel-3
As another former user of the GHC API, I'd say my two biggest complaints were the relative instability of the API and the lack of documentation. I haven't used the API in at least three years though, so it's possible much has changed since my experience.

I remember often having to do significant work to adapt LiquidHaskell to new versions of GHC due to small changes in the API. We often felt like we had to import internal modules (ie other than 'GHC') to get key bits of functionality we needed, which might explain the churn. But it also points to Iavor's point that the public API grew organically and might benefit from a bit of top-down design to make sure it's complete enough for typical use cases.

For documentation, the issue was less the API docs but a lack of "How do I do X?" docs and examples. One problem that I remember being particularly vexing was resolving names in a particular scope (in my case it was always module-level scopes, but I can easily imagine clients that would want to resolve names in local scopes).

I don't know if the API needs to go through something like the Steering Committee, but a stronger focus on API stability and perhaps a broader view of what constitutes (or should be included in) the public-facing API would be welcome!

On Mon, Jul 27, 2020, at 11:45, Iavor Diatchki wrote:

> In principle, I think we should treat the GHC API like any other
> library, and try not to break code unnecessarily.  However, my
> impression is that the GHC API grew somewhat organically, so we may
> want to put some additional work before we stabilize things too much.  
> It's been a while since I used it, so I might be out of date, but last
> I looked the GHC API was a module exporting some high-level functions
> from GHC.   I think that a single module is too small of an API for a
> project as large as GHC.  In fact, it probably makes sense to define
> more than one API.  For example, each plugin point should probably have
> its own API, and that's likely different to the GHC API that exposes
> functionality such as "load and type check this module here", or "parse
> and evaluate this string".
>
> -Iavor
>
>
>
> On Mon, Jul 27, 2020 at 4:05 AM Simon Peyton Jones via ghc-devs
> <[hidden email]> wrote:
> > What I’m after is a clear opportunity for informed debate, and a clear yes/no decision.  That need not be high overhead.____
>
> > __ __
>
> > It means paying some upfront cost for design changes.  But that’s better than the legacy cost of dealing with things that turn out, in retrospect, to be less well designed than they could be.____
>
> > __ __
>
> > We tend to think of APIs as implementation details.  But they are deeply significant, and express key abstractions, just like language designs do.  I think we should treat them just as seriously.____
>
> > __ __
>
> > Simon____
>
> > __ __
>
> > *From:* Mathieu Boespflug <[hidden email]>
> > *Sent:* 27 July 2020 11:11
> > *To:* Simon Peyton Jones <[hidden email]>
> > *Cc:* [hidden email] Devs <[hidden email]>
> > *Subject:* Re: How should we treat changes to the GHC API?____
>
> > __ __
>
> > I would just point out that decision by committee, and in particular the GHC Proposals process, has a high cost in terms of both total human brain cycles and latency. The cost is entirely justified when it comes to things that are a) hard to revert and b) extremely hard to get right the first time, like new extensions to the language, or c) very sensitive (like governance, say). For things like breaking changes to API's, it's worth writing out what the current problems are. Are users complaining that the API churn is too high? Are they concerned about endemic quality problems with the API?____
>
> > __ __
>
> > It may be enough to make sure to know who the main users of the API are and tag them as reviewers on these types of changes in GitLab. Or to avoid extra process but enshrine principles that might be necessary to adopt, like saying that existing API functions should always be kept as-is during some deprecation period and new functionality should be exposed in new additions to the API. Principles to be upheld by reviewers.____
>
> > __ __
>
> > On Mon, Jul 27, 2020 at 10:45:50, Simon Peyton Jones <[hidden email]> wrote:____
>
> >> A recent MR for GHC <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.haskell.org%2Fghc%2Fghc%2F-%2Fmerge_requests%2F3758&data=02%7C01%7Csimonpj%40microsoft.com%7C730e52088cb64dcebe1408d8321567e1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637314414803966186&sdata=GCGrNinXtNxPcJfEu%2F%2BifrJJ22BB2bkIy4E9d5IWOuo%3D&reserved=0> (adding machinery for plugins to write data to extensible interface files) made me wonder:  ____
>
> >> how we should treat significant changes to the GHC API?____
>
> >> Changes to the GHC API, especially to bits used by plugins or by IDEs, are clearly user-visible to an important class of users – they are not just internal to GHC itself.   So, how should we review them?  Should they perhaps be part of the GHC proposals process?  Or some other similar process?   (The collection of experts on the GHC API, plugins, IDEs etc, is rather different to the membership of the GHC steering group.)____
>
> >> I'm asking, not to be obstructive, but because the GHC API deserves to be thought of as a whole; in the past it has grown incrementally, without much discussion, and that has not served us well.  But at the moment there is no process, no group to consult.____
>
> >> Any views?____
>
> >> Simon____
>
> >> _______________________________________________
> >> ghc-devs mailing list
> >> [hidden email]
> >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs <https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.haskell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-devs&data=02%7C01%7Csimonpj%40microsoft.com%7C730e52088cb64dcebe1408d8321567e1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637314414803966186&sdata=2TwuVzxKm88aevbTooLG3sLeakrSYZziFPNDozFCvHo%3D&reserved=0>____
>
> > __ __
>
> > _______________________________________________
> > ghc-devs mailing list
> > [hidden email]
> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: How should we treat changes to the GHC API?

John Ericson-2
I would say the current API is....absolutely terrible and so we should
have no qualms about breaking it. Talking to the people who work on the
IDE---i.e. the people who are the most adversely impacted by
churn---they also don't like it and not want it stabilized in its
current form.

I think the only good long term solution is to modularize GHC into
multiple packages, giving us conceptually distinct interfaces which
break at different rates. This is similar to modularizing base as e.g.
GHC.* breaks far more often than e.g. Data.List.

How we evaluate changes is less important to me as my main priority is
to get the pace of development much higher so we have a chance on fixing
all the technical debt to get us out of the current situation.
Basically, you could count me in "team heat bath"; I think that should
have the right connotations.

John

On 7/27/20 2:07 PM, Eric Seidel wrote:

> As another former user of the GHC API, I'd say my two biggest complaints were the relative instability of the API and the lack of documentation. I haven't used the API in at least three years though, so it's possible much has changed since my experience.
>
> I remember often having to do significant work to adapt LiquidHaskell to new versions of GHC due to small changes in the API. We often felt like we had to import internal modules (ie other than 'GHC') to get key bits of functionality we needed, which might explain the churn. But it also points to Iavor's point that the public API grew organically and might benefit from a bit of top-down design to make sure it's complete enough for typical use cases.
>
> For documentation, the issue was less the API docs but a lack of "How do I do X?" docs and examples. One problem that I remember being particularly vexing was resolving names in a particular scope (in my case it was always module-level scopes, but I can easily imagine clients that would want to resolve names in local scopes).
>
> I don't know if the API needs to go through something like the Steering Committee, but a stronger focus on API stability and perhaps a broader view of what constitutes (or should be included in) the public-facing API would be welcome!
>
> On Mon, Jul 27, 2020, at 11:45, Iavor Diatchki wrote:
>> In principle, I think we should treat the GHC API like any other
>> library, and try not to break code unnecessarily.  However, my
>> impression is that the GHC API grew somewhat organically, so we may
>> want to put some additional work before we stabilize things too much.
>> It's been a while since I used it, so I might be out of date, but last
>> I looked the GHC API was a module exporting some high-level functions
>> from GHC.   I think that a single module is too small of an API for a
>> project as large as GHC.  In fact, it probably makes sense to define
>> more than one API.  For example, each plugin point should probably have
>> its own API, and that's likely different to the GHC API that exposes
>> functionality such as "load and type check this module here", or "parse
>> and evaluate this string".
>>
>> -Iavor
>>
>>
>>
>> On Mon, Jul 27, 2020 at 4:05 AM Simon Peyton Jones via ghc-devs
>> <[hidden email]> wrote:
>>> What I’m after is a clear opportunity for informed debate, and a clear yes/no decision.  That need not be high overhead.____
>>> __ __
>>> It means paying some upfront cost for design changes.  But that’s better than the legacy cost of dealing with things that turn out, in retrospect, to be less well designed than they could be.____
>>> __ __
>>> We tend to think of APIs as implementation details.  But they are deeply significant, and express key abstractions, just like language designs do.  I think we should treat them just as seriously.____
>>> __ __
>>> Simon____
>>> __ __
>>> *From:* Mathieu Boespflug <[hidden email]>
>>> *Sent:* 27 July 2020 11:11
>>> *To:* Simon Peyton Jones <[hidden email]>
>>> *Cc:* [hidden email] Devs <[hidden email]>
>>> *Subject:* Re: How should we treat changes to the GHC API?____
>>> __ __
>>> I would just point out that decision by committee, and in particular the GHC Proposals process, has a high cost in terms of both total human brain cycles and latency. The cost is entirely justified when it comes to things that are a) hard to revert and b) extremely hard to get right the first time, like new extensions to the language, or c) very sensitive (like governance, say). For things like breaking changes to API's, it's worth writing out what the current problems are. Are users complaining that the API churn is too high? Are they concerned about endemic quality problems with the API?____
>>> __ __
>>> It may be enough to make sure to know who the main users of the API are and tag them as reviewers on these types of changes in GitLab. Or to avoid extra process but enshrine principles that might be necessary to adopt, like saying that existing API functions should always be kept as-is during some deprecation period and new functionality should be exposed in new additions to the API. Principles to be upheld by reviewers.____
>>> __ __
>>> On Mon, Jul 27, 2020 at 10:45:50, Simon Peyton Jones <[hidden email]> wrote:____
>>>> A recent MR for GHC <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.haskell.org%2Fghc%2Fghc%2F-%2Fmerge_requests%2F3758&data=02%7C01%7Csimonpj%40microsoft.com%7C730e52088cb64dcebe1408d8321567e1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637314414803966186&sdata=GCGrNinXtNxPcJfEu%2F%2BifrJJ22BB2bkIy4E9d5IWOuo%3D&reserved=0> (adding machinery for plugins to write data to extensible interface files) made me wonder:  ____
>>>> how we should treat significant changes to the GHC API?____
>>>> Changes to the GHC API, especially to bits used by plugins or by IDEs, are clearly user-visible to an important class of users – they are not just internal to GHC itself.   So, how should we review them?  Should they perhaps be part of the GHC proposals process?  Or some other similar process?   (The collection of experts on the GHC API, plugins, IDEs etc, is rather different to the membership of the GHC steering group.)____
>>>> I'm asking, not to be obstructive, but because the GHC API deserves to be thought of as a whole; in the past it has grown incrementally, without much discussion, and that has not served us well.  But at the moment there is no process, no group to consult.____
>>>> Any views?____
>>>> Simon____
>>>> _______________________________________________
>>>> ghc-devs mailing list
>>>> [hidden email]
>>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs <https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.haskell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-devs&data=02%7C01%7Csimonpj%40microsoft.com%7C730e52088cb64dcebe1408d8321567e1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637314414803966186&sdata=2TwuVzxKm88aevbTooLG3sLeakrSYZziFPNDozFCvHo%3D&reserved=0>____
>>> __ __
>>> _______________________________________________
>>> ghc-devs mailing list
>>> [hidden email]
>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>> _______________________________________________
>> ghc-devs mailing list
>> [hidden email]
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: How should we treat changes to the GHC API?

Richard Eisenberg-5
In reply to this post by Eric Seidel-3


On Jul 27, 2020, at 2:07 PM, Eric Seidel <[hidden email]> wrote:

a stronger focus on API stability and perhaps a broader view of what constitutes (or should be included in) the public-facing API would be welcome!

I agree with this in the abstract, but I'm not sure about agreeing with it in the concrete. (I don't mean to pick on Eric here -- but this was a nice sentence I could quote.) The problem is that stability in the API has a very real cost. It means that GHC developers maintain some interface indefinitely, even if the implementation drifts in a different direction. Folks doing the internal GHC work often don't have extensive experience with the API, which means that a move toward API stability would mean that GHC devs would be poorly equipped to decide when an interface is important to preserve or unimportant. This leads to the possibility that devs would spend a lot of time maintaining particular behavior that is not needed. And, of course, time spent holding the API stable is time not spent doing other tasks, and thus a move toward stability would slow down development. This might be a small difference -- I'm not predicting calamity -- but it's a real cost.

On the other hand, we could imagine a dedicated GHC API volunteer who maintains the API on top of the shifting sands of GHC. Having a central project to deal with the API would satisfy clients by projecting a stable interface, and it wouldn't hinder GHC development, as the API volunteer would take on the compatibility burden. Of course, success here would require a dedicated, competent volunteer.

To be clear: I'm not against API stability, but I do want to make clear that this choice has a cost.

Richard

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: How should we treat changes to the GHC API?

John Ericson-2
On 7/27/20 4:57 PM, Richard Eisenberg wrote
> On the other hand, we could imagine a dedicated GHC API volunteer who
> maintains the API on top of the shifting sands of GHC. Having a
> central project to deal with the API would satisfy clients by
> projecting a stable interface, and it wouldn't hinder GHC development,
> as the API volunteer would take on the compatibility burden. Of
> course, success here would require a dedicated, competent volunteer.

I think that basically exists:
https://hackage.haskell.org/package/ghc-lib ? As permanent solution I
don't like it, but as a stop gap to relieve any pressure for stability
until we have sane interfaces, I *love* it.

John

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: How should we treat changes to the GHC API?

Shayne Fletcher


On Mon, Jul 27, 2020 at 5:24 PM John Ericson <[hidden email]> wrote:

I think that basically exists:
https://hackage.haskell.org/package/ghc-lib ? As permanent solution I
don't like it, but as a stop gap to relieve any pressure for stability
until we have sane interfaces, I *love* it.

`ghc-lib` is a literal subset of GHC files: `ghc-lib-parser` that set of files sufficient to produce abstract syntax trees (~200 files), `ghc-lib` the remaining set of files enabling Core generation from parsed syntax trees (~300 of those) (more detail in the project README here https://github.com/digital-asset/ghc-lib/blob/master/README.md).

There is at this time, no attempt to provide any additional interface, "higher level" or otherwise, in these packages.

The key property of `ghc-lib-parser`/`ghc-lib` that makes them useful to tool developers (e.g. HLint) is that they can utilize the GHC API without being bound to a specific compiler version to do so. 
 
--
Shayne Fletcher

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: How should we treat changes to the GHC API?

Eric Seidel-3
In reply to this post by Richard Eisenberg-5
Thanks Richard, you raise some important points. 

API stability does impose a real cost on developers (working at a company with a firm-wide integration build for the past three years has made me keenly aware of that). But API instability also imposes a real cost on clients.

I think there are a couple things that could be done though, to help lessen the burden, and to share it between devs and clients. One is to define a clear separation between the public API and internal modules. GHC does this to an extent, but as I mentioned earlier, I think it needs a more comprehensive public API (I did not mean that we should just declare more modules as part of the public API, but provide stable wrappers around that functionality). You’re right that GHC devs are not the best suited to define the public API, I think it should be done as a collaboration between clients and devs. 

Another option could be to have a transition period for APIs like we do for changes to the language. GHC already commits to waiting a few revisions to use new language features in the name of shortening bootstrapping chains, waiting a few revisions to drop an old API doesn’t feel all that different and would give clients a lot of breathing room. 

I don’t want to put the burden of API stability all on GHC devs either, but I do think it could probably be balanced better. And this is where having something like the Steering Committee could help. It doesn’t need to be as formal either, perhaps a small group of stakeholders that get tagged on MRs to the public API, or a mailing list where you solicit feedback. 

Sent from my iPhone

On Jul 27, 2020, at 15:57, Richard Eisenberg <[hidden email]> wrote:



On Jul 27, 2020, at 2:07 PM, Eric Seidel <[hidden email]> wrote:

a stronger focus on API stability and perhaps a broader view of what constitutes (or should be included in) the public-facing API would be welcome!

I agree with this in the abstract, but I'm not sure about agreeing with it in the concrete. (I don't mean to pick on Eric here -- but this was a nice sentence I could quote.) The problem is that stability in the API has a very real cost. It means that GHC developers maintain some interface indefinitely, even if the implementation drifts in a different direction. Folks doing the internal GHC work often don't have extensive experience with the API, which means that a move toward API stability would mean that GHC devs would be poorly equipped to decide when an interface is important to preserve or unimportant. This leads to the possibility that devs would spend a lot of time maintaining particular behavior that is not needed. And, of course, time spent holding the API stable is time not spent doing other tasks, and thus a move toward stability would slow down development. This might be a small difference -- I'm not predicting calamity -- but it's a real cost.

On the other hand, we could imagine a dedicated GHC API volunteer who maintains the API on top of the shifting sands of GHC. Having a central project to deal with the API would satisfy clients by projecting a stable interface, and it wouldn't hinder GHC development, as the API volunteer would take on the compatibility burden. Of course, success here would require a dedicated, competent volunteer.

To be clear: I'm not against API stability, but I do want to make clear that this choice has a cost.

Richard

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: How should we treat changes to the GHC API?

Richard Eisenberg-5
In reply to this post by John Ericson-2


On Jul 27, 2020, at 5:24 PM, John Ericson <[hidden email]> wrote:

As permanent solution I don't like it, but as a stop gap to relieve any pressure for stability until we have sane interfaces, I *love* it.

I guess I'm arguing that ghc-lib (or something like it) should be the permanent solution. GHC will always be in flux internally.

Richard

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: How should we treat changes to the GHC API?

Daniel Trstenjak-2
In reply to this post by Richard Eisenberg-5
On Mon, Jul 27, 2020 at 08:57:12PM +0000, Richard Eisenberg wrote:
> On the other hand, we could imagine a dedicated GHC API volunteer who
> maintains the API on top of the shifting sands of GHC.

Looking at other compilers that have been successful in having a stable
API - like clang with the libclang - that's pretty much how they
achieved it.

The manpower of GHC devs is already pretty small, so putting another
burden on them with a stable API won't work out that well.

Also as a compiler dev you want to have the freedom to change your AST
in the ways you need it, to incorporate new features. Having workarounds
at this level to ensure API stability just seems to be the wrong place and
will only increase the complexity in an already quite complex project.
However a high level API on top of the AST is the perfect place for such
special cases between different compiler versions.

Greetings,
Daniel
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: How should we treat changes to the GHC API?

Neil Mitchell
The API changing regularly is a real hassle for people building on top of it. But, the people building on top of it are also some of the people changing it most. And the API isn't nice and beautiful as it stands, with lots of horrible work-arounds for downstream users.

I'd suggest letting people change the API freely. But avoid making bikeshed API changes (those which are minor improvements at best, and really just a different flavour of the same). And perhaps loop in heavy API users when removing functionality.

Thanks, Neil

On Tue, Jul 28, 2020 at 7:38 AM Daniel Trstenjak <[hidden email]> wrote:
On Mon, Jul 27, 2020 at 08:57:12PM +0000, Richard Eisenberg wrote:
> On the other hand, we could imagine a dedicated GHC API volunteer who
> maintains the API on top of the shifting sands of GHC.

Looking at other compilers that have been successful in having a stable
API - like clang with the libclang - that's pretty much how they
achieved it.

The manpower of GHC devs is already pretty small, so putting another
burden on them with a stable API won't work out that well.

Also as a compiler dev you want to have the freedom to change your AST
in the ways you need it, to incorporate new features. Having workarounds
at this level to ensure API stability just seems to be the wrong place and
will only increase the complexity in an already quite complex project.
However a high level API on top of the AST is the perfect place for such
special cases between different compiler versions.

Greetings,
Daniel
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: How should we treat changes to the GHC API?

Shayne Fletcher
In reply to this post by Shayne Fletcher


On Mon, Jul 27, 2020 at 7:13 PM Shayne Fletcher <[hidden email]> wrote:


On Mon, Jul 27, 2020 at 5:24 PM John Ericson <[hidden email]> wrote:

I think that basically exists:
https://hackage.haskell.org/package/ghc-lib ? As permanent solution I
don't like it, but as a stop gap to relieve any pressure for stability
until we have sane interfaces, I *love* it.

`ghc-lib` is a literal subset of GHC files: `ghc-lib-parser` that set of files sufficient to produce abstract syntax trees (~200 files), `ghc-lib` the remaining set of files enabling Core generation from parsed syntax trees (~300 of those) (more detail in the project README here https://github.com/digital-asset/ghc-lib/blob/master/README.md).

There is at this time, no attempt to provide any additional interface, "higher level" or otherwise, in these packages.

[...]

I should mention that the related project `ghc-lib-parser-ex` (README here https://github.com/shayne-fletcher/ghc-lib-parser-ex/blob/master/README.md) IS in large part motivated by the need to provide stable interfaces over `ghc-lib-parser`.

--
Shayne Fletcher

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: How should we treat changes to the GHC API?

John Ericson-2
In reply to this post by Richard Eisenberg-5
On 7/27/20 10:57 PM, Richard Eisenberg wrote:
> I guess I'm arguing that ghc-lib (or something like it) should be the
> permanent solution. GHC will always be in flux internally.

So to be clear I don't disagree that GHC will always be in flux. But I
think things like ghc-lib largely work today because GHC is so imperative.

Once it (hopefully!) is more functional, we'll have the problem less
that functions change[1], and more that data structures change. I think
that will be harder to abstract over---often to migrate an interface
properly, data structure will need migrations in both directions *and*
that those migrations form an isomorphism, which can be an impossible
requirement. It's just a fact of life that rich data structures make
migrations harder, but that's no reason we shouldn't have rich data
structures!

I'm probably too deeply exploring what's only one possible future at
this point, so let me rewind out of that exact hypothesis and just make
the broader point that if the interfaces change a lot, a lot of
organizational stuff can/should too. If the interface changes *don't*
engender cultural shifts around working with GHC, we're probably doing
them wrong and they are vapid churn after all!

John

[1]: there should be way fewer functions than today. we currently have a
gratuitous combinatorial explosion of little helpers which obscures what
the underlying orthogonal concepts are.

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

RE: How should we treat changes to the GHC API?

GHC - devs mailing list
In reply to this post by Neil Mitchell

My real question is this:

  • Who “owns” (as in: takes responsibility for) the GHC API?

 

If no person, or no group of people, feels responsible for it, it’ll just grow in a disorganised fashion, which is why we are where we are. 

 

This discussion thread has focused mainly on stability, but there is more to it than that.  For example:

  • What kind of stability strikes a reasonable balance between developers and clients?  Personally I think we should have
    • A public API that is pretty stable, plus
    • Access to many internal GHC functions that change quite a bit.

Heavily used internal functions can get “blessed” and moved to the public API.   Someone needs to do the blessing.

  • What is a good public API?  Where is it written down?
  • What should GHC’s extensibility interface be like?   Plugins and all that.  What is a good design for (say) extensible interface files?  What “hooks” should the GHC API afford?  This is more than just “what arguments should this function take”… it’s a matter of fundamental design.   But design questions like this belong in the GHC-API world (not the core GHC world) because they are all about extension points.
  • What is a good design for the plugins story more generally?

I’d be quite content with a fairly informal group to act as the “owner”.  But I think we would be better served if we had such a group, and knew who they were.

 

Simon

 

From: ghc-devs <[hidden email]> On Behalf Of Neil Mitchell
Sent: 28 July 2020 09:19
To: GHC developers <[hidden email]>
Subject: Re: How should we treat changes to the GHC API?

 

The API changing regularly is a real hassle for people building on top of it. But, the people building on top of it are also some of the people changing it most. And the API isn't nice and beautiful as it stands, with lots of horrible work-arounds for downstream users.

 

I'd suggest letting people change the API freely. But avoid making bikeshed API changes (those which are minor improvements at best, and really just a different flavour of the same). And perhaps loop in heavy API users when removing functionality.

 

Thanks, Neil

 

On Tue, Jul 28, 2020 at 7:38 AM Daniel Trstenjak <[hidden email]> wrote:

On Mon, Jul 27, 2020 at 08:57:12PM +0000, Richard Eisenberg wrote:
> On the other hand, we could imagine a dedicated GHC API volunteer who
> maintains the API on top of the shifting sands of GHC.

Looking at other compilers that have been successful in having a stable
API - like clang with the libclang - that's pretty much how they
achieved it.

The manpower of GHC devs is already pretty small, so putting another
burden on them with a stable API won't work out that well.

Also as a compiler dev you want to have the freedom to change your AST
in the ways you need it, to incorporate new features. Having workarounds
at this level to ensure API stability just seems to be the wrong place and
will only increase the complexity in an already quite complex project.
However a high level API on top of the AST is the perfect place for such
special cases between different compiler versions.

Greetings,
Daniel
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: How should we treat changes to the GHC API?

Moritz Angermann-2
I think this is the core issue here:
> What should GHC’s extensibility interface be like?   Plugins and all that.  What is a good design for (say) extensible interface files?  What “hooks” should the GHC API afford?  This is more than just “what arguments should this function take”… it’s a matter of fundamental design.   But design questions like this belong in the GHC-API world (not the core GHC world) because they are all about extension points.
I don't think we know this apriori, and it will be discovered over
time as more and more producers and consumers start making use of it.
Is the current design the best one? I have my doubts, is it a first
approximation? I think so.

I think these features are more discovered than designed. It's easier
to iterate over concrete implementations in this area than over
abstract ideas.

I would propose having some EXPERIMENTAL markers for these kinds of features.

I do agree that a group of people who feel strongly about this should
be listed in the CODEOWNERS file for the respective parts of the
codebase, and take an active part in code review.
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: How should we treat changes to the GHC API?

John Ericson-2
In reply to this post by GHC - devs mailing list
On 7/31/20 9:27 AM, Simon Peyton Jones via ghc-devs wrote:

My real question is this:

  • Who “owns” (as in: takes responsibility for) the GHC API?

I guess one thing very important to me is that the architecture of GHC and it's public interfaces will always be deeply intertwined; or put a different way, efforts to manager the public interface as a thin veneer over a big black box will not work.

  • What is a good public API?  Where is it written down?

This I agree is important to discuss. I do agree with Moritz that a lot of this stuff can be evolved, but the general direction can be still be discussed. For example, the basic objects I offered are a *huge* departure from what we have today, and any huge change should be discussed up front a bit.

  • What should GHC’s extensibility interface be like?   Plugins and all that.  What is a good design for (say) extensible interface files?  What “hooks” should the GHC API afford?  This is more than just “what arguments should this function take”… it’s a matter of fundamental design.   But design questions like this belong in the GHC-API world (not the core GHC world) because they are all about extension points.
  • What is a good design for the plugins story more generally?

I've mentioned this before but I think plugins/hooks/etc. are a terrible way to reuse GHC for other purposes, and exist as a symptom of the rest of the compiler not being at all modular. It's fine that we continue to support them in the short term, but in the long term I'd really like to have a plan to obviate them completely. (One can search "composition over configuration" and variations on that slogan to find much ink has been spilled on this general principle.)

John



_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs