Re: [GHC DevOps Group] Making GHC's fast release cadence work

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [GHC DevOps Group] Making GHC's fast release cadence work

Boespflug, Mathieu
A related discussion came up previously: https://mail.haskell.org/pipermail/ghc-devops-group/2017-December/thread.html#118. From the moment that GHC accepts unreleased dependencies on its release branch, GHC developers lose full control over their own release timeline. Because a new GHC release can only ship when the dependencies have been released, the release process could block on upstream for indefinite periods of time, like is currently happening with at least one GHC dependency that has been blocked on the maintainer for at least 27 days (and counting).

It was suggested then that GHC should only use release versions of its dependencies on the master or release branch, and that if this isn't practical due to co-dependency, then that suggests said dependencies should have the same release cycle as GHC (i.e. same people can cut releases of both) or even be merged in GHC itself.

--
Mathieu Boespflug
Founder at http://tweag.io.


On Mon, 1 Jul 2019 at 21:03, David Feuer <[hidden email]> wrote:
I realize that my response came off as rather unfriendly. I'm sorry about that. Thinking a bit more about it, I think GHC HQ can help Core Library maintainers in several unintrusive ways. Here are a couple I would appreciate myself; others may disagree.

1. Work with Herbert, Ryan, etc., to make it easy to use -Werror in Travis scripts. See https://github.com/haskell/containers/pull/645

2. Give us a nudge when you have a release timeframe in mind. "We're hoping to have the core libraries ready for release by December 5" (several weeks beforehand, repeated a couple times thereafter with appropriate adjustments) is much less stressful than "We're basically ready for release but we're waiting on core libraries."

On Mon, Jul 1, 2019, 1:30 PM David Feuer <[hidden email]> wrote:
I think it would be nice to start by letting the core library maintainers know your preferred release deadlines.

On Mon, Jul 1, 2019, 1:22 PM Ben Gamari <[hidden email]> wrote:
Hi everyone,

GHC's core libraries are a critical part of the Haskell ecosystem. I
want to thank you for overseeing the maintenance of this infrastructure.

However, for the last three weeks the release candidate for GHC 8.8.1
has been ready aside from releases of a couple of our core libraries.

Naturally, delays like this make it hard for GHC to maintain its faster
release cycle. At the same time, we do not want this cadence to impose
an undue burden on our core library maintainers.

How do you think we might speed up this process?

For instance, perhaps the GHC release manager could pick up
some of the "boring parts" of core library maintenance limited to:

 * Version bound bumps

 * Changes of CPP conditionals to accommodate changes in the
   compiler and other core libraries

 * Changelog entries to describe these releases

 * Uploading these releases or revisions to Hackage

Of course, this would merely be an offer to maintainers; this would be
GHC's way of carrying some of the burden that our release process
imposes.

In general, I am interested in a discussion on how to make this faster
release pace work. Ideas?

Cheers,

- Ben

_______________________________________________
Ghc-devops-group mailing list
[hidden email]
https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group

_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: [GHC DevOps Group] Making GHC's fast release cadence work

Carter Schonwald
what lib didn't have a release that wouldn't work with the release candidate?



On Mon, Jul 1, 2019 at 4:51 PM Boespflug, Mathieu <[hidden email]> wrote:
A related discussion came up previously: https://mail.haskell.org/pipermail/ghc-devops-group/2017-December/thread.html#118. From the moment that GHC accepts unreleased dependencies on its release branch, GHC developers lose full control over their own release timeline. Because a new GHC release can only ship when the dependencies have been released, the release process could block on upstream for indefinite periods of time, like is currently happening with at least one GHC dependency that has been blocked on the maintainer for at least 27 days (and counting).

It was suggested then that GHC should only use release versions of its dependencies on the master or release branch, and that if this isn't practical due to co-dependency, then that suggests said dependencies should have the same release cycle as GHC (i.e. same people can cut releases of both) or even be merged in GHC itself.

--
Mathieu Boespflug
Founder at http://tweag.io.


On Mon, 1 Jul 2019 at 21:03, David Feuer <[hidden email]> wrote:
I realize that my response came off as rather unfriendly. I'm sorry about that. Thinking a bit more about it, I think GHC HQ can help Core Library maintainers in several unintrusive ways. Here are a couple I would appreciate myself; others may disagree.

1. Work with Herbert, Ryan, etc., to make it easy to use -Werror in Travis scripts. See https://github.com/haskell/containers/pull/645

2. Give us a nudge when you have a release timeframe in mind. "We're hoping to have the core libraries ready for release by December 5" (several weeks beforehand, repeated a couple times thereafter with appropriate adjustments) is much less stressful than "We're basically ready for release but we're waiting on core libraries."

On Mon, Jul 1, 2019, 1:30 PM David Feuer <[hidden email]> wrote:
I think it would be nice to start by letting the core library maintainers know your preferred release deadlines.

On Mon, Jul 1, 2019, 1:22 PM Ben Gamari <[hidden email]> wrote:
Hi everyone,

GHC's core libraries are a critical part of the Haskell ecosystem. I
want to thank you for overseeing the maintenance of this infrastructure.

However, for the last three weeks the release candidate for GHC 8.8.1
has been ready aside from releases of a couple of our core libraries.

Naturally, delays like this make it hard for GHC to maintain its faster
release cycle. At the same time, we do not want this cadence to impose
an undue burden on our core library maintainers.

How do you think we might speed up this process?

For instance, perhaps the GHC release manager could pick up
some of the "boring parts" of core library maintenance limited to:

 * Version bound bumps

 * Changes of CPP conditionals to accommodate changes in the
   compiler and other core libraries

 * Changelog entries to describe these releases

 * Uploading these releases or revisions to Hackage

Of course, this would merely be an offer to maintainers; this would be
GHC's way of carrying some of the burden that our release process
imposes.

In general, I am interested in a discussion on how to make this faster
release pace work. Ideas?

Cheers,

- Ben

_______________________________________________
Ghc-devops-group mailing list
[hidden email]
https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group
_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries

_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: [GHC DevOps Group] Making GHC's fast release cadence work

Carter Schonwald
err: which library lacked a release that worked with the 8.8 RC?

On Tue, Jul 2, 2019 at 12:25 AM Carter Schonwald <[hidden email]> wrote:
what lib didn't have a release that wouldn't work with the release candidate?



On Mon, Jul 1, 2019 at 4:51 PM Boespflug, Mathieu <[hidden email]> wrote:
A related discussion came up previously: https://mail.haskell.org/pipermail/ghc-devops-group/2017-December/thread.html#118. From the moment that GHC accepts unreleased dependencies on its release branch, GHC developers lose full control over their own release timeline. Because a new GHC release can only ship when the dependencies have been released, the release process could block on upstream for indefinite periods of time, like is currently happening with at least one GHC dependency that has been blocked on the maintainer for at least 27 days (and counting).

It was suggested then that GHC should only use release versions of its dependencies on the master or release branch, and that if this isn't practical due to co-dependency, then that suggests said dependencies should have the same release cycle as GHC (i.e. same people can cut releases of both) or even be merged in GHC itself.

--
Mathieu Boespflug
Founder at http://tweag.io.


On Mon, 1 Jul 2019 at 21:03, David Feuer <[hidden email]> wrote:
I realize that my response came off as rather unfriendly. I'm sorry about that. Thinking a bit more about it, I think GHC HQ can help Core Library maintainers in several unintrusive ways. Here are a couple I would appreciate myself; others may disagree.

1. Work with Herbert, Ryan, etc., to make it easy to use -Werror in Travis scripts. See https://github.com/haskell/containers/pull/645

2. Give us a nudge when you have a release timeframe in mind. "We're hoping to have the core libraries ready for release by December 5" (several weeks beforehand, repeated a couple times thereafter with appropriate adjustments) is much less stressful than "We're basically ready for release but we're waiting on core libraries."

On Mon, Jul 1, 2019, 1:30 PM David Feuer <[hidden email]> wrote:
I think it would be nice to start by letting the core library maintainers know your preferred release deadlines.

On Mon, Jul 1, 2019, 1:22 PM Ben Gamari <[hidden email]> wrote:
Hi everyone,

GHC's core libraries are a critical part of the Haskell ecosystem. I
want to thank you for overseeing the maintenance of this infrastructure.

However, for the last three weeks the release candidate for GHC 8.8.1
has been ready aside from releases of a couple of our core libraries.

Naturally, delays like this make it hard for GHC to maintain its faster
release cycle. At the same time, we do not want this cadence to impose
an undue burden on our core library maintainers.

How do you think we might speed up this process?

For instance, perhaps the GHC release manager could pick up
some of the "boring parts" of core library maintenance limited to:

 * Version bound bumps

 * Changes of CPP conditionals to accommodate changes in the
   compiler and other core libraries

 * Changelog entries to describe these releases

 * Uploading these releases or revisions to Hackage

Of course, this would merely be an offer to maintainers; this would be
GHC's way of carrying some of the burden that our release process
imposes.

In general, I am interested in a discussion on how to make this faster
release pace work. Ideas?

Cheers,

- Ben

_______________________________________________
Ghc-devops-group mailing list
[hidden email]
https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group
_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries

_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

RE: [GHC DevOps Group] Making GHC's fast release cadence work

Haskell - Libraries mailing list
In reply to this post by Boespflug, Mathieu

It seems to me that both questions are sensible ones:

 

  1. What should GHC’s release cadence be?  For example, would annual be better than semi-annual?  Herbert and Michael both think so, and they know a thing or two.   I’d love to hear from more people on this question.

 

  1. When we do make a release, whenever that is, how we can work most productively with library authors, so that the release comes out in a timely way?  This was Ben’s original question, and it applies regardless of cadence.

 

Simon

 

From: Ghc-devops-group <[hidden email]> On Behalf Of Michael Snoyman
Sent: 02 July 2019 05:54
To: Herbert Valerio Riedel <[hidden email]>
Cc: David Feuer <[hidden email]>; [hidden email]; Ashley Yakeley <[hidden email]>; Judah Jacobson <[hidden email]>; David Terei <[hidden email]>; Herbert Valerio Riedel <[hidden email]>; Haskell Libraries ([hidden email]) <[hidden email]>
Subject: Re: [GHC DevOps Group] Making GHC's fast release cadence work

 

 

 

On Tue, Jul 2, 2019 at 12:12 AM Herbert Valerio Riedel <[hidden email]> wrote:

On Mon, Jul 1, 2019 at 8:09 PM Ben Gamari <[hidden email]> wrote:
> Naturally, delays like this make it hard for GHC to maintain its faster
release cycle
> ...
> How do you think we might speed up this process?

IMO You're asking the wrong question.

This seems based on a premise that everyone agreed with a faster
release cycle... to me the downsides on the ecosystem and
infrastructure of a faster release churn significantly outweight the
modest benefit some people might perceive; and the issue's we've been
observing (not only boot libraries, but also the 10k packages on
Hackage) with the overspeeded release cycle are IMO a sign that we're
moving faster than the ecosystem can accommodate. Just because GHC HQ
managed to optimize their release processes (and I have to say, at the
expense of the release quality -- GHC 8.6.5 was the first major
release since GHC's beginning to require five attempts -- and I have
to note that this results in annoying busy work for GHC packagers like
myself) doesn't mean that everyone else has the time and energy to
adapt to this new order as well.

 

+1 


_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: [GHC DevOps Group] Making GHC's fast release cadence work

Matthew Pickering
Six months is even quite a long time between releases. What do other
projects do?

IMO, the main cause of this churn is the very incremental painful
refactoring of the core type classes. Are there any other changes
which regularly require library intervention?

Cheers,

Matt

On Tue, Jul 2, 2019 at 1:16 PM Simon Peyton Jones via Libraries
<[hidden email]> wrote:

>
> It seems to me that both questions are sensible ones:
>
>
>
> What should GHC’s release cadence be?  For example, would annual be better than semi-annual?  Herbert and Michael both think so, and they know a thing or two.   I’d love to hear from more people on this question.
>
>
>
> When we do make a release, whenever that is, how we can work most productively with library authors, so that the release comes out in a timely way?  This was Ben’s original question, and it applies regardless of cadence.
>
>
>
> Simon
>
>
>
> From: Ghc-devops-group <[hidden email]> On Behalf Of Michael Snoyman
> Sent: 02 July 2019 05:54
> To: Herbert Valerio Riedel <[hidden email]>
> Cc: David Feuer <[hidden email]>; [hidden email]; Ashley Yakeley <[hidden email]>; Judah Jacobson <[hidden email]>; David Terei <[hidden email]>; Herbert Valerio Riedel <[hidden email]>; Haskell Libraries ([hidden email]) <[hidden email]>
> Subject: Re: [GHC DevOps Group] Making GHC's fast release cadence work
>
>
>
>
>
>
>
> On Tue, Jul 2, 2019 at 12:12 AM Herbert Valerio Riedel <[hidden email]> wrote:
>
> On Mon, Jul 1, 2019 at 8:09 PM Ben Gamari <[hidden email]> wrote:
> > Naturally, delays like this make it hard for GHC to maintain its faster
> release cycle
> > ...
> > How do you think we might speed up this process?
>
> IMO You're asking the wrong question.
>
> This seems based on a premise that everyone agreed with a faster
> release cycle... to me the downsides on the ecosystem and
> infrastructure of a faster release churn significantly outweight the
> modest benefit some people might perceive; and the issue's we've been
> observing (not only boot libraries, but also the 10k packages on
> Hackage) with the overspeeded release cycle are IMO a sign that we're
> moving faster than the ecosystem can accommodate. Just because GHC HQ
> managed to optimize their release processes (and I have to say, at the
> expense of the release quality -- GHC 8.6.5 was the first major
> release since GHC's beginning to require five attempts -- and I have
> to note that this results in annoying busy work for GHC packagers like
> myself) doesn't mean that everyone else has the time and energy to
> adapt to this new order as well.
>
>
>
> +1
>
> _______________________________________________
> Libraries mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

RE: [GHC DevOps Group] Making GHC's fast release cadence work

Henning Thielemann
In reply to this post by Haskell - Libraries mailing list

On Tue, 2 Jul 2019, Simon Peyton Jones via Libraries wrote:

>  1. What should GHC’s release cadence be?  For example, would annual be
>     better than semi-annual?  Herbert and
>     Michael both think so, and they know a thing or two.   I’d love to
>     hear from more people on this question.

Annual is better for me. I do not need the latest and greatest GHC
features immediately. I am happy to get bugfixes quickly and easily via
hvr's ghc ppa.

However, I'd prefer that libraries are separated from GHC and 'base' would
be split.
_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: [GHC DevOps Group] Making GHC's fast release cadence work

John Ericson-2
Yeah there are many interlocking issues here. I've been thinking about
writing one of those ecosystem proposals about this. Here's some
observations about the status quo for now.

- Frequent releases are a really nice for feature for the community. I
know I, for one, would be less motivated to work on GHC if it would take
a year in some case for my contributions to trickle back to my day job.
Rust releases every 6 *weeks*. I see no reason why we shouldn't aim for
that.

- The GitLab move and switch to CI have made GHC development more
accessible and, temporary Marge Bot issues aside, better scaling. I am
immensely grateful for those that put in the work for that every MR I
write. Going back to annual releases would feel like a regression,
cancelling out some of those gains.

- Base really is a PVP mess. Only the messy guts actually make sense to
pair with GHC releases, a breaking change every time. The "pure" bits
would naturally have *fewer* breaking releases, but *more* non-breaking
releases.

- AMP-like proposals could probably move faster if they weren't tied to
GHC releases. I.e. if multiple GHC releases supported multiple class
hierarchies, people could move at their own pace to some extent, and we
could support faster *and* slower migrations.

- I hope GHC can move in a more modular direction, which by definition
means farming off work to libraries. In other words, I think GHC should
ultimately have the same degree of code reuse as any other Haskell
project. But the current experience with submodules is excruciating, so
no one dealing with GHC every day rightfully has any appetite for that.
It's a pity because once it's done I think it would be *less* work for
the core GHC team, there's just the awkward transition out of today's
monolithic local maximum.

So all this needs leads me to conclude a few things

- Base must be broken up, so it can be released more often than GHC.
Base would continue to exist reexport code in other libraries
exclusively. New users, executables, and legacy code can continue to
depend on base as before, but libraries are encouraged to adopt more
fine-grained dependencies on the underlying libraries.

- the kicker: Any library that wishes to be included in GHC / base *must
share maintainership* with the GHC release team / libraries committee,
respectively/./ That means those groups can release things whenever they
want to unblock themselves.
//

It's fairly clear-cut to me: having your library being included in GHC
or base is a big honor, and so you should not be able to hold up the
release process of these important projects unilaterally. If you want to
keep full custody of your baby, that's fine, but it's worth not the
precious time of the GHC/base developers to use it then. [Aim for, say,
Haskell Platform inclusion rather than base inclusion.]

Granted, GHC, unlike Base, there is the alternative to use Cabal's new
support for multiple versions of a package in the same package DB. But
until will have private dependencies that would place some probably
unpopular constraints on consumers of GHC-the-library. If GHC does
migrate from ~10 to ~100 dependencies, maybe it might make sense to
float that as an alternative, but seeing that is many GHC releases away
no matter what, I think it's best to start with the shared
maintainership policy.

  John

On 7/2/19 8:36 AM, Henning Thielemann wrote:

>
> On Tue, 2 Jul 2019, Simon Peyton Jones via Libraries wrote:
>
>>  1. What should GHC’s release cadence be? For example, would annual be
>>     better than semi-annual?  Herbert and
>>     Michael both think so, and they know a thing or two.   I’d love to
>>     hear from more people on this question.
>
> Annual is better for me. I do not need the latest and greatest GHC
> features immediately. I am happy to get bugfixes quickly and easily
> via hvr's ghc ppa.
>
> However, I'd prefer that libraries are separated from GHC and 'base'
> would be split.
>
> _______________________________________________
> Libraries mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

RE: [GHC DevOps Group] Making GHC's fast release cadence work

Haskell - Libraries mailing list
| - the kicker: Any library that wishes to be included in GHC / base *must
| share maintainership* with the GHC release team / libraries committee,
| respectively/./ That means those groups can release things whenever they
| want to unblock themselves.

That seems very plausible to me.   But "sharing maintainership" could be limited to:

  The GHC release team is free to make a minor (patch-level)
  release of an existing released version of the library L,
  embodying any changes necessary to make the library work
  with the new version of GHC

I think that is all that's needed. In general GHC should not rely
on a bleeding-edge release of a library.  But it's not uncommon
for tiny changes to be needed, and by sharing maintainership, the
library author would not need to be bothered about making and
releasing those changes.

It's worth noting that it's not so much "libraries that want to
be included in GHC", but rather "libraries that GHC's authors
would like to import in GHC's own source code, rather than
re-implementing that functionality (which would be silly)".

Simon

| -----Original Message-----
| From: John Ericson <[hidden email]>
| Sent: 02 July 2019 17:00
| To: Henning Thielemann <[hidden email]>; Simon Peyton Jones
| <[hidden email]>
| Cc: [hidden email]; Michael Snoyman <[hidden email]>;
| David Terei <[hidden email]>; Ashley Yakeley <[hidden email]>;
| Herbert Valerio Riedel <[hidden email]>; Haskell Libraries
| ([hidden email]) <[hidden email]>
| Subject: Re: [GHC DevOps Group] Making GHC's fast release cadence work
|
| Yeah there are many interlocking issues here. I've been thinking about
| writing one of those ecosystem proposals about this. Here's some
| observations about the status quo for now.
|
| - Frequent releases are a really nice for feature for the community. I know
| I, for one, would be less motivated to work on GHC if it would take a year
| in some case for my contributions to trickle back to my day job.
| Rust releases every 6 *weeks*. I see no reason why we shouldn't aim for
| that.
|
| - The GitLab move and switch to CI have made GHC development more
| accessible and, temporary Marge Bot issues aside, better scaling. I am
| immensely grateful for those that put in the work for that every MR I
| write. Going back to annual releases would feel like a regression,
| cancelling out some of those gains.
|
| - Base really is a PVP mess. Only the messy guts actually make sense to
| pair with GHC releases, a breaking change every time. The "pure" bits would
| naturally have *fewer* breaking releases, but *more* non-breaking releases.
|
| - AMP-like proposals could probably move faster if they weren't tied to GHC
| releases. I.e. if multiple GHC releases supported multiple class
| hierarchies, people could move at their own pace to some extent, and we
| could support faster *and* slower migrations.
|
| - I hope GHC can move in a more modular direction, which by definition
| means farming off work to libraries. In other words, I think GHC should
| ultimately have the same degree of code reuse as any other Haskell project.
| But the current experience with submodules is excruciating, so no one
| dealing with GHC every day rightfully has any appetite for that.
| It's a pity because once it's done I think it would be *less* work for the
| core GHC team, there's just the awkward transition out of today's
| monolithic local maximum.
|
| So all this needs leads me to conclude a few things
|
| - Base must be broken up, so it can be released more often than GHC.
| Base would continue to exist reexport code in other libraries exclusively.
| New users, executables, and legacy code can continue to depend on base as
| before, but libraries are encouraged to adopt more fine-grained
| dependencies on the underlying libraries.
|
| - the kicker: Any library that wishes to be included in GHC / base *must
| share maintainership* with the GHC release team / libraries committee,
| respectively/./ That means those groups can release things whenever they
| want to unblock themselves.
| //
|
| It's fairly clear-cut to me: having your library being included in GHC or
| base is a big honor, and so you should not be able to hold up the release
| process of these important projects unilaterally. If you want to keep full
| custody of your baby, that's fine, but it's worth not the precious time of
| the GHC/base developers to use it then. [Aim for, say, Haskell Platform
| inclusion rather than base inclusion.]
|
| Granted, GHC, unlike Base, there is the alternative to use Cabal's new
| support for multiple versions of a package in the same package DB. But
| until will have private dependencies that would place some probably
| unpopular constraints on consumers of GHC-the-library. If GHC does migrate
| from ~10 to ~100 dependencies, maybe it might make sense to float that as
| an alternative, but seeing that is many GHC releases away no matter what, I
| think it's best to start with the shared maintainership policy.
|
|   John
|
| On 7/2/19 8:36 AM, Henning Thielemann wrote:
| >
| > On Tue, 2 Jul 2019, Simon Peyton Jones via Libraries wrote:
| >
| >>  1. What should GHC’s release cadence be? For example, would annual be
| >>     better than semi-annual?  Herbert and
| >>     Michael both think so, and they know a thing or two.   I’d love to
| >>     hear from more people on this question.
| >
| > Annual is better for me. I do not need the latest and greatest GHC
| > features immediately. I am happy to get bugfixes quickly and easily
| > via hvr's ghc ppa.
| >
| > However, I'd prefer that libraries are separated from GHC and 'base'
| > would be split.
| >
| > _______________________________________________
| > Libraries mailing list
| > [hidden email]
| >
| https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.haske
| ll.org%2Fcgi-
| bin%2Fmailman%2Flistinfo%2Flibraries&amp;data=02%7C01%7Csimonpj%40microsoft
| .com%7C6ae01104c4b345984a0208d6ff06464e%7C72f988bf86f141af91ab2d7cd011db47%
| 7C1%7C1%7C636976799775801107&amp;sdata=YEf6lczTdY9Mp%2B6lFHzCdwcagiQWZ3VgxI
| QTfqH5ZXk%3D&amp;reserved=0
_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: [GHC DevOps Group] Making GHC's fast release cadence work

David Feuer
The biggest problems are for packages like containers, that are not only used by GHC but also exposed to users through the GHC API. These libraries aren't part of GHC or base, but pretty much have to move in lock step.

On Tue, Jul 2, 2019, 12:19 PM Simon Peyton Jones via Libraries <[hidden email]> wrote:
| - the kicker: Any library that wishes to be included in GHC / base *must
| share maintainership* with the GHC release team / libraries committee,
| respectively/./ That means those groups can release things whenever they
| want to unblock themselves.

That seems very plausible to me.   But "sharing maintainership" could be limited to:

  The GHC release team is free to make a minor (patch-level)
  release of an existing released version of the library L,
  embodying any changes necessary to make the library work
  with the new version of GHC

I think that is all that's needed. In general GHC should not rely
on a bleeding-edge release of a library.  But it's not uncommon
for tiny changes to be needed, and by sharing maintainership, the
library author would not need to be bothered about making and
releasing those changes.

It's worth noting that it's not so much "libraries that want to
be included in GHC", but rather "libraries that GHC's authors
would like to import in GHC's own source code, rather than
re-implementing that functionality (which would be silly)".

Simon

| -----Original Message-----
| From: John Ericson <[hidden email]>
| Sent: 02 July 2019 17:00
| To: Henning Thielemann <[hidden email]>; Simon Peyton Jones
| <[hidden email]>
| Cc: [hidden email]; Michael Snoyman <[hidden email]>;
| David Terei <[hidden email]>; Ashley Yakeley <[hidden email]>;
| Herbert Valerio Riedel <[hidden email]>; Haskell Libraries
| ([hidden email]) <[hidden email]>
| Subject: Re: [GHC DevOps Group] Making GHC's fast release cadence work
|
| Yeah there are many interlocking issues here. I've been thinking about
| writing one of those ecosystem proposals about this. Here's some
| observations about the status quo for now.
|
| - Frequent releases are a really nice for feature for the community. I know
| I, for one, would be less motivated to work on GHC if it would take a year
| in some case for my contributions to trickle back to my day job.
| Rust releases every 6 *weeks*. I see no reason why we shouldn't aim for
| that.
|
| - The GitLab move and switch to CI have made GHC development more
| accessible and, temporary Marge Bot issues aside, better scaling. I am
| immensely grateful for those that put in the work for that every MR I
| write. Going back to annual releases would feel like a regression,
| cancelling out some of those gains.
|
| - Base really is a PVP mess. Only the messy guts actually make sense to
| pair with GHC releases, a breaking change every time. The "pure" bits would
| naturally have *fewer* breaking releases, but *more* non-breaking releases.
|
| - AMP-like proposals could probably move faster if they weren't tied to GHC
| releases. I.e. if multiple GHC releases supported multiple class
| hierarchies, people could move at their own pace to some extent, and we
| could support faster *and* slower migrations.
|
| - I hope GHC can move in a more modular direction, which by definition
| means farming off work to libraries. In other words, I think GHC should
| ultimately have the same degree of code reuse as any other Haskell project.
| But the current experience with submodules is excruciating, so no one
| dealing with GHC every day rightfully has any appetite for that.
| It's a pity because once it's done I think it would be *less* work for the
| core GHC team, there's just the awkward transition out of today's
| monolithic local maximum.
|
| So all this needs leads me to conclude a few things
|
| - Base must be broken up, so it can be released more often than GHC.
| Base would continue to exist reexport code in other libraries exclusively.
| New users, executables, and legacy code can continue to depend on base as
| before, but libraries are encouraged to adopt more fine-grained
| dependencies on the underlying libraries.
|
| - the kicker: Any library that wishes to be included in GHC / base *must
| share maintainership* with the GHC release team / libraries committee,
| respectively/./ That means those groups can release things whenever they
| want to unblock themselves.
| //
|
| It's fairly clear-cut to me: having your library being included in GHC or
| base is a big honor, and so you should not be able to hold up the release
| process of these important projects unilaterally. If you want to keep full
| custody of your baby, that's fine, but it's worth not the precious time of
| the GHC/base developers to use it then. [Aim for, say, Haskell Platform
| inclusion rather than base inclusion.]
|
| Granted, GHC, unlike Base, there is the alternative to use Cabal's new
| support for multiple versions of a package in the same package DB. But
| until will have private dependencies that would place some probably
| unpopular constraints on consumers of GHC-the-library. If GHC does migrate
| from ~10 to ~100 dependencies, maybe it might make sense to float that as
| an alternative, but seeing that is many GHC releases away no matter what, I
| think it's best to start with the shared maintainership policy.
|
|   John
|
| On 7/2/19 8:36 AM, Henning Thielemann wrote:
| >
| > On Tue, 2 Jul 2019, Simon Peyton Jones via Libraries wrote:
| >
| >>  1. What should GHC’s release cadence be? For example, would annual be
| >>     better than semi-annual?  Herbert and
| >>     Michael both think so, and they know a thing or two.   I’d love to
| >>     hear from more people on this question.
| >
| > Annual is better for me. I do not need the latest and greatest GHC
| > features immediately. I am happy to get bugfixes quickly and easily
| > via hvr's ghc ppa.
| >
| > However, I'd prefer that libraries are separated from GHC and 'base'
| > would be split.
| >
| > _______________________________________________
| > Libraries mailing list
| > [hidden email]
| >
| https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.haske
| ll.org%2Fcgi-
| bin%2Fmailman%2Flistinfo%2Flibraries&amp;data=02%7C01%7Csimonpj%40microsoft
| .com%7C6ae01104c4b345984a0208d6ff06464e%7C72f988bf86f141af91ab2d7cd011db47%
| 7C1%7C1%7C636976799775801107&amp;sdata=YEf6lczTdY9Mp%2B6lFHzCdwcagiQWZ3VgxI
| QTfqH5ZXk%3D&amp;reserved=0
_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries

_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: [GHC DevOps Group] Making GHC's fast release cadence work

Ashley Yakeley
In reply to this post by Haskell - Libraries mailing list
For the time library, I would rather the GHC team make a pull request first, and let me do the release. At the very least I want to know what's going on and what the needs are.

-- Ashley

On Tue, 2019-07-02 at 16:18 +0000, Simon Peyton Jones wrote:

  The GHC release team is free to make a minor (patch-level)
  release of an existing released version of the library L,
  embodying any changes necessary to make the library work
  with the new version of GHC



_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

RE: [GHC DevOps Group] Making GHC's fast release cadence work

Ben Gamari-2
In reply to this post by Haskell - Libraries mailing list
Simon Peyton Jones via Libraries <[hidden email]> writes:

> | - the kicker: Any library that wishes to be included in GHC / base *must
> | share maintainership* with the GHC release team / libraries committee,
> | respectively/./ That means those groups can release things whenever they
> | want to unblock themselves.
>
> That seems very plausible to me.   But "sharing maintainership" could be limited to:
>
>   The GHC release team is free to make a minor (patch-level)
>   release of an existing released version of the library L,
>   embodying any changes necessary to make the library work
>   with the new version of GHC
>
> I think that is all that's needed. In general GHC should not rely
> on a bleeding-edge release of a library.  But it's not uncommon
> for tiny changes to be needed, and by sharing maintainership, the
> library author would not need to be bothered about making and
> releasing those changes.
>
Yes, this is precisely what I was trying to articulate in my original
email (although failed to do so clearly). We would not want to be
responsible for managing major releases; merely incremental patch-level
releases to patch things up to be shippable with GHC.

Cheers,

- Ben


_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [GHC DevOps Group] Making GHC's fast release cadence work

Ben Gamari-2
In reply to this post by David Feuer
David Feuer <[hidden email]> writes:

> The biggest problems are for packages like containers, that are not only
> used by GHC but also exposed to users through the GHC API. These libraries
> aren't part of GHC or base, but pretty much have to move in lock step.
>
I'm not sure I understand why this is so. Yes, install plans involving
the GHC library are forced to use the same version of containers that
GHC uses, but I would think that this is not the common case.

Assuming most people aren't linking against the GHC library then I don't
see the harm in GHC staying a bit behind upstreams like containers.

Cheers,

- Ben


_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [GHC DevOps Group] Making GHC's fast release cadence work

Ben Gamari-2
In reply to this post by Ashley Yakeley
Ashley Yakeley <[hidden email]> writes:

> For the time library, I would rather the GHC team make a pull request
> first, and let me do the release. At the very least I want to know
> what's going on and what the needs are.
>
As long as the review turnaround is reasonably fast (which it always has
been in the case of time; thank you!), I am perfectly content with this.

Cheers,

- Ben

_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [GHC DevOps Group] Making GHC's fast release cadence work

David Feuer
In reply to this post by Ben Gamari-2
It seems to mostly be an issue for stack, which wants a fully consistent package set.

On Tue, Jul 2, 2019, 12:54 PM Ben Gamari <[hidden email]> wrote:
David Feuer <[hidden email]> writes:

> The biggest problems are for packages like containers, that are not only
> used by GHC but also exposed to users through the GHC API. These libraries
> aren't part of GHC or base, but pretty much have to move in lock step.
>
I'm not sure I understand why this is so. Yes, install plans involving
the GHC library are forced to use the same version of containers that
GHC uses, but I would think that this is not the common case.

Assuming most people aren't linking against the GHC library then I don't
see the harm in GHC staying a bit behind upstreams like containers.

Cheers,

- Ben


_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: [GHC DevOps Group] Making GHC's fast release cadence work

John Ericson-2
In reply to this post by Ben Gamari-2
> The GHC release team is free to make a minor (patch-level) release of
> an existing released version of the library L, embodying any changes
> necessary to make the library work with the new version of GHC

Yeah I like that limitation. It is good to make my proposal less
draconian :) Happy to hear that's what you wanted all along Ben!


Ashley:

> For the time library, I would rather the GHC team make a pull request
> first, and let me do the release. At the very least I want to know
> what's going on and what the needs are.

Adding to what Ben said, I think the way submodules work today mean that
unmerged (e.g. just in the GHC fork) changes to a submodule block a PR?
So only in repos where the PR happens to live in the same repo as master
can GHC use the PR branch and still pass CI, and the usual case is the
PR must be merged before GHC relies on it. I hope in the majority of
cases these GHC-driven releases would just take stuff that is on master
but unreleased, and therefore already reviewed the normal way.


David:

> It seems to mostly be an issue for stack, which wants a fully
> consistent package set.

Stackage should be comfortable building GHC itself with a newer
containers then, I think. (And likewise for anyone that needs a newer
containers in their build of the GHC library.)

John

On 7/2/19 12:46 PM, Ben Gamari wrote:

> Simon Peyton Jones via Libraries <[hidden email]> writes:
>
>> | - the kicker: Any library that wishes to be included in GHC / base *must
>> | share maintainership* with the GHC release team / libraries committee,
>> | respectively/./ That means those groups can release things whenever they
>> | want to unblock themselves.
>>
>> That seems very plausible to me.   But "sharing maintainership" could be limited to:
>>
>>
>>
>> I think that is all that's needed. In general GHC should not rely
>> on a bleeding-edge release of a library.  But it's not uncommon
>> for tiny changes to be needed, and by sharing maintainership, the
>> library author would not need to be bothered about making and
>> releasing those changes.
>>
> Yes, this is precisely what I was trying to articulate in my original
> email (although failed to do so clearly). We would not want to be
> responsible for managing major releases; merely incremental patch-level
> releases to patch things up to be shippable with GHC.
>
> Cheers,
>
> - Ben
>
_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: [GHC DevOps Group] Making GHC's fast release cadence work

Henning Thielemann
In reply to this post by John Ericson-2

On Tue, 2 Jul 2019, John Ericson wrote:

> Yeah there are many interlocking issues here. I've been thinking about
> writing one of those ecosystem proposals about this. Here's some observations
> about the status quo for now.
>
> - Frequent releases are a really nice for feature for the community. I know
> I, for one, would be less motivated to work on GHC if it would take a year in
> some case for my contributions to trickle back to my day job. Rust releases
> every 6 *weeks*. I see no reason why we shouldn't aim for that.

In the past we had nightly builds. They are not quite releases but if
provided as, say, Debian packages it would be easy to install and upgrade
them.
_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: [GHC DevOps Group] Making GHC's fast release cadence work

Ben Gamari-3
For what it's worth, we do provide nightly builds as a natural result of our CI infrastructure. Perhaps we should draw more attention to this fact.

Cheers,

Ben

On July 6, 2019 2:21:33 AM EDT, Henning Thielemann <[hidden email]> wrote:

On Tue, 2 Jul 2019, John Ericson wrote:

Yeah there are many interlocking issues here. I've been thinking about
writing one of those ecosystem proposals about this. Here's some observations
about the status quo for now.

- Frequent releases are a really nice for feature for the community. I know
I, for one, would be less motivated to work on GHC if it would take a year in
some case for my contributions to trickle back to my day job. Rust releases
every 6 *weeks*. I see no reason why we shouldn't aim for that.

In the past we had nightly builds. They are not quite releases but if
provided as, say, Debian packages it would be easy to install and upgrade
them.
Ghc-devops-group mailing list
[hidden email]
https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: [GHC DevOps Group] Making GHC's fast release cadence work

Eric Seidel-3
In reply to this post by David Feuer
Not just stack, system package managers want fully consistent package sets too. This is something I have to deal with occasionally when I do updates to our internal Haskell package set at work.

On Tue, Jul 2, 2019, at 12:58, David Feuer wrote:

> It seems to mostly be an issue for stack, which wants a fully
> consistent package set.
>
> On Tue, Jul 2, 2019, 12:54 PM Ben Gamari <[hidden email]> wrote:
> > David Feuer <[hidden email]> writes:
> >
> >  > The biggest problems are for packages like containers, that are not only
> >  > used by GHC but also exposed to users through the GHC API. These libraries
> >  > aren't part of GHC or base, but pretty much have to move in lock step.
> >  >
> >  I'm not sure I understand why this is so. Yes, install plans involving
> >  the GHC library are forced to use the same version of containers that
> >  GHC uses, but I would think that this is not the common case.
> >
> >  Assuming most people aren't linking against the GHC library then I don't
> >  see the harm in GHC staying a bit behind upstreams like containers.
> >
> >  Cheers,
> >
> >  - Ben
> >
> _______________________________________________
> Libraries mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
>
_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: [GHC DevOps Group] Making GHC's fast release cadence work

Herbert Valerio Riedel-3
In reply to this post by David Feuer
> The biggest problems are for packages like containers, that are not only used by GHC but also exposed to users through the GHC API. These libraries aren't part of GHC or base, but pretty much have to move in lock step.

fwiw, this what
https://mail.haskell.org/pipermail/ghc-devs/2017-July/014424.html aims
to address and would help to reduce this artificial boot-library
version pinning for consumers. Moritz has been pushing this effort
further via https://gitlab.haskell.org/ghc/ghc/merge_requests/490 but
it didn't make it into GHC 8.8 yet.
_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries