GHCs dependencies (libraries) and maintenance

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

GHCs dependencies (libraries) and maintenance

Moritz Angermann-2
Hi there!

so this comes up periodically, and I think we need to discuss this.  This is not
related to anything right now, so if you wonder if I'm writing this because of
something that just happened that I'm involved and you might have missed
something, you probably did not.  It came up on the #ghc IRC channel a
few day ago.

GHC depends on quite a set of libraries, and ships those in releases. When ever
a new GHC release is cut, all these dependencies need to be on hackage and
have release versions.  We do not want to produce a GHC release which depends
on in-flight packages.  In-flight might happen for example due to GHC having to
patch dependencies to make them work with HEAD.

Everyone who maintains any kind of software online, knows that maintenance can
be a drag, and then life happens, and what not.  There are many very responsive
maintainers and we all owe them a grate amount of gratitude towards their
relentless work, keeping those libraries up to date and responding to questions,
patches, ...

I therefore would like to float the following idea to make the GHC
release processes
a bit more reliable.  GHCHQ (that is those in charge of producing GHC
releases for
us all), are made co-maintainers on each library GHC depends on, to guarantee
that GHC can move forward in the worst of circumstances.  Now I would
hope that in
almost all cases GHCHQ would never have to maintain any of the dependencies
actively, they deal with GHC already, so let's try to keep it that
way.  However GHCHQ
can, after a 14day notification period, exercise the co-maintainance
and cut releases
(and upload them to hackage), should the maintainer not be able to do
so on his own
for various reasons.

I'd like to see this as an insurance policy for GHC continuous
development.  The only
alternative that I see would be that GHCHQ starts forking dependencies
and initiates
the hackage maintainer takeover protocol, which will cause additional
delays, and
incurs an extra burden on the GHC maintainers.

I hope we can all agree that libraries that end up being dependencies
of GHC should
be held in high regards and form the very foundation GHC is built
upon.  As such it
should be an honour to have GHCHQ being a co-maintainer for ones library, as it
signifies that importances of the library for the continuous development of GHC.

Again I don't expect much to change, except for GHCHQ becoming co-maintainers
for libraries GHC depends on. The baseline expectation will remain as
it is.  However we
will have ensured the frictionless development of GHC going forward.

Cheers,
 Moritz
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: GHCs dependencies (libraries) and maintenance

John Cotton Ericson
I completely agree. I would like GHC be able to use *more* of the
ecosystem than it does today, and there's no way we can can do that
unless we dramatically decrease the pain per dependency.

John

On 6/1/20 5:23 AM, Moritz Angermann wrote:

> Hi there!
>
> so this comes up periodically, and I think we need to discuss this.  This is not
> related to anything right now, so if you wonder if I'm writing this because of
> something that just happened that I'm involved and you might have missed
> something, you probably did not.  It came up on the #ghc IRC channel a
> few day ago.
>
> GHC depends on quite a set of libraries, and ships those in releases. When ever
> a new GHC release is cut, all these dependencies need to be on hackage and
> have release versions.  We do not want to produce a GHC release which depends
> on in-flight packages.  In-flight might happen for example due to GHC having to
> patch dependencies to make them work with HEAD.
>
> Everyone who maintains any kind of software online, knows that maintenance can
> be a drag, and then life happens, and what not.  There are many very responsive
> maintainers and we all owe them a grate amount of gratitude towards their
> relentless work, keeping those libraries up to date and responding to questions,
> patches, ...
>
> I therefore would like to float the following idea to make the GHC
> release processes
> a bit more reliable.  GHCHQ (that is those in charge of producing GHC
> releases for
> us all), are made co-maintainers on each library GHC depends on, to guarantee
> that GHC can move forward in the worst of circumstances.  Now I would
> hope that in
> almost all cases GHCHQ would never have to maintain any of the dependencies
> actively, they deal with GHC already, so let's try to keep it that
> way.  However GHCHQ
> can, after a 14day notification period, exercise the co-maintainance
> and cut releases
> (and upload them to hackage), should the maintainer not be able to do
> so on his own
> for various reasons.
>
> I'd like to see this as an insurance policy for GHC continuous
> development.  The only
> alternative that I see would be that GHCHQ starts forking dependencies
> and initiates
> the hackage maintainer takeover protocol, which will cause additional
> delays, and
> incurs an extra burden on the GHC maintainers.
>
> I hope we can all agree that libraries that end up being dependencies
> of GHC should
> be held in high regards and form the very foundation GHC is built
> upon.  As such it
> should be an honour to have GHCHQ being a co-maintainer for ones library, as it
> signifies that importances of the library for the continuous development of GHC.
>
> Again I don't expect much to change, except for GHCHQ becoming co-maintainers
> for libraries GHC depends on. The baseline expectation will remain as
> it is.  However we
> will have ensured the frictionless development of GHC going forward.
>
> Cheers,
>   Moritz
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: GHCs dependencies (libraries) and maintenance

Oleg Grenrus
In reply to this post by Moritz Angermann-2
I want to ask a honest question: Who is GHC HQ? There is a page listing
who is on

- Haskell.org committee,
- on core libraries committee (CLC),
- GHC steering committee;
- there is a list of named maintainers for core libraries,
- and it is relatively easily to deduce who maintains other tools and
libraries relevant to GHC (e.g. Cabal or say `shake` for hadrian).

But who are in GHC HQ? Neither google nor gitlab.haskell.org search give
any hints.
---

I think the core issue here is bad or insufficient communication.
GHC-8.12 beta release is planned to happen in three months. There are
patches still in flight (e.g. for process, Cabal has tracking issue
open). I'm sure that maintainers are able to ack on that. Maybe start by
requiring these explicit acknowledgments?

I also want to remind about first GHC-8.10 schedule posted on this list

     October  18 2019:  start of one week freeze in preparation for
branching
     October  25 2019:  ghc-8.10 branch cut
     November 8  2019:  8.10.1-alpha1
     November 22 2019:  8.10.1-alpha2
     December 6  2019:  8.10.1-alpha3
     December 20 2019:  8.10.1-rc1
     January  10 2020:  Final 8.10.1 release

I haven't seen *any* updates to that. GHC-8.10.1 was released in end of
March 2020, *three months later*, text issue was urgent in beginning of
February.
The plan for GHC-8.12 is to

  * Mid-June 2020: Aim to have all major features in the tree
  * Late-June 2020: Cut the ghc-8.12 branch
  * June - August 2020: 3 alpha releases
  * 1 September 2020: beta release
  * 25 September 2020: Final 8.12.1 release

are we still on track?

What I'm trying to say, it is that it is hard to believe that issue is
urgent for GHC. It's not the first time (relating to ghc-8.10) when
schedules are not been kept.
And this is one of reasons why I opened an issue about "Annual release
cycle, its structure and long-term-support".

- Oleg

On 1.6.2020 12.23, Moritz Angermann wrote:

> Hi there!
>
> so this comes up periodically, and I think we need to discuss this.  This is not
> related to anything right now, so if you wonder if I'm writing this because of
> something that just happened that I'm involved and you might have missed
> something, you probably did not.  It came up on the #ghc IRC channel a
> few day ago.
>
> GHC depends on quite a set of libraries, and ships those in releases. When ever
> a new GHC release is cut, all these dependencies need to be on hackage and
> have release versions.  We do not want to produce a GHC release which depends
> on in-flight packages.  In-flight might happen for example due to GHC having to
> patch dependencies to make them work with HEAD.
>
> Everyone who maintains any kind of software online, knows that maintenance can
> be a drag, and then life happens, and what not.  There are many very responsive
> maintainers and we all owe them a grate amount of gratitude towards their
> relentless work, keeping those libraries up to date and responding to questions,
> patches, ...
>
> I therefore would like to float the following idea to make the GHC
> release processes
> a bit more reliable.  GHCHQ (that is those in charge of producing GHC
> releases for
> us all), are made co-maintainers on each library GHC depends on, to guarantee
> that GHC can move forward in the worst of circumstances.  Now I would
> hope that in
> almost all cases GHCHQ would never have to maintain any of the dependencies
> actively, they deal with GHC already, so let's try to keep it that
> way.  However GHCHQ
> can, after a 14day notification period, exercise the co-maintainance
> and cut releases
> (and upload them to hackage), should the maintainer not be able to do
> so on his own
> for various reasons.
>
> I'd like to see this as an insurance policy for GHC continuous
> development.  The only
> alternative that I see would be that GHCHQ starts forking dependencies
> and initiates
> the hackage maintainer takeover protocol, which will cause additional
> delays, and
> incurs an extra burden on the GHC maintainers.
>
> I hope we can all agree that libraries that end up being dependencies
> of GHC should
> be held in high regards and form the very foundation GHC is built
> upon.  As such it
> should be an honour to have GHCHQ being a co-maintainer for ones library, as it
> signifies that importances of the library for the continuous development of GHC.
>
> Again I don't expect much to change, except for GHCHQ becoming co-maintainers
> for libraries GHC depends on. The baseline expectation will remain as
> it is.  However we
> will have ensured the frictionless development of GHC going forward.
>
> Cheers,
>   Moritz
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: GHCs dependencies (libraries) and maintenance

Ben Gamari-3
Oleg Grenrus <[hidden email]> writes:

> I want to ask a honest question: Who is GHC HQ? There is a page listing
> who is on
>
> - Haskell.org committee,
> - on core libraries committee (CLC),
> - GHC steering committee;
> - there is a list of named maintainers for core libraries,
> - and it is relatively easily to deduce who maintains other tools and
> libraries relevant to GHC (e.g. Cabal or say `shake` for hadrian).
>
> But who are in GHC HQ? Neither google nor gitlab.haskell.org search give
> any hints.
I frankly have never liked the term GHC HQ for this reason and generally
avoid using it. I would guess that in the context of Moritz's proposal
it means "the GHC release manager", which is currently me.

> I think the core issue here is bad or insufficient communication.
> GHC-8.12 beta release is planned to happen in three months. There are
> patches still in flight (e.g. for process, Cabal has tracking issue
> open). I'm sure that maintainers are able to ack on that. Maybe start by
> requiring these explicit acknowledgments?
>
If this is what it would take I would be happy to perform such
one-on-one handshakes. However, I will say that most maintainers
(thankfully) are very responsive under the current scheme. There are
only a few which have extremely variable communications latencies,
despite many attempts at reaching them. I'm skeptical that adding more
attempts will somehow change this situation.

> I also want to remind about first GHC-8.10 schedule posted on this list
>
>      October  18 2019:  start of one week freeze in preparation for
> branching
>      October  25 2019:  ghc-8.10 branch cut
>      November 8  2019:  8.10.1-alpha1
>      November 22 2019:  8.10.1-alpha2
>      December 6  2019:  8.10.1-alpha3
>      December 20 2019:  8.10.1-rc1
>      January  10 2020:  Final 8.10.1 release
>
> I haven't seen *any* updates to that. GHC-8.10.1 was released in end of
> March 2020, *three months later*, text issue was urgent in beginning of
> February.
>
Frankly, the text issue was urgent well before February. I made several
attempts over weeks to get in touch with Herbert to no avail. It
was only in February when it was essentially the *only*
record.

That being said, I will say that there should have been more
communication about the state of the release when the initial release
date came and went. 8.10.1-rc1 was cut two weeks after the scheduled
release date; at this point I expected to be a week or so away from
final. However, it then took months to get releases for a few
submodules. All of this should have been clearly communicated.

> The plan for GHC-8.12 is to
>
>   * Mid-June 2020: Aim to have all major features in the tree
>   * Late-June 2020: Cut the ghc-8.12 branch
>   * June - August 2020: 3 alpha releases
>   * 1 September 2020: beta release
>   * 25 September 2020: Final 8.12.1 release
>
> are we still on track?
>
Yes, we are. This is precisely why I am trying to start this process
earlier. I currently have the base version bump underway (!3335) and am
working on hammering out some remaining Windows issues so we can have a
stable fork point.

However, in general the problem is that it is incredibly hard to know
whether we are *actually* on track schedule-wise when something as small
as a bounds bump may take weeks or months to be merged.

> What I'm trying to say, it is that it is hard to believe that issue is
> urgent for GHC. It's not the first time (relating to ghc-8.10) when
> schedules are not been kept.
>
I would be happy to send bi-weekly release engineering updates if this
would help upstreams better understand where we are in the release
process.

> And this is one of reasons why I opened an issue about "Annual release
> cycle, its structure and long-term-support".
>
I have had an editor with my draft response open for the last week.
Apologies that it has taken so long to finish but things have been busy
over the last week due to the imminent fork.

Cheers,

- Ben

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: GHCs dependencies (libraries) and maintenance

Ben Gamari-3
In reply to this post by Moritz Angermann-2
Moritz Angermann <[hidden email]> writes:

> Hi there!
>
> Again I don't expect much to change, except for GHCHQ becoming
> co-maintainers for libraries GHC depends on. The baseline expectation
> will remain as it is. However we will have ensured the frictionless
> development of GHC going forward.
>
While I think this would indeed help and wouldn't mind seeing this idea
implemented, last time it was floated there was quite some resistance
from some maintainers.

Really, I think the problems with core library maintenance that GHC
feels are bigger than GHC. Afterall, if it takes weeks for a patch from
the GHC release manager to make its way into a core library, it will
almost certainly take the same time, if not longer, for a patch from a
non-GHC contributor to be accepted. We see this in the large number of
outstanding merge requests in some of the core library repositories.

This isn't a problem that GHC can solve; the bus number for many of our
core packages is simply too low. I think the way to fix this is to
ensure that core libraries have a sufficiently large pool of maintainers
that responsiveness is not a question.

Of course, fixing this isn't easy. Our core libraries *need* to maintain
a high standard of quality and enforcing that standard requires effort
on the part of skilled maintainers. Finding maintainers who have the
desire, time, and skill to fill this role is hard. That being said, I do
feel that currently outside contributors don't feel welcome enough to
even offer to help, largely *because* of the rather minimal maintenance
that some core packages see. There is something of a chicken-and-egg
problem here.

I would also like to reiterate that the above doesn't describe all core
libraries. Most of our core library maintainers are impressively
responsive and this is something for which I am very grateful.

Moreover, while I do believe that the discussion above points to a real
problem, I want to make it very clear that all of our core library
maintainers deserve recognition for the difficult role that they fill:
hand-holding contributors, triaging tickets, reviewing merge requests is
hard work and the success of our entire ecosystem rests of their
efforts. The fact that Haskell has seen such growth in the past years is
in large part the direct result of their work which has made Haskell
such a joy to use.

Cheers,

- Ben

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (497 bytes) Download Attachment