Nightlies

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Nightlies

Mateusz Kowalczyk
Hi all,

I'd just like to query the status of the nightly builds. Is anything
happening in that area? [1] is right on the front page of the GHC Trac
even though no builds were ran for ~5 months. Perhaps it should be
moved out of the way if there's no plan to resume these in the near
future.

Does anything specific need doing to get these to run again?

[1]: https://ghc.haskell.org/trac/ghc/wiki/Builder
--
Mateusz K.

Reply | Threaded
Open this post in threaded view
|

Nightlies

Austin Seipp-5
As of right now, Pali's FreeBSD builds seem to be the only nightly
that is still consistently running (and thanks to him for that!)

The build infrastructure in its current status is mainly just
'unmaintained'. Furthermore there's not really a good roster of
machines that were/were not part of the system AFAIK aside from the
old list, and it's unclear what the status of many of those machines
are (as you said, many haven't checked in in a while.)

There is much interest in a better nightly infrastructure and people
have asked me several times about setting one up on IRC. We have
historically had some problems with the nightly infrastructure, mainly
things like network disconnectivity or firewalling policies, since
most people aren't running dedicated internet facing machines (or even
a dedicated machine at all. Firewalls have been a problem for places
like MSR from what I understand.)

Several individual people run Jenkins individually, and I like it, but
I'm not sure how well it does when spread across the globe in terms of
networking (and realistically builders will look like that, as we
can't possibly have a dedicated farm somewhere.) I was also at one
point worried about the size of such a tool on systems like ARM
machines where resources are at a premium, but in hindsight this looks
OK. I'd like any opinions on this if people have deployed things in
these highly distributed scenarios.

I have had some ideas for an extremely-minimal nightly build
infrastructure that would ideally require minimal setup and let
clients have power over choosing how and when to build, but I have yet
to find the time to finish the basic implementation to try it.


On Sat, Jan 25, 2014 at 8:22 PM, Mateusz Kowalczyk
<fuuzetsu at fuuzetsu.co.uk> wrote:

> Hi all,
>
> I'd just like to query the status of the nightly builds. Is anything
> happening in that area? [1] is right on the front page of the GHC Trac
> even though no builds were ran for ~5 months. Perhaps it should be
> moved out of the way if there's no plan to resume these in the near
> future.
>
> Does anything specific need doing to get these to run again?
>
> [1]: https://ghc.haskell.org/trac/ghc/wiki/Builder
> --
> Mateusz K.
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://www.haskell.org/mailman/listinfo/ghc-devs
>



--
Regards,

Austin Seipp, Haskell Consultant
Well-Typed LLP, http://www.well-typed.com/

Reply | Threaded
Open this post in threaded view
|

Nightlies

Mateusz Kowalczyk
On 26/01/14 03:29, Austin Seipp wrote:

> As of right now, Pali's FreeBSD builds seem to be the only nightly
> that is still consistently running (and thanks to him for that!)
>
> The build infrastructure in its current status is mainly just
> 'unmaintained'. Furthermore there's not really a good roster of
> machines that were/were not part of the system AFAIK aside from the
> old list, and it's unclear what the status of many of those machines
> are (as you said, many haven't checked in in a while.)
>
> There is much interest in a better nightly infrastructure and people
> have asked me several times about setting one up on IRC. We have
> historically had some problems with the nightly infrastructure, mainly
> things like network disconnectivity or firewalling policies, since
> most people aren't running dedicated internet facing machines (or even
> a dedicated machine at all. Firewalls have been a problem for places
> like MSR from what I understand.)

Why not simply have the clients post the results once a night? If the
builds are nightly, is there really any need to have an open daemon
listening? From what I can tell from
http://darcs.haskell.org/ghcBuilder/builders/ it is simply the matter of
building once a day/night and then posting the results in an e-mail to
the list and uploading the binaries and test results elsewhere. Could we
not simply have a wrapper script around GHC build process that in the
end posts all these results to relevant places? The clients could simply
have a nightly cron job and it'd be up to the slave owner to keep these
builds going as often or as rarely as they want. The only downside is
that you guys can't tell the clients precisely when to run but looking
at build times, it's only once a day anyway.

> Several individual people run Jenkins individually, and I like it, but
> I'm not sure how well it does when spread across the globe in terms of
> networking (and realistically builders will look like that, as we
> can't possibly have a dedicated farm somewhere.) I was also at one
> point worried about the size of such a tool on systems like ARM
> machines where resources are at a premium, but in hindsight this looks
> OK. I'd like any opinions on this if people have deployed things in
> these highly distributed scenarios.
>
> I have had some ideas for an extremely-minimal nightly build
> infrastructure that would ideally require minimal setup and let
> clients have power over choosing how and when to build, but I have yet
> to find the time to finish the basic implementation to try it.
>
>
> On Sat, Jan 25, 2014 at 8:22 PM, Mateusz Kowalczyk
> <fuuzetsu at fuuzetsu.co.uk> wrote:
>> Hi all,
>>
>> I'd just like to query the status of the nightly builds. Is anything
>> happening in that area? [1] is right on the front page of the GHC Trac
>> even though no builds were ran for ~5 months. Perhaps it should be
>> moved out of the way if there's no plan to resume these in the near
>> future.
>>
>> Does anything specific need doing to get these to run again?
>>
>> [1]: https://ghc.haskell.org/trac/ghc/wiki/Builder
>> --
>> Mateusz K.
>> _______________________________________________
>> ghc-devs mailing list
>> ghc-devs at haskell.org
>> http://www.haskell.org/mailman/listinfo/ghc-devs
>>
>
>
>


--
Mateusz K.

Reply | Threaded
Open this post in threaded view
|

Nightlies

Joachim Breitner-2
In reply to this post by Mateusz Kowalczyk
Hi,

Am Sonntag, den 26.01.2014, 02:22 +0000 schrieb Mateusz Kowalczyk:
> I'd just like to query the status of the nightly builds. Is anything
> happening in that area? [1] is right on the front page of the GHC Trac
> even though no builds were ran for ~5 months. Perhaps it should be
> moved out of the way if there's no plan to resume these in the near
> future.

just to clarify: For what purpose do you want the nightlies? To check
whether GHC validates cleanly, to compare performance numbers, or to get
hold of up-to-date binary distributions?

For the first, I?d really really like to see something that runs before
a change enters master, so that non-validating mistakes like
http://git.haskell.org/ghc.git/commitdiff/b26e2f92c5c6f77fe361293a128da637e728959c
(without the corresponding change in
http://git.haskell.org/ghc.git/commitdiff/59f491a933ec7380698b776e14c3753c2a318a89)
do not reach master in the first place.

I?m happy to help setting up such an infrastructure, including designing
the precise workflow.


For the second and third, a build farm like the builders would of course
be great. I actually once got a Igloo snowboall from Linaro for that
purpose, but never finished setting it up properly. So once the builders
are going to be revived, I?d like to finally do that.


Greetings,
Joachim

--
Joachim ?nomeata? Breitner
  mail at joachim-breitner.de ? http://www.joachim-breitner.de/
  Jabber: nomeata at joachim-breitner.de  ? GPG-Key: 0x4743206C
  Debian Developer: nomeata at debian.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20140126/d06f1af8/attachment.sig>

Reply | Threaded
Open this post in threaded view
|

Nightlies

Páli Gábor János
On Sun, Jan 26, 2014 at 12:16 PM, Joachim Breitner
<mail at joachim-breitner.de> wrote:
> just to clarify: For what purpose do you want the nightlies? To check
> whether GHC validates cleanly, to compare performance numbers, or to get
> hold of up-to-date binary distributions?

Well, I run those clients primarily because that is (was?) one of the
primary requirements for Tier-1 platforms :-)  And this indeed greatly
helps me to see if something has gone wrong on FreeBSD -- so I can
track down the problems and fix them gradually continuously, therefore
birthing a new release becomes a bit easier.  But yes, I also feel
useful to offer daily snapshots for the interested parties as a side
effect.

> For the first, I?d really really like to see something that runs before
> a change enters master

I am afraid that you may not want to pass each change through all the
supported platforms before moving it to master.  Of course, that is
the ideal case, but it adds some operational cost, and can easily
frustrate developers who do not have access to the given platform
where it fails.

> For the second and third, a build farm like the builders would of course
> be great.

I believe Ian's original project (the builder-server I use) [1] was to
have a distributed farm of builders where anybody is allowed to
dedicate a machine.  Therefore GHC may be built on various platforms
while the cost maintenance is shared between the operators of the
respective platforms.  I think it worked pretty well until the
disappearance of the coordinator machine.

We also used the binary tarballs produced by the builders for the
latest releases -- Ian just set the release flag, waited for the next
day, picked the release tarballs and published them, without any
further interaction.

[1] https://ghc.haskell.org/trac/ghc/wiki/Builder

Reply | Threaded
Open this post in threaded view
|

Nightlies

Mateusz Kowalczyk
In reply to this post by Joachim Breitner-2
On 26/01/14 11:16, Joachim Breitner wrote:

> Hi,
>
> Am Sonntag, den 26.01.2014, 02:22 +0000 schrieb Mateusz Kowalczyk:
>> I'd just like to query the status of the nightly builds. Is anything
>> happening in that area? [1] is right on the front page of the GHC Trac
>> even though no builds were ran for ~5 months. Perhaps it should be
>> moved out of the way if there's no plan to resume these in the near
>> future.
>
> just to clarify: For what purpose do you want the nightlies? To check
> whether GHC validates cleanly, to compare performance numbers, or to get
> hold of up-to-date binary distributions?

Personally it's to see what validates that day and while we're at it, I
don't see the reason to not use this to get the nightly binaries as
well. I agree with pretty much everything that P?li said in his reply.

If we can get the validate results from other people's machines, at the
very least we have a sanity check: does it only fail for me or for
everyone else too? I think that if we have a list of platforms with
angry red everywhere, accessible to everyone, people are more likely to
react to build failures and we're less likely to have e-mails on
ghc-devs from people going ?is it just me or is it failing for everyone??.

> For the first, I?d really really like to see something that runs before
> a change enters master, so that non-validating mistakes like
> http://git.haskell.org/ghc.git/commitdiff/b26e2f92c5c6f77fe361293a128da637e728959c
> (without the corresponding change in
> http://git.haskell.org/ghc.git/commitdiff/59f491a933ec7380698b776e14c3753c2a318a89)
> do not reach master in the first place.
>
> I?m happy to help setting up such an infrastructure, including designing
> the precise workflow.

I think doing a per-commit validate before something enters master would
be difficult simply because one would have to wait a long time before
their commit is allowed in. Even on the fast boxes, the quick validate
from clean checkout seems to take about an hour at best.

> For the second and third, a build farm like the builders would of course
> be great. I actually once got a Igloo snowboall from Linaro for that
> purpose, but never finished setting it up properly. So once the builders
> are going to be revived, I?d like to finally do that.
>
>
> Greetings,
> Joachim
>


--
Mateusz K.

Reply | Threaded
Open this post in threaded view
|

Nightlies

Austin Seipp-5
In reply to this post by Joachim Breitner-2
On Sun, Jan 26, 2014 at 5:16 AM, Joachim Breitner
<mail at joachim-breitner.de> wrote:
> just to clarify: For what purpose do you want the nightlies? To check
> whether GHC validates cleanly, to compare performance numbers, or to get
> hold of up-to-date binary distributions?

In practice: all three. Developers want logs to see what went wrong.
Users want snapshot consistent distribution of snapshots to test
against. Both are legitimate uses that are covered by such
infrastructure.

> For the first, I?d really really like to see something that runs before
> a change enters master, so that non-validating mistakes like
> http://git.haskell.org/ghc.git/commitdiff/b26e2f92c5c6f77fe361293a128da637e728959c
> (without the corresponding change in
> http://git.haskell.org/ghc.git/commitdiff/59f491a933ec7380698b776e14c3753c2a318a89)
> do not reach master in the first place.
>
> I?m happy to help setting up such an infrastructure, including designing
> the precise workflow.

This is doable, but the question is to what extent? There are
literally dozens of build configurations that could break with any
given patch, without others breaking:

 * Profiling could break. Or profiling GHC (but not other smaller
things) could break.
 * Dynamic linking could break.
 * Rarer configurations could break but only for some cases, e.g.
threaded + profiling. Or LLVM + Profiling, or LLVM + dynamic linking,
etc etc.
 * Static linking for GHCi could break on platforms that now use
dynamic linking by default (as we saw happen when I broke it.)
 * GHC may only expose certain faulty behavior at certain optimization
levels (both in bootstrapping itself and in the tests - so maybe
./validate looks mostly OK, but -O2 is not.)
 * Bootstrapping the build with different compilers may break (i.e. an
unintentional backwards incompatible change is introduced in the
stage1 build)
 * Any of these could theoretically break depending on things like the
host platform.
 * The testsuite runs 'fast' by default. It would need to run slowly
to potentially uncover more problems, but this greatly increases the
runtime.
 * Not all machines are equal, and some will take dramatically longer
or shorter amounts of time to build (and subsequently) uncover these
problems.

In my experience, all of the above are absolutely possible scenarios
for something wrong to happen.

Also, in practice, a lot of these things either need an incredible
amount of cross-communication to fix (between the bot runner and the
developer,) or require direct access to the machine in order to debug.
Not everyone has that hardware, and not everyone will even be willing
to give access (for legitimate reasons - some people have offered to
run build bots, but behind corporate infrastructure at places like
IBM.)

And with the amount of time that many configurations requires, the
turnaround time for some things could become incredibly large and
frustrating.

I think if we were to introduce pre-push validation, the only thing it
could reasonably test would be ./validate and nothing else. And even
then, e.g. on high-powered ARM platforms, this will still seriously
take *hours*, and that's a significantly longer time-to-wait than most
people are used to.

> For the second and third, a build farm like the builders would of course
> be great. I actually once got a Igloo snowboall from Linaro for that
> purpose, but never finished setting it up properly. So once the builders
> are going to be revived, I?d like to finally do that.

If you're willing to contribute ARM builders, both Ben Gamari and I
would be very happy to have you do so (me and him are the only people
actively doing lots of ARM work, and frankly, Ben is doing most of
it.)

> Greetings,
> Joachim
>
> --
> Joachim ?nomeata? Breitner
>   mail at joachim-breitner.de ? http://www.joachim-breitner.de/
>   Jabber: nomeata at joachim-breitner.de  ? GPG-Key: 0x4743206C
>   Debian Developer: nomeata at debian.org
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://www.haskell.org/mailman/listinfo/ghc-devs
>

--
Regards,

Austin Seipp, Haskell Consultant
Well-Typed LLP, http://www.well-typed.com/

Reply | Threaded
Open this post in threaded view
|

Nightlies

Austin Seipp-5
In reply to this post by Mateusz Kowalczyk
FWIW, this is pretty much what it was going to do. Except it also
needs to host things like publicly accessible binary snapshots, as
people want to use them. And that's a no-go for firewalls or otherwise
non-controllable infrastructure that people may have bots on, so it
must have a central place for the results to be located on
haskell.org. It can then send summarized reports to the list based
e.g. on a cron job (asking bot runners to manage their own emails for
bots is annoying and painful, and doesn't scale nicely for them as we
add more.) And also, of course, there needs to be some aspect of
non-repudiation to the results, so that people know builds and emails
are legitimate (i.e. signed by an GPG pubkey and verified on the
server.)

On Sat, Jan 25, 2014 at 9:50 PM, Mateusz Kowalczyk
<fuuzetsu at fuuzetsu.co.uk> wrote:

> On 26/01/14 03:29, Austin Seipp wrote:
>> As of right now, Pali's FreeBSD builds seem to be the only nightly
>> that is still consistently running (and thanks to him for that!)
>>
>> The build infrastructure in its current status is mainly just
>> 'unmaintained'. Furthermore there's not really a good roster of
>> machines that were/were not part of the system AFAIK aside from the
>> old list, and it's unclear what the status of many of those machines
>> are (as you said, many haven't checked in in a while.)
>>
>> There is much interest in a better nightly infrastructure and people
>> have asked me several times about setting one up on IRC. We have
>> historically had some problems with the nightly infrastructure, mainly
>> things like network disconnectivity or firewalling policies, since
>> most people aren't running dedicated internet facing machines (or even
>> a dedicated machine at all. Firewalls have been a problem for places
>> like MSR from what I understand.)
>
> Why not simply have the clients post the results once a night? If the
> builds are nightly, is there really any need to have an open daemon
> listening? From what I can tell from
> http://darcs.haskell.org/ghcBuilder/builders/ it is simply the matter of
> building once a day/night and then posting the results in an e-mail to
> the list and uploading the binaries and test results elsewhere. Could we
> not simply have a wrapper script around GHC build process that in the
> end posts all these results to relevant places? The clients could simply
> have a nightly cron job and it'd be up to the slave owner to keep these
> builds going as often or as rarely as they want. The only downside is
> that you guys can't tell the clients precisely when to run but looking
> at build times, it's only once a day anyway.
>
>> Several individual people run Jenkins individually, and I like it, but
>> I'm not sure how well it does when spread across the globe in terms of
>> networking (and realistically builders will look like that, as we
>> can't possibly have a dedicated farm somewhere.) I was also at one
>> point worried about the size of such a tool on systems like ARM
>> machines where resources are at a premium, but in hindsight this looks
>> OK. I'd like any opinions on this if people have deployed things in
>> these highly distributed scenarios.
>>
>> I have had some ideas for an extremely-minimal nightly build
>> infrastructure that would ideally require minimal setup and let
>> clients have power over choosing how and when to build, but I have yet
>> to find the time to finish the basic implementation to try it.
>>
>>
>> On Sat, Jan 25, 2014 at 8:22 PM, Mateusz Kowalczyk
>> <fuuzetsu at fuuzetsu.co.uk> wrote:
>>> Hi all,
>>>
>>> I'd just like to query the status of the nightly builds. Is anything
>>> happening in that area? [1] is right on the front page of the GHC Trac
>>> even though no builds were ran for ~5 months. Perhaps it should be
>>> moved out of the way if there's no plan to resume these in the near
>>> future.
>>>
>>> Does anything specific need doing to get these to run again?
>>>
>>> [1]: https://ghc.haskell.org/trac/ghc/wiki/Builder
>>> --
>>> Mateusz K.
>>> _______________________________________________
>>> ghc-devs mailing list
>>> ghc-devs at haskell.org
>>> http://www.haskell.org/mailman/listinfo/ghc-devs
>>>
>>
>>
>>
>
>
> --
> Mateusz K.
>



--
Regards,

Austin Seipp, Haskell Consultant
Well-Typed LLP, http://www.well-typed.com/

Reply | Threaded
Open this post in threaded view
|

Nightlies

Austin Seipp-5
In reply to this post by Mateusz Kowalczyk
On Sun, Jan 26, 2014 at 6:35 PM, Mateusz Kowalczyk
<fuuzetsu at fuuzetsu.co.uk> wrote:
> If we can get the validate results from other people's machines, at the
> very least we have a sanity check: does it only fail for me or for
> everyone else too? I think that if we have a list of platforms with
> angry red everywhere, accessible to everyone, people are more likely to
> react to build failures and we're less likely to have e-mails on
> ghc-devs from people going ?is it just me or is it failing for everyone??.

And for the record, I do agree with this. I think a historic problem
is the results have never been public enough to most developers, and
unfortunately not everyone is trained to respond to just the emails
sent to ghc-builds at haskell.org to diagnose a problem. Seeing a
gigantic angry red build failure that blames you directly is likely
much easier to for most people as opposed to sorting through emails
from bots every morning.

--
Regards,

Austin Seipp, Haskell Consultant
Well-Typed LLP, http://www.well-typed.com/

Reply | Threaded
Open this post in threaded view
|

Nightlies

Austin Seipp-5
In reply to this post by Austin Seipp-5
On Sun, Jan 26, 2014 at 6:57 PM, Austin Seipp <austin at well-typed.com> wrote:

>  * Profiling could break. Or profiling GHC (but not other smaller
> things) could break.
>  * Dynamic linking could break.
>  * Rarer configurations could break but only for some cases, e.g.
> threaded + profiling. Or LLVM + Profiling, or LLVM + dynamic linking,
> etc etc.
>  * Static linking for GHCi could break on platforms that now use
> dynamic linking by default (as we saw happen when I broke it.)
>  * GHC may only expose certain faulty behavior at certain optimization
> levels (both in bootstrapping itself and in the tests - so maybe
> ./validate looks mostly OK, but -O2 is not.)
>  * Bootstrapping the build with different compilers may break (i.e. an
> unintentional backwards incompatible change is introduced in the
> stage1 build)
>  * Any of these could theoretically break depending on things like the
> host platform.
>  * The testsuite runs 'fast' by default. It would need to run slowly
> to potentially uncover more problems, but this greatly increases the
> runtime.

Disregard all this, upon closer inspection I see you only wanted
./validate anyway.* :)

* But it still will hurt more when you add in low-powered builders.

--
Regards,

Austin Seipp, Haskell Consultant
Well-Typed LLP, http://www.well-typed.com/

Reply | Threaded
Open this post in threaded view
|

Nightlies

Karel Gardas
In reply to this post by Austin Seipp-5

Austin,

On 01/26/14 04:29 AM, Austin Seipp wrote:
> As of right now, Pali's FreeBSD builds seem to be the only nightly
> that is still consistently running (and thanks to him for that!)
>
> The build infrastructure in its current status is mainly just
> 'unmaintained'. Furthermore there's not really a good roster of
> machines that were/were not part of the system AFAIK aside from the
> old list, and it's unclear what the status of many of those machines
> are (as you said, many haven't checked in in a while.)

honestly speaking, last message from Ian was that builder server waits
for "abbot" update. That's IIRC. So my i.MX/ARM buildbot and solaris
buildbot waits for abbot to be update to connect again.

> There is much interest in a better nightly infrastructure and people
> have asked me several times about setting one up on IRC. We have
> historically had some problems with the nightly infrastructure, mainly
> things like network disconnectivity or firewalling policies, since

I got those disconnectivity issue on builder client v2, I've not seen
them on v1, but this may be just a coincidence.

> Several individual people run Jenkins individually, and I like it, but
> I'm not sure how well it does when spread across the globe in terms of
> networking (and realistically builders will look like that, as we
> can't possibly have a dedicated farm somewhere.) I was also at one
> point worried about the size of such a tool on systems like ARM
> machines where resources are at a premium, but in hindsight this looks
> OK. I'd like any opinions on this if people have deployed things in
> these highly distributed scenarios.

ARM is all right, at least cortex-Ax boards provides 1GB usually and
sometimes even more. Using NFS or attached drive I've been able to
perform ghc build as dictated by the builder server in several days (4-5
IIRC). Pandaboard would be a lot faster (2 days IIRC) but is not that
stable and I don't have modern cortex-a15 boards here or quad A9, those
would be even faster. Anyway, if you do not require build every night
than this is doable on one board. If you require better coverage, then
more than one board will be needed.

> I have had some ideas for an extremely-minimal nightly build
> infrastructure that would ideally require minimal setup and let
> clients have power over choosing how and when to build, but I have yet
> to find the time to finish the basic implementation to try it.

Why to waste your precious time on something which was basically done
already several times in the past and what in its last incarnation done
by Ian worked quite well? Just please start the venerable builder server
and let's see people connect again and buildbots running...

Thanks!
Karel

Reply | Threaded
Open this post in threaded view
|

Nightlies

Joachim Breitner-2
In reply to this post by Austin Seipp-5
Good morning,

Am Sonntag, den 26.01.2014, 19:18 -0600 schrieb Austin Seipp:
> Disregard all this, upon closer inspection I see you only wanted
> ./validate anyway.* :)
>
> * But it still will hurt more when you add in low-powered builders.

right, I did not want to highjack the thread with my wishlist.

And even then I would not want low-powered builders, but rather one
strong, ?most typical? setup. I think it would already by a big win if
we ensured statically (heh) that every change to master has been
validated completely once somewhere. And if for changes like the one I
liked (removing dead code), as a developer I don?t have to laboriously
enforce this invariant (which I obviously then didn?t do), but can rely
on the safety nets of the infrastructure.

Greetings,
Joachim

PS: I?m subscribed to the list, no need to mail me personally.

--
Joachim ?nomeata? Breitner
  mail at joachim-breitner.de ? http://www.joachim-breitner.de/
  Jabber: nomeata at joachim-breitner.de  ? GPG-Key: 0x4743206C
  Debian Developer: nomeata at debian.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20140127/8791ee5a/attachment.sig>

Reply | Threaded
Open this post in threaded view
|

Nightlies

Simon Peyton Jones
Guys,

I want just to say thank you for thinking about this.  We badly need better nightly-builds for GHC, on a variety of platforms
a) to identify regressions, preferably to the actually commit that
   caused it, so Austin's big red message can go to the right person
b) to create binary snapshots that people can use without having to
   build GHC from scratch

Ian invested quite a bit of effort in a home-grown system to do just that, and I have no idea of how it stacks up against more widely-supported systems.

These days GHC HQ is really just ghc-devs.  So please go right ahead... if you can converge to a solution and implement it, that'd be great.

Thanks

Simon

| -----Original Message-----
| From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of
| Joachim Breitner
| Sent: 27 January 2014 09:47
| To: ghc-devs at haskell.org
| Subject: Re: Nightlies
|
| Good morning,
|
| Am Sonntag, den 26.01.2014, 19:18 -0600 schrieb Austin Seipp:
| > Disregard all this, upon closer inspection I see you only wanted
| > ./validate anyway.* :)
| >
| > * But it still will hurt more when you add in low-powered builders.
|
| right, I did not want to highjack the thread with my wishlist.
|
| And even then I would not want low-powered builders, but rather one
| strong, ?most typical? setup. I think it would already by a big win if
| we ensured statically (heh) that every change to master has been
| validated completely once somewhere. And if for changes like the one I
| liked (removing dead code), as a developer I don?t have to laboriously
| enforce this invariant (which I obviously then didn?t do), but can rely
| on the safety nets of the infrastructure.
|
| Greetings,
| Joachim
|
| PS: I?m subscribed to the list, no need to mail me personally.
|
| --
| Joachim ?nomeata? Breitner
|   mail at joachim-breitner.de ? http://www.joachim-breitner.de/
|   Jabber: nomeata at joachim-breitner.de  ? GPG-Key: 0x4743206C
|   Debian Developer: nomeata at debian.org

Reply | Threaded
Open this post in threaded view
|

Pre-Master checks (Was: Nightlies)

Joachim Breitner-2
Hi,

Am Dienstag, den 04.02.2014, 08:04 +0000 schrieb Simon Peyton Jones:
> I want just to say thank you for thinking about this.  We badly need better nightly-builds for GHC, on a variety of platforms
> a) to identify regressions, preferably to the actually commit that
>    caused it, so Austin's big red message can go to the right person

I believe we can do better, so that Austin?s big red message does not
have to be written in the first place.

Here is what I have in mind, and I?ll volunteer to implement it if
people think it is a good idea and I get the resources/permissions:

Proposal
~~~~~~~~

Nobody gets to push to master directly. Instead, every push to master is
diverted? to a temporary branch "validating/<some id>". One of our
servers detects the appearance of such a branch and will
 * check it out,
 * validate it,
 * if ok: check if master can still be fast-forward?ed to it,
 * if yes: push to master.

If it does not validate, or if master has changed in between, the branch
will be moved to failed/<some id>, and a message is sent to the pushing
developer?, including a tail of the log and a link to the full log.

Systems can fail, and maybe nothing validates anymore for reasons not
easily fixable. For that case, a backdoor is available: Pushes to the
branch "master-unchecked" will be moved to master, well, unchecked.

Benefits:
 * It is guaranteed that master has validated at least once somewhere.
   I.e. no need to ask on the mailing list ?does master validate for you
   right now??
 * It is now ok to do changes that are ?obviously correct? (comment
   changes, removing dead code, code reformatting) without having
   to validate manually, which _saves developer time_ (our most precious
   resource).

Downsides:
 * When two commits are racing for master, one will be rejected for
   being a non-fast-forward commit. The user will then have to merge
   or rebase and try again.
   But: The same would be true if he was validating locally (unless he
   does not validate the merge/rebase, in which case we are again where
   we don?t want to be: Unvalidated versions in master.)


Is this something you would what, or could live with?

If it is technically feasible (given our hardware constraints,
repository structure and git?s feature) is a different question, which
needs to be discussed afterwards.

Greetings,
Joachim



? Might not be possible transparently
(http://stackoverflow.com/questions/21362833), but for the sake of
argument and workflow design, assume it was.
? As an approximation: The committer of the latest patch.

PS: I?m also considering (but not pushing hard) for a stronger variant
as follows. We do not need to discuss that now and should, if at all,
start the with the proposal above. I?m just adding it to show where this
is going ...
       
Stronger proposal
~~~~~~~~~~~~~~~~~

        Every commit in master needs to be validated!
I tend to make sure that all patches on my branch validate individually
(git rebase -i -x "./validate" is a great tool here, you should use it!
). Contributors who do not want to go through that trouble should then
use "git merge --squash" to produce a single commit from their branch.

This would make the git history more useful for things like bitsecting.



--
Joachim Breitner
  e-Mail: mail at joachim-breitner.de
  Homepage: http://www.joachim-breitner.de
  Jabber-ID: nomeata at joachim-breitner.de

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20140204/b30a3c80/attachment.sig>

Reply | Threaded
Open this post in threaded view
|

Pre-Master checks (Was: Nightlies)

Richard Eisenberg-2
Yay!

I, for one, would probably submit more bugfix patches with this structure in place. I take validating seriously, but my daily schedule often involves switching machines and tasks, and once I press "validate", it's often hours before I actually look at the results... by which time, some other commit may have come through. My need (real or perceived) for an uninterrupted chunk of time to make a patch stops me from doing them, sometimes.

With Joachim's structure, I could spend a half hour doing a quick patch (especially comments!) and push and move on.

Please do it! :)

Thanks!
Richard

PS: Transparency in the git redirects would be nice, of course, but is in no way necessary. We could I suppose add a "push" script to the repo that handles some of the overhead.

On Feb 4, 2014, at 4:41 AM, Joachim Breitner <mail at joachim-breitner.de> wrote:

> Hi,
>
> Am Dienstag, den 04.02.2014, 08:04 +0000 schrieb Simon Peyton Jones:
>> I want just to say thank you for thinking about this.  We badly need better nightly-builds for GHC, on a variety of platforms
>> a) to identify regressions, preferably to the actually commit that
>>   caused it, so Austin's big red message can go to the right person
>
> I believe we can do better, so that Austin?s big red message does not
> have to be written in the first place.
>
> Here is what I have in mind, and I?ll volunteer to implement it if
> people think it is a good idea and I get the resources/permissions:
>
> Proposal
> ~~~~~~~~
>
> Nobody gets to push to master directly. Instead, every push to master is
> diverted? to a temporary branch "validating/<some id>". One of our
> servers detects the appearance of such a branch and will
> * check it out,
> * validate it,
> * if ok: check if master can still be fast-forward?ed to it,
> * if yes: push to master.
>
> If it does not validate, or if master has changed in between, the branch
> will be moved to failed/<some id>, and a message is sent to the pushing
> developer?, including a tail of the log and a link to the full log.
>
> Systems can fail, and maybe nothing validates anymore for reasons not
> easily fixable. For that case, a backdoor is available: Pushes to the
> branch "master-unchecked" will be moved to master, well, unchecked.
>
> Benefits:
> * It is guaranteed that master has validated at least once somewhere.
>   I.e. no need to ask on the mailing list ?does master validate for you
>   right now??
> * It is now ok to do changes that are ?obviously correct? (comment
>   changes, removing dead code, code reformatting) without having
>   to validate manually, which _saves developer time_ (our most precious
>   resource).
>
> Downsides:
> * When two commits are racing for master, one will be rejected for
>   being a non-fast-forward commit. The user will then have to merge
>   or rebase and try again.
>   But: The same would be true if he was validating locally (unless he
>   does not validate the merge/rebase, in which case we are again where
>   we don?t want to be: Unvalidated versions in master.)
>
>
> Is this something you would what, or could live with?
>
> If it is technically feasible (given our hardware constraints,
> repository structure and git?s feature) is a different question, which
> needs to be discussed afterwards.
>
> Greetings,
> Joachim
>
>
>
> ? Might not be possible transparently
> (http://stackoverflow.com/questions/21362833), but for the sake of
> argument and workflow design, assume it was.
> ? As an approximation: The committer of the latest patch.
>
> PS: I?m also considering (but not pushing hard) for a stronger variant
> as follows. We do not need to discuss that now and should, if at all,
> start the with the proposal above. I?m just adding it to show where this
> is going ...
>
> Stronger proposal
> ~~~~~~~~~~~~~~~~~
>
>        Every commit in master needs to be validated!
> I tend to make sure that all patches on my branch validate individually
> (git rebase -i -x "./validate" is a great tool here, you should use it!
> ). Contributors who do not want to go through that trouble should then
> use "git merge --squash" to produce a single commit from their branch.
>
> This would make the git history more useful for things like bitsecting.
>
>
>
> --
> Joachim Breitner
>  e-Mail: mail at joachim-breitner.de
>  Homepage: http://www.joachim-breitner.de
>  Jabber-ID: nomeata at joachim-breitner.de
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://www.haskell.org/mailman/listinfo/ghc-devs


Reply | Threaded
Open this post in threaded view
|

Pre-Master checks (Was: Nightlies)

Mateusz Kowalczyk
In reply to this post by Joachim Breitner-2
On 04/02/14 09:41, Joachim Breitner wrote:

> Hi,
>
> Am Dienstag, den 04.02.2014, 08:04 +0000 schrieb Simon Peyton Jones:
>> I want just to say thank you for thinking about this.  We badly need better nightly-builds for GHC, on a variety of platforms
>> a) to identify regressions, preferably to the actually commit that
>>    caused it, so Austin's big red message can go to the right person
>
> I believe we can do better, so that Austin?s big red message does not
> have to be written in the first place.
>
> Here is what I have in mind, and I?ll volunteer to implement it if
> people think it is a good idea and I get the resources/permissions:
>
> Proposal
> ~~~~~~~~
>
> Nobody gets to push to master directly. Instead, every push to master is
> diverted? to a temporary branch "validating/<some id>". One of our
> servers detects the appearance of such a branch and will
>  * check it out,
>  * validate it,
>  * if ok: check if master can still be fast-forward?ed to it,
>  * if yes: push to master.
>
> If it does not validate, or if master has changed in between, the branch
> will be moved to failed/<some id>, and a message is sent to the pushing
> developer?, including a tail of the log and a link to the full log.
>
> Systems can fail, and maybe nothing validates anymore for reasons not
> easily fixable. For that case, a backdoor is available: Pushes to the
> branch "master-unchecked" will be moved to master, well, unchecked.
>
> Benefits:
>  * It is guaranteed that master has validated at least once somewhere.
>    I.e. no need to ask on the mailing list ?does master validate for you
>    right now??
>  * It is now ok to do changes that are ?obviously correct? (comment
>    changes, removing dead code, code reformatting) without having
>    to validate manually, which _saves developer time_ (our most precious
>    resource).
>
> Downsides:
>  * When two commits are racing for master, one will be rejected for
>    being a non-fast-forward commit. The user will then have to merge
>    or rebase and try again.
>    But: The same would be true if he was validating locally (unless he
>    does not validate the merge/rebase, in which case we are again where
>    we don?t want to be: Unvalidated versions in master.)
>
>
> Is this something you would what, or could live with?
>
> If it is technically feasible (given our hardware constraints,
> repository structure and git?s feature) is a different question, which
> needs to be discussed afterwards.
>
> Greetings,
> Joachim
>
>
>
> ? Might not be possible transparently
> (http://stackoverflow.com/questions/21362833), but for the sake of
> argument and workflow design, assume it was.
> ? As an approximation: The committer of the latest patch.
>
> PS: I?m also considering (but not pushing hard) for a stronger variant
> as follows. We do not need to discuss that now and should, if at all,
> start the with the proposal above. I?m just adding it to show where this
> is going ...
>        
> Stronger proposal
> ~~~~~~~~~~~~~~~~~
>
>         Every commit in master needs to be validated!
> I tend to make sure that all patches on my branch validate individually
> (git rebase -i -x "./validate" is a great tool here, you should use it!
> ). Contributors who do not want to go through that trouble should then
> use "git merge --squash" to produce a single commit from their branch.
>
> This would make the git history more useful for things like bitsecting.
>
>
>
>
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://www.haskell.org/mailman/listinfo/ghc-devs
>

You mention that it's enough for one machine to validate the patch. What
if the change was for example to fix something on ARM (where builds take
a long time) and has no effect on x86? The x86 machine is likely to come
through first and give you the OK, pushing to master something that's
effectively not checked. In fact, the fastest slave is likely to come
through first every time and if it validates there (for example, I hear
that 64-bit Linux is the golden platform for validate), it will be
pushed to master.

I think the proposal needs an enhancement: ability to specify which
platform the commit needs to validate on (i.e. even if i686 comes
through first, we'll wait for ARM anyway) before being pushed in.

--
Mateusz K.

Reply | Threaded
Open this post in threaded view
|

Pre-Master checks (Was: Nightlies)

Carter Schonwald
This actually raises an important question: we don't have any good policy
for making sure validate / etc stays clean on all the platforms.   There's
enough variations that it might take an implausible amount of resources to
test all the targets we want to support on every commit. (Eg given how long
builds apparently take on certain platforms like arm, we'd need quite a few
wee arm machines to test arm at every commit.  Or maybe not?)

Point being: I agree that support tooling to make sure builds don't stay
broken and don't get broken aren't where it should be. And I  agree that
something like you describe should be done.  I'm just wondering if we have
the resources to do that correctly.

On Tuesday, February 4, 2014, Mateusz Kowalczyk <fuuzetsu at fuuzetsu.co.uk>
wrote:

> On 04/02/14 09:41, Joachim Breitner wrote:
> > Hi,
> >
> > Am Dienstag, den 04.02.2014, 08:04 +0000 schrieb Simon Peyton Jones:
> >> I want just to say thank you for thinking about this.  We badly need
> better nightly-builds for GHC, on a variety of platforms
> >> a) to identify regressions, preferably to the actually commit that
> >>    caused it, so Austin's big red message can go to the right person
> >
> > I believe we can do better, so that Austin?s big red message does not
> > have to be written in the first place.
> >
> > Here is what I have in mind, and I?ll volunteer to implement it if
> > people think it is a good idea and I get the resources/permissions:
> >
> > Proposal
> > ~~~~~~~~
> >
> > Nobody gets to push to master directly. Instead, every push to master is
> > diverted? to a temporary branch "validating/<some id>". One of our
> > servers detects the appearance of such a branch and will
> >  * check it out,
> >  * validate it,
> >  * if ok: check if master can still be fast-forward?ed to it,
> >  * if yes: push to master.
> >
> > If it does not validate, or if master has changed in between, the branch
> > will be moved to failed/<some id>, and a message is sent to the pushing
> > developer?, including a tail of the log and a link to the full log.
> >
> > Systems can fail, and maybe nothing validates anymore for reasons not
> > easily fixable. For that case, a backdoor is available: Pushes to the
> > branch "master-unchecked" will be moved to master, well, unchecked.
> >
> > Benefits:
> >  * It is guaranteed that master has validated at least once somewhere.
> >    I.e. no need to ask on the mailing list ?does master validate for you
> >    right now??
> >  * It is now ok to do changes that are ?obviously correct? (comment
> >    changes, removing dead code, code reformatting) without having
> >    to validate manually, which _saves developer time_ (our most precious
> >    resource).
> >
> > Downsides:
> >  * When two commits are racing for master, one will be rejected for
> >    being a non-fast-forward commit. The user will then have to merge
> >    or rebase and try again.
> >    But: The same would be true if he was validating locally (unless he
> >    does not validate the merge/rebase, in which case we are again where
> >    we don?t want to be: Unvalidated versions in master.)
> >
> >
> > Is this something you would what, or could live with?
> >
> > If it is technically feasible (given our hardware constraints,
> > repository structure and git?s feature) is a different question, which
> > needs to be discussed afterwards.
> >
> > Greetings,
> > Joachim
> >
> >
> >
> > ? Might not be possible transparently
> > (http://stackoverflow.com/questions/21362833), but for the sake of
> > argument and workflow design, assume it was.
> > ? As an approximation: The committer of the latest patch.
> >
> > PS: I?m also considering (but not pushing hard) for a stronger variant
> > as follows. We do not need to discuss that now and should, if at all,
> > start the with the proposal above. I?m just adding it to show where this
> > is going ...
> >
> > Stronger proposal
> > ~~~~~~~~~~~~~~~~~
> >
> >         Every commit in master needs to be validated!
> > I tend to make sure that all patches on my branch validate individually
> > (git rebase -i -x "./validate" is a great tool here, you should use it!
> > ). Contributors who do not want to go through that trouble should then
> > use "git merge --squash" to produce a single commit from their branch.
> >
> > This would make the git history more useful for things like bitsecting.
> >
> >
> >
> >
> >
> > _______________________________________________
> > ghc-devs mailing list
> > ghc-devs at haskell.org <javascript:;>
> > http://www.haskell.org/mailman/listinfo/ghc-devs
> >
>
> You mention that it's enough for one machine to validate the patch. What
> if the change was for example to fix something on ARM (where builds take
> a long time) and has no effect on x86? The x86 machine is likely to come
> through first and give you the OK, pushing to master something that's
> effectively not checked. In fact, the fastest slave is likely to come
> through first every time and if it validates there (for example, I hear
> that 64-bit Linux is the golden platform for validate), it will be
> pushed to master.
>
> I think the proposal needs an enhancement: ability to specify which
> platform the commit needs to validate on (i.e. even if i686 comes
> through first, we'll wait for ARM anyway) before being pushed in.
>
> --
> Mateusz K.
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org <javascript:;>
> http://www.haskell.org/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20140204/279dba55/attachment.html>

Reply | Threaded
Open this post in threaded view
|

Pre-Master checks (Was: Nightlies)

Joachim Breitner-2
In reply to this post by Mateusz Kowalczyk
Hi,

Am Dienstag, den 04.02.2014, 15:26 +0000 schrieb Mateusz Kowalczyk:
> You mention that it's enough for one machine to validate the patch. What
> if the change was for example to fix something on ARM (where builds take
> a long time) and has no effect on x86? The x86 machine is likely to come
> through first and give you the OK, pushing to master something that's
> effectively not checked. In fact, the fastest slave is likely to come
> through first every time and if it validates there (for example, I hear
> that 64-bit Linux is the golden platform for validate), it will be
> pushed to master.

sorry, I was not clear. There would be _one_ dedicated machine for doing
pre-master checks, and its main purpose is to detect obvious mistakes,
i.e. broken code, forgotten test output updates etc. It will not replace
the build farm that does daily checks on other systems.

If someone fixes a problem on an different architecture, I have no
doubts that we will have tested this fix. It is the forgotten update of
the import list (which will make validate fail for others, independent o
of the architecture) that I worry about.

Greetings,
Joachim



--
Joachim ?nomeata? Breitner
  mail at joachim-breitner.de ? http://www.joachim-breitner.de/
  Jabber: nomeata at joachim-breitner.de  ? GPG-Key: 0x4743206C
  Debian Developer: nomeata at debian.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20140204/51e4d1dc/attachment-0001.sig>

Reply | Threaded
Open this post in threaded view
|

Pre-Master checks (Was: Nightlies)

Simon Peyton Jones
In reply to this post by Joachim Breitner-2
I can see advantages here, similar to Richard.  I'm just a bit worried about keeping all those branches straight in my head.

Who names all these temporary branches?  Does someone delete them so they don't accumulate and dominate the branch list?

Would need careful documenting, so that new people know exactly what to do.

Simon

| -----Original Message-----
| From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of
| Joachim Breitner
| Sent: 04 February 2014 09:42
| To: ghc-devs at haskell.org
| Subject: Re: Pre-Master checks (Was: Nightlies)
|
| Hi,
|
| Am Dienstag, den 04.02.2014, 08:04 +0000 schrieb Simon Peyton Jones:
| > I want just to say thank you for thinking about this.  We badly need
| better nightly-builds for GHC, on a variety of platforms
| > a) to identify regressions, preferably to the actually commit that
| >    caused it, so Austin's big red message can go to the right person
|
| I believe we can do better, so that Austin?s big red message does not
| have to be written in the first place.
|
| Here is what I have in mind, and I?ll volunteer to implement it if
| people think it is a good idea and I get the resources/permissions:
|
| Proposal
| ~~~~~~~~
|
| Nobody gets to push to master directly. Instead, every push to master
| is
| diverted? to a temporary branch "validating/<some id>". One of our
| servers detects the appearance of such a branch and will
|  * check it out,
|  * validate it,
|  * if ok: check if master can still be fast-forward?ed to it,
|  * if yes: push to master.
|
| If it does not validate, or if master has changed in between, the
| branch
| will be moved to failed/<some id>, and a message is sent to the pushing
| developer?, including a tail of the log and a link to the full log.
|
| Systems can fail, and maybe nothing validates anymore for reasons not
| easily fixable. For that case, a backdoor is available: Pushes to the
| branch "master-unchecked" will be moved to master, well, unchecked.
|
| Benefits:
|  * It is guaranteed that master has validated at least once somewhere.
|    I.e. no need to ask on the mailing list ?does master validate for
| you
|    right now??
|  * It is now ok to do changes that are ?obviously correct? (comment
|    changes, removing dead code, code reformatting) without having
|    to validate manually, which _saves developer time_ (our most
| precious
|    resource).
|
| Downsides:
|  * When two commits are racing for master, one will be rejected for
|    being a non-fast-forward commit. The user will then have to merge
|    or rebase and try again.
|    But: The same would be true if he was validating locally (unless he
|    does not validate the merge/rebase, in which case we are again where
|    we don?t want to be: Unvalidated versions in master.)
|
|
| Is this something you would what, or could live with?
|
| If it is technically feasible (given our hardware constraints,
| repository structure and git?s feature) is a different question, which
| needs to be discussed afterwards.
|
| Greetings,
| Joachim
|
|
|
| ? Might not be possible transparently
| (http://stackoverflow.com/questions/21362833), but for the sake of
| argument and workflow design, assume it was.
| ? As an approximation: The committer of the latest patch.
|
| PS: I?m also considering (but not pushing hard) for a stronger variant
| as follows. We do not need to discuss that now and should, if at all,
| start the with the proposal above. I?m just adding it to show where
| this
| is going ...
|
| Stronger proposal
| ~~~~~~~~~~~~~~~~~
|
|         Every commit in master needs to be validated!
| I tend to make sure that all patches on my branch validate individually
| (git rebase -i -x "./validate" is a great tool here, you should use it!
| ). Contributors who do not want to go through that trouble should then
| use "git merge --squash" to produce a single commit from their branch.
|
| This would make the git history more useful for things like bitsecting.
|
|
|
| --
| Joachim Breitner
|   e-Mail: mail at joachim-breitner.de
|   Homepage: http://www.joachim-breitner.de
|   Jabber-ID: nomeata at joachim-breitner.de


Reply | Threaded
Open this post in threaded view
|

Pre-Master checks (Was: Nightlies)

Carter Schonwald
one point in the design space (mind you one i'm not going to advocate) is
the testing / merging workflow that the Mozilla folks are doing for rust.
 Every patch / pull request gets tested by their build bot before merging.
Does make the commits on master look pretty mess / mergy though.

mind you its a bit tightly coupled to github, but some of the ideas *might*
be a good strawman.  Likewise, how does eg LLVM/Clang manage its
testing infrastructure?  How does GCC manage it?


On Tue, Feb 4, 2014 at 12:56 PM, Simon Peyton Jones
<simonpj at microsoft.com>wrote:

> I can see advantages here, similar to Richard.  I'm just a bit worried
> about keeping all those branches straight in my head.
>
> Who names all these temporary branches?  Does someone delete them so they
> don't accumulate and dominate the branch list?
>
> Would need careful documenting, so that new people know exactly what to do.
>
> Simon
>
> | -----Original Message-----
> | From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of
> | Joachim Breitner
> | Sent: 04 February 2014 09:42
> | To: ghc-devs at haskell.org
> | Subject: Re: Pre-Master checks (Was: Nightlies)
> |
> | Hi,
> |
> | Am Dienstag, den 04.02.2014, 08:04 +0000 schrieb Simon Peyton Jones:
> | > I want just to say thank you for thinking about this.  We badly need
> | better nightly-builds for GHC, on a variety of platforms
> | > a) to identify regressions, preferably to the actually commit that
> | >    caused it, so Austin's big red message can go to the right person
> |
> | I believe we can do better, so that Austin?s big red message does not
> | have to be written in the first place.
> |
> | Here is what I have in mind, and I?ll volunteer to implement it if
> | people think it is a good idea and I get the resources/permissions:
> |
> | Proposal
> | ~~~~~~~~
> |
> | Nobody gets to push to master directly. Instead, every push to master
> | is
> | diverted? to a temporary branch "validating/<some id>". One of our
> | servers detects the appearance of such a branch and will
> |  * check it out,
> |  * validate it,
> |  * if ok: check if master can still be fast-forward?ed to it,
> |  * if yes: push to master.
> |
> | If it does not validate, or if master has changed in between, the
> | branch
> | will be moved to failed/<some id>, and a message is sent to the pushing
> | developer?, including a tail of the log and a link to the full log.
> |
> | Systems can fail, and maybe nothing validates anymore for reasons not
> | easily fixable. For that case, a backdoor is available: Pushes to the
> | branch "master-unchecked" will be moved to master, well, unchecked.
> |
> | Benefits:
> |  * It is guaranteed that master has validated at least once somewhere.
> |    I.e. no need to ask on the mailing list ?does master validate for
> | you
> |    right now??
> |  * It is now ok to do changes that are ?obviously correct? (comment
> |    changes, removing dead code, code reformatting) without having
> |    to validate manually, which _saves developer time_ (our most
> | precious
> |    resource).
> |
> | Downsides:
> |  * When two commits are racing for master, one will be rejected for
> |    being a non-fast-forward commit. The user will then have to merge
> |    or rebase and try again.
> |    But: The same would be true if he was validating locally (unless he
> |    does not validate the merge/rebase, in which case we are again where
> |    we don?t want to be: Unvalidated versions in master.)
> |
> |
> | Is this something you would what, or could live with?
> |
> | If it is technically feasible (given our hardware constraints,
> | repository structure and git?s feature) is a different question, which
> | needs to be discussed afterwards.
> |
> | Greetings,
> | Joachim
> |
> |
> |
> | ? Might not be possible transparently
> | (http://stackoverflow.com/questions/21362833), but for the sake of
> | argument and workflow design, assume it was.
> | ? As an approximation: The committer of the latest patch.
> |
> | PS: I?m also considering (but not pushing hard) for a stronger variant
> | as follows. We do not need to discuss that now and should, if at all,
> | start the with the proposal above. I?m just adding it to show where
> | this
> | is going ...
> |
> | Stronger proposal
> | ~~~~~~~~~~~~~~~~~
> |
> |         Every commit in master needs to be validated!
> | I tend to make sure that all patches on my branch validate individually
> | (git rebase -i -x "./validate" is a great tool here, you should use it!
> | ). Contributors who do not want to go through that trouble should then
> | use "git merge --squash" to produce a single commit from their branch.
> |
> | This would make the git history more useful for things like bitsecting.
> |
> |
> |
> | --
> | Joachim Breitner
> |   e-Mail: mail at joachim-breitner.de
> |   Homepage: http://www.joachim-breitner.de
> |   Jabber-ID: nomeata at joachim-breitner.de
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://www.haskell.org/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20140204/b60134f8/attachment.html>

12