Discussion: Hadrian's defaults

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Discussion: Hadrian's defaults

Spiwack, Arnaud

I’ve been using Hadrian for a while. And I really like it. But I
think, based on my experience and also the reading of the soon to be
freshly reforged newcomers’ guide, that it could be greatly improved
by changing the default of some of the more common options:

  • The -c option should be the default. One of the cool thing about
    Hadrian is that it needs fewer steps to completion, as it knows how
    to run the boot and configuration script. It’s not always what you
    want, but if it isn’t, it should befall to you to specify it. The
    most common use is the default (-c is, for instance, used in the
    newcomers’ guide). The reverse flag could be --no-configure.
  • --freeze1 should be the default. Ok, this one is harder. It’s
    almost always the right thing to call --freeze1 and forgetting it
    can have big consequences. So let’s make --freeze1 the default,
    add a flag --rebuild1 to force the rebuilding of stage 1. However,
    this requires an exception: if stage 1 was never built, we should
    build it, even without the flag --rebuild1.
  • Eventually, the new --share option should be the default as
    well. But maybe the paint is a bit too fresh on this.

If I missed something (other defaults which should be changed, or
reason to keep the defaults as they currently are) reply to this
thread.


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Discussion: Hadrian's defaults

Matthew Pickering
My personal experience is that

1. -c doesn't work very well compared to ./configure for me and picks
up the wrong versions of alex/happy as installed by hadrian rather
than the ones provisioned by ghc.nix/. I have tried to avoid it anyway
after encountering some problems.
2. I have the impression that freeze1 being the default could be quite safe.
3. In my experience --share is too unreliable at the moment. I tried
using it for 2-3 days but it didn't seem ready.

The most friendly option to developers would be not building the perf
build by default but this has been repeatedly shot down in the past.

Cheers,

Matt

On Thu, Mar 14, 2019 at 3:20 PM Spiwack, Arnaud <[hidden email]> wrote:

>
> I’ve been using Hadrian for a while. And I really like it. But I
> think, based on my experience and also the reading of the soon to be
> freshly reforged newcomers’ guide, that it could be greatly improved
> by changing the default of some of the more common options:
>
> The -c option should be the default. One of the cool thing about
> Hadrian is that it needs fewer steps to completion, as it knows how
> to run the boot and configuration script. It’s not always what you
> want, but if it isn’t, it should befall to you to specify it. The
> most common use is the default (-c is, for instance, used in the
> newcomers’ guide). The reverse flag could be --no-configure.
> --freeze1 should be the default. Ok, this one is harder. It’s
> almost always the right thing to call --freeze1 and forgetting it
> can have big consequences. So let’s make --freeze1 the default,
> add a flag --rebuild1 to force the rebuilding of stage 1. However,
> this requires an exception: if stage 1 was never built, we should
> build it, even without the flag --rebuild1.
> Eventually, the new --share option should be the default as
> well. But maybe the paint is a bit too fresh on this.
>
> If I missed something (other defaults which should be changed, or
> reason to keep the defaults as they currently are) reply to this
> thread.
>
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Discussion: Hadrian's defaults

Spiwack, Arnaud


The most friendly option to developers would be not building the perf
build by default but this has been repeatedly shot down in the past.
 
I agree. But I also find it reasonable to make it so one won't accidentally build a dev build when trying to make a release. (that being said, in the Ocaml world, things have been going the other way, where, if I understand correctly, `dune build` builds development version, while `dune build @install` builds release artifact). I can't say I have a strong opinion on this.

1. -c doesn't work very well compared to ./configure for me and picks
up the wrong versions of alex/happy as installed by hadrian rather
than the ones provisioned by ghc.nix/. I have tried to avoid it anyway
after encountering some problems.

If anybody else have such issues with `-c`, please share your experience in the thread. I haven't had any problem so far, myself.

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Discussion: Hadrian's defaults

Herbert Valerio Riedel-3
In reply to this post by Spiwack, Arnaud


On Thu, Mar 14, 2019 at 4:20 PM Spiwack, Arnaud <[hidden email]> wrote:
  • The -c option should be the default. 
Very strong -1 from me on this one; I've been quite vocal on the Hadrian issue tracker early on and multiple times against having Hadrian invoke ./configure at all, even more so against having it do so by default. I don't have the ticket number at my fingertips but it should be fairly easy to find.



_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Discussion: Hadrian's defaults

Moritz Angermann-2


> On Mar 15, 2019, at 2:19 AM, Herbert Valerio Riedel <[hidden email]> wrote:
> On Thu, Mar 14, 2019 at 4:20 PM Spiwack, Arnaud <[hidden email]> wrote:
> • The -c option should be the default.
> Very strong -1 from me on this one; I've been quite vocal on the Hadrian issue tracker early on and multiple times against having Hadrian invoke ./configure at all, even more so against having it do so by default. I don't have the ticket number at my fingertips but it should be fairly easy to find.

I'm with Herbert here. I think that the `-c` flag should be mandatory if you want
hadrian to invoke autoconf magic.

I believe the confusion might stem from the newcomers guide[1]?  I'd rather see the
newcomers guide *not* use `-c`, and instead make it obvious to call `boot` and
`configure`.  These are essential steps and hiding them makes them less obvious.

`boot` does
- (1) checks that url rewrites are in place.
- (2) checks that all bootpackages are available
- (3) run autoreconf as needed
- (4) and generates a bunch of make files for the make based build system.

(4) can be disabled by passing `--hadrian` to `boot`.  (1) is needed due to
the relative submodules I beleive.

`configure` generates the necessary configuration files based on the configure flags
passed. And hadrian does the actual build step that `make` used to do.

It's magically conflating two different phases with `-c`. The configure phase and
the build phase. Making this the default means it's always magic. I don't like magic!

Cheers,
 Moritz

--
[1]: https://github.com/tdammers/ghc-wiki/blob/wip/newcomers/newcomers-tutorial.md

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Discussion: Hadrian's defaults

Spiwack, Arnaud
On Thu, Mar 14, 2019 at 7:20 PM Herbert Valerio Riedel <[hidden email]> wrote:
I don't have the ticket number at my fingertips but it should be fairly easy to find.

I'm afraid it doesn't appear to be. Could you share your arguments in this thread?

On Fri, Mar 15, 2019 at 3:10 AM Moritz Angermann <[hidden email]> wrote:
It's magically conflating two different phases with `-c`. The configure phase and
the build phase. Making this the default means it's always magic. I don't like magic!

Unfortunately, I really don't understand what you are saying. What's magic about combining the phases?

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Discussion: Hadrian's defaults

Moritz Angermann-2
Hi Arnaud,

> On Mar 15, 2019, at 8:32 PM, Spiwack, Arnaud <[hidden email]> wrote:
>
> On Thu, Mar 14, 2019 at 7:20 PM Herbert Valerio Riedel <[hidden email]> wrote:
> I don't have the ticket number at my fingertips but it should be fairly easy to find.
>
> I'm afraid it doesn't appear to be. Could you share your arguments in this thread?
This was the last one that lead to the current `-c` state:
- https://github.com/snowleopard/hadrian/issues/457
There is also
- https://github.com/snowleopard/hadrian/issues/655

if you look through the issues on snowleopard/hadrian and sort by comment frequency
you'll likely find quite a lot of further discussion about not making configure and
boot the default.

>
> On Fri, Mar 15, 2019 at 3:10 AM Moritz Angermann <[hidden email]> wrote:
> It's magically conflating two different phases with `-c`. The configure phase and
> the build phase. Making this the default means it's always magic. I don't like magic!
>
> Unfortunately, I really don't understand what you are saying. What's magic about combining the phases?

We have two phases:

Phase 1: autoconf

  This phase is essentially a code-generation phase, where specific templates are
  instantiated to configure time value.  Which again can be split into two specific
  subproblems:

  - Generation of the configure script from the configure.ac and aclocal.m4 files
    using autoconf.
  - Generating code using the configure script by computing configure time calues
    and filling those into the `.in` files producing the files that lack the `.in`
    extension.

Phase 2: building

  This has been traditionally the job of make, and this is what hadrian should
  replace.


By subsuming the configure phase (by invoking ./configure) from hadrian we loose
the phase distinction and if the `-c` flag is optional, users will *not even see*
a flag that indicates that the system will run `./configure` for them. This is the
magic I'm referring to and to which I strongly object.  If we can retire autoconf
and do the whole configuration in hadrian, that story may change.  But as long as
we are using an autoconf based configuration we should *not* run that magically.
The `-c` flag is at least there to show that hadrian is explicilty instructed to
run configure.

./configure supports its own set of flags, if hadrian subsumes those, we'd need
some generic way of passing flags to ./configure, at which point I have to ask
why do we do this in the first place and try to call ./configure from within hadrian?

Unless you want to reconfigure ghc, or hack on it's autoconf part, you are likely
going to run the following only:

./boot --hadrian
./configure <your flags of choice>
./hadrian/build.sh -j<N> ...
./hadrian/build.sh -j<N> ...
./hadrian/build.sh -j<N> ...
./hadrian/build.sh -j<N> ...
...

the configure step is required, and should be explicit. That is where you configure
your ghc build. Set host/build/target values, and other configure flags that
influence how you want your ghc to be configure. Hadrian is there to build that
configuration.  Mixing both may be convenient but hides the fact that there is a
./configure step.  I consider this hiding to be magic which is meant to benefit the
user but hides what's really going on.  And again I don't like magic.

Cheers,
 Moritz

PS: we also don't hide the `./configure` step in the usual `./configure <args> && make -j`
    instructions when building other software, even though you could surely hack that into
    your Makefile if you so wanted to.  Why start with ghc now?

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Discussion: Hadrian's defaults

Andrey Mokhov
In reply to this post by Spiwack, Arnaud
Hi Arnaud,

Great to hear you've been using Hadrian for a while and like it!

As others have already pointed, -c used to be default. I personally was always inclined to run boot and configure by default, because to me they seem like an unnecessary complication on the way to the first GHC build (they confused me a lot when I did my first GHC build). On the other hand, people who strongly oppose -c being default are all expert users of configure, and should be capable of passing --skip-configure when need be (or always).

It's good to discuss this and see what proportion of GHC developers would prefer to include -c by default. So far the opponents were much more vocal.

Making --freeze1 default is probably fine. If we go this way, there should be a big warning in the documentation saying that this default setting is convenient but unsafe, i.e. can produce wrong binaries.

I think --share is not ready to become default. Hopefully, it will be in 1-2 months!

Cheers,
Andrey


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Discussion: Hadrian's defaults

Alp Mestanogullari-2
In reply to this post by Moritz Angermann-2

I have to admit I sympathize with Moritz's view. Since `-c` only "subsumes" the case where we call 'configure' with no extra env var or argument, and in the absence of a generic way to pass options to 'configure' when using -c, I'd quite like to keep -c as a "cherry on top", for users who just need to boot and configure with the default arguments and for whom hadrian provides a way to save a few keystrokes. Just like Moritz, I'm not even sure it would make all that much sense to provide a way to pass configure arguments through hadrian, I have a hard time seeing a better UX than just running configure with the extra arguments directly and _then_ calling hadrian to start the build.

On 15/03/2019 14:35, Moritz Angermann wrote:
Hi Arnaud,

On Mar 15, 2019, at 8:32 PM, Spiwack, Arnaud [hidden email] wrote:

On Thu, Mar 14, 2019 at 7:20 PM Herbert Valerio Riedel [hidden email] wrote:
I don't have the ticket number at my fingertips but it should be fairly easy to find.

I'm afraid it doesn't appear to be. Could you share your arguments in this thread?
This was the last one that lead to the current `-c` state:
- https://github.com/snowleopard/hadrian/issues/457
There is also
- https://github.com/snowleopard/hadrian/issues/655

if you look through the issues on snowleopard/hadrian and sort by comment frequency
you'll likely find quite a lot of further discussion about not making configure and
boot the default.

On Fri, Mar 15, 2019 at 3:10 AM Moritz Angermann [hidden email] wrote:
It's magically conflating two different phases with `-c`. The configure phase and
the build phase. Making this the default means it's always magic. I don't like magic!

Unfortunately, I really don't understand what you are saying. What's magic about combining the phases?
We have two phases:

Phase 1: autoconf

  This phase is essentially a code-generation phase, where specific templates are
  instantiated to configure time value.  Which again can be split into two specific
  subproblems:

  - Generation of the configure script from the configure.ac and aclocal.m4 files
    using autoconf.
  - Generating code using the configure script by computing configure time calues
    and filling those into the `.in` files producing the files that lack the `.in`
    extension.

Phase 2: building

  This has been traditionally the job of make, and this is what hadrian should
  replace.


By subsuming the configure phase (by invoking ./configure) from hadrian we loose
the phase distinction and if the `-c` flag is optional, users will *not even see*
a flag that indicates that the system will run `./configure` for them. This is the
magic I'm referring to and to which I strongly object.  If we can retire autoconf
and do the whole configuration in hadrian, that story may change.  But as long as
we are using an autoconf based configuration we should *not* run that magically.
The `-c` flag is at least there to show that hadrian is explicilty instructed to
run configure.

./configure supports its own set of flags, if hadrian subsumes those, we'd need
some generic way of passing flags to ./configure, at which point I have to ask
why do we do this in the first place and try to call ./configure from within hadrian?

Unless you want to reconfigure ghc, or hack on it's autoconf part, you are likely
going to run the following only:

./boot --hadrian
./configure <your flags of choice>
./hadrian/build.sh -j<N> ...
./hadrian/build.sh -j<N> ...
./hadrian/build.sh -j<N> ...
./hadrian/build.sh -j<N> ...
...

the configure step is required, and should be explicit. That is where you configure
your ghc build. Set host/build/target values, and other configure flags that
influence how you want your ghc to be configure. Hadrian is there to build that
configuration.  Mixing both may be convenient but hides the fact that there is a
./configure step.  I consider this hiding to be magic which is meant to benefit the
user but hides what's really going on.  And again I don't like magic.

Cheers,
 Moritz

PS: we also don't hide the `./configure` step in the usual `./configure <args> && make -j`
    instructions when building other software, even though you could surely hack that into
    your Makefile if you so wanted to.  Why start with ghc now?

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
-- 
Alp Mestanogullari, Haskell Consultant
Well-Typed LLP, https://www.well-typed.com/

Registered in England and Wales, OC335890
118 Wymering Mansions, Wymering Road, London, W9 2NF, England

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Discussion: Hadrian's defaults

Brandon Allbery
In reply to this post by Moritz Angermann-2
Expanding on Moritz a bit:

configure creates a build plan, hadrian runs it.

Newcomers to autoconf-based ecosystems often add a rule to run configure to their Makefiles, as a "shortcut". At some time thereafter, they discover that there's a problem when configure alters the build plan make is executing. There's no sane way to integrate this properly if the build plan changes enough, even if configure is always the very first thing executed as part of the original build plan. This generally results in forcing make to abort after running autoconf, since trying to be more clever results in looping over configure. The only other way that works is arranging for 2-stage Makefiles where the outer one amounts to "configure && make -f Makefile.real". Which makes it noticeably more error-prone to tweak something and do a fast rebuild, since configure will often cause it to be a full rebuild instead. If you try to give configure dependencies so that it doesn't get re-run, you discover that it's dependent in some sense on its own outputs and again can cause loops.

Trying to integrate configure with hadrian runs the same risk, heightened by hadrian not using anything like Makefiles, so it would be even easier for it to either miss changes made by configure or end up looping on it. Or worse, the same with autoconf — or for ghc, the boot script which runs autoconf, among other things, and thus has even nastier risks in terms of missing important changes made thereby or causing loops.

Really, don't try to conflate these; someone is going to end up very unhappy.

On Fri, Mar 15, 2019 at 9:35 AM Moritz Angermann <[hidden email]> wrote:
Hi Arnaud,

> On Mar 15, 2019, at 8:32 PM, Spiwack, Arnaud <[hidden email]> wrote:
>
> On Thu, Mar 14, 2019 at 7:20 PM Herbert Valerio Riedel <[hidden email]> wrote:
> I don't have the ticket number at my fingertips but it should be fairly easy to find.
>
> I'm afraid it doesn't appear to be. Could you share your arguments in this thread?
This was the last one that lead to the current `-c` state:
- https://github.com/snowleopard/hadrian/issues/457
There is also
- https://github.com/snowleopard/hadrian/issues/655

if you look through the issues on snowleopard/hadrian and sort by comment frequency
you'll likely find quite a lot of further discussion about not making configure and
boot the default.

>
> On Fri, Mar 15, 2019 at 3:10 AM Moritz Angermann <[hidden email]> wrote:
> It's magically conflating two different phases with `-c`. The configure phase and
> the build phase. Making this the default means it's always magic. I don't like magic!
>
> Unfortunately, I really don't understand what you are saying. What's magic about combining the phases?

We have two phases:

Phase 1: autoconf

  This phase is essentially a code-generation phase, where specific templates are
  instantiated to configure time value.  Which again can be split into two specific
  subproblems:

  - Generation of the configure script from the configure.ac and aclocal.m4 files
    using autoconf.
  - Generating code using the configure script by computing configure time calues
    and filling those into the `.in` files producing the files that lack the `.in`
    extension.

Phase 2: building

  This has been traditionally the job of make, and this is what hadrian should
  replace.


By subsuming the configure phase (by invoking ./configure) from hadrian we loose
the phase distinction and if the `-c` flag is optional, users will *not even see*
a flag that indicates that the system will run `./configure` for them. This is the
magic I'm referring to and to which I strongly object.  If we can retire autoconf
and do the whole configuration in hadrian, that story may change.  But as long as
we are using an autoconf based configuration we should *not* run that magically.
The `-c` flag is at least there to show that hadrian is explicilty instructed to
run configure.

./configure supports its own set of flags, if hadrian subsumes those, we'd need
some generic way of passing flags to ./configure, at which point I have to ask
why do we do this in the first place and try to call ./configure from within hadrian?

Unless you want to reconfigure ghc, or hack on it's autoconf part, you are likely
going to run the following only:

./boot --hadrian
./configure <your flags of choice>
./hadrian/build.sh -j<N> ...
./hadrian/build.sh -j<N> ...
./hadrian/build.sh -j<N> ...
./hadrian/build.sh -j<N> ...
...

the configure step is required, and should be explicit. That is where you configure
your ghc build. Set host/build/target values, and other configure flags that
influence how you want your ghc to be configure. Hadrian is there to build that
configuration.  Mixing both may be convenient but hides the fact that there is a
./configure step.  I consider this hiding to be magic which is meant to benefit the
user but hides what's really going on.  And again I don't like magic.

Cheers,
 Moritz

PS: we also don't hide the `./configure` step in the usual `./configure <args> && make -j`
    instructions when building other software, even though you could surely hack that into
    your Makefile if you so wanted to.  Why start with ghc now?
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


--
brandon s allbery kf8nh

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Discussion: Hadrian's defaults

Ben Gamari-3
In reply to this post by Moritz Angermann-2
Moritz Angermann <[hidden email]> writes:

> Hi Arnaud,
>
...


I second everything that Moritz said. Having spent countless hours
fighting other projects' "non-traditional" build systems in the past, my
heart fills with dread whenever I see a project trying to be "clever"
with autoconf. Inevitably such cleverness ends up being a source
poorly-documented, unpredictable, and at times broken behavior which
inevitably complicates the task of building the project in any
non-trivial configuration (for instance, cross-compilation).

We should keep in mind that users and packagers, not GHC developers, are
the primary audience of GHC's build system. For their sake we should
strive to keep GHC's build system as close to the usual
configure/make/make install workflow as possible.

In fact, I have at times wondered whether when we switch to Hadrian as
our primary build system we should ship our source distributions with a
stub makefile to build and run hadrian, allowing downstreams to treat
the distribution as a normal autotools-based package.

Cheers,

-  Ben


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (497 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Discussion: Hadrian's defaults

Phyx
In reply to this post by Alp Mestanogullari-2
Agreed, I was also an opponent of -c as the default, and as you pointed out it only works for the case where the default is used. But even if the defaults are used it is still harmful to do it automatically as the user's environment could have changed resulting in different configure output if you automatically rerun it. 

Not only isn't there a sane UX for this, there isn't even a safe way to do this. 

Sent from my Mobile

On Fri, Mar 15, 2019, 16:51 Alp Mestanogullari <[hidden email]> wrote:

I have to admit I sympathize with Moritz's view. Since `-c` only "subsumes" the case where we call 'configure' with no extra env var or argument, and in the absence of a generic way to pass options to 'configure' when using -c, I'd quite like to keep -c as a "cherry on top", for users who just need to boot and configure with the default arguments and for whom hadrian provides a way to save a few keystrokes. Just like Moritz, I'm not even sure it would make all that much sense to provide a way to pass configure arguments through hadrian, I have a hard time seeing a better UX than just running configure with the extra arguments directly and _then_ calling hadrian to start the build.

On 15/03/2019 14:35, Moritz Angermann wrote:
Hi Arnaud,

On Mar 15, 2019, at 8:32 PM, Spiwack, Arnaud [hidden email] wrote:

On Thu, Mar 14, 2019 at 7:20 PM Herbert Valerio Riedel [hidden email] wrote:
I don't have the ticket number at my fingertips but it should be fairly easy to find.

I'm afraid it doesn't appear to be. Could you share your arguments in this thread?
This was the last one that lead to the current `-c` state:
- https://github.com/snowleopard/hadrian/issues/457
There is also
- https://github.com/snowleopard/hadrian/issues/655

if you look through the issues on snowleopard/hadrian and sort by comment frequency
you'll likely find quite a lot of further discussion about not making configure and
boot the default.

On Fri, Mar 15, 2019 at 3:10 AM Moritz Angermann [hidden email] wrote:
It's magically conflating two different phases with `-c`. The configure phase and
the build phase. Making this the default means it's always magic. I don't like magic!

Unfortunately, I really don't understand what you are saying. What's magic about combining the phases?
We have two phases:

Phase 1: autoconf

  This phase is essentially a code-generation phase, where specific templates are
  instantiated to configure time value.  Which again can be split into two specific
  subproblems:

  - Generation of the configure script from the configure.ac and aclocal.m4 files
    using autoconf.
  - Generating code using the configure script by computing configure time calues
    and filling those into the `.in` files producing the files that lack the `.in`
    extension.

Phase 2: building

  This has been traditionally the job of make, and this is what hadrian should
  replace.


By subsuming the configure phase (by invoking ./configure) from hadrian we loose
the phase distinction and if the `-c` flag is optional, users will *not even see*
a flag that indicates that the system will run `./configure` for them. This is the
magic I'm referring to and to which I strongly object.  If we can retire autoconf
and do the whole configuration in hadrian, that story may change.  But as long as
we are using an autoconf based configuration we should *not* run that magically.
The `-c` flag is at least there to show that hadrian is explicilty instructed to
run configure.

./configure supports its own set of flags, if hadrian subsumes those, we'd need
some generic way of passing flags to ./configure, at which point I have to ask
why do we do this in the first place and try to call ./configure from within hadrian?

Unless you want to reconfigure ghc, or hack on it's autoconf part, you are likely
going to run the following only:

./boot --hadrian
./configure <your flags of choice>
./hadrian/build.sh -j<N> ...
./hadrian/build.sh -j<N> ...
./hadrian/build.sh -j<N> ...
./hadrian/build.sh -j<N> ...
...

the configure step is required, and should be explicit. That is where you configure
your ghc build. Set host/build/target values, and other configure flags that
influence how you want your ghc to be configure. Hadrian is there to build that
configuration.  Mixing both may be convenient but hides the fact that there is a
./configure step.  I consider this hiding to be magic which is meant to benefit the
user but hides what's really going on.  And again I don't like magic.

Cheers,
 Moritz

PS: we also don't hide the `./configure` step in the usual `./configure <args> && make -j`
    instructions when building other software, even though you could surely hack that into
    your Makefile if you so wanted to.  Why start with ghc now?

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
-- 
Alp Mestanogullari, Haskell Consultant
Well-Typed LLP, https://www.well-typed.com/

Registered in England and Wales, OC335890
118 Wymering Mansions, Wymering Road, London, W9 2NF, England
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Discussion: Hadrian's defaults

Spiwack, Arnaud
In reply to this post by Brandon Allbery
On Fri, Mar 15, 2019 at 4:52 PM Brandon Allbery <[hidden email]> wrote:
Newcomers to autoconf-based ecosystems often add a rule to run configure to their Makefiles, as a "shortcut". At some time thereafter, they discover that there's a problem when configure alters the build plan make is executing. There's no sane way to integrate this properly if the build plan changes enough, even if configure is always the very first thing executed as part of the original build plan. This generally results in forcing make to abort after running autoconf, since trying to be more clever results in looping over configure. The only other way that works is arranging for 2-stage Makefiles where the outer one amounts to "configure && make -f Makefile.real". Which makes it noticeably more error-prone to tweak something and do a fast rebuild, since configure will often cause it to be a full rebuild instead. If you try to give configure dependencies so that it doesn't get re-run, you discover that it's dependent in some sense on its own outputs and again can cause loops.

Trying to integrate configure with hadrian runs the same risk, heightened by hadrian not using anything like Makefiles, so it would be even easier for it to either miss changes made by configure or end up looping on it. Or worse, the same with autoconf — or for ghc, the boot script which runs autoconf, among other things, and thus has even nastier risks in terms of missing important changes made thereby or causing loops.

 Aha! But is Hadrian doesn't use Make. And the shortcoming of Make which make autoconf necessary as a separate tools are ones that Shake fix. Making it a breeze to integrate a configure script without hassle.

I'll be honest, so far, so far, the thread doesn't have much convincing arguments in favour of keeping `-c`. The argument which I understand is Ben's in [ https://github.com/snowleopard/hadrian/issues/457 ], who saw the arguments he passed to ./configure overwritten by Hadrian. This is surprising and unnecessary.

But you have to realise what you are advocating for: the most typical workflow (not passing special arguments to ./configure) is the one which is special cased. We should rather take this as a challenge: how do we fix the limitations of `-c` so that it can be made the default; rather than giving up. A possibility would be to record the arguments passed to the configure script in a file, and make Hadrian pass these arguments to `./configure`. For instance.

I'd also like to impart a bit of urgency to this: we are breaking everybody's habit now, so now is the time to discuss changes. When we're used to the new build system and it isn't new anymore, it will become much harder to justify change.

By the way, on the default flavour: in addition to Ocaml's Dune, it appears that Bazel (Google's newish multi-language build system) also has the developers' build be the default. It's not enough data to establish that this default is the consensus. But I'm quite warming up to the position myself.

Best,
Arnaud

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Discussion: Hadrian's defaults

Herbert Valerio Riedel-3
In reply to this post by Ben Gamari-3
On 2019-03-15 at 13:28:21 -0400, Ben Gamari wrote:

[...]

> We should keep in mind that users and packagers, not GHC developers, are
> the primary audience of GHC's build system. For their sake we should
> strive to keep GHC's build system as close to the usual
> configure/make/make install workflow as possible.

Actually, even as a GHC developer, I strongly prefer the IMO quite
intuitive configure/make phase-separated workflow I've been used to for
over 2 decades. It's deeply ingrained into my muscle memory and my
tooling automatically knows what to do when it discovers a `configure`
script and/or a `GNUmakefile`/`Makefile` file in the filesystem and I'm
not excited to have to adapt all my dev scripts/integrations to make
them Hadrian-aware which is effectively a bespoken build-system used by
exactly one project and starting with a specific version of the project
while the previous ones still require Makefiles.

> In fact, I have at times wondered whether when we switch to Hadrian as
> our primary build system we should ship our source distributions with a
> stub makefile to build and run hadrian, allowing downstreams to treat
> the distribution as a normal autotools-based package.

Definitely, I've been wanting this as well! It's perfectly fine IMO to
have it be a best-effort leaky abstraction. It'd be great if the
following standard invocations kept working, which represents the usage
pattern I use both in packaging scripts as well as in my dev-workflow
scripts.

The less I have to be aware of Hadrian pulling the strings behind the
curtains for my 80% use-cases the better. And if I need to do something
from the remaining 20% of use-cases that's when I'd be willing to invoke
Hadrian directly, as that's also when I'd be doing something
non-standard/non-routine anyway and I'd be consciously paying attention
anyway...


,----
| # an example build.mk with some of the keys I typically set one way or
| # another; just support a flat simple key/value grammar as text-file
| # config input to Hadrian
|
| cat > mk/build.mk  <<EOF
| #INTEGER_LIBRARY   = integer-simple
| BUILD_DOCBOOK_HTML = NO
| BUILD_DOCBOOK_PDF  = NO
| BUILD_DOCBOOK_PS   = NO
| BUILD_MAN          = NO
| BUILD_PROF_LIBS    = NO
| BUILD_SPHINX_HTML  = NO
| BUILD_SPHINX_PDF   = NO
| DYNAMIC_BY_DEFAULT   = NO
| DYNAMIC_GHC_PROGRAMS = NO
| GhcLibWays         = v dyn p
| HADDOCK_DOCS       = NO
| HSCOLOUR_SRCS      = NO
| LATEX_DOCS         = NO
| V                  = 0
| EOF
|
| ./boot
| ./configure --prefix /opt/ghc/HEAD ...otherflags...
|
| # the usual targets I use
|
| make clean
| make distclean
| make all
| make install
| make sdist
| make binary-dist
`----
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Discussion: Hadrian's defaults

Phyx
In reply to this post by Spiwack, Arnaud
> I'll be honest, so far, so far, the thread doesn't have much convincing arguments in favour of keeping `-c`. The argument which I understand is Ben's in [ https://github.com/snowleopard/hadrian/issues/457 ], who saw the arguments he passed to ./configure overwritten by Hadrian. This is surprising and unnecessary.

How about my reply in that it does not work. Configure isn't make.it never was. The whole point of configure is to perform configuration checks based on your environment. 

There are two real technical reasons that Hadrian cannot and should not run configure again on its own. 

1) you cannot tell what arguments the configure was originally called with. Your assumption here is that everyone calls make with no arguments. This is simply not true. Yes the arguments are stored somewhat mangled in config.status but extracting them will prove very fragile. 

2) Hadrian cannot restore the environment to what it was when configure was first called. Simple example CC=/foo/custom/compiler ./configure. 

Hadrain's -c by default would blow away this CC value and produce a config that would use something else. Giving me an abi issue that's harder to track down than the amount of effort it takes for someone to put -c. 

Configure by default has a much larger chance of producing invalid code than not having it by default. 

There is a very good reason why autoconf and automake are different. They have different goals. Hadrian is a replacement for make and shouldn't try to dictate what autoconf does. 

Tamar. 


On Tue, Mar 19, 2019, 08:42 Spiwack, Arnaud <[hidden email]> wrote:
On Fri, Mar 15, 2019 at 4:52 PM Brandon Allbery <[hidden email]> wrote:
Newcomers to autoconf-based ecosystems often add a rule to run configure to their Makefiles, as a "shortcut". At some time thereafter, they discover that there's a problem when configure alters the build plan make is executing. There's no sane way to integrate this properly if the build plan changes enough, even if configure is always the very first thing executed as part of the original build plan. This generally results in forcing make to abort after running autoconf, since trying to be more clever results in looping over configure. The only other way that works is arranging for 2-stage Makefiles where the outer one amounts to "configure && make -f Makefile.real". Which makes it noticeably more error-prone to tweak something and do a fast rebuild, since configure will often cause it to be a full rebuild instead. If you try to give configure dependencies so that it doesn't get re-run, you discover that it's dependent in some sense on its own outputs and again can cause loops.

Trying to integrate configure with hadrian runs the same risk, heightened by hadrian not using anything like Makefiles, so it would be even easier for it to either miss changes made by configure or end up looping on it. Or worse, the same with autoconf — or for ghc, the boot script which runs autoconf, among other things, and thus has even nastier risks in terms of missing important changes made thereby or causing loops.

 Aha! But is Hadrian doesn't use Make. And the shortcoming of Make which make autoconf necessary as a separate tools are ones that Shake fix. Making it a breeze to integrate a configure script without hassle.

I'll be honest, so far, so far, the thread doesn't have much convincing arguments in favour of keeping `-c`. The argument which I understand is Ben's in [ https://github.com/snowleopard/hadrian/issues/457 ], who saw the arguments he passed to ./configure overwritten by Hadrian. This is surprising and unnecessary.

But you have to realise what you are advocating for: the most typical workflow (not passing special arguments to ./configure) is the one which is special cased. We should rather take this as a challenge: how do we fix the limitations of `-c` so that it can be made the default; rather than giving up. A possibility would be to record the arguments passed to the configure script in a file, and make Hadrian pass these arguments to `./configure`. For instance.

I'd also like to impart a bit of urgency to this: we are breaking everybody's habit now, so now is the time to discuss changes. When we're used to the new build system and it isn't new anymore, it will become much harder to justify change.

By the way, on the default flavour: in addition to Ocaml's Dune, it appears that Bazel (Google's newish multi-language build system) also has the developers' build be the default. It's not enough data to establish that this default is the consensus. But I'm quite warming up to the position myself.

Best,
Arnaud
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Discussion: Hadrian's defaults

Brandon Allbery
In reply to this post by Spiwack, Arnaud
I feel like you didn't read this completely. I used make as the example, but hadrian is also a build system and will have some version of the same problem: configure's dependencies are also its outputs. Hadrian might be able to handle this, if it recomputes the entire dependency tree after every operation — but any mistakes in the dependencies will lead to loops.

And none of this addresses the point that if configure is in any way automatic, including as a separate initialization operation before dependencies are computed, it's going to make life difficult for developers who want to restart a build. Granting the viewpoint that says that end-builder ease is more important than developer ease, but even there if you make things too difficult for developers, a project that is already short on developers may become even more short on developers and/or developer time.

(I also think decisions like whether and how to run configure and/or the build tooling should be part of my local configuration, because my own needs will differ from others'. We already see this with devs who prefer e.g. quick vs. perf builds.)

By far the safest approach is to keep configure separate.

On Tue, Mar 19, 2019 at 4:42 AM Spiwack, Arnaud <[hidden email]> wrote:
On Fri, Mar 15, 2019 at 4:52 PM Brandon Allbery <[hidden email]> wrote:
Newcomers to autoconf-based ecosystems often add a rule to run configure to their Makefiles, as a "shortcut". At some time thereafter, they discover that there's a problem when configure alters the build plan make is executing. There's no sane way to integrate this properly if the build plan changes enough, even if configure is always the very first thing executed as part of the original build plan. This generally results in forcing make to abort after running autoconf, since trying to be more clever results in looping over configure. The only other way that works is arranging for 2-stage Makefiles where the outer one amounts to "configure && make -f Makefile.real". Which makes it noticeably more error-prone to tweak something and do a fast rebuild, since configure will often cause it to be a full rebuild instead. If you try to give configure dependencies so that it doesn't get re-run, you discover that it's dependent in some sense on its own outputs and again can cause loops.

Trying to integrate configure with hadrian runs the same risk, heightened by hadrian not using anything like Makefiles, so it would be even easier for it to either miss changes made by configure or end up looping on it. Or worse, the same with autoconf — or for ghc, the boot script which runs autoconf, among other things, and thus has even nastier risks in terms of missing important changes made thereby or causing loops.

 Aha! But is Hadrian doesn't use Make. And the shortcoming of Make which make autoconf necessary as a separate tools are ones that Shake fix. Making it a breeze to integrate a configure script without hassle.

I'll be honest, so far, so far, the thread doesn't have much convincing arguments in favour of keeping `-c`. The argument which I understand is Ben's in [ https://github.com/snowleopard/hadrian/issues/457 ], who saw the arguments he passed to ./configure overwritten by Hadrian. This is surprising and unnecessary.

But you have to realise what you are advocating for: the most typical workflow (not passing special arguments to ./configure) is the one which is special cased. We should rather take this as a challenge: how do we fix the limitations of `-c` so that it can be made the default; rather than giving up. A possibility would be to record the arguments passed to the configure script in a file, and make Hadrian pass these arguments to `./configure`. For instance.

I'd also like to impart a bit of urgency to this: we are breaking everybody's habit now, so now is the time to discuss changes. When we're used to the new build system and it isn't new anymore, it will become much harder to justify change.

By the way, on the default flavour: in addition to Ocaml's Dune, it appears that Bazel (Google's newish multi-language build system) also has the developers' build be the default. It's not enough data to establish that this default is the consensus. But I'm quite warming up to the position myself.

Best,
Arnaud


--
brandon s allbery kf8nh

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs