Build system idea

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
50 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Build system idea

Simon Marlow-7
Simon PJ and I had a talk about the build system earlier today, I thought
I'd float the idea we discussed (I should admit that the idea was mine,
lest Simon PJ think I'm attributing bad ideas to him :-).  This is not
completely thought through, but I'm pretty sure a solution exists along
these lines that would improve things for us.

Ok, the starting point is this:

  - Cabal has code to generate Makefiles.  Almost nobody uses it except
    for the GHC build system.  It essentially duplicates the build system
    for compiling Haskell source (but not for installation, haddocking,
    registration, configuration, etc.)

  - Cabal is a library

I propose we do this:

  - Extract the code from Cabal that generates Makefiles, and treat it as
    part of the GHC build system.  Rather than generating a Makefile
    complete with build rules, we generate a Makefile that just
    has the package-specific metadata (list of modules, etc.), and put
    the code to actually build the package in the GHC build system.

This means we still get to use 'make', we still get to use the .cabal files
as metadata, but the build system is more private to GHC, more extensible,
and hopefully more understandable and modifiable.  We can express
dependencies that Cabal currently doesn't know about.  It would let us
avoid the current uncomfortable situation where we have to feed all kinds
of configuration information from the GHC build system into Cabal - Cabal
would be essentially just a mechanism for translating the .cabal file into
Makefile bindings and package metadata for ghc-pkg.

There will undoubtedly be some sticking points where we have to tradeoff
duplicating things from Cabal against re-using parts of Cabal which might
require modifying Cabal itself.  For instance, we could use Cabal for
installation, but that means that our build system has to leave everything
in the places that Cabal's installation code expects, so it might be more
feasible to do installation ourselves, but that means duplicating parts of
Cabal.

It will probably mean that we have a tighter dependency on Cabal, because
we use it as a library rather than a black box; but hopefully we can keep
our branch of Cabal more stable and not have to update it so often.

Anyway, this is an idea that I think is interesting.  Obviously it needs a
lot more fleshing out to be a real proposal, but I'm interested in whether
anyone thinks this idea is worth persuing, or whether there are better
alternatives.

Cheers,
        Simon
_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Build system idea

Duncan Coutts
On Tue, 2008-08-12 at 11:11 +0100, Simon Marlow wrote:

> I propose we do this:
>
>   - Extract the code from Cabal that generates Makefiles, and treat it as
>     part of the GHC build system.  Rather than generating a Makefile
>     complete with build rules, we generate a Makefile that just
>     has the package-specific metadata (list of modules, etc.), and put
>     the code to actually build the package in the GHC build system.

As you know, I've been trying to get rid of that code ever since it
arrived :-)

> It will probably mean that we have a tighter dependency on Cabal, because
> we use it as a library rather than a black box; but hopefully we can keep
> our branch of Cabal more stable and not have to update it so often.

If you don't need to update so often it would make life easier for Cabal
hackers and Manuel would be pleased :-)

> Anyway, this is an idea that I think is interesting.  Obviously it needs a
> lot more fleshing out to be a real proposal, but I'm interested in whether
> anyone thinks this idea is worth persuing, or whether there are better
> alternatives.

Right, so probably the crucial thing is how much you end up having to
duplicate and of how much of said duplicated infrastructure has to be
kept in sync. For example if the path layout is different does that make
Cabal's haddocking support not work forcing that to be duplicated too?

Duncan

_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Build system idea

Malcolm Wallace
In reply to this post by Simon Marlow-7
Simon Marlow <[hidden email]> wrote:

> This means we still get to use 'make', we still get to use the .cabal
> files  as metadata, but the build system is more private to GHC, more
> extensible,  and hopefully more understandable and modifiable.

This is essentially the same approach that nhc98 currently takes to
building libraries.  The Cabal file holds all the metadata, but the
build system is Makefile-driven.  There is a small separate tool
(CabalParse) that extracts metadata from the cabal file.  The Cabal
*library* could be used to implement that extraction tool, but currently
ours is hand-rolled.  (One of the benefits of open specifications of
file formats is that you can have multiple implementations for different
purposes.)

Here is an example of how it works:

CABALFILE  = $(shell ls *.cabal | head -n 1 )
READ       = $(CABALPARSE) $(CABALFILE) -quiet
MAP        = $(LOCAL)map

THISPKG    = $(shell $(READ) name | cut -c2- )
VERSION    = $(shell $(READ) version)
SEARCH     = $(shell $(READ) build-depends  | $(MAP) "echo -package" ) \
             $(shell $(READ) include-dirs   | $(MAP) "echo -i" | cut -c1,2,4-) \
             $(shell $(READ) hs-source-dir  | $(MAP) "echo -I" | cut -c1,2,4-) \
             $(shell $(READ) hs-source-dirs | $(MAP) "echo -I" | cut -c1,2,4-)
CINCLUDES  = $(shell $(READ) include-dirs   | $(MAP) "echo -I" | cut -c1,2,4-)
SRCS       = $(shell $(READ) -slash exposed-modules)
EXTRA_SRCS = $(shell $(READ) -slash other-modules)
SRCS_C     = $(shell $(READ) c-sources)
DIRS       = $(shell $(READ) -slash exposed-modules other-modules \
                     | $(MAP) dirname | sort | uniq )
EXTRA_C_FLAGS = $(shell $(READ) cc-options)
EXTRA_H_FLAGS = $(shell $(READ) nhc98-options)


Regards,
    Malcolm
_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Build system idea

Simon Marlow-7
Malcolm Wallace wrote:
> Simon Marlow <[hidden email]> wrote:
>
>> This means we still get to use 'make', we still get to use the .cabal
>> files  as metadata, but the build system is more private to GHC, more
>> extensible,  and hopefully more understandable and modifiable.
>
> This is essentially the same approach that nhc98 currently takes to
> building libraries.

Right, I was aware that nhc98 uses this method but forgot to mention it.
Thanks for pointing it out.

I think it makes a lot more sense for us to re-use parts of Cabal than to
re-implement the whole thing, although the balance is probably different
for nhc98.  Cabal generates the InstalledPackageInfo from the .cabal file,
for example, and this is certainly something we don't want to re-implement.

Cheers,
        Simon
_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Build system idea

Norman Ramsey
In reply to this post by Simon Marlow-7
 > Simon PJ and I had a talk about the build system earlier today, I thought
 > I'd float the idea we discussed...
 > I propose we do this:
 >
 >   - Extract the code from Cabal that generates Makefiles, and treat it as
 >     part of the GHC build system.  Rather than generating a Makefile
 >     complete with build rules, we generate a Makefile that just
 >     has the package-specific metadata (list of modules, etc.), and put
 >     the code to actually build the package in the GHC build system.
 >
 > This means we still get to use 'make', we still get to use the .cabal files
 > as metadata, but the build system is more private to GHC, more extensible,
 > and hopefully more understandable and modifiable...
 >
 > ... I'm interested in whether anyone thinks this idea is worth
 > persuing, or whether there are better alternatives.

Simon,

This direction sounds very promising.  I hope you will keep us posted.


Norman
_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Build system idea

Manuel M T Chakravarty
In reply to this post by Duncan Coutts
Duncan Coutts:

> On Tue, 2008-08-12 at 11:11 +0100, Simon Marlow wrote:
>
>> I propose we do this:
>>
>>  - Extract the code from Cabal that generates Makefiles, and treat  
>> it as
>>    part of the GHC build system.  Rather than generating a Makefile
>>    complete with build rules, we generate a Makefile that just
>>    has the package-specific metadata (list of modules, etc.), and put
>>    the code to actually build the package in the GHC build system.
>
> As you know, I've been trying to get rid of that code ever since it
> arrived :-)
>
>> It will probably mean that we have a tighter dependency on Cabal,  
>> because
>> we use it as a library rather than a black box; but hopefully we  
>> can keep
>> our branch of Cabal more stable and not have to update it so often.
>
> If you don't need to update so often it would make life easier for  
> Cabal
> hackers and Manuel would be pleased :-)

Yes!

>> Anyway, this is an idea that I think is interesting.  Obviously it  
>> needs a
>> lot more fleshing out to be a real proposal, but I'm interested in  
>> whether
>> anyone thinks this idea is worth persuing, or whether there are  
>> better
>> alternatives.

I think this is definitely an interesting idea.  At the moment, it  
seems to me that all the metadata handling of Cabal is what's most  
useful to GHC, whereas the actual build procedure and its  
inflexibility causes a lot of grief, especially if you want to do  
something non-standard.  The proposed idea would pick the best of both  
worlds (Cabal's metadata handling and make's build flexibility plus  
the fact that many more people know how to tweak makefiles even if it  
is a pain, but its pretty well understood pain).

Manuel

_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Build system idea

Roman Leshchinskiy
In reply to this post by Simon Marlow-7
On 12/08/2008, at 20:11, Simon Marlow wrote:

> - Extract the code from Cabal that generates Makefiles, and treat it  
> as
>   part of the GHC build system.  Rather than generating a Makefile
>   complete with build rules, we generate a Makefile that just
>   has the package-specific metadata (list of modules, etc.), and put
>   the code to actually build the package in the GHC build system.

Sounds good. It would be nice if the .cabal parser from Cabal could be  
made into a separate, stable library which ghc (and nhc?) could use.

This makes me wonder, though. Wouldn't this model make more sense for  
Cabal in general than the current approach of duplicating the  
functionality of autoconf, make and other stuff? If it works ghc, it  
ought to work for other projects, too. Cabal as a preprocessor seems  
much more attractive to me than as a universal build system.

Roman


_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Build system idea

Simon Marlow-7
Roman Leshchinskiy wrote:

> On 12/08/2008, at 20:11, Simon Marlow wrote:
>
>> - Extract the code from Cabal that generates Makefiles, and treat it as
>>   part of the GHC build system.  Rather than generating a Makefile
>>   complete with build rules, we generate a Makefile that just
>>   has the package-specific metadata (list of modules, etc.), and put
>>   the code to actually build the package in the GHC build system.
>
> Sounds good. It would be nice if the .cabal parser from Cabal could be
> made into a separate, stable library which ghc (and nhc?) could use.
>
> This makes me wonder, though. Wouldn't this model make more sense for
> Cabal in general than the current approach of duplicating the
> functionality of autoconf, make and other stuff? If it works ghc, it
> ought to work for other projects, too. Cabal as a preprocessor seems
> much more attractive to me than as a universal build system.

So packages would be required to provide their own build system?  That
sounds like it would make it a lot harder for people to just create a
package that others can use.  The ease of making a Cabal package has I
think a lot to do with the wealth of software available on Hackage.

GHC is a special case: we already need a build system for other reasons.

It was a design decision early on with Cabal that we didn't want to rely
on the target system having a Unix-like build environment.  You might
disagree with this, but it certainly has some value: a Windows user can
download GHC and immediately start building and installing external
packages without having to install Cygwin.

Cheers,
        Simon
_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Build system idea

Roman Leshchinskiy
On 13/08/2008, at 17:47, Simon Marlow wrote:

> Roman Leshchinskiy wrote:
>> On 12/08/2008, at 20:11, Simon Marlow wrote:
>>> - Extract the code from Cabal that generates Makefiles, and treat  
>>> it as
>>>  part of the GHC build system.  Rather than generating a Makefile
>>>  complete with build rules, we generate a Makefile that just
>>>  has the package-specific metadata (list of modules, etc.), and put
>>>  the code to actually build the package in the GHC build system.
>> Sounds good. It would be nice if the .cabal parser from Cabal could  
>> be made into a separate, stable library which ghc (and nhc?) could  
>> use.
>> This makes me wonder, though. Wouldn't this model make more sense  
>> for Cabal in general than the current approach of duplicating the  
>> functionality of autoconf, make and other stuff? If it works ghc,  
>> it ought to work for other projects, too. Cabal as a preprocessor  
>> seems much more attractive to me than as a universal build system.
>
> So packages would be required to provide their own build system?  
> That sounds like it would make it a lot harder for people to just  
> create a package that others can use.  The ease of making a Cabal  
> package has I think a lot to do with the wealth of software  
> available on Hackage.

Of course there should be a standard build system for simple packages.  
It could be part of Cabal or a separate tool (for which Cabal could,  
again, act as a preprocessor).

> GHC is a special case: we already need a build system for other  
> reasons.

I agree. I just don't think that adding a full-fledged build system to  
Cabal is the solution. In my experience, huge monolithic tools which  
try to do everything never work well. I much prefer small, modular  
tools. A Haskell-based build system is an interesting project but why  
does it have to be a part of Cabal?

> It was a design decision early on with Cabal that we didn't want to  
> rely on the target system having a Unix-like build environment.  You  
> might disagree with this, but it certainly has some value: a Windows  
> user can download GHC and immediately start building and installing  
> external packages without having to install Cygwin.

I agree with this decision but IIUC, this only really works for simple  
(wrt building) packages which don't even use configure. Making Cabal  
into a modular preprocessor and providing a thin wrapper for ghc --
make which can act as a target for Cabal would achieve this just as  
well.

Roman


_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Build system idea

Simon Marlow-7
Roman Leshchinskiy wrote:

> Of course there should be a standard build system for simple packages.
> It could be part of Cabal or a separate tool (for which Cabal could,
> again, act as a preprocessor).
>
>> GHC is a special case: we already need a build system for other reasons.
>
> I agree. I just don't think that adding a full-fledged build system to
> Cabal is the solution. In my experience, huge monolithic tools which try
> to do everything never work well. I much prefer small, modular tools. A
> Haskell-based build system is an interesting project but why does it
> have to be a part of Cabal?

Hmm, but you said above "there should be a standard build system for simple
packages.  It could be part of Cabal...".

Cabal has two parts: some generic infrastructure, and a "simple" build
system (under Distribution.Simple) that suffices for most packages.  We
distribute them together only because it's convenient; you don't have to
use the simple build system if you don't want to.

I think perhaps you're objecting to the fact that the "simple" build system
isn't so simple, and we keep adding more functionality to it.  This is
true, but the alternative - forcing some packages to provide their own
build system - seems worse to me.

Cheers,
        Simon
_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Build system idea

Duncan Coutts
On Wed, 2008-08-13 at 11:34 +0100, Simon Marlow wrote:

> Cabal has two parts: some generic infrastructure, and a "simple" build
> system (under Distribution.Simple) that suffices for most packages.  We
> distribute them together only because it's convenient; you don't have to
> use the simple build system if you don't want to.

The two parts also have different degrees of stability. In particular
there are lots of tools that rely on the declarative parts, the types
and parsers so we try not to break those so often. Roman asks for a
"separate, stable library which ghc (and nhc?) could use" but since this
part of Cabal is fairly stable I don't think it needs to be separate. I
don't think it'd be more stable by being in a different package. The
reasons to change it are usually to add new fields, and that usually
does not affect clients that do not need to know about the new fields.

> I think perhaps you're objecting to the fact that the "simple" build system
> isn't so simple, and we keep adding more functionality to it.  This is
> true, but the alternative - forcing some packages to provide their own
> build system - seems worse to me.

As Isaac used to say, it's not the Simple build system because it's
simple. It's because it does complex things to simple packages.

The Make build type was supposed to let people wrap existing make-based
build systems. Unfortunately it's not used much so is not well developed
and for ghc is really the wrong way round. I think your approach of
exporting the info in make syntax makes more sense for ghc, since it's
not trying to pretend it's a cabal package anyway (which is what the
Make build type was for, wrapping things up so people building packages
didn't need to know what was used underneath).

Duncan

_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Build system idea

Roman Leshchinskiy
In reply to this post by Simon Marlow-7
On 13/08/2008, at 20:34, Simon Marlow wrote:

> Roman Leshchinskiy wrote:
>
>> Of course there should be a standard build system for simple  
>> packages. It could be part of Cabal or a separate tool (for which  
>> Cabal could, again, act as a preprocessor).
>>> GHC is a special case: we already need a build system for other  
>>> reasons.
>> I agree. I just don't think that adding a full-fledged build system  
>> to Cabal is the solution. In my experience, huge monolithic tools  
>> which try to do everything never work well. I much prefer small,  
>> modular tools. A Haskell-based build system is an interesting  
>> project but why does it have to be a part of Cabal?
>
> Hmm, but you said above "there should be a standard build system for  
> simple packages.  It could be part of Cabal...".

On second thought, it shouldn't be part of Cabal :-)

> Cabal has two parts: some generic infrastructure, and a "simple"  
> build system (under Distribution.Simple) that suffices for most  
> packages.  We distribute them together only because it's convenient;  
> you don't have to use the simple build system if you don't want to.

My impression of Cabal is that it is a build system with a bit of  
generic infrastructure. In particular, a large part of the .cabal  
syntax is specified in terms of this build system and some of it only  
really makes sense for this build system.

> I think perhaps you're objecting to the fact that the "simple" build  
> system isn't so simple, and we keep adding more functionality to  
> it.  This is true, but the alternative - forcing some packages to  
> provide their own build system - seems worse to me.


Cabal packages do provide their own build system; it's just that they  
use Cabal syntax instead of, say, make. The advantage of doing this  
is, of course, that Cabal's syntax is simpler. Adding things to the  
"simple" build system erodes this advantage. Complex projects will  
still have complex build systems - the complexity will be in  
the .cabal files. If Cabal's goal is to be able to build any project  
it will basically have to duplicate the functionality of autoconf,  
automake, libtool, make and a couple of other tools *and* be just as  
flexible. I think this is neither realistic nor necessary. So where do  
we stop? And what about the packages that Cabal won't support when we  
stop?

IMO, we should have stopped some time ago. A .cabal file should  
describe a package, not how to build it. Building should be handled by  
different tools with a clear interface between them and Cabal. If the  
build system of choice needs additional information, then that  
information should be provided in a separate file and not in the  
package description.

Again, I'm not arguing against a build system written in Haskell. I'd  
just like it to be completely separated from Haskell's packaging  
system. In particular, "polluting" a package description with build  
information seems wrong to me.

Roman


_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Build system idea

Duncan Coutts
On Wed, 2008-08-13 at 22:47 +1000, Roman Leshchinskiy wrote:

> Again, I'm not arguing against a build system written in Haskell. I'd  
> just like it to be completely separated from Haskell's packaging  
> system. In particular, "polluting" a package description with build  
> information seems wrong to me.

There is a huge overlap of course. The things needed to build a package
tend to be the dependencies. The ability to automatically extract the
dependencies from a package description is crucial as it is what enables
automatic package management either directly or by conversion to distro
packages. Tools like automake + autoconf do not give us that.

There is of course some separation possible, which in Cabal roughly
corresponds to the stuff under Distribution.Simple vs everything else.
We could split those two aspects into separate packages but it's not
clear to me that we'd gain much by doing that.

There is still the Make build type which we could improve if people want
it. That allows the declarative stuff to be given in the .cabal file (so
that package managers can do their thing) and all the building is
delegated to make. People have not shown any interest in this so it's
never been improved much. The obvious disadvantage of using it is that
you have to do a lot of work to make your build system do all the things
that users expect.

Duncan

_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Build system idea

Roman Leshchinskiy
On 14/08/2008, at 06:32, Duncan Coutts wrote:

> On Wed, 2008-08-13 at 22:47 +1000, Roman Leshchinskiy wrote:
>
>> Again, I'm not arguing against a build system written in Haskell. I'd
>> just like it to be completely separated from Haskell's packaging
>> system. In particular, "polluting" a package description with build
>> information seems wrong to me.
>
> There is a huge overlap of course. The things needed to build a  
> package
> tend to be the dependencies. The ability to automatically extract the
> dependencies from a package description is crucial as it is what  
> enables
> automatic package management either directly or by conversion to  
> distro
> packages. Tools like automake + autoconf do not give us that.

Right. Dependencies are part of a package description. That's what  
Cabal should do. It should provide a nice clean interface to the  
dependencies stuff for the build system to use. I don't think it does  
that at the moment; IIUC, it is all done by Distribution.Simple.

> There is of course some separation possible, which in Cabal roughly
> corresponds to the stuff under Distribution.Simple vs everything else.
> We could split those two aspects into separate packages but it's not
> clear to me that we'd gain much by doing that.

My point isn't really about distribution, it's about coupling. My  
concern is that the syntax of .cabal files is increasingly based on  
what Distribution.Simple needs. This effectively makes all other build  
systems second class. It also loses us clean package descriptions  
which is what .cabal files should be. It's not too bad at the moment  
but will get worse as Distribution.Simple gets more complex since it  
will need more and more information.

Just as an example, consider something like ld-options. This is  
obviously not a dependency and is basically only documented by how it  
is used by Distribution.Simple. It shouldn't be in .cabal, IMO. If a  
build system needs this information, it should be provided somewhere  
else.

> There is still the Make build type which we could improve if people  
> want
> it. That allows the declarative stuff to be given in the .cabal file  
> (so
> that package managers can do their thing) and all the building is
> delegated to make. People have not shown any interest in this so it's
> never been improved much. The obvious disadvantage of using it is that
> you have to do a lot of work to make your build system do all the  
> things
> that users expect.

But that is precisely my (other) point. A lot of that work is really  
unnecessary and could be done by Cabal since it only or mostly depends  
on the package information. Instead, it is implemented somewhere in  
Distribution.Simple and not really usable from the outside. For  
instance, a lot of the functionality of setup sdist, setup register  
and so on could be implemented generically and used by a make-based  
build system as well. Also, there is no easy way for build systems to  
work with the declarative stuff because a lot of that functionality  
is, again, part of Distribution.Simple. IMO, this is a direct result  
of the tight coupling between the package management and build system  
parts of Cabal.

The other problem, of course, is that it isn't clear what exactly a  
build system should provide. IIUC, that's what "Building and  
installing a package" in the Cabal manual defines but there, we have  
things like this:

setup test

Run the test suite specified by the runTests field of  
Distribution.Simple.UserHooks. See Distribution.Simple for information  
about creating hooks and using defaultMainWithHooks.

As a matter of fact, a lot of Cabal is documented in terms of what  
Distribution.Simple does. Again, this effectively shuts out other  
build systems.

I'm sorry if this all sounds too negative, it shouldn't really. I  
think you guys have done a great job in implementing a system which is  
obviously very important to the community. I just somewhat disagree  
with the direction in which it is heading now.

Roman


_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Build system idea

Simon Marlow-7
Roman Leshchinskiy wrote:

> But that is precisely my (other) point. A lot of that work is really
> unnecessary and could be done by Cabal since it only or mostly depends
> on the package information. Instead, it is implemented somewhere in
> Distribution.Simple and not really usable from the outside. For
> instance, a lot of the functionality of setup sdist, setup register and
> so on could be implemented generically and used by a make-based build
> system as well.

That's exactly what I'm proposing we do in GHC: re-use Cabal's setup
register and some of the other parts of the simple build system in a
make-based build system for packages.  It might require a bit of
refactoring of Cabal, but I don't expect it to be a major upheaval at all.

I think what you're proposing is mostly a matter of abstracting parts of
Cabal with cleaner and more modular APIs, which is absolutely a good thing,
but doesn't require a fundamental redesign.  The tight coupling and lack of
separation between Cabal's generic parts and the simple build system is
somewhat accidental (lazy implementors :-), and is actually a lot better
than it used to be thanks to the work Duncan has put in.  I'm sure it'll
improve further over time.

The other part of your complaint is that the BuildInfo is in the .cabal
file along with the PackageDescription (the types are pretty well separated
internally).  Again I don't think there's anything fundamental here, and in
fact some packages have separate .buildinfo files.

Cheers,
        Simon
_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Build system idea

Roman Leshchinskiy
On 14/08/2008, at 18:01, Simon Marlow wrote:

> Roman Leshchinskiy wrote:
>
>> But that is precisely my (other) point. A lot of that work is  
>> really unnecessary and could be done by Cabal since it only or  
>> mostly depends on the package information. Instead, it is  
>> implemented somewhere in Distribution.Simple and not really usable  
>> from the outside. For instance, a lot of the functionality of setup  
>> sdist, setup register and so on could be implemented generically  
>> and used by a make-based build system as well.
>
> That's exactly what I'm proposing we do in GHC: re-use Cabal's setup  
> register and some of the other parts of the simple build system in a  
> make-based build system for packages.  It might require a bit of  
> refactoring of Cabal, but I don't expect it to be a major upheaval  
> at all.

Ah! I hadn't realised that you are going to reuse Cabal functionality.  
You wrote "Extract the code from Cabal that generates Makefiles" so I  
thought you won't be really using anything from Cabal.

> I think what you're proposing is mostly a matter of abstracting  
> parts of Cabal with cleaner and more modular APIs, which is  
> absolutely a good thing, but doesn't require a fundamental  
> redesign.  The tight coupling and lack of separation between Cabal's  
> generic parts and the simple build system is somewhat accidental  
> (lazy implementors :-), and is actually a lot better than it used to  
> be thanks to the work Duncan has put in.  I'm sure it'll improve  
> further over time.

IMO, getting this right is absolutely crucial for Cabal's usability  
and should be the primary short-term goal. Then again, I guess I  
should contribute code instead of opinions already :-)

> The other part of your complaint is that the BuildInfo is in  
> the .cabal file along with the PackageDescription (the types are  
> pretty well separated internally).  Again I don't think there's  
> anything fundamental here, and in fact some packages have  
> separate .buildinfo files.

Well, it is fundamental in the sense that this is how Cabal is used  
(and is supposed to be used) at the moment. It is good that Cabal  
separates these things internally but the separation should be  
enforced in the external interface, as well.

Roman


_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Build system idea

John Meacham
In reply to this post by Roman Leshchinskiy
On Wed, Aug 13, 2008 at 01:31:55PM +1000, Roman Leshchinskiy wrote:
> This makes me wonder, though. Wouldn't this model make more sense for  
> Cabal in general than the current approach of duplicating the  
> functionality of autoconf, make and other stuff? If it works ghc, it  
> ought to work for other projects, too. Cabal as a preprocessor seems  
> much more attractive to me than as a universal build system.

I can't tell you how much I agree with this. the fact that cabal wants
to be my build system as well as my configuration system means it is
pretty much unusable to me in my projects.

Features are something that _hurts_ a system such as this. between a
build system, a configuration manager, a packaging system, etc, it is
rare for any large project that at least one isn't imposed on you by
some external constrant or just a better choice for the job. I would
much rather see cabals functionality split among a variety of different
programs so the pieces can be used when appropriate, not as an all or
nothing thing. (bring back hmake! :) ).

        John

--
John Meacham - ⑆repetae.net⑆john⑈
_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Build system idea

Malcolm Wallace
John Meacham <[hidden email]> wrote:

>                          (bring back hmake! :) ).

It never went away...
    http://www.cs.york.ac.uk/fp/hmake

I even have the idea to allow hmake to read the .cabal file format for
configuration data (although that is waiting for a delivery of round
tuits).

Regards,
    Malcolm
_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Build system idea

Duncan Coutts
In reply to this post by John Meacham
On Wed, 2008-08-27 at 03:04 -0700, John Meacham wrote:

> On Wed, Aug 13, 2008 at 01:31:55PM +1000, Roman Leshchinskiy wrote:
> > This makes me wonder, though. Wouldn't this model make more sense for  
> > Cabal in general than the current approach of duplicating the  
> > functionality of autoconf, make and other stuff? If it works ghc, it  
> > ought to work for other projects, too. Cabal as a preprocessor seems  
> > much more attractive to me than as a universal build system.
>
> I can't tell you how much I agree with this. the fact that cabal wants
> to be my build system as well as my configuration system means it is
> pretty much unusable to me in my projects.
>
> Features are something that _hurts_ a system such as this. between a
> build system, a configuration manager, a packaging system, etc, it is
> rare for any large project that at least one isn't imposed on you by
> some external constrant or just a better choice for the job. I would
> much rather see cabals functionality split among a variety of different
> programs so the pieces can be used when appropriate, not as an all or
> nothing thing. (bring back hmake! :) ).

People are of course still free to use autoconf and make to implement
their own build system and have it still be a Cabal package (which has
the advantage of presenting the same meta-data and command interface to
packaging tools). It's been that way since the original design. Quite a
few packages to use autoconf though the use seems to be slightly on the
decline as people try and make their packages portable to Windows. Very
few packages use make as it involves re-implementing their own build
system which is a lot of work. That's partly a self-fulfilling prophecy
of course because nobody uses that interface so it does not get improved
so nobody uses it etc. Also, as far as I'm aware hmake still works, at
least for nhc, I've not used it recently for building with ghc. So
there's nothing stopping people from using that (except hard work), even
as part of a cabal package.

The different parts of the system are relatively separated. The
declarative bits that deal with package meta-data (.cabal files) are
available through the Cabal library (Distribution.*) and many tools make
use of this. Then the 'Simple' build system is in the same library but
fairly cleanly separated (Distribution.Simple.*). As I mentioned, you
do not have to use the 'Simple' build system, but the vast majority of
packages do. Then there are the packaging tools like the tools for
converting to native packages and cabal-install which use the Cabal
library and the command line interface that Cabal packages present.

I'm not saying it's perfect, but it's not as monolithic as some would
suggest.

Duncan

_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Build system idea

John Meacham
The problem with the way cabal wants to mix with make/autoconf is that
it is the wrong way round. make is very good at managing pre-processors,
dependency tracking and calling external programs in the right order, in
parallel, and as needed. cabal is generally good at building a single
library or executable given relatively straightforward haskell source.
(I know it _can_ do more, but this is mainly what it is good at).

The way this should work is that make determines what haskell libraries
need to be built, and what haskell files need to be generated to allow
cabal to run and calls cabal to build just the ones needed. cabal as a
build tool that make calls is much more flexible and in tune with each
tools capabilities.

The other issue is with cabal files themselves which are somewhat
conflicted in purpose. on one hand, you have declarative stuff about a
package. name, version, etc... information you want before you start to
build something. but then you have build-depends, which is something
that you cannot know until after your configuration manager (whatever it
may be, autoconf being a popular one) is run. What packages you depend
on are going to depend on things like what compiler you have installed,
your configuration options, which packages are installed, what operating
system you are running on, which kernel version you are running, which c
libraries you have installed. etc. things that cannot be predicted
before the configuration is actually run.

Then you have cabal as a packaging system (or perhaps hackage/cabal
considered together). Which has its own warts, if it is meant to live in
the niche of package managers such as rpm or deb, where are the
'release' version numbers that rpms and debs have for one example? If it is
meant to be a tarball like format, where is the distinction between
'distribution' and 'source' tarballs? For instance, jhc from darcs for
developers requires perl,ghc,DrIFT,pandoc,autotools, and happy.  however the jhc
tarball requires _only_ ghc. nothing else. This is because the make dist
target is more interesting than just taring up the source. (and
posthooks/prehooks don't really help. they are sort of equivalent to
saying 'write your own build system'.)

One of the biggest sources of conflict arise from using cabal as a
configuration manager. A configuration managers entire purpose is to
examine the system and figure out how to adapt your programs build to
the system. this is completely 100% at odds with the idea of users
having to 'upgrade' cabal. Figuring out how to adapt your build to
whatever cabal is installed or failing gracefully if you can't is
exactly the job of the configuration manager. something like autoconf.
This is why _users_ need not install autoconf, just developers. since
autoconf generates a portable script is so that users are never told to
upgrade their autoconf. if a developer wants to use new features, he
gets the new autoconf and reruns 'autoreconf'. The user is never
asked to update anything that isn't actually needed for the project
itself. This distinction is key fora configuration manager and really
conflicts with cabal wanting to also be a build system and package
manager. It is also what is needed for forwards and backwards
compatibility.

All in all, I think these conflicting goals of cabal make it hard to use
in projects and have led to very odd design choices. I think external
tools should not be the exception but rather the rule. Not that cabal
shouldn't come with a full set of said tools. But as long as they are
integrated I don't see cabal's design problems being fixed, meerly
augmented with various work-arounds.

        John


--
John Meacham - ⑆repetae.net⑆john⑈
_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
123