Shipments in Cabal

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

Shipments in Cabal

Krasimir Angelov-2
Hello Guys,

I have started developing Shipments in Cabal. It is working like it
was discussed previously. When there are multiple *.cabal files, in
the directory where the Setup.lhs lies, then they all will be build in
dependency order. Packages that contain both executables and libraries
are deprecated. In order to keep the backward compatibility the
package description parser returns a list of PackageDescription. In
the list there is one element for the library and one element for each
executable. The package name for the executables is equal to the
"executable" field in the corresponding stanza in the package
description file.

The trouble that I see is with the bindir, libdir, etc directory
names. They all may depend on $pkgid value which is different for each
package in the shipment. The consequence is that by default all
binaries will be installed in their own directories.
I think it is better to install them in common directory instead. It
might be better if we were able to use $shipment variable instead of
$pkgid. Another possible usage of is in "sdist" command. By default
the "sdist" command is building the ${pkgid}.tar.gz archive. In
presence of multiple packages I think it should build one common
package: ${shipment}.tar.gz.

The problem is how to guess the $shipment value. There are some ideas:

a> The shipment name might be kept in some special file.

b> In HSQL I am using package names like: hsql, hsql-odbc, hsql-mysql, .....
One solution is to use the common prefix in the package names as
shipment name. When there is only one package, then the shipment name
will be equal to the package name. The advantage is that, when the
shipment name is encoded in the package name, then for the user it
will be easier to find the package sources.
The drawback is that we have to force some restriction on the package names.

c> Another solution is to use the name of the top level directory. The
drawback is that the user can't rename the top level directory.

Any ideas?

Cheers,
  Krasimir
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Shipments in Cabal

Isaac Jones-2
Krasimir Angelov <[hidden email]> writes:

> Hello Guys,
>
> I have started developing Shipments in Cabal.

Awesome!  FWIW, you should probably take a look at the package
dependency code in cabal-get, and you also might want to look at the
new cabal-install tool, as I expect that when we ship a cabal w/
cabal-install, people will use it a lot.  Although having shipments
makes cabal-install less important.

The smarts in cabal-get is that it checks to see what's already
installed, and this might not really be desirable for a shipment
(unless we want to have a flag for only installing the necessary
stuff).

BTW, could you open a new ticket or wiki page at the cabal wiki /
ticket tracker:
http://hackage.haskell.org/cgi-bin/trac/trac.cgi/report/3

> It is working like it was discussed previously. When there are
> multiple *.cabal files, in the directory where the Setup.lhs lies,
> then they all will be build in dependency order.

Are you using a temporary package-conf file?

> Packages that contain both executables and libraries are deprecated.

Sounds good.

> In order to keep the backward compatibility the package description
> parser returns a list of PackageDescription. In the list there is
> one element for the library and one element for each executable. The
> package name for the executables is equal to the "executable" field
> in the corresponding stanza in the package description file.

I see.  You didn't want to break backward compatibility w/ the
PackageDescription type?

> The trouble that I see is with the bindir, libdir, etc directory
> names. They all may depend on $pkgid value which is different for each
> package in the shipment. The consequence is that by default all
> binaries will be installed in their own directories.
> I think it is better to install them in common directory instead. It
> might be better if we were able to use $shipment variable instead of
> $pkgid. Another possible usage of is in "sdist" command. By default
> the "sdist" command is building the ${pkgid}.tar.gz archive. In
> presence of multiple packages I think it should build one common
> package: ${shipment}.tar.gz.
>
> The problem is how to guess the $shipment value. There are some ideas:
>
> a> The shipment name might be kept in some special file.

I kinda like this, package.cabalship (or whatever) might look like:

shipment: shipmentname
version: 1.1 -- inherited by the packages if they have no version num
copyright -- ditto

The only down side I see to this is that if version can be in the
shipment, and need not be in the .cabal files, then the .cabal files
are no longer stand-alone (they need the shipment file).

OTOH, if you have a scheme where you want to have version numbers the
same across all packages in your shipment, well then it would be very
nice to not have to specify it in each .cabal file.

What are the advantages of having standalone .cabal files within a
shipment?  I don't see anything too big unless someone wants to be
able to grab just part of the shipment, but in that case, they could
just get the .cabalship file anyway.

The other advantage is that .cabal files would not be backward
compatible w/ older cabals but I guess that older cabals will already
break in the presence of multiple .cabal files in a directory, so
maybe it doesn't matter.  Opinions.

> b> In HSQL I am using package names like: hsql, hsql-odbc, hsql-mysql, .....
> One solution is to use the common prefix in the package names as
> shipment name. When there is only one package, then the shipment name
> will be equal to the package name. The advantage is that, when the
> shipment name is encoded in the package name, then for the user it
> will be easier to find the package sources.
> The drawback is that we have to force some restriction on the package names.

I don't really like the idea of forcing them to name their packages in
a particular way.  For instance, you might have a package cabal but
want an executable install-cabal.  We don't want to force people to
use particular names for their binaries.

> c> Another solution is to use the name of the top level directory. The
> drawback is that the user can't rename the top level directory.

Another simple idea is that we could add a shipment: field to the
.cabal file.  This would cause some redundancy between .cabal files,
but that's nicer than b> or c>.

Of course, shipment: wouldn't have to be in each file; it would just
be required that one of the files has that field in it, but that's not
very neat / symmetric.

Also, have you thought about how one might specify that they only want
to operate on one part of the package (a single .cabal file?).  Maybe
defaultSetup could take optional command-line option which is the
.cabal file(s) to operate on, and if none are specified, then use
*.cabal.

This is where the cabal-get dependency code could come in handy.
Let's say I have a shipment with A.cabal, B.cabal, and C.cabal where A
depends on B and B depends on C.  I already have C installed for some
strange reason, it would be nice to say:

./setup configure  A.cabal B.cabal
./setup build
./setup install

and that should work just fine, because C is already installed.  OTOH,
if you said:

./setup configure  A.cabal
./setup build
./setup install

then setup should complain that you don't have B.

This is something like how cabal-install works.  I can say:
cabal-install foo/foo.cabal bar/bar.cabal --user --prefix=/home/ijones/usr



peace,

  isaac
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Shipments in Cabal

Duncan Coutts
On Thu, 2005-12-08 at 10:02 -0800, Isaac Jones wrote:

> Krasimir Angelov <[hidden email]> writes:
>
> > Hello Guys,
> >
> > I have started developing Shipments in Cabal.
>
> Awesome!  FWIW, you should probably take a look at the package
> dependency code in cabal-get, and you also might want to look at the
> new cabal-install tool, as I expect that when we ship a cabal w/
> cabal-install, people will use it a lot.  Although having shipments
> makes cabal-install less important.

I also wrote a bit about this issue:

http://haskell.org/hawiki/Cabal_2fMultiPackageDistributables

I started off from the assumption that "shipments" are necessary and
obviously beneficial. However after thinking about it for a while and
thinking through some of the implications (especially on dependency
resolution) I'm not not so convinced that we need shipments as such.

There is certainly a case for making it easier to work with a
system/project that consists of multiple libraries and executables,
however I'm convinced that the cabal package should be the fundamental
unit and that "shipments" should be merely for the convenience of the
developer.

I would certainly like to see better support in Cabal for larger
projects (for one thing I'd love to be able to convert Gtk2Hs to use
Cabal) but I think we should be careful to fully understand the changes
we intend to make to support larger projects.

Duncan

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Shipments in Cabal

Isaac Jones-2
Duncan Coutts <[hidden email]> writes:

> On Thu, 2005-12-08 at 10:02 -0800, Isaac Jones wrote:
>> Krasimir Angelov <[hidden email]> writes:
>>
>> > Hello Guys,
>> >
>> > I have started developing Shipments in Cabal.
>>
>> Awesome!  FWIW, you should probably take a look at the package
>> dependency code in cabal-get, and you also might want to look at the
>> new cabal-install tool, as I expect that when we ship a cabal w/
>> cabal-install, people will use it a lot.  Although having shipments
>> makes cabal-install less important.
>
> I also wrote a bit about this issue:
>
> http://haskell.org/hawiki/Cabal_2fMultiPackageDistributables
>
> I started off from the assumption that "shipments" are necessary and
> obviously beneficial. However after thinking about it for a while and
> thinking through some of the implications (especially on dependency
> resolution) I'm not not so convinced that we need shipments as such.
>
> There is certainly a case for making it easier to work with a
> system/project that consists of multiple libraries and executables,
>
> however I'm convinced that the cabal package should be the fundamental
> unit and that "shipments" should be merely for the convenience of the
> developer.

This is actually why I wrote cabal-install, I wanted to try out the
idea of leaving .cabal packages as the fundamental unit and building
this layered tool for convenience.

It doesn't do dependency analysis yet, and it doesn't install in a
temporary location, but you can say:

cabal-install foo.tgz bar.tgz -- etc

(it'll also accept .tar, .tar.gz, .cabal, and a directory name)

The main advantage of shipments, IMO, is that they're lighter-weight
than multiple packages.  I'm still somewhat torn, but leaning toward
shipments, especially if it can pretty much be a layer on top of
.cabal files.  Moving the version into .cabalship breaks this
somewhat.

I was planning to gain some experiences w/ cabal-install before
pushing into shipments, but Krasimir jumped into it.  I think they're
more important for the visual studio plugin than for other stuff.

peace,

  isaac
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Shipments in Cabal

Krasimir Angelov-2
In reply to this post by Isaac Jones-2
2005/12/8, Isaac Jones <[hidden email]>:

> Krasimir Angelov <[hidden email]> writes:
>
> > Hello Guys,
> >
> > I have started developing Shipments in Cabal.
>
> Awesome!  FWIW, you should probably take a look at the package
> dependency code in cabal-get, and you also might want to look at the
> new cabal-install tool, as I expect that when we ship a cabal w/
> cabal-install, people will use it a lot.  Although having shipments
> makes cabal-install less important.
>
> The smarts in cabal-get is that it checks to see what's already
> installed, and this might not really be desirable for a shipment
> (unless we want to have a flag for only installing the necessary
> stuff).
>
> BTW, could you open a new ticket or wiki page at the cabal wiki /
> ticket tracker:
> http://hackage.haskell.org/cgi-bin/trac/trac.cgi/report/3

Yes. I will.

> > It is working like it was discussed previously. When there are
> > multiple *.cabal files, in the directory where the Setup.lhs lies,
> > then they all will be build in dependency order.
>
> Are you using a temporary package-conf file?

It should but the dependency checking isn't implemented yet.

> > In order to keep the backward compatibility the package description
> > parser returns a list of PackageDescription. In the list there is
> > one element for the library and one element for each executable. The
> > package name for the executables is equal to the "executable" field
> > in the corresponding stanza in the package description file.
>
> I see.  You didn't want to break backward compatibility w/ the
> PackageDescription type?

The PackageDescription type is changed. The library and executables
fields are removed. There is a new field content :: PackageContent.
The PackageContent is something like:

data PackageContent
  = Library { ..... }
  | Executable { .... }

I just want keep backward compatibility in the package description file syntax.

> I kinda like this, package.cabalship (or whatever) might look like:
>
> shipment: shipmentname
> version: 1.1 -- inherited by the packages if they have no version num
> copyright -- ditto
>
> The only down side I see to this is that if version can be in the
> shipment, and need not be in the .cabal files, then the .cabal files
> are no longer stand-alone (they need the shipment file).

I would like to avoid the need for extra description files. The
version and copyright fields doesn't make sense for me providing that
each package has its own version, copyright, etc.

> > b> In HSQL I am using package names like: hsql, hsql-odbc, hsql-mysql, .....
> > One solution is to use the common prefix in the package names as
> > shipment name. When there is only one package, then the shipment name
> > will be equal to the package name. The advantage is that, when the
> > shipment name is encoded in the package name, then for the user it
> > will be easier to find the package sources.
> > The drawback is that we have to force some restriction on the package names.
>
> I don't really like the idea of forcing them to name their packages in
> a particular way.  For instance, you might have a package cabal but
> want an executable install-cabal.  We don't want to force people to
> use particular names for their binaries.
>
> > c> Another solution is to use the name of the top level directory. The
> > drawback is that the user can't rename the top level directory.
>
> Another simple idea is that we could add a shipment: field to the
> .cabal file.  This would cause some redundancy between .cabal files,
> but that's nicer than b> or c>.

I like the idea for "shipment" field since it is a light weight
solution. The redundancy is not high and I can't see any reason why we
shouldn't allow to have .cabal files from different shipments in one
directory. In this case the sdist command will simply create multiple
.tar.gz files. If the "shipment" field is missing then the default
value will be equal to $pkgid.


> Also, have you thought about how one might specify that they only want
> to operate on one part of the package (a single .cabal file?).  Maybe
> defaultSetup could take optional command-line option which is the
> .cabal file(s) to operate on, and if none are specified, then use
> *.cabal.
>
> This is where the cabal-get dependency code could come in handy.
> Let's say I have a shipment with A.cabal, B.cabal, and C.cabal where A
> depends on B and B depends on C.  I already have C installed for some
> strange reason, it would be nice to say:
>
> ./setup configure  A.cabal B.cabal
> ./setup build
> ./setup install
>
> and that should work just fine, because C is already installed.  OTOH,
> if you said:
>
> ./setup configure  A.cabal
> ./setup build
> ./setup install
>
> then setup should complain that you don't have B.

Yes. In Visual Haskell, I will need some way to specify that I want to
build only one of the packages.

Cheers,
  Krasimir
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Shipments in Cabal

Isaac Jones-2
In reply to this post by Isaac Jones-2
Isaac Jones <[hidden email]> writes:

>> a> The shipment name might be kept in some special file.
>
> I kinda like this, package.cabalship (or whatever) might look like:
>
> shipment: shipmentname
> version: 1.1 -- inherited by the packages if they have no version num
> copyright -- ditto

Another thing we could do w/ a shipment file is specify the location
of .cabal files so you can have them in their own source tree.  So if
you had a source tree like:
src/foo/
        foo.hs
        foo.cabal

src/bar/
        bar.hs
        bar.cabal

then you could have

src/foobar.cabalship:
shipment: foobar
cabal-files: bar/bar.cabal, foo/foo.cabal

peace,

  isaac

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

RE: Shipments in Cabal

Simon Peyton Jones
In reply to this post by Krasimir Angelov-2
| I started off from the assumption that "shipments" are necessary and
| obviously beneficial. However after thinking about it for a while and
| thinking through some of the implications (especially on dependency
| resolution) I'm not not so convinced that we need shipments as such.

I have not been following this thread much, but my original
understanding was precisely that

        A Cabal package *is* the unit of distribution
        Indeed, that's what it is for

I'm entirely unclear why new need some other larger unit.  Yes, a Cabal
package may depend on other Cabal package, but those dependencies are
explicit in the package descr file, and Cabal-get can follow the links.
Any larger unit (shipments) will also have dependencies that need to be
followed, in just the same way, so I don't understand what has been
gained.  

I'm probably missing something, but perhaps someone can write down the
reasoning on Duncan's Wiki page?
http://haskell.org/hawiki/Cabal_2fMultiPackageDistributables

Simon

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Shipments in Cabal

Krasimir Angelov-2
Cabal already supports multiple executables in one package but only
one library.
The problem is that the current implementation is just a hack. The
shipment allows to pack multiple packages in one tar.gz file and to
build them together. Each package will have only one library or only
one executable and this resumes the symmetry between executables and
libraries. I found that this simplifies the code a lot.

Cheers,
  Krasimir

2005/12/9, Simon Peyton-Jones <[hidden email]>:

> | I started off from the assumption that "shipments" are necessary and
> | obviously beneficial. However after thinking about it for a while and
> | thinking through some of the implications (especially on dependency
> | resolution) I'm not not so convinced that we need shipments as such.
>
> I have not been following this thread much, but my original
> understanding was precisely that
>
>        A Cabal package *is* the unit of distribution
>        Indeed, that's what it is for
>
> I'm entirely unclear why new need some other larger unit.  Yes, a Cabal
> package may depend on other Cabal package, but those dependencies are
> explicit in the package descr file, and Cabal-get can follow the links.
> Any larger unit (shipments) will also have dependencies that need to be
> followed, in just the same way, so I don't understand what has been
> gained.
>
> I'm probably missing something, but perhaps someone can write down the
> reasoning on Duncan's Wiki page?
> http://haskell.org/hawiki/Cabal_2fMultiPackageDistributables
>
> Simon
>
> _______________________________________________
> Libraries mailing list
> [hidden email]
> http://www.haskell.org/mailman/listinfo/libraries
>
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

RE: Shipments in Cabal

Simon Peyton Jones
In reply to this post by Krasimir Angelov-2
That's helpful, but I still don't understand.  I'm just Joe User.  I'd
like to install GTK.  I'd like to download one thing and install it. I
don't much mind whether it's called a "package" or a "shipment".

Here are three things we can do

a) Make it possible for a "package" to have multiple executables
b) Invent "shipments", which can have multiple executables, and get rid
of "packages"
c) Have both "packages" and "shipments".

Clearly (a) and (b) are essentially identical, modulo nomenclature.
They seem simpler than (c).  What is the merit of (c)?  Would Joe User
ever see or know about a package?

Adding a summary of the reply to Duncan's wiki page would be useful I
think

Simon

| -----Original Message-----
| From: Krasimir Angelov [mailto:[hidden email]]
| Sent: 09 December 2005 08:43
| To: Simon Peyton-Jones
| Cc: Duncan Coutts; Haskell libraries
| Subject: Re: Shipments in Cabal
|
| Cabal already supports multiple executables in one package but only
| one library.
| The problem is that the current implementation is just a hack. The
| shipment allows to pack multiple packages in one tar.gz file and to
| build them together. Each package will have only one library or only
| one executable and this resumes the symmetry between executables and
| libraries. I found that this simplifies the code a lot.
|
| Cheers,
|   Krasimir
|
| 2005/12/9, Simon Peyton-Jones <[hidden email]>:
| > | I started off from the assumption that "shipments" are necessary
and
| > | obviously beneficial. However after thinking about it for a while
and
| > | thinking through some of the implications (especially on
dependency
| > | resolution) I'm not not so convinced that we need shipments as
such.
| >
| > I have not been following this thread much, but my original
| > understanding was precisely that
| >
| >        A Cabal package *is* the unit of distribution
| >        Indeed, that's what it is for
| >
| > I'm entirely unclear why new need some other larger unit.  Yes, a
Cabal
| > package may depend on other Cabal package, but those dependencies
are
| > explicit in the package descr file, and Cabal-get can follow the
links.
| > Any larger unit (shipments) will also have dependencies that need to
be
| > followed, in just the same way, so I don't understand what has been
| > gained.
| >
| > I'm probably missing something, but perhaps someone can write down
the
| > reasoning on Duncan's Wiki page?
| > http://haskell.org/hawiki/Cabal_2fMultiPackageDistributables
| >
| > Simon
| >
| > _______________________________________________
| > Libraries mailing list
| > [hidden email]
| > http://www.haskell.org/mailman/listinfo/libraries
| >
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Shipments in Cabal

Duncan Coutts
In reply to this post by Krasimir Angelov-2
On Fri, 2005-12-09 at 10:42 +0200, Krasimir Angelov wrote:
> Cabal already supports multiple executables in one package but only
> one library.

I think that's because we track dependencies on libraries but not
executables. So having one lib per package allows us to track library
deps correctly.

Having more than one library in a package breaks dependency checking as
described in the wiki page.

If a shipment is merely an aggregation of cabal packages and the package
remains the dependency tracking unit then it's ok.

However a key point is this:

You must be able to build the individual packages within a shipment
separately (though respecting their dependencies). We do not want to end
up in the situation where to build another optional package from a
shipment you have to rebuild every other dependant package in that
shipment.

As an example, suppose Gtk2Hs is a shipment with 2 large core packages
and 6 small optional ones. If to satisfy a dependency you need to
install one of the extra optional packages but you've already got the
core packages installed then you really do not want to have to rebuild
those core packages (which take 10 times longer to build than any of the
optional packages).

The danger of a shipment as opposed to merely a collection of cabal
packages (with deps between them) is that there can be hidden
dependencies. If they are all built together then people will be tempted
to communicate information from one build to another within the build
tree, for example by sharing the results of ./configure or by picking
up .chi files from another package. All these things would break if we
try to build the packages in a shipment separately (which is exactly
what cabal-get and other package managers will want to do).

It would certainly be useful to have a way of managing and building a
collection of related packages but I'm not sure that there is any
benefit on the users end.

For example a developer will want an easy way of building a project that
consists of several packages and many of those package might share the
same information (eg version number, copyright, homepage etc).

However is there any reason for that collection of packages to be
distributed to end users in one tarball and built together in one build
tree, rather than just using the dependency resolution mechanism of
cabal-get or the system package manager?

Duncan

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

RE: Shipments in Cabal

Duncan Coutts
In reply to this post by Simon Peyton Jones
On Fri, 2005-12-09 at 08:54 +0000, Simon Peyton-Jones wrote:
> That's helpful, but I still don't understand.  I'm just Joe User.  I'd
> like to install GTK.  I'd like to download one thing and install it. I
> don't much mind whether it's called a "package" or a "shipment".

Hmm, but it's not just "GTK" it's actually a whole collection of
packages, some core and some optional. Some will require other C libs
that you might not have installed on your system (eg if you run Windows
or Mac, or if you run Linux/Solaris/BSD but not Gnome).

At the moment Gtk2Hs is distributed as one big package and the configure
script figures out which packages can be build given the C libs that
you've got installed.

I'm not sure how best to deal with that if it's split into multiple
cabal packages.

Duncan

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

RE: Shipments in Cabal

Simon Peyton Jones
In reply to this post by Krasimir Angelov-2

| I think that's because we track dependencies on libraries but not
| executables. So having one lib per package allows us to track library
| deps correctly.
|
| Having more than one library in a package breaks dependency checking
as
| described in the wiki page.

Crumbs.  What is "a library"?  Previously I'd understood that a package
*was* a library.  I think you are saying that

        Shipments contain N packages
        Packages contain N executables and M libraries
        Libraries contain N modules

Do we really need FOUR levels?   (Shipments, packages, libraries,
modules)

Joe is getting worried.

Simon
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

RE: Shipments in Cabal

Duncan Coutts
On Fri, 2005-12-09 at 09:55 +0000, Simon Peyton-Jones wrote:
> | I think that's because we track dependencies on libraries but not
> | executables. So having one lib per package allows us to track library
> | deps correctly.
> |
> | Having more than one library in a package breaks dependency checking
> as
> | described in the wiki page.
>
> Crumbs.  What is "a library"?

A quick clarification of terminology:

With GHC a package is described by an entry in the package.conf
database, it usually has just one libHSthing.a file. "base", "haskell98"
etc are ghc packages.

With Cabal a package is described by a .cabal file. It can contain at
most one GHC package (though it is of course not compiler specific) and
zero or more executables.

> Previously I'd understood that a package *was* a library.  I think you
> are saying that
>
> Shipments contain N packages
> Packages contain N executables and M libraries

So Cabal packages currently contain N executables and 0 or 1 libraries.

> Libraries contain N modules
>
> Do we really need FOUR levels?   (Shipments, packages, libraries,
> modules)
>
> Joe is getting worried.

I'm not sure that we need shipments, just some convenient tools for
maintaining a collection of related packages.

On the other hand how does one conveniently talk about a
"project/system/thing" that has various optional bits.

As you said earlier, Joe wants to install "HSQL" but the fact is that
there isn't HSQL, there the hsql core bit (which is useless on it's own)
and backends for various databases. So if we've just got packages then
Joe has to know to install hsql and hsql-postgres or whatever.

Some Linux distros have the notion of a meta-package that ties together
a collection of packages that may have some optional dependencies.

Duncan

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

RE: Shipments in Cabal

Simon Marlow
In reply to this post by Krasimir Angelov-2
I've read the thread, and I must admit I've been thinking along the same
lines as Duncan.  We don't need shipments.  The problem is purely with
developing aggregate packages, which we can solve with either tools or
small additions to Cabal.  (sorry, I intended this to be a short
message, but it's turned out rather long :-/)

I don't propose to change much, in fact I want to make the situation
simpler:

  Cabal package
        == one library or one executable
      == one distro package
      == one distributable

To aggregate packages, just create another package which contains only
dependencies.  This is how many distro packaging systems aggregate eg.
all the KDE components into a single KDE package.

1. End users: they are served by this just fine.  They just
cabal-install
   the top-level package, and all the dependencies are magically
   downloaded and installed.  Or they use their distro's packaging
system
   to do the same thing.  Dependencies between libraries are maintained
   correctly.

2. Developers.  A developer working on an aggregate package needs to be
   able to edit & rebuild an aggregate package without rebuilding
   everything every time.  This is the scenario I believe we should
solve
   with a few small additions to Cabal (see below).

3. Visual studio.  VS has solutions, which are containers for multiple
   packages, but the problem is that it doesn't want to *install* each
   package as it builds, but it somehow needs to make each package
   available to the others when building a chain of dependencies.
Again,
   this is solved by the same small additions to Cabal.


As a developer, here is how I'd like to work on an aggregate package.
Suppose I have three packages mylib, myexe1, myexe2.  I aggregate these
using a package bigpkg.  I then arrange my tree like this:

  .../bigpkg/
       Setup.lhs
       bigpkg.cabal   -- build-depends: mylib, myexe1, myexe2
       mylib/
          Setup.lhs
          mylib.cabal
          src/
       myexe1/
          Setup.lhs
          myexe1.cabal -- build-depends: mylib
          src/
       myexe2/
          Setup.lhs
          myexe2.cabal -- build-depends: mylib
          src/

and I want to work on the whole blob like this:

  $ cd .../bigpkg
  $ runghc Setup.lhs configure --local
  $ runghc Setup.lhs build

what does --local do? (we can choose a better name, incedentally):

  - it implies
        --local-deps
        --prefix=`pwd`/install
        --packagedb=`pwd`/install/package.conf

what does --local-deps do?

  - "configure" looks for dependencies in subdirectories of `pwd`

  - "configure" configures recursively.  It passes
    --prefix and other settings to each subdir

  - "build" command recursively builds dependencies that
    are found in subdirectories, in dependency order.  After building
    each library dependency, it runs "setup install" in that subdir.

  - other commands just operate recursively.

what does --packagedb do?

  - specifies a package database into which packages are registered
    by "setup register" and "setup install"

  - you probably want to set GHC_PACKAGE_PATH too


-=-=- Summary -=-=-

  - Add --packagedb flag to Cabal.  This just adds a -package-conf
    to each invocation of GHC and ghc-pkg.

  - Add --local-deps flag to "setup configure" in the simple build
    sysetm, which looks for dependencies in subdirectories, and causes
    other setup commands to recurse into dependencies in the right
    order.

  - Add --local flag which is just a macro


Pretty simple to implement, and does everything we want, I think.

Visual Studio would do things slightly differently, but I believe these
facilities provide everything VS needs from Cabal.

Incedentally, I want --local for converting GHC's libraries to Cabal.
We need to build & use directly from the build tree, which is what
--local does.

Cheers,
        Simon
   
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

RE: Shipments in Cabal

Simon Marlow
In reply to this post by Krasimir Angelov-2
This message got garbled a bit by my mailer.  It's easier to read on the
wiki:

http://haskell.org/hawiki/Cabal_2fAggregatePackages

Cheers,
        Simon

On 09 December 2005 11:07, Simon Marlow wrote:

> I've read the thread, and I must admit I've been thinking along the
> same lines as Duncan.  We don't need shipments.  The problem is
> purely with developing aggregate packages, which we can solve with
> either tools or small additions to Cabal.  (sorry, I intended this to
> be a short message, but it's turned out rather long :-/)
>
> I don't propose to change much, in fact I want to make the situation
> simpler:
>
>   Cabal package
> == one library or one executable
>       == one distro package
>       == one distributable
>
> To aggregate packages, just create another package which contains only
> dependencies.  This is how many distro packaging systems aggregate eg.
> all the KDE components into a single KDE package.
>
> 1. End users: they are served by this just fine.  They just
> cabal-install
>    the top-level package, and all the dependencies are magically
>    downloaded and installed.  Or they use their distro's packaging
> system
>    to do the same thing.  Dependencies between libraries are
>    maintained correctly.
>
> 2. Developers.  A developer working on an aggregate package needs to
>    be able to edit & rebuild an aggregate package without rebuilding
>    everything every time.  This is the scenario I believe we should
> solve
>    with a few small additions to Cabal (see below).
>
> 3. Visual studio.  VS has solutions, which are containers for multiple
>    packages, but the problem is that it doesn't want to *install* each
>    package as it builds, but it somehow needs to make each package
>    available to the others when building a chain of dependencies.
> Again,
>    this is solved by the same small additions to Cabal.
>
>
> As a developer, here is how I'd like to work on an aggregate package.
> Suppose I have three packages mylib, myexe1, myexe2.  I aggregate
> these using a package bigpkg.  I then arrange my tree like this:
>
>   .../bigpkg/
>        Setup.lhs
>        bigpkg.cabal   -- build-depends: mylib, myexe1, myexe2
>        mylib/
>           Setup.lhs
>           mylib.cabal
>           src/
>        myexe1/
>           Setup.lhs
>           myexe1.cabal -- build-depends: mylib
>           src/
>        myexe2/
>           Setup.lhs
>           myexe2.cabal -- build-depends: mylib
>           src/
>
> and I want to work on the whole blob like this:
>
>   $ cd .../bigpkg
>   $ runghc Setup.lhs configure --local
>   $ runghc Setup.lhs build
>
> what does --local do? (we can choose a better name, incedentally):
>
>   - it implies
>         --local-deps
>         --prefix=`pwd`/install
>         --packagedb=`pwd`/install/package.conf
>
> what does --local-deps do?
>
>   - "configure" looks for dependencies in subdirectories of `pwd`
>
>   - "configure" configures recursively.  It passes
>     --prefix and other settings to each subdir
>
>   - "build" command recursively builds dependencies that
>     are found in subdirectories, in dependency order.  After building
>     each library dependency, it runs "setup install" in that subdir.
>
>   - other commands just operate recursively.
>
> what does --packagedb do?
>
>   - specifies a package database into which packages are registered
>     by "setup register" and "setup install"
>
>   - you probably want to set GHC_PACKAGE_PATH too
>
>
> -=-=- Summary -=-=-
>
>   - Add --packagedb flag to Cabal.  This just adds a -package-conf
>     to each invocation of GHC and ghc-pkg.
>
>   - Add --local-deps flag to "setup configure" in the simple build
>     sysetm, which looks for dependencies in subdirectories, and causes
>     other setup commands to recurse into dependencies in the right
>     order.
>
>   - Add --local flag which is just a macro
>
>
> Pretty simple to implement, and does everything we want, I think.
>
> Visual Studio would do things slightly differently, but I believe
> these facilities provide everything VS needs from Cabal.
>
> Incedentally, I want --local for converting GHC's libraries to Cabal.
> We need to build & use directly from the build tree, which is what
> --local does.
>
> Cheers,
> Simon
>
> _______________________________________________
> Libraries mailing list
> [hidden email]
> http://www.haskell.org/mailman/listinfo/libraries

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Shipments in Cabal

Krasimir Angelov-2
For the end user the presence of shipment isn't important. The
shipment infrastructure should be as simple as possible and in the
best case the end user shouldn't know anything about it.

The shipment is more important for the developer. In HSQL I have to
move from one directory to another just to build the corresponding
package. The shipment as I see it is just a simple extension to Cabal.
Currently when Setup.lhs finds more than one .cabal file in the
current directory then it will raise an error. Instead I would like to
allow it to build all packages in the directory. In this case I will
be able to build all packages with one go.

I will be lucky with Simon's proposal too. It will allow to build
automatically all packages but I think that it is more complicated. At
least everything that have to be added to Cabal for this proposal will
be required in my proposal too. The aggregate package description is
in fact just a replacement for the shipment description which Isaac
proposed. I don't think that it is required to have a separate
description.

I have to write an another aggregate package. Will it be distributed
separatelly in Hackage?

Cheers,
  Krasimir


2005/12/9, Simon Marlow <[hidden email]>:

> This message got garbled a bit by my mailer.  It's easier to read on the
> wiki:
>
> http://haskell.org/hawiki/Cabal_2fAggregatePackages
>
> Cheers,
>        Simon
>
> On 09 December 2005 11:07, Simon Marlow wrote:
>
> > I've read the thread, and I must admit I've been thinking along the
> > same lines as Duncan.  We don't need shipments.  The problem is
> > purely with developing aggregate packages, which we can solve with
> > either tools or small additions to Cabal.  (sorry, I intended this to
> > be a short message, but it's turned out rather long :-/)
> >
> > I don't propose to change much, in fact I want to make the situation
> > simpler:
> >
> >   Cabal package
> >       == one library or one executable
> >       == one distro package
> >       == one distributable
> >
> > To aggregate packages, just create another package which contains only
> > dependencies.  This is how many distro packaging systems aggregate eg.
> > all the KDE components into a single KDE package.
> >
> > 1. End users: they are served by this just fine.  They just
> > cabal-install
> >    the top-level package, and all the dependencies are magically
> >    downloaded and installed.  Or they use their distro's packaging
> > system
> >    to do the same thing.  Dependencies between libraries are
> >    maintained correctly.
> >
> > 2. Developers.  A developer working on an aggregate package needs to
> >    be able to edit & rebuild an aggregate package without rebuilding
> >    everything every time.  This is the scenario I believe we should
> > solve
> >    with a few small additions to Cabal (see below).
> >
> > 3. Visual studio.  VS has solutions, which are containers for multiple
> >    packages, but the problem is that it doesn't want to *install* each
> >    package as it builds, but it somehow needs to make each package
> >    available to the others when building a chain of dependencies.
> > Again,
> >    this is solved by the same small additions to Cabal.
> >
> >
> > As a developer, here is how I'd like to work on an aggregate package.
> > Suppose I have three packages mylib, myexe1, myexe2.  I aggregate
> > these using a package bigpkg.  I then arrange my tree like this:
> >
> >   .../bigpkg/
> >        Setup.lhs
> >        bigpkg.cabal   -- build-depends: mylib, myexe1, myexe2
> >        mylib/
> >           Setup.lhs
> >           mylib.cabal
> >           src/
> >        myexe1/
> >           Setup.lhs
> >           myexe1.cabal -- build-depends: mylib
> >           src/
> >        myexe2/
> >           Setup.lhs
> >           myexe2.cabal -- build-depends: mylib
> >           src/
> >
> > and I want to work on the whole blob like this:
> >
> >   $ cd .../bigpkg
> >   $ runghc Setup.lhs configure --local
> >   $ runghc Setup.lhs build
> >
> > what does --local do? (we can choose a better name, incedentally):
> >
> >   - it implies
> >         --local-deps
> >         --prefix=`pwd`/install
> >         --packagedb=`pwd`/install/package.conf
> >
> > what does --local-deps do?
> >
> >   - "configure" looks for dependencies in subdirectories of `pwd`
> >
> >   - "configure" configures recursively.  It passes
> >     --prefix and other settings to each subdir
> >
> >   - "build" command recursively builds dependencies that
> >     are found in subdirectories, in dependency order.  After building
> >     each library dependency, it runs "setup install" in that subdir.
> >
> >   - other commands just operate recursively.
> >
> > what does --packagedb do?
> >
> >   - specifies a package database into which packages are registered
> >     by "setup register" and "setup install"
> >
> >   - you probably want to set GHC_PACKAGE_PATH too
> >
> >
> > -=-=- Summary -=-=-
> >
> >   - Add --packagedb flag to Cabal.  This just adds a -package-conf
> >     to each invocation of GHC and ghc-pkg.
> >
> >   - Add --local-deps flag to "setup configure" in the simple build
> >     sysetm, which looks for dependencies in subdirectories, and causes
> >     other setup commands to recurse into dependencies in the right
> >     order.
> >
> >   - Add --local flag which is just a macro
> >
> >
> > Pretty simple to implement, and does everything we want, I think.
> >
> > Visual Studio would do things slightly differently, but I believe
> > these facilities provide everything VS needs from Cabal.
> >
> > Incedentally, I want --local for converting GHC's libraries to Cabal.
> > We need to build & use directly from the build tree, which is what
> > --local does.
> >
> > Cheers,
> >       Simon
> >
> > _______________________________________________
> > Libraries mailing list
> > [hidden email]
> > http://www.haskell.org/mailman/listinfo/libraries
>
>
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

RE: Shipments in Cabal

Simon Marlow
In reply to this post by Krasimir Angelov-2
I just made some changes to my proposal:

http://haskell.org/hawiki/Cabal_2fAggregatePackages

Mainly I made it so that the sub-packages don't have to be in a
subdirectory, this means it will be easier to tar up the aggregate
package.

On 09 December 2005 14:19, Krasimir Angelov wrote:

> For the end user the presence of shipment isn't important.
[snip]
> The shipment is more important for the developer.

yes, exactly what I said!

[snip]
> I will be lucky with Simon's proposal too. It will allow to build
> automatically all packages but I think that it is more complicated. At
> least everything that have to be added to Cabal for this proposal will
> be required in my proposal too. The aggregate package description is
> in fact just a replacement for the shipment description which Isaac
> proposed.

the point is that an aggregate package is just a package; we don't need
a separate concept for it.

> I don't think that it is required to have a separate description.

The problem with putting all the .cabal files in a single place is that
you then have to distribute the whole set of packages together; and it
becomes some kind of larger entity for which we need a name
("shipment").     Shipments will be visible at the Hackage level,
because Hackage will have to know the set of packages contained in a
shipment.

I don't think we need to do this.  All that is needed is an empty
package that depends on all the others.  Hackage doesn't need to learn
anything new, neither do the other tools.

> I have to write an another aggregate package. Will it be distributed
> separatelly in Hackage?

Yes - but that's exactly how other package systems work.  It's flexible
and simple.

Cheers,
        Simon
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Shipments in Cabal

Isaac Jones-2
"Simon Marlow" <[hidden email]> writes:

> I just made some changes to my proposal:
>
> http://haskell.org/hawiki/Cabal_2fAggregatePackages
>
> Mainly I made it so that the sub-packages don't have to be in a
> subdirectory, this means it will be easier to tar up the aggregate
> package.

Can you clarify the terminology a bit?  A distro package is an OS
package?  Like RPM?  And a distributable is a kind of meta-package
that does nothing but depend on other packages?  (If so, let's call it
a meta-package or something).

I can't edit the wiki right now :(

I like the idea of having a meta package that can be only
dependencies, but I don't think I really like the --local idea.  I
don't really like the idea of one cabal package going to build another
package, invoking another setup script.  I think that should be left
for layered tools.

Although I'm all for adding flags to cabal to allow packages to be
used in-place more effectively, I'd rather have a tool where I can
say:

cabal-install: metapackage/metapackage.cabal --searchpath=.

and maybe

cabal-sdist metapackage/metapackage.cabal --searchpath=.

will go and build .tar.gz files for everything it can find.  That way
hackage won't have to change, it still works on the cabal-package
level.

Incidentally, we would need to separate the concept of depends and
build-depends again, I guess.  Remember that all build-depends cause
-package flags to be added to the command line of GHC.  If these
dependencies are executables, then that's bad.

Krasimir, would meta-packages and cabal-install solve the problems w/
visual studio?  We could add flags to cabal to improve in-place usage,
and to do something w/ a particular packageDB (though I want to figure
out a way to solve this for Hugs too).

We could have meta-packages which are just a minor refactoring of
packages to say that they can just depend on other packages.  This is
definitely something that cabal needs anyway.  We clean up the
multi-executable thing, which never worked well anyway.

But the actual handling of meta-packages should be left to layered
tools like cabal-install.  Cabal-get should be refactored to work on
top of cabal-install; cabal-get should just download the packages and
invoke cabal-install to do the dirty work.

The nice thing about this solution is that if we need to add a
shipment: field or some of these other suggestions, we can do it
later and everything will still make sense.

What do you guys think?

peace,

  isaac
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

RE: Shipments in Cabal

Simon Marlow
In reply to this post by Krasimir Angelov-2
On 09 December 2005 19:50, Isaac Jones wrote:

> "Simon Marlow" <[hidden email]> writes:
>
>> I just made some changes to my proposal:
>>
>> http://haskell.org/hawiki/Cabal_2fAggregatePackages
>>
>> Mainly I made it so that the sub-packages don't have to be in a
>> subdirectory, this means it will be easier to tar up the aggregate
>> package.
>
> Can you clarify the terminology a bit?  A distro package is an OS
> package?  Like RPM?

Yes.

> And a distributable is a kind of meta-package
> that does nothing but depend on other packages?  (If so, let's call it
> a meta-package or something).

No, by "distrutable" I meant a source distribution, i.e. whatever it is
you grab from Hackage.

> I like the idea of having a meta package that can be only
> dependencies, but I don't think I really like the --local idea.  I
> don't really like the idea of one cabal package going to build another
> package, invoking another setup script.  I think that should be left
> for layered tools.

That's a shame, I thought it was quite cool :-(  But I see your point.  

It seems slightly strange to be able to build *one* package using
Setup.lhs, but as soon as I need to split my package into two I have to
use some other tool to work on it.  Not a big deal though.

> Although I'm all for adding flags to cabal to allow packages to be
> used in-place more effectively, I'd rather have a tool where I can
> say:
>
> cabal-install: metapackage/metapackage.cabal --searchpath=.
>
> and maybe
>
> cabal-sdist metapackage/metapackage.cabal --searchpath=.
>
> will go and build .tar.gz files for everything it can find.  That way
> hackage won't have to change, it still works on the cabal-package
> level.

Ok, but remember we're only solving a problem that occurs for developers
here.  Users don't need to know about search paths - they just need to
install a package and its dependencies.  A developer, on the other hand,
needs to be able to rebuild a set of packages in dependency order,
without building everything each time.  That is, the tool needs a way to
ask for a rebuild, not just a full configure/build/install.

And specifying the .cabal file for the metapackage isn't really enough:
you should really go through the Setup.lhs script (it might have some
optional dependencies, for example).

> Incidentally, we would need to separate the concept of depends and
> build-depends again, I guess.  Remember that all build-depends cause
> -package flags to be added to the command line of GHC.  If these
> dependencies are executables, then that's bad.

Ok, I'm agnostic about depends.
 
> Krasimir, would meta-packages and cabal-install solve the problems w/
> visual studio?  We could add flags to cabal to improve in-place usage,
> and to do something w/ a particular packageDB (though I want to figure
> out a way to solve this for Hugs too).

In-place usage is the main thing.  If we can agree on --packagedb that's
a step forward (simulating it with symlinks for Hugs?).  This is the
level that VS will hook into.

Then as long as we have a way to build a set of packages in dependency
order, without rebuilding everything each time, installing to a
temporary location, I think the non-VS developer's needs will be met
too.

[snip]

Cheers,
        Simon
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries