ANN: HDBC v2.0 now available

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: ANN: HDBC v2.0 now available

Duncan Coutts
On Sun, 2009-02-01 at 15:56 +0100, Niklas Broberg wrote:
> > So in the next cabal-install release (which should be pretty soon now)
> > configure will do the same thing and pick base 3 unless you specify
> > build-depends base >= 4.
>
> ... and so there will never be any incentive for these many packages
> to migrate to base-4, which also has consequences for packages that do
> want to use base-4, but also want to depend on such packages.

Actually a package that uses base 4 can depend on other packages that
use base 3. They all co-exist fine. This already happens now.

> I would suggest as a less stagnating approach to issue a warning/hint
> when a package with no explicit version dependency for base fails to
> build.

So my plan is to make hackage require an upper bound on the version of
base for all new packages. That should avoid the need to use the
preferences hack the next time around.

As for what mechanisms we use to persuade package authors to use base 4
over base 3 for new releases, I'm open to suggestions.

One possibility is to warn when configuring if we used the preference to
select base 3 rather than 4. That is, if the solver had a free choice
between 3 and 4 then that's probably a sign that it needs updating. Of
course that doesn't help for packages that work with 3 or 4 perfectly
well.

We could try a mechanism for base not using the preference system but
something more special. For example we could only apply the base 3
preference if there is no upper bound on the version of base. So if the
package says:

build-depends: base >= 3 && < 5

then we say great, and don't use any default preference (and thus pick
base 4). If on the other hand the package says:

build-depends: base

Then we pick base 3 and we reject all new packages uploaded to hackage
like this. They must all specify an upper bound. We could also warn at
configuration time.

So that's the suggestion, we'd only use the base 3 preference if there
is no upper bound on the version of base. That means it should continue
to work for old packages and new ones will default to base 4.

What do you think?

Duncan

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: ANN: HDBC v2.0 now available

Niklas Broberg
In reply to this post by Duncan Coutts
>> I really really think this is the wrong way to go. Occasional
>> destruction is desperately needed for progress, else things will
>> invariably stagnate.
>
> I disagree. Having everything fail (we measured it as ~90% of hackage)
> when people upgraded to ghc-6.10 would have been a disaster. Do you
> recall the screaming, wailing and gnashing of teeth after the release of
> ghc-6.8 when most of hackage broke? We (I mean ghc and cabal hackers)
> got a lot of flak for not making the upgrade process easier and
> needlessly breaking everyone's  perfectly good packages.

90%? That sounds like an awful lot for the relatively minor changes
with base-4. I guess a number of packages will use exceptions, and
others with use Data and Typeable, but 90%? I don't doubt you when you
say that you measured it though, just marvelling.

If 90% of hackage would truly break, then I agree that we might need
more caution than my radical approach. But I'm not fully convinced
there either. After all, unlike with the ghc-6.8 release, all that's
needed for a package to work again is to upload a new .cabal that
makes the dependency on base-3 explicit, if an author really doesn't
want to update the codebase. And even for those packages where library
authors don't do that simple step, all that's needed of the user of
the library is to specify the base-3 constraint when running
cabal-install.

My main complaint is really that there is currently no incentive
whatsoever for library authors to migrate. If we make base-4 the
default, it will require just a little bit of work to make packages
that depend on base-3 work anyway, as seen above. It's not so much
work that it should incur any screaming, wailing and teeth gnashing.
But it should be just enough work to encourage action in one way or
another, either truly migrating the code or just making the dependency
explicit as I noted above. I think it would be hard to find a more
accurate, and non-intrusive, incentive. :-)

I definitely agree with your suggestion to make hackage require an
upper bound on base. But that's to make us future proof, it won't
solve the issue here and now.

Cheers,

/Niklas
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: ANN: HDBC v2.0 now available

Duncan Coutts
On Sun, 2009-02-01 at 17:22 +0100, Niklas Broberg wrote:

> >> I really really think this is the wrong way to go. Occasional
> >> destruction is desperately needed for progress, else things will
> >> invariably stagnate.
> >
> > I disagree. Having everything fail (we measured it as ~90% of hackage)
> > when people upgraded to ghc-6.10 would have been a disaster. Do you
> > recall the screaming, wailing and gnashing of teeth after the release of
> > ghc-6.8 when most of hackage broke? We (I mean ghc and cabal hackers)
> > got a lot of flak for not making the upgrade process easier and
> > needlessly breaking everyone's  perfectly good packages.
>
> 90%? That sounds like an awful lot for the relatively minor changes
> with base-4. I guess a number of packages will use exceptions, and
> others with use Data and Typeable, but 90%? I don't doubt you when you
> say that you measured it though, just marvelling.

http://www.haskell.org/pipermail/glasgow-haskell-users/2008-October/015654.html

Most non-trivial packages either use exceptions or depend on a package
that uses exceptions (or syb).

The 90% is from memory however. I can't find our exact measurements. I
think we may have given up trying to measure precisely after it became
obvious that nothing was working.

> If 90% of hackage would truly break, then I agree that we might need
> more caution than my radical approach. But I'm not fully convinced
> there either.

I think in the end we managed to get it down to around 5% breakage which
I felt was a pretty good result.

> After all, unlike with the ghc-6.8 release, all that's
> needed for a package to work again is to upload a new .cabal that
> makes the dependency on base-3 explicit,

For the most part that was all that was required for 6.8 too, to add
dependencies on the new packages split out of base. People still
complained. A lot. :-)

> if an author really doesn't want to update the codebase. And even for
> those packages where library authors don't do that simple step, all
> that's needed of the user of the library is to specify the base-3
> constraint when running cabal-install.

But people do not know that. It has to work by default (even if it
warns).

> My main complaint is really that there is currently no incentive
> whatsoever for library authors to migrate. If we make base-4 the
> default, it will require just a little bit of work to make packages
> that depend on base-3 work anyway, as seen above. It's not so much
> work that it should incur any screaming, wailing and teeth gnashing.
> But it should be just enough work to encourage action in one way or
> another, either truly migrating the code or just making the dependency
> explicit as I noted above. I think it would be hard to find a more
> accurate, and non-intrusive, incentive. :-)

I think the right place to put incentives is in checks in new uploads to
hackage. Making existing packages break should be avoided when possible,
even when the changes the developer needs to make are trivial. Many many
end users have no idea how to make these trivial changes and it is not
our intention to punish them. From the perspective of an end user just
trying to install something it either works or it doesn't. When a large
proportion stop working is when the teeth gnashing starts. Also remember
that many packages do not have very active maintainers.

> I definitely agree with your suggestion to make hackage require an
> upper bound on base. But that's to make us future proof,

True.

> it won't solve the issue here and now.

What do you think about the suggestions in my other reply? I hope that
is a reasonable suggestion to encourage a transition to base 4 without
punishing end users.

Duncan

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: ANN: HDBC v2.0 now available

John Goerzen-3
In reply to this post by Duncan Coutts
Duncan Coutts wrote:

> On Sun, 2009-02-01 at 15:56 +0100, Niklas Broberg wrote:
>>> So in the next cabal-install release (which should be pretty soon now)
>>> configure will do the same thing and pick base 3 unless you specify
>>> build-depends base >= 4.
>> ... and so there will never be any incentive for these many packages
>> to migrate to base-4, which also has consequences for packages that do
>> want to use base-4, but also want to depend on such packages.
>
> Actually a package that uses base 4 can depend on other packages that
> use base 3. They all co-exist fine. This already happens now.
>
>> I would suggest as a less stagnating approach to issue a warning/hint
>> when a package with no explicit version dependency for base fails to
>> build.
>
> So my plan is to make hackage require an upper bound on the version of
> base for all new packages. That should avoid the need to use the
> preferences hack the next time around.

Hrm.  I can see why you might do that, if you keep the old base around.

On the other hand, what if base 5 introduces less invasive changes?

We have had, for awhile, a number of Haskell packages in Debian that
require a version of GHC greater than version x and less than version
x+1.  This has not worked out entirely well for us.  Granted, Cabal is
different because GHC 6.10 has two versions of base.

But still, I feel queasy about stating that my package won't work with
base 5 when I haven't even seen base 5 yet.

While we're at it, isn't this a more general problem that could occur
elsewhere?  Does base really need to be a special case?  What if, say,
utf8-string, binary, or some other commonly-used package had to break API?

> As for what mechanisms we use to persuade package authors to use base 4
> over base 3 for new releases, I'm open to suggestions.

Well, I'd say this will be difficult to achieve for at least two years,
since GHC 6.8 is in current and future shipping versions of numerous
Linux distributions, and such may well be the version of GHC most
readily accessible to quite a few users.

I am taking the approach of supporting *both*, but that is certainly not
the most simple approach (it's not all that complex either, once you
know how).

-- John
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: ANN: HDBC v2.0 now available

Duncan Coutts
On Sun, 2009-02-01 at 13:58 -0600, John Goerzen wrote:

> > So my plan is to make hackage require an upper bound on the version of
> > base for all new packages. That should avoid the need to use the
> > preferences hack the next time around.
>
> Hrm.  I can see why you might do that, if you keep the old base around.
>
> On the other hand, what if base 5 introduces less invasive changes?

Then it would not be base 5, it would be 4.something. That's the point
of the versioning policy.

http://haskell.org/haskellwiki/Package_versioning_policy

> We have had, for awhile, a number of Haskell packages in Debian that
> require a version of GHC greater than version x and less than version
> x+1.  This has not worked out entirely well for us.  Granted, Cabal is
> different because GHC 6.10 has two versions of base.
>
> But still, I feel queasy about stating that my package won't work with
> base 5 when I haven't even seen base 5 yet.

But it's easy to change it when you do see base 5 but it's annoying for
everyone when it fails messily. If you take the conservative approach
it's clear to end users and everyone else. We can track the status much
more easily.

> While we're at it, isn't this a more general problem that could occur
> elsewhere?  Does base really need to be a special case?  What if, say,
> utf8-string, binary, or some other commonly-used package had to break API?

That is what the package versioning policy is for. Any package can
opt-in to following it. In which case other packages that depend on said
package can rely on it and use appropriate version constraints.

All the core and platform packages follow the versioning policy.

The plan is to add tool support to let packages declare that they follow
the policy (eg: version-policy: PVP-1) and then we can check that the
package really is following the policy and we can also make helpful
suggestions to other packages that depend on it (ie tell them what
version constraints to use).

http://hackage.haskell.org/trac/hackage/ticket/434#comment:1

> > As for what mechanisms we use to persuade package authors to use base 4
> > over base 3 for new releases, I'm open to suggestions.
>
> Well, I'd say this will be difficult to achieve for at least two years,
> since GHC 6.8 is in current and future shipping versions of numerous
> Linux distributions, and such may well be the version of GHC most
> readily accessible to quite a few users.

However I'm not sure that it will be possible for ghc-6.12 to support
base 3 4 and 5. Some kinds of changes (especially to types) make
supporting multiple versions impossible.

> I am taking the approach of supporting *both*, but that is certainly not
> the most simple approach (it's not all that complex either, once you
> know how).

I'm not advocating dropping support for base 3. So I guess I am
advocating supporting both. I'm also not sure we need everyone to switch
now, but it'll become more important when we get nearer to ghc-6.12 next
autumn.

Duncan

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: ANN: HDBC v2.0 now available

Yitzchak Gale
In reply to this post by Duncan Coutts
Duncan Coutts wrote:
>>> So in the next cabal-install release (which should be pretty soon now)
>>> configure will do the same thing and pick base 3 unless you specify
>>> build-depends base >= 4.

Niklas Broberg wrote:
>> I really really think this is the wrong way to go. Occasional
>> destruction is desperately needed for progress, else things will
>> invariably stagnate.

> I disagree. Having everything fail... would have been a disaster...
> during the lifespan of base 4 we need to encourage new
> releases to start working with it...
> Doing that with warnings hints etc is the way to go.

No, that's not good enough either. Existing packages
will just stay with old versions to avoid the work, and
new packages will then also use old versions for
maximum compatibility.

The incentive is not strong enough. Those warnings
and hints must have teeth.

Comparing with what has happened in other languages,
which have stagnated or not stagnated at various rates,
it is clear that what we need is a well-defined deprecation
process.

The warnings should say something like: you had better
upgrade this, otherwise it will stop working in the next
version. Both maintainers and users should be aware of
this threat.

That way, code that is maintained will not stop working.
There will be plenty of time to make the change. Code that
is used but not maintained will generate a hue and cry
among the users that will hopefully motivate someone to
do something about it. Code that is not maintained and
little used will eventually be destroyed, but that code is
probably bitrotting in any case.

The deprecation process can be as slow and fine-grained
as we like. But there must be a well-defined point in
the future when old unmaintained code will be allowed
to stop working.

> Destruction is not such a friendly approach. We do not
> need to make the users suffer

When done carefully, gradually, and with plenty of warning
(at least one full version cycle), destruction is indeed
friendly and helpful. It allows users to understand precisely
what versions should be used, and when, in old, current, and
future projects, while permitting Haskell to march steadily
onward.

-Yitz
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: ANN: HDBC v2.0 now available

Duncan Coutts
On Mon, 2009-02-02 at 15:22 +0200, Yitzchak Gale wrote:

> Duncan Coutts wrote:
> >>> So in the next cabal-install release (which should be pretty soon now)
> >>> configure will do the same thing and pick base 3 unless you specify
> >>> build-depends base >= 4.
>
> Niklas Broberg wrote:
> >> I really really think this is the wrong way to go. Occasional
> >> destruction is desperately needed for progress, else things will
> >> invariably stagnate.
>
> > I disagree. Having everything fail... would have been a disaster...
> > during the lifespan of base 4 we need to encourage new
> > releases to start working with it...
> > Doing that with warnings hints etc is the way to go.
>
> No, that's not good enough either. Existing packages
> will just stay with old versions to avoid the work, and
> new packages will then also use old versions for
> maximum compatibility.
>
> The incentive is not strong enough. Those warnings
> and hints must have teeth.
>
> Comparing with what has happened in other languages,
> which have stagnated or not stagnated at various rates,
> it is clear that what we need is a well-defined deprecation
> process.

> The warnings should say something like: you had better
> upgrade this, otherwise it will stop working in the next
> version. Both maintainers and users should be aware of
> this threat.

I shouldn't worry. The next major version of ghc probably will not have
base 3 any more. So the question is how we check and what warnings we
give and to who.

My current suggestion is to require upper bounds on the version of base
and to warn when we end up selecting base 3 for older packages that do
not specify an upper bound.

Perhaps at some point we should start warning when the upper bound
people specify for new uploads is < 4.

What do you think the policy and mechanism should be?

Duncan

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
12