Hackage is flooded with old package versions reuploads

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
32 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Hackage is flooded with old package versions reuploads

Kyra-3
Hi, guys,

It looks old (and even ancient) versions of many packages gets uploaded
to hackage over and over again in ever increasing amounts. The username
of uploader for vast majority of these uploads is HerbertValerioRiedel.

While this is harmless I wonder what idea stands behind this?

Cheers,
Kyra
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Brandon Allbery
On Sun, Jan 18, 2015 at 12:56 PM, kyra <[hidden email]> wrote:
It looks old (and even ancient) versions of many packages gets uploaded to hackage over and over again in ever increasing amounts. The username of uploader for vast majority of these uploads is HerbertValerioRiedel.

While this is harmless I wonder what idea stands behind this?

Correcting bad package dependency limits to avoid things trying to use packages that can never work but whose loose versioning claims that it will? 

--
brandon s allbery kf8nh                               sine nomine associates
[hidden email]                                  [hidden email]
unix, openafs, kerberos, infrastructure, xmonad        http://sinenomine.net

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Edward Kmett-2
In reply to this post by Kyra-3
Herbert patches old versions to fix up build plans and put in upper bounds that are shown to cause problems.

He's doing an awesome job and has greatly simplified my life.

Sent from my iPhone

> On Jan 18, 2015, at 12:56 PM, kyra <[hidden email]> wrote:
>
> Hi, guys,
>
> It looks old (and even ancient) versions of many packages gets uploaded to hackage over and over again in ever increasing amounts. The username of uploader for vast majority of these uploads is HerbertValerioRiedel.
>
> While this is harmless I wonder what idea stands behind this?
>
> Cheers,
> Kyra
> _______________________________________________
> Libraries mailing list
> [hidden email]
> http://www.haskell.org/mailman/listinfo/libraries
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Vincent Hanquez
In reply to this post by Kyra-3

On 18/01/2015 09:56, kyra wrote:
> Hi, guys,
>
> It looks old (and even ancient) versions of many packages gets
> uploaded to hackage over and over again in ever increasing amounts.
> The username of uploader for vast majority of these uploads is
> HerbertValerioRiedel.
>
> While this is harmless I wonder what idea stands behind this?
This is not harmless. This is a security issue by itself, as now
packages get changes transparently given a url, you might have a
different package one day, which trigger hash check failure. or signed
tag verification failure.

This has also the effect of not changing the bounds in the repository,
so for example, next time you upload a tweak'ed packages, you
effectively revert the change done on hackage only.

This is also done without the consent of the maintainer of a given
package, nor that the maintainer is actually notified when that happens,
or allow to prevent it happening. This is pretty big start from the
other similar policy for taking over packages, that insist on a very
long period of repeated communication with the author and then the
community.

The whole thing is at best ill advised,
--
Vincent
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Edward Kmett-2
The alternative is just that cabal will continue indefinitely to try to install completely broken combinations, and more people will be driven to a fixed package set like stackage LTS. 

Most of these problems are caused by people being too optimistic about upper bounds and when they realize their mistake and upload a new version, they'll often leave the old versions with the lying bounds intact, which causes cabal to pick old versions without bug fixes, and then give strange build errors.

To my knowledge, the few cases where Herbert has actively done a patch to the .cabal file like this without author communication is because the package is in very very widespread use and the author has been incommunicado for many months. As I recall, Max Bolingbroke has a some packages that fit this bill.

At least in my case, and in the case of the Haskell core libraries, Herbert has been very conscientious about talking to me, finding problems, auditing what builds across all versions of GHC in recent and not-so-recent memory, and working with me to find the best fix on a case by case basis. He has my explicit consent for any tweaks he has had to make to the build dependencies of my packages and has worked with the core libraries committee very closely for patches to the core libraries.

If you have an example of a package you've written that he's patched that you'd rather he left alone, I'm sure he'd be happy to oblige. I am, however, as of yet unaware of any such overreach and I'm rather disinclined to view the enormous amount of effort Herbert has poured into keeping the ecosystem working smoothly as anything but a good thing. The price of doing nothing here is quite high.

-Edward


On Sun, Jan 18, 2015 at 6:05 PM, Vincent Hanquez <[hidden email]> wrote:

On 18/01/2015 09:56, kyra wrote:
Hi, guys,

It looks old (and even ancient) versions of many packages gets uploaded to hackage over and over again in ever increasing amounts. The username of uploader for vast majority of these uploads is HerbertValerioRiedel.

While this is harmless I wonder what idea stands behind this?
This is not harmless. This is a security issue by itself, as now packages get changes transparently given a url, you might have a different package one day, which trigger hash check failure. or signed tag verification failure.

This has also the effect of not changing the bounds in the repository, so for example, next time you upload a tweak'ed packages, you effectively revert the change done on hackage only.

This is also done without the consent of the maintainer of a given package, nor that the maintainer is actually notified when that happens, or allow to prevent it happening. This is pretty big start from the other similar policy for taking over packages, that insist on a very long period of repeated communication with the author and then the community.

The whole thing is at best ill advised,
--
Vincent

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries


_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

David Feuer
In reply to this post by Vincent Hanquez

It would be best to be sure to make the maintainer (if there is one) aware of such changes. That said, not every package has a responsive maintainer, and *someone* has to do this work, and do it promptly. A signed hash failure does not introduce a security hole, unless you count a sort of semi-manual, avoidable denial of service. If you don't trust Herbert and Austin, you probably shouldn't bother trying to use Haskell anyway.

On Jan 18, 2015 6:05 PM, "Vincent Hanquez" <[hidden email]> wrote:

On 18/01/2015 09:56, kyra wrote:
Hi, guys,

It looks old (and even ancient) versions of many packages gets uploaded to hackage over and over again in ever increasing amounts. The username of uploader for vast majority of these uploads is HerbertValerioRiedel.

While this is harmless I wonder what idea stands behind this?
This is not harmless. This is a security issue by itself, as now packages get changes transparently given a url, you might have a different package one day, which trigger hash check failure. or signed tag verification failure.

This has also the effect of not changing the bounds in the repository, so for example, next time you upload a tweak'ed packages, you effectively revert the change done on hackage only.

This is also done without the consent of the maintainer of a given package, nor that the maintainer is actually notified when that happens, or allow to prevent it happening. This is pretty big start from the other similar policy for taking over packages, that insist on a very long period of repeated communication with the author and then the community.

The whole thing is at best ill advised,
--
Vincent
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Vincent Hanquez
In reply to this post by Edward Kmett-2

On 18/01/2015 15:49, Edward Kmett wrote:
> The alternative is just that cabal will continue indefinitely to try
> to install completely broken combinations, and more people will be
> driven to a fixed package set like stackage LTS.
That strikes me as probably-a-good-thing.

> Most of these problems are caused by people being too optimistic about
> upper bounds and when they realize their mistake and upload a new
> version, they'll often leave the old versions with the lying bounds
> intact, which causes cabal to pick old versions without bug fixes, and
> then give strange build errors.
>
> To my knowledge, the few cases where Herbert has actively done a patch
> to the .cabal file like this without author communication is because
> the package is in very very widespread use and the author has been
> incommunicado for many months. As I recall, Max Bolingbroke has a some
> packages that fit this bill.
A simple counter example, that I noticed after the fact [1]
> If you have an example of a package you've written that he's patched
> that you'd rather he left alone, I'm sure he'd be happy to oblige. I
> am, however, as of yet unaware of any such overreach and I'm rather
> disinclined to view the enormous amount of effort Herbert has poured
> into keeping the ecosystem working smoothly as anything but a good
> thing. The price of doing nothing here is quite high.
I strongly object to the current mechanism of silent updates (for the
downloader), and I would much rather have all my packages left alone
until this changes at the very least (if ever). The maintenance
overriding is another sad point (which might be warranted in some case,
although haskell already have a procedure in place for that), but in any
case not as critical as the first point.

[1] https://hackage.haskell.org/package/hourglass

--
Vincent
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Vincent Hanquez
In reply to this post by David Feuer

On 18/01/2015 15:51, David Feuer wrote:
>
> It would be best to be sure to make the maintainer (if there is one)
> aware of such changes. That said, not every package has a responsive
> maintainer, and *someone* has to do this work, and do it promptly. A
> signed hash failure does not introduce a security hole, unless you
> count a sort of semi-manual, avoidable denial of service.
>
Not sure how you got "security hole" from what I said, but a failing
hash or signature, means that the build system breaks while cabal
install stuff and that I have to manually inspect what the change is. If
you can't pin down a special tarball when doing a download (i.e. it can
changes under your feet, one day to the other), then it's an issue.

Lots of people would be *horrified* to download some
{c,c++,python,ruby,...} library-a.b.c.tar.gz and found anything changed
inside without changing the exact name for it.
>
> If you don't trust Herbert and Austin, you probably shouldn't bother
> trying to use Haskell anyway.
>
lol ? Do you mean that I should switch language, if security is remotely
important to me ?

As much as Herbert and Austin are doing awesome work in general, I
certainly do not blindly trust them.

--
Vincent
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Brandon Allbery
On Sun, Jan 18, 2015 at 7:57 PM, Vincent Hanquez <[hidden email]> wrote:
Lots of people would be *horrified* to download some {c,c++,python,ruby,...} library-a.b.c.tar.gz and found anything changed inside without changing the exact name for it.

Funny thing, that, because it happens pretty often and packagers have to have "silent update" (upstream silently repackaged without changing the version) rules to deal with it.

--
brandon s allbery kf8nh                               sine nomine associates
[hidden email]                                  [hidden email]
unix, openafs, kerberos, infrastructure, xmonad        http://sinenomine.net

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Eric Mertens

Wow, I thought this was a joke at first. HVRs work is terribly useful for enabling old and new versions of GHC to work with hackage, and I've very much appreciated the build reports he generates.

I'm sure he'd be happy to leave packages alone for anyone who prefers their packages remain broken.

Thanks hvr for fixing the dozens and dozens of versions of just the lens package in order to sort the base-4.8 rollout in particular that would have taken an age to do manually.


On Sun, Jan 18, 2015, 5:00 PM Brandon Allbery <[hidden email]> wrote:
On Sun, Jan 18, 2015 at 7:57 PM, Vincent Hanquez <[hidden email]> wrote:
Lots of people would be *horrified* to download some {c,c++,python,ruby,...} library-a.b.c.tar.gz and found anything changed inside without changing the exact name for it.

Funny thing, that, because it happens pretty often and packagers have to have "silent update" (upstream silently repackaged without changing the version) rules to deal with it.


--
brandon s allbery kf8nh                               sine nomine associates
[hidden email]                                  [hidden email]
unix, openafs, kerberos, infrastructure, xmonad        http://sinenomine.net
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Roman Cheplyaka-2
In reply to this post by Vincent Hanquez
On 19/01/15 01:05, Vincent Hanquez wrote:
> This is not harmless. This is a security issue by itself, as now
> packages get changes transparently given a url, you might have a
> different package one day, which trigger hash check failure. or signed
> tag verification failure.

Correct me if I'm wrong, but editing version bounds on hackage doesn't
actually affect the tarball (and its checksum). The modified cabal file
is downloaded separately as part of the index.

Not saying it doesn't introduce its own problems, but the hash check
should continue to pass.

Roman

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Vincent Hanquez

On 18/01/2015 20:23, Roman Cheplyaka wrote:
> On 19/01/15 01:05, Vincent Hanquez wrote:
>> This is not harmless. This is a security issue by itself, as now
>> packages get changes transparently given a url, you might have a
>> different package one day, which trigger hash check failure. or signed
>> tag verification failure.
> Correct me if I'm wrong, but editing version bounds on hackage doesn't
> actually affect the tarball (and its checksum). The modified cabal file
> is downloaded separately as part of the index.
yes, that's right. I meant to say that what you're downloading through
cabal get
tweaked by cabal, but the end result is pretty much the same
> Not saying it doesn't introduce its own problems, but the hash check
> should continue to pass.
of the tarball yes, not of your compilation tree, and maybe not the
resulting binary.

--
Vincent
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Herbert Valerio Riedel
In reply to this post by Kyra-3
On 2015-01-18 at 18:56:42 +0100, kyra wrote:
> It looks old (and even ancient) versions of many packages gets
> uploaded to hackage over and over again in ever increasing
> amounts. The username of uploader for vast majority of these uploads
> is HerbertValerioRiedel.

That would be me...

Just to clarify: I those aren't re-uploads of existing package versions,
but rather meta-data revisions. The actual source-code is *not* touched,
(and Hackage is very restricting in what's allowed to be changed in a
.cabal file)

This is also mentioned on http://hackage.haskell.org/packages/trustees/

> While this is harmless I wonder what idea stands behind this?

The idea, as others in this thread have already pointed out, is to add
missing lower/upper package bounds to packages so that the Cabal
install-plan solver can fulfill its task.

Over the last week, I've been analyzing the current situation (motivated
by the upcoming GHC 7.10 release which trips up quite a few packages
with missing bounds on base/deepseq/time), and trying to come up with
some tooling[1] to help the otherwise IMO rather unmanageable job of a
Hackage trustee, as well as surgically fixing up the meta-data where I
had empirical proof that the .cabal meta-data was incorrect (and thus
leading the solver to select a broken install-plan).

Here's a very simple buildreport for illustration:

 https://ghc.haskell.org/~hvr/buildreports/pretty.html

The purpose of my meta-data edits were/are (in order of decreasing
priority):

 1. First and foremost, no currently working/valid install-plan was lost.

 2. A simple constraint-less `cabal install pretty` ought to result
    *either* in a valid install-plan, *or* cabal failing to find any
    install-plan.

 3. An major-version constraint `cabal install 'pretty == a.b.*'` ought
    to result *either* in a valid install-plan, *or* cabal failing to find
    any install-plan.

 4. A package shall not cause indirect *build*-failures of other
    packages directly or indirectly depending on this package[3]

The build-report linked helps directly with 2. and 3. as it shows that
`pretty` failed to `cabal install pretty` (as well as `cabal install
'pretty == 1.1.*'`) for GHC 7.0 and GHC 7.2.  Case 4. could be detected
by running the buildtool on many Hackage packages depending on `pretty`,
and look for failures to build `pretty` during the `cabal install --dep`
step (which would result in a "FAIL (deps)" cells).


I've encountered a couple of situations, where fixing the meta-data of
an important package low in the dependency graph fixed a couple of other
packages depending on that package. However, this also means that the
more popular a package is, the higher its responsibility is to make sure
its build-depend meta-data is correct, as such packages have the
potential to instantly break large branches of Hackage:

One example for that was the release of deepseq-1.4, which changes the
default-implementation of `rnf` which a few packages relied on, but the
rather important `text` package was among of those. This affected most
of the released ~50 `text` package versions, and since `deepseq` is
compatible with all GHC 7.* versions, as soon an install-plan was
selected which included `deepseq-1.4` with a non-latest version of
`text` the build would fail. And since it's not uncommon to restrict the
upper bound of `text`[2], all those packages depending on a non-latest
`text` would have broken install-plans (unless `deepseq < 1.4` was
selected).

I hope this answers your question.

Cheers,
  hvr


 [1]: https://github.com/hvr/hackage-matrix-builder

      (WIP example reports can be seen in

       https://ghc.haskell.org/~hvr/buildreports/0INDEX.html)


 [2]: http://packdeps.haskellers.com/reverse/text

 [3]: This can happen due to additional package-version constraints
      causing other (but broken) configurations to be selected by the
      solver not detected in case 2. and 3.
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Herbert Valerio Riedel
In reply to this post by Vincent Hanquez
On 2015-01-19 at 00:05:31 +0100, Vincent Hanquez wrote:

[...]

>> While this is harmless I wonder what idea stands behind this?

> This is not harmless. This is a security issue by itself, as now
> packages get changes transparently given a url, you might have a
> different package one day, which trigger hash check failure. or signed
> tag verification failure.

Just to clarify: the source tarball are *never* modified to specifically
make sure hashes don't randomly change (which would trip all sorts of
warnings for distribution packagers). In fact, there's also a
`cabal get` option if you want to unpack with the original unaltered .cabal
file:

    --pristine    Unpack the original pristine tarball,
                  rather than updating the .cabal file
                  with the latest revision from the
                  package archive.

> This has also the effect of not changing the bounds in the repository,
> so for example, next time you upload a tweak'ed packages, you
> effectively revert the change done on hackage only.
> This is also done without the consent of the maintainer of a given
> package, nor that the maintainer is actually notified when that
> happens, or allow to prevent it happening.

Fwiw, there's plans for an automated notification feature:

  https://github.com/haskell/hackage-server/issues/230

We're actively working on refining the system, and finding out where it
needs to be improved. For most of my edits I did in fact inform upstream
to make sure the broken install-plan issue was known so future uploads
could address the issue (or at least add missing bounds to avoid
regressing again).

> This is pretty big start from the other similar policy for taking over
> packages, that insist on a very long period of repeated communication
> with the author and then the community.

We probably need a broader discussion to clarify what Hackage's goal
shall be in terms of (various dimensions of) stability, what policies
Hackage trustees shall follow to help achieve these goals, and most
importantly, which responsibilities a package author agrees to (and
which (s)he delegates to trustees), when (s)he uploads a package to
Hackage, which is a shared and vital resource of our ecosystem and IMO
ought to be kept in good condition to ensure a good experience for all
its users.

Right now there's varying degrees of expectations of what Hackage is,
with one extreme being that Hackage is simply considered a storage-space
for dumping package source-tarballs to (and where .cabal meta-data
hygiene has low to no priority), and the other end of the spectrum being
the vision of Hackage as a somewhat curated (and yet bleeding-edge)
package repository.

The latter is not unlike Stackage (or the LTSHaskell idea), but with a
different priority of eliminating broken install plans (while retaining
the degree of freedom version-ranges for dependencies provide) rather
than providing a single vetted cross-section of Hackage.

Even with the legitimate existence of Stackage (and possibly
LTSHaskell), there's still the need for keeping Hackage in good shape
meta-data-wise:


 1) There's a significant amount of users which are stuck with an older
    GHC (Debian Stable currently ships with GHC 7.4.2, Debian Jessie
    will most likely ship with GHC 7.6.3 later this year; I've seen
    reports of RHEL6 users being stuck with GHC 7.0.4)

    Also, the Haskell Platform lags behind one major GHC version for
    some time till it picks up the latest stable GHC version. It'll be
    quite some time till we see a HP release featuring GHC 7.10.x after
    GHC 7.10.1 is released.

    While I don't consider it paramount for every latest package version
    to still be compatible with GHC 7.4.2 (or GHC 7.0), I do consider it
    important that a `cabal install $PKG` picks the latest still
    compatible package version (if one exists). The more popular $PKG
    is, the more important I consider this.

    Just consider a new Haskell user who doesn't know all the issues
    with Hackage who reads a blog-post (I'm thinking of something like
    "24 Days of Hackage"), then goes on to `cabal install $PKG` and is
    confronted with a compile-error (rather than picking an older but
    working version)... is that the first impression we want new users
    to have with Haskell & Hackage?
   
 2) Before a new package version can be cherry-picked into
    Stackage/LTSHaskell, it needs to be on something like Hackage first,
    also to allow other packages to be updated against the new version
    (which can result in a cascade of package version updates throughout
    the dependency-graph).

    This is IMO one killer feature Hackage has with its version-ranges
    instead of single-version pinned down package-sets. I think such
    update scenarios would be quite harder to perform if Stackage didn't
    have Hackage as the staging/integration area for version-update
    waves.

 3) As an extension of 2), library maintainers also need properly
    working build-plans for older GHCs for latest-versions of the
    build-deps they rely on, as otherwise they have a hard time
    providing and ensuring backward compatibility for their packages
    during development if e.g. their Travis-job-matrices start falling
    apart due to suddenly beaking install-plans.

    The alternative would be to have package versions restrict their
    support to single GHC major versions and provide bugfixes (and
    improvements) for older GHCs via backports to previous
    package-version branches.

    Personally, I consider it less effort to have the latest
    package-version support a couple of recent GHC version, and slowly
    phase out support for older GHCs if it becomes to costly to support
    those (because e.g. features from newer GHCs are exploited)

    That results in a "diagonal support corridor" in the build-reports
    as in e.g.

      https://ghc.haskell.org/~hvr/buildreports/either.html

    Ideally, if you can support older GHCs, then the pattern looks like
    a lower-left support triangle as in

      https://ghc.haskell.org/~hvr/buildreports/unordered-containers.html


This email's already gotten much longer than I intended to and I started
digressing... so I'm better stop here... :-)

Cheers,
  hvr
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Herbert Valerio Riedel
In reply to this post by Eric Mertens
Hello,

On 2015-01-19 at 02:25:56 +0100, Eric Mertens wrote:

[...]

> I'm sure he'd be happy to leave packages alone for anyone who prefers
> their packages remain broken.

I wouldn't say I'm "happy" to, but I'll certainly comply with the
request.

The reason I wouldn't be happy is that the effects of a "broken" package
(especially the more popular it becomes) can't be contained easily. The
breakage cascades through the build-dep graph (I already mentioned the
`text`-example in another reply). In some sense, "no package is an
island"...

Cheers,
  hvr
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Brandon Allbery
On Mon, Jan 19, 2015 at 7:06 AM, Herbert Valerio Riedel <[hidden email]> wrote:
The reason I wouldn't be happy is that the effects of a "broken" package
(especially the more popular it becomes) can't be contained easily. The

I wonder if Hackage can be extended to support an out-of-band "broken" flag that can be applied to such packages, and cabal-install then refuse to use those packages (possibly with an option to override).

--
brandon s allbery kf8nh                               sine nomine associates
[hidden email]                                  [hidden email]
unix, openafs, kerberos, infrastructure, xmonad        http://sinenomine.net

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Carter Schonwald
I for one am immensely grateful for Herberts awesome work, and appreciate it immensely and support it emphatically. (and I do a bit of similar work in miniature as well as a hackage trustee). I dont think people realize how many breakages/old issues have been resolved via the hard work of the various hackage trustees. hackage trustee's doing their jobs well results in a huge reduction in build problems by end users. 

On Mon, Jan 19, 2015 at 9:59 AM, Brandon Allbery <[hidden email]> wrote:
On Mon, Jan 19, 2015 at 7:06 AM, Herbert Valerio Riedel <[hidden email]> wrote:
The reason I wouldn't be happy is that the effects of a "broken" package
(especially the more popular it becomes) can't be contained easily. The

I wonder if Hackage can be extended to support an out-of-band "broken" flag that can be applied to such packages, and cabal-install then refuse to use those packages (possibly with an option to override).

--
brandon s allbery kf8nh                               sine nomine associates
[hidden email]                                  [hidden email]
unix, openafs, kerberos, infrastructure, xmonad        http://sinenomine.net

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries



_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Gabriel Gonzalez
In reply to this post by Herbert Valerio Riedel
I just want to say that I appreciate your work.  You're a hero for doing
this very thankless but very beneficial work.

On 01/19/2015 12:31 AM, Herbert Valerio Riedel wrote:

> On 2015-01-18 at 18:56:42 +0100, kyra wrote:
>> It looks old (and even ancient) versions of many packages gets
>> uploaded to hackage over and over again in ever increasing
>> amounts. The username of uploader for vast majority of these uploads
>> is HerbertValerioRiedel.
> That would be me...
>
> Just to clarify: I those aren't re-uploads of existing package versions,
> but rather meta-data revisions. The actual source-code is *not* touched,
> (and Hackage is very restricting in what's allowed to be changed in a
> .cabal file)
>
> This is also mentioned on http://hackage.haskell.org/packages/trustees/
>
>> While this is harmless I wonder what idea stands behind this?
> The idea, as others in this thread have already pointed out, is to add
> missing lower/upper package bounds to packages so that the Cabal
> install-plan solver can fulfill its task.
>
> Over the last week, I've been analyzing the current situation (motivated
> by the upcoming GHC 7.10 release which trips up quite a few packages
> with missing bounds on base/deepseq/time), and trying to come up with
> some tooling[1] to help the otherwise IMO rather unmanageable job of a
> Hackage trustee, as well as surgically fixing up the meta-data where I
> had empirical proof that the .cabal meta-data was incorrect (and thus
> leading the solver to select a broken install-plan).
>
> Here's a very simple buildreport for illustration:
>
>   https://ghc.haskell.org/~hvr/buildreports/pretty.html
>
> The purpose of my meta-data edits were/are (in order of decreasing
> priority):
>
>   1. First and foremost, no currently working/valid install-plan was lost.
>
>   2. A simple constraint-less `cabal install pretty` ought to result
>      *either* in a valid install-plan, *or* cabal failing to find any
>      install-plan.
>
>   3. An major-version constraint `cabal install 'pretty == a.b.*'` ought
>      to result *either* in a valid install-plan, *or* cabal failing to find
>      any install-plan.
>
>   4. A package shall not cause indirect *build*-failures of other
>      packages directly or indirectly depending on this package[3]
>
> The build-report linked helps directly with 2. and 3. as it shows that
> `pretty` failed to `cabal install pretty` (as well as `cabal install
> 'pretty == 1.1.*'`) for GHC 7.0 and GHC 7.2.  Case 4. could be detected
> by running the buildtool on many Hackage packages depending on `pretty`,
> and look for failures to build `pretty` during the `cabal install --dep`
> step (which would result in a "FAIL (deps)" cells).
>
>
> I've encountered a couple of situations, where fixing the meta-data of
> an important package low in the dependency graph fixed a couple of other
> packages depending on that package. However, this also means that the
> more popular a package is, the higher its responsibility is to make sure
> its build-depend meta-data is correct, as such packages have the
> potential to instantly break large branches of Hackage:
>
> One example for that was the release of deepseq-1.4, which changes the
> default-implementation of `rnf` which a few packages relied on, but the
> rather important `text` package was among of those. This affected most
> of the released ~50 `text` package versions, and since `deepseq` is
> compatible with all GHC 7.* versions, as soon an install-plan was
> selected which included `deepseq-1.4` with a non-latest version of
> `text` the build would fail. And since it's not uncommon to restrict the
> upper bound of `text`[2], all those packages depending on a non-latest
> `text` would have broken install-plans (unless `deepseq < 1.4` was
> selected).
>
> I hope this answers your question.
>
> Cheers,
>    hvr
>
>
>   [1]: https://github.com/hvr/hackage-matrix-builder
>
>        (WIP example reports can be seen in
>
>         https://ghc.haskell.org/~hvr/buildreports/0INDEX.html)
>
>
>   [2]: http://packdeps.haskellers.com/reverse/text
>
>   [3]: This can happen due to additional package-version constraints
>        causing other (but broken) configurations to be selected by the
>        solver not detected in case 2. and 3.
> _______________________________________________
> Libraries mailing list
> [hidden email]
> http://www.haskell.org/mailman/listinfo/libraries

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Edward Kmett-2
In reply to this post by Brandon Allbery
We have a deprecation flag on hackage now, it just doesn't do anything at last check.

On Mon, Jan 19, 2015 at 9:59 AM, Brandon Allbery <[hidden email]> wrote:
On Mon, Jan 19, 2015 at 7:06 AM, Herbert Valerio Riedel <[hidden email]> wrote:
The reason I wouldn't be happy is that the effects of a "broken" package
(especially the more popular it becomes) can't be contained easily. The

I wonder if Hackage can be extended to support an out-of-band "broken" flag that can be applied to such packages, and cabal-install then refuse to use those packages (possibly with an option to override).

--
brandon s allbery kf8nh                               sine nomine associates
[hidden email]                                  [hidden email]
unix, openafs, kerberos, infrastructure, xmonad        http://sinenomine.net

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries



_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Brandon Allbery
On Mon, Jan 19, 2015 at 5:07 PM, Edward Kmett <[hidden email]> wrote:
We have a deprecation flag on hackage now, it just doesn't do anything at last check.

Which is exactly why I specified that it needs to be something that cabal honors by default.

As a sysadmin, I'd actually prefer the flag to overwriting packages --- but I find the Haskell community usually goes in other directions than what makes sense to me from the standpoint of stability vs. usability vs. flexibility. And given that the deprecation flag was added as more or less documentation, I don't really expect that a proper disable option will be added....

--
brandon s allbery kf8nh                               sine nomine associates
[hidden email]                                  [hidden email]
unix, openafs, kerberos, infrastructure, xmonad        http://sinenomine.net

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
12