Hi, guys,
It looks old (and even ancient) versions of many packages gets uploaded to hackage over and over again in ever increasing amounts. The username of uploader for vast majority of these uploads is HerbertValerioRiedel. While this is harmless I wonder what idea stands behind this? Cheers, Kyra _______________________________________________ Libraries mailing list [hidden email] http://www.haskell.org/mailman/listinfo/libraries |
On Sun, Jan 18, 2015 at 12:56 PM, kyra <[hidden email]> wrote: It looks old (and even ancient) versions of many packages gets uploaded to hackage over and over again in ever increasing amounts. The username of uploader for vast majority of these uploads is HerbertValerioRiedel. Correcting bad package dependency limits to avoid things trying to use packages that can never work but whose loose versioning claims that it will? brandon s allbery kf8nh sine nomine associates unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net _______________________________________________ Libraries mailing list [hidden email] http://www.haskell.org/mailman/listinfo/libraries |
In reply to this post by Kyra-3
Herbert patches old versions to fix up build plans and put in upper bounds that are shown to cause problems.
He's doing an awesome job and has greatly simplified my life. Sent from my iPhone > On Jan 18, 2015, at 12:56 PM, kyra <[hidden email]> wrote: > > Hi, guys, > > It looks old (and even ancient) versions of many packages gets uploaded to hackage over and over again in ever increasing amounts. The username of uploader for vast majority of these uploads is HerbertValerioRiedel. > > While this is harmless I wonder what idea stands behind this? > > Cheers, > Kyra > _______________________________________________ > Libraries mailing list > [hidden email] > http://www.haskell.org/mailman/listinfo/libraries Libraries mailing list [hidden email] http://www.haskell.org/mailman/listinfo/libraries |
In reply to this post by Kyra-3
On 18/01/2015 09:56, kyra wrote: > Hi, guys, > > It looks old (and even ancient) versions of many packages gets > uploaded to hackage over and over again in ever increasing amounts. > The username of uploader for vast majority of these uploads is > HerbertValerioRiedel. > > While this is harmless I wonder what idea stands behind this? This is not harmless. This is a security issue by itself, as now packages get changes transparently given a url, you might have a different package one day, which trigger hash check failure. or signed tag verification failure. This has also the effect of not changing the bounds in the repository, so for example, next time you upload a tweak'ed packages, you effectively revert the change done on hackage only. This is also done without the consent of the maintainer of a given package, nor that the maintainer is actually notified when that happens, or allow to prevent it happening. This is pretty big start from the other similar policy for taking over packages, that insist on a very long period of repeated communication with the author and then the community. The whole thing is at best ill advised, -- Vincent _______________________________________________ Libraries mailing list [hidden email] http://www.haskell.org/mailman/listinfo/libraries |
The alternative is just that cabal will continue indefinitely to try to install completely broken combinations, and more people will be driven to a fixed package set like stackage LTS. Most of these problems are caused by people being too optimistic about upper bounds and when they realize their mistake and upload a new version, they'll often leave the old versions with the lying bounds intact, which causes cabal to pick old versions without bug fixes, and then give strange build errors. To my knowledge, the few cases where Herbert has actively done a patch to the .cabal file like this without author communication is because the package is in very very widespread use and the author has been incommunicado for many months. As I recall, Max Bolingbroke has a some packages that fit this bill. At least in my case, and in the case of the Haskell core libraries, Herbert has been very conscientious about talking to me, finding problems, auditing what builds across all versions of GHC in recent and not-so-recent memory, and working with me to find the best fix on a case by case basis. He has my explicit consent for any tweaks he has had to make to the build dependencies of my packages and has worked with the core libraries committee very closely for patches to the core libraries. If you have an example of a package you've written that he's patched that you'd rather he left alone, I'm sure he'd be happy to oblige. I am, however, as of yet unaware of any such overreach and I'm rather disinclined to view the enormous amount of effort Herbert has poured into keeping the ecosystem working smoothly as anything but a good thing. The price of doing nothing here is quite high. -Edward On Sun, Jan 18, 2015 at 6:05 PM, Vincent Hanquez <[hidden email]> wrote:
_______________________________________________ Libraries mailing list [hidden email] http://www.haskell.org/mailman/listinfo/libraries |
In reply to this post by Vincent Hanquez
It would be best to be sure to make the maintainer (if there is one) aware of such changes. That said, not every package has a responsive maintainer, and *someone* has to do this work, and do it promptly. A signed hash failure does not introduce a security hole, unless you count a sort of semi-manual, avoidable denial of service. If you don't trust Herbert and Austin, you probably shouldn't bother trying to use Haskell anyway. On Jan 18, 2015 6:05 PM, "Vincent Hanquez" <[hidden email]> wrote:
_______________________________________________ Libraries mailing list [hidden email] http://www.haskell.org/mailman/listinfo/libraries |
In reply to this post by Edward Kmett-2
On 18/01/2015 15:49, Edward Kmett wrote: > The alternative is just that cabal will continue indefinitely to try > to install completely broken combinations, and more people will be > driven to a fixed package set like stackage LTS. That strikes me as probably-a-good-thing. > Most of these problems are caused by people being too optimistic about > upper bounds and when they realize their mistake and upload a new > version, they'll often leave the old versions with the lying bounds > intact, which causes cabal to pick old versions without bug fixes, and > then give strange build errors. > > To my knowledge, the few cases where Herbert has actively done a patch > to the .cabal file like this without author communication is because > the package is in very very widespread use and the author has been > incommunicado for many months. As I recall, Max Bolingbroke has a some > packages that fit this bill. > If you have an example of a package you've written that he's patched > that you'd rather he left alone, I'm sure he'd be happy to oblige. I > am, however, as of yet unaware of any such overreach and I'm rather > disinclined to view the enormous amount of effort Herbert has poured > into keeping the ecosystem working smoothly as anything but a good > thing. The price of doing nothing here is quite high. I strongly object to the current mechanism of silent updates (for the downloader), and I would much rather have all my packages left alone until this changes at the very least (if ever). The maintenance overriding is another sad point (which might be warranted in some case, although haskell already have a procedure in place for that), but in any case not as critical as the first point. [1] https://hackage.haskell.org/package/hourglass -- Vincent _______________________________________________ Libraries mailing list [hidden email] http://www.haskell.org/mailman/listinfo/libraries |
In reply to this post by David Feuer
On 18/01/2015 15:51, David Feuer wrote: > > It would be best to be sure to make the maintainer (if there is one) > aware of such changes. That said, not every package has a responsive > maintainer, and *someone* has to do this work, and do it promptly. A > signed hash failure does not introduce a security hole, unless you > count a sort of semi-manual, avoidable denial of service. > Not sure how you got "security hole" from what I said, but a failing hash or signature, means that the build system breaks while cabal install stuff and that I have to manually inspect what the change is. If you can't pin down a special tarball when doing a download (i.e. it can changes under your feet, one day to the other), then it's an issue. Lots of people would be *horrified* to download some {c,c++,python,ruby,...} library-a.b.c.tar.gz and found anything changed inside without changing the exact name for it. > > If you don't trust Herbert and Austin, you probably shouldn't bother > trying to use Haskell anyway. > lol ? Do you mean that I should switch language, if security is remotely important to me ? As much as Herbert and Austin are doing awesome work in general, I certainly do not blindly trust them. -- Vincent _______________________________________________ Libraries mailing list [hidden email] http://www.haskell.org/mailman/listinfo/libraries |
On Sun, Jan 18, 2015 at 7:57 PM, Vincent Hanquez <[hidden email]> wrote:
Funny thing, that, because it happens pretty often and packagers have to have "silent update" (upstream silently repackaged without changing the version) rules to deal with it. brandon s allbery kf8nh sine nomine associates unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net _______________________________________________ Libraries mailing list [hidden email] http://www.haskell.org/mailman/listinfo/libraries |
Wow, I thought this was a joke at first. HVRs work is terribly useful for enabling old and new versions of GHC to work with hackage, and I've very much appreciated the build reports he generates. I'm sure he'd be happy to leave packages alone for anyone who prefers their packages remain broken. Thanks hvr for fixing the dozens and dozens of versions of just the lens package in order to sort the base-4.8 rollout in particular that would have taken an age to do manually. On Sun, Jan 18, 2015, 5:00 PM Brandon Allbery <[hidden email]> wrote:
_______________________________________________ Libraries mailing list [hidden email] http://www.haskell.org/mailman/listinfo/libraries |
In reply to this post by Vincent Hanquez
On 19/01/15 01:05, Vincent Hanquez wrote:
> This is not harmless. This is a security issue by itself, as now > packages get changes transparently given a url, you might have a > different package one day, which trigger hash check failure. or signed > tag verification failure. Correct me if I'm wrong, but editing version bounds on hackage doesn't actually affect the tarball (and its checksum). The modified cabal file is downloaded separately as part of the index. Not saying it doesn't introduce its own problems, but the hash check should continue to pass. Roman _______________________________________________ Libraries mailing list [hidden email] http://www.haskell.org/mailman/listinfo/libraries |
On 18/01/2015 20:23, Roman Cheplyaka wrote: > On 19/01/15 01:05, Vincent Hanquez wrote: >> This is not harmless. This is a security issue by itself, as now >> packages get changes transparently given a url, you might have a >> different package one day, which trigger hash check failure. or signed >> tag verification failure. > Correct me if I'm wrong, but editing version bounds on hackage doesn't > actually affect the tarball (and its checksum). The modified cabal file > is downloaded separately as part of the index. yes, that's right. I meant to say that what you're downloading through cabal get tweaked by cabal, but the end result is pretty much the same > Not saying it doesn't introduce its own problems, but the hash check > should continue to pass. of the tarball yes, not of your compilation tree, and maybe not the resulting binary. -- Vincent _______________________________________________ Libraries mailing list [hidden email] http://www.haskell.org/mailman/listinfo/libraries |
In reply to this post by Kyra-3
On 2015-01-18 at 18:56:42 +0100, kyra wrote:
> It looks old (and even ancient) versions of many packages gets > uploaded to hackage over and over again in ever increasing > amounts. The username of uploader for vast majority of these uploads > is HerbertValerioRiedel. That would be me... Just to clarify: I those aren't re-uploads of existing package versions, but rather meta-data revisions. The actual source-code is *not* touched, (and Hackage is very restricting in what's allowed to be changed in a .cabal file) This is also mentioned on http://hackage.haskell.org/packages/trustees/ > While this is harmless I wonder what idea stands behind this? The idea, as others in this thread have already pointed out, is to add missing lower/upper package bounds to packages so that the Cabal install-plan solver can fulfill its task. Over the last week, I've been analyzing the current situation (motivated by the upcoming GHC 7.10 release which trips up quite a few packages with missing bounds on base/deepseq/time), and trying to come up with some tooling[1] to help the otherwise IMO rather unmanageable job of a Hackage trustee, as well as surgically fixing up the meta-data where I had empirical proof that the .cabal meta-data was incorrect (and thus leading the solver to select a broken install-plan). Here's a very simple buildreport for illustration: https://ghc.haskell.org/~hvr/buildreports/pretty.html The purpose of my meta-data edits were/are (in order of decreasing priority): 1. First and foremost, no currently working/valid install-plan was lost. 2. A simple constraint-less `cabal install pretty` ought to result *either* in a valid install-plan, *or* cabal failing to find any install-plan. 3. An major-version constraint `cabal install 'pretty == a.b.*'` ought to result *either* in a valid install-plan, *or* cabal failing to find any install-plan. 4. A package shall not cause indirect *build*-failures of other packages directly or indirectly depending on this package[3] The build-report linked helps directly with 2. and 3. as it shows that `pretty` failed to `cabal install pretty` (as well as `cabal install 'pretty == 1.1.*'`) for GHC 7.0 and GHC 7.2. Case 4. could be detected by running the buildtool on many Hackage packages depending on `pretty`, and look for failures to build `pretty` during the `cabal install --dep` step (which would result in a "FAIL (deps)" cells). I've encountered a couple of situations, where fixing the meta-data of an important package low in the dependency graph fixed a couple of other packages depending on that package. However, this also means that the more popular a package is, the higher its responsibility is to make sure its build-depend meta-data is correct, as such packages have the potential to instantly break large branches of Hackage: One example for that was the release of deepseq-1.4, which changes the default-implementation of `rnf` which a few packages relied on, but the rather important `text` package was among of those. This affected most of the released ~50 `text` package versions, and since `deepseq` is compatible with all GHC 7.* versions, as soon an install-plan was selected which included `deepseq-1.4` with a non-latest version of `text` the build would fail. And since it's not uncommon to restrict the upper bound of `text`[2], all those packages depending on a non-latest `text` would have broken install-plans (unless `deepseq < 1.4` was selected). I hope this answers your question. Cheers, hvr [1]: https://github.com/hvr/hackage-matrix-builder (WIP example reports can be seen in https://ghc.haskell.org/~hvr/buildreports/0INDEX.html) [2]: http://packdeps.haskellers.com/reverse/text [3]: This can happen due to additional package-version constraints causing other (but broken) configurations to be selected by the solver not detected in case 2. and 3. _______________________________________________ Libraries mailing list [hidden email] http://www.haskell.org/mailman/listinfo/libraries |
In reply to this post by Vincent Hanquez
On 2015-01-19 at 00:05:31 +0100, Vincent Hanquez wrote:
[...] >> While this is harmless I wonder what idea stands behind this? > This is not harmless. This is a security issue by itself, as now > packages get changes transparently given a url, you might have a > different package one day, which trigger hash check failure. or signed > tag verification failure. Just to clarify: the source tarball are *never* modified to specifically make sure hashes don't randomly change (which would trip all sorts of warnings for distribution packagers). In fact, there's also a `cabal get` option if you want to unpack with the original unaltered .cabal file: --pristine Unpack the original pristine tarball, rather than updating the .cabal file with the latest revision from the package archive. > This has also the effect of not changing the bounds in the repository, > so for example, next time you upload a tweak'ed packages, you > effectively revert the change done on hackage only. > This is also done without the consent of the maintainer of a given > package, nor that the maintainer is actually notified when that > happens, or allow to prevent it happening. Fwiw, there's plans for an automated notification feature: https://github.com/haskell/hackage-server/issues/230 We're actively working on refining the system, and finding out where it needs to be improved. For most of my edits I did in fact inform upstream to make sure the broken install-plan issue was known so future uploads could address the issue (or at least add missing bounds to avoid regressing again). > This is pretty big start from the other similar policy for taking over > packages, that insist on a very long period of repeated communication > with the author and then the community. We probably need a broader discussion to clarify what Hackage's goal shall be in terms of (various dimensions of) stability, what policies Hackage trustees shall follow to help achieve these goals, and most importantly, which responsibilities a package author agrees to (and which (s)he delegates to trustees), when (s)he uploads a package to Hackage, which is a shared and vital resource of our ecosystem and IMO ought to be kept in good condition to ensure a good experience for all its users. Right now there's varying degrees of expectations of what Hackage is, with one extreme being that Hackage is simply considered a storage-space for dumping package source-tarballs to (and where .cabal meta-data hygiene has low to no priority), and the other end of the spectrum being the vision of Hackage as a somewhat curated (and yet bleeding-edge) package repository. The latter is not unlike Stackage (or the LTSHaskell idea), but with a different priority of eliminating broken install plans (while retaining the degree of freedom version-ranges for dependencies provide) rather than providing a single vetted cross-section of Hackage. Even with the legitimate existence of Stackage (and possibly LTSHaskell), there's still the need for keeping Hackage in good shape meta-data-wise: 1) There's a significant amount of users which are stuck with an older GHC (Debian Stable currently ships with GHC 7.4.2, Debian Jessie will most likely ship with GHC 7.6.3 later this year; I've seen reports of RHEL6 users being stuck with GHC 7.0.4) Also, the Haskell Platform lags behind one major GHC version for some time till it picks up the latest stable GHC version. It'll be quite some time till we see a HP release featuring GHC 7.10.x after GHC 7.10.1 is released. While I don't consider it paramount for every latest package version to still be compatible with GHC 7.4.2 (or GHC 7.0), I do consider it important that a `cabal install $PKG` picks the latest still compatible package version (if one exists). The more popular $PKG is, the more important I consider this. Just consider a new Haskell user who doesn't know all the issues with Hackage who reads a blog-post (I'm thinking of something like "24 Days of Hackage"), then goes on to `cabal install $PKG` and is confronted with a compile-error (rather than picking an older but working version)... is that the first impression we want new users to have with Haskell & Hackage? 2) Before a new package version can be cherry-picked into Stackage/LTSHaskell, it needs to be on something like Hackage first, also to allow other packages to be updated against the new version (which can result in a cascade of package version updates throughout the dependency-graph). This is IMO one killer feature Hackage has with its version-ranges instead of single-version pinned down package-sets. I think such update scenarios would be quite harder to perform if Stackage didn't have Hackage as the staging/integration area for version-update waves. 3) As an extension of 2), library maintainers also need properly working build-plans for older GHCs for latest-versions of the build-deps they rely on, as otherwise they have a hard time providing and ensuring backward compatibility for their packages during development if e.g. their Travis-job-matrices start falling apart due to suddenly beaking install-plans. The alternative would be to have package versions restrict their support to single GHC major versions and provide bugfixes (and improvements) for older GHCs via backports to previous package-version branches. Personally, I consider it less effort to have the latest package-version support a couple of recent GHC version, and slowly phase out support for older GHCs if it becomes to costly to support those (because e.g. features from newer GHCs are exploited) That results in a "diagonal support corridor" in the build-reports as in e.g. https://ghc.haskell.org/~hvr/buildreports/either.html Ideally, if you can support older GHCs, then the pattern looks like a lower-left support triangle as in https://ghc.haskell.org/~hvr/buildreports/unordered-containers.html This email's already gotten much longer than I intended to and I started digressing... so I'm better stop here... :-) Cheers, hvr _______________________________________________ Libraries mailing list [hidden email] http://www.haskell.org/mailman/listinfo/libraries |
In reply to this post by Eric Mertens
Hello,
On 2015-01-19 at 02:25:56 +0100, Eric Mertens wrote: [...] > I'm sure he'd be happy to leave packages alone for anyone who prefers > their packages remain broken. I wouldn't say I'm "happy" to, but I'll certainly comply with the request. The reason I wouldn't be happy is that the effects of a "broken" package (especially the more popular it becomes) can't be contained easily. The breakage cascades through the build-dep graph (I already mentioned the `text`-example in another reply). In some sense, "no package is an island"... Cheers, hvr _______________________________________________ Libraries mailing list [hidden email] http://www.haskell.org/mailman/listinfo/libraries |
On Mon, Jan 19, 2015 at 7:06 AM, Herbert Valerio Riedel <[hidden email]> wrote:
I wonder if Hackage can be extended to support an out-of-band "broken" flag that can be applied to such packages, and cabal-install then refuse to use those packages (possibly with an option to override). brandon s allbery kf8nh sine nomine associates unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net _______________________________________________ Libraries mailing list [hidden email] http://www.haskell.org/mailman/listinfo/libraries |
I for one am immensely grateful for Herberts awesome work, and appreciate it immensely and support it emphatically. (and I do a bit of similar work in miniature as well as a hackage trustee). I dont think people realize how many breakages/old issues have been resolved via the hard work of the various hackage trustees. hackage trustee's doing their jobs well results in a huge reduction in build problems by end users. On Mon, Jan 19, 2015 at 9:59 AM, Brandon Allbery <[hidden email]> wrote:
_______________________________________________ Libraries mailing list [hidden email] http://www.haskell.org/mailman/listinfo/libraries |
In reply to this post by Herbert Valerio Riedel
I just want to say that I appreciate your work. You're a hero for doing
this very thankless but very beneficial work. On 01/19/2015 12:31 AM, Herbert Valerio Riedel wrote: > On 2015-01-18 at 18:56:42 +0100, kyra wrote: >> It looks old (and even ancient) versions of many packages gets >> uploaded to hackage over and over again in ever increasing >> amounts. The username of uploader for vast majority of these uploads >> is HerbertValerioRiedel. > That would be me... > > Just to clarify: I those aren't re-uploads of existing package versions, > but rather meta-data revisions. The actual source-code is *not* touched, > (and Hackage is very restricting in what's allowed to be changed in a > .cabal file) > > This is also mentioned on http://hackage.haskell.org/packages/trustees/ > >> While this is harmless I wonder what idea stands behind this? > The idea, as others in this thread have already pointed out, is to add > missing lower/upper package bounds to packages so that the Cabal > install-plan solver can fulfill its task. > > Over the last week, I've been analyzing the current situation (motivated > by the upcoming GHC 7.10 release which trips up quite a few packages > with missing bounds on base/deepseq/time), and trying to come up with > some tooling[1] to help the otherwise IMO rather unmanageable job of a > Hackage trustee, as well as surgically fixing up the meta-data where I > had empirical proof that the .cabal meta-data was incorrect (and thus > leading the solver to select a broken install-plan). > > Here's a very simple buildreport for illustration: > > https://ghc.haskell.org/~hvr/buildreports/pretty.html > > The purpose of my meta-data edits were/are (in order of decreasing > priority): > > 1. First and foremost, no currently working/valid install-plan was lost. > > 2. A simple constraint-less `cabal install pretty` ought to result > *either* in a valid install-plan, *or* cabal failing to find any > install-plan. > > 3. An major-version constraint `cabal install 'pretty == a.b.*'` ought > to result *either* in a valid install-plan, *or* cabal failing to find > any install-plan. > > 4. A package shall not cause indirect *build*-failures of other > packages directly or indirectly depending on this package[3] > > The build-report linked helps directly with 2. and 3. as it shows that > `pretty` failed to `cabal install pretty` (as well as `cabal install > 'pretty == 1.1.*'`) for GHC 7.0 and GHC 7.2. Case 4. could be detected > by running the buildtool on many Hackage packages depending on `pretty`, > and look for failures to build `pretty` during the `cabal install --dep` > step (which would result in a "FAIL (deps)" cells). > > > I've encountered a couple of situations, where fixing the meta-data of > an important package low in the dependency graph fixed a couple of other > packages depending on that package. However, this also means that the > more popular a package is, the higher its responsibility is to make sure > its build-depend meta-data is correct, as such packages have the > potential to instantly break large branches of Hackage: > > One example for that was the release of deepseq-1.4, which changes the > default-implementation of `rnf` which a few packages relied on, but the > rather important `text` package was among of those. This affected most > of the released ~50 `text` package versions, and since `deepseq` is > compatible with all GHC 7.* versions, as soon an install-plan was > selected which included `deepseq-1.4` with a non-latest version of > `text` the build would fail. And since it's not uncommon to restrict the > upper bound of `text`[2], all those packages depending on a non-latest > `text` would have broken install-plans (unless `deepseq < 1.4` was > selected). > > I hope this answers your question. > > Cheers, > hvr > > > [1]: https://github.com/hvr/hackage-matrix-builder > > (WIP example reports can be seen in > > https://ghc.haskell.org/~hvr/buildreports/0INDEX.html) > > > [2]: http://packdeps.haskellers.com/reverse/text > > [3]: This can happen due to additional package-version constraints > causing other (but broken) configurations to be selected by the > solver not detected in case 2. and 3. > _______________________________________________ > Libraries mailing list > [hidden email] > http://www.haskell.org/mailman/listinfo/libraries _______________________________________________ Libraries mailing list [hidden email] http://www.haskell.org/mailman/listinfo/libraries |
In reply to this post by Brandon Allbery
We have a deprecation flag on hackage now, it just doesn't do anything at last check. On Mon, Jan 19, 2015 at 9:59 AM, Brandon Allbery <[hidden email]> wrote:
_______________________________________________ Libraries mailing list [hidden email] http://www.haskell.org/mailman/listinfo/libraries |
On Mon, Jan 19, 2015 at 5:07 PM, Edward Kmett <[hidden email]> wrote: We have a deprecation flag on hackage now, it just doesn't do anything at last check. Which is exactly why I specified that it needs to be something that cabal honors by default. As a sysadmin, I'd actually prefer the flag to overwriting packages --- but I find the Haskell community usually goes in other directions than what makes sense to me from the standpoint of stability vs. usability vs. flexibility. And given that the deprecation flag was added as more or less documentation, I don't really expect that a proper disable option will be added.... brandon s allbery kf8nh sine nomine associates unix, openafs, kerberos, infrastructure, xmonad http://sinenomine.net _______________________________________________ Libraries mailing list [hidden email] http://www.haskell.org/mailman/listinfo/libraries |
Free forum by Nabble | Edit this page |