Hackage is flooded with old package versions reuploads

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
32 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Greg Weber

On Mon, Jan 19, 2015 at 2:51 AM, Herbert Valerio Riedel <[hidden email]> wrote:
Just to clarify: the source tarball are *never* modified to specifically
make sure hashes don't randomly change (which would trip all sorts of
warnings for distribution packagers). In fact, there's also a
`cabal get` option if you want to unpack with the original unaltered .cabal
file:

    --pristine    Unpack the original pristine tarball,
                  rather than updating the .cabal file
                  with the latest revision from the
                  package archive.

Why is this option needed if the tarball is never modified?
This documentation states that cabal modifies the cabal file on the local system.

We all appreciate Herbert's efforts. The problem is the tools he is working with are limited.
The local cabal should operate the same way as hackage: always keeping the meta-data separate.

Please correct me if I am mis-representing the situation.

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Mikhail Glushenkov-2
Hi,

On 20 January 2015 at 04:03, Greg Weber <[hidden email]> wrote:
>>     --pristine    Unpack the original pristine tarball,
>>                   rather than updating the .cabal file
>>                   with the latest revision from the
>>                   package archive.
>
>
> Why is this option needed if the tarball is never modified?
> This documentation states that cabal modifies the cabal file on the local
> system.

By default, `cabal unpack` replaces the .cabal file in the unpacked
tarball with the one from the index. With '--pristine' it doesn't do
that.
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Duncan Coutts-4
In reply to this post by Kyra-3
On Sun, 2015-01-18 at 20:56 +0300, kyra wrote:
> Hi, guys,
>
> It looks old (and even ancient) versions of many packages gets uploaded
> to hackage over and over again in ever increasing amounts. The username
> of uploader for vast majority of these uploads is HerbertValerioRiedel.

BTW, where do you see these changes? I thought we had fixed the "latest
uploads" page and rss to only show real uploads, and not edits to
metadata.

Not that we want to hide the edits, they need to be visible, but we
shouldn't be confusing them with new package uploads in the UI anywhere.

Duncan

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Duncan Coutts-4
In reply to this post by Vincent Hanquez
On Sun, 2015-01-18 at 15:05 -0800, Vincent Hanquez wrote:
> On 18/01/2015 09:56, kyra wrote:
> > Hi, guys,
> >
> > It looks old (and even ancient) versions of many packages gets
> > uploaded to hackage over and over again in ever increasing amounts.
> > The username of uploader for vast majority of these uploads is
> > HerbertValerioRiedel.
> >
> > While this is harmless I wonder what idea stands behind this?

> This is not harmless. This is a security issue by itself, as now
> packages get changes transparently given a url, you might have a
> different package one day, which trigger hash check failure. or signed
> tag verification failure.

Note that hackage never changes the content of package tarballs. The
checksums on those are stable. Guaranteed.

> This has also the effect of not changing the bounds in the repository,
> so for example, next time you upload a tweak'ed packages, you
> effectively revert the change done on hackage only.

Communicating changes upstream is certainly something we need to work on
to be able to use this as widely as it'd be helpful.

Up until recently we've only used the metadata editing feature with core
packages (or the maintainers themselves have done it). Recently Herbert
has been going a bit wider and if we are now running into issues of
communication with maintainers then I think this says that now is the
time to address that properly.

So that includes:
      * this discussion
      * wider communication with maintainers of just what is and is not
        possible (since we're actually deliberately rather conservative)
      * a proper notification and opt-in/opt-out system for maintainers
        to avail themselves of the helpful service that the trustees can
        provide.

Duncan

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Kyra-3
In reply to this post by Duncan Coutts-4
On 20.01.2015 14:47, Duncan Coutts wrote:

> On Sun, 2015-01-18 at 20:56 +0300, kyra wrote:
>> Hi, guys,
>>
>> It looks old (and even ancient) versions of many packages gets uploaded
>> to hackage over and over again in ever increasing amounts. The username
>> of uploader for vast majority of these uploads is HerbertValerioRiedel.
> BTW, where do you see these changes? I thought we had fixed the "latest
> uploads" page and rss to only show real uploads, and not edits to
> metadata.
>
> Not that we want to hide the edits, they need to be visible, but we
> shouldn't be confusing them with new package uploads in the UI anywhere.
>
> Duncan
>
> .
>
I've wrote a small program producing old-hackage-log-like-file from
hackage index file ("Recent additions" page is too short).

Regads,
Kyra

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Duncan Coutts-4
In reply to this post by Vincent Hanquez
On Sun, 2015-01-18 at 16:57 -0800, Vincent Hanquez wrote:

> On 18/01/2015 15:51, David Feuer wrote:
> >
> > It would be best to be sure to make the maintainer (if there is one)
> > aware of such changes. That said, not every package has a responsive
> > maintainer, and *someone* has to do this work, and do it promptly. A
> > signed hash failure does not introduce a security hole, unless you
> > count a sort of semi-manual, avoidable denial of service.
> >
> Not sure how you got "security hole" from what I said, but a failing
> hash or signature, means that the build system breaks while cabal
> install stuff and that I have to manually inspect what the change is. If
> you can't pin down a special tarball when doing a download (i.e. it can
> changes under your feet, one day to the other), then it's an issue.
>
> Lots of people would be *horrified* to download some
> {c,c++,python,ruby,...} library-a.b.c.tar.gz and found anything changed
> inside without changing the exact name for it.

Indeed they would be horrified and quite rightly so.

Fear not, we're not completely nuts. We have been working with packaging
with distros and cabal/hackage for some years now :-)

Hackage has always followed the rule that package tarballs never change.
Distros must be able to rely on this. I used to work on gentoo packaging
and that was one of their big issues. Some annoying upstream
distributors would have a policy of changing their tarballs and that
would just force all downstream distributors to make their own snapshots
and mirrors. All very annoying. So, no, that's why hackage has always
and will always provide immutable package tarballs.

The metadata editing works by keeping the metadata separate from the
tarballs.

As for security, Austin and I are currently working on an improvement
for the IHG. We're following the approach of server-based index signing,
which will also rely on those stable tarball checksums.

Duncan

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Duncan Coutts-4
In reply to this post by Brandon Allbery
On Mon, 2015-01-19 at 09:59 -0500, Brandon Allbery wrote:
> On Mon, Jan 19, 2015 at 7:06 AM, Herbert Valerio Riedel <[hidden email]> wrote:
>
> > The reason I wouldn't be happy is that the effects of a "broken" package
> > (especially the more popular it becomes) can't be contained easily. The
> >
>
> I wonder if Hackage can be extended to support an out-of-band "broken" flag
> that can be applied to such packages, and cabal-install then refuse to use
> those packages (possibly with an option to override).

This can be achieved by editing the .cabal file, and Herbert has done so
in at least one case. It's just a matter of making the constraints
impossible, e.g. base > 1 && < 1. It could possibly be done more
obviously or directly, e.g. adding a dep on something impossible (though
note that we don't currently allow adding deps).

Deprecation is orthogonal I'd say. People may need to rely on deprecated
packages while they migrate away.

Duncan

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Duncan Coutts-4
In reply to this post by Vincent Hanquez
On Sun, 2015-01-18 at 21:31 -0800, Vincent Hanquez wrote:
> On 18/01/2015 20:23, Roman Cheplyaka wrote:
> > On 19/01/15 01:05, Vincent Hanquez wrote:
> >> This is not harmless. This is a security issue by itself, as now
> >> packages get changes transparently given a url, you might have a
> >> different package one day, which trigger hash check failure. or signed
> >> tag verification failure.
> > Correct me if I'm wrong, but editing version bounds on hackage doesn't
> > actually affect the tarball (and its checksum). The modified cabal file
> > is downloaded separately as part of the index.

> yes, that's right. I meant to say that what you're downloading through
> cabal get
> tweaked by cabal, but the end result is pretty much the same

> > Not saying it doesn't introduce its own problems, but the hash check
> > should continue to pass.
> of the tarball yes, not of your compilation tree, and maybe not the
> resulting binary.

So we have thought about it a bit when designing this feature and we
don't think that is correct.

If you freeze your solution (e.g. with cabal freeze) then changes to the
metadata should have no effect on compilation. We're pretty restrictive
about what changes are allowed and we don't think there's anything that
can cause changes except for the versions of deps picked (and hence if
you freeze then nothing).

Apart from version constraints we pretty much only allow tweaks to
metadata like description, maintainer email etc.

If you don't freeze then currently you have no guarantee that you'll not
get different solutions each time you "cabal update" to a new hackage
snapshot (or indeed installing things can change what solutions you get
since we prefer already installed instances).

One caveat is that it's possible to accidentally constrain deps and make
an existing working solution impossible. This could make a frozen
solution no longer build, which is annoying but doesn't make a broken
binary. If this proves to be a real issue then we can deal with it
(simply ignore constraints for frozen solutions).

Duncan

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Herbert Valerio Riedel
In reply to this post by Duncan Coutts-4
On 2015-01-20 at 17:43:24 +0100, Duncan Coutts wrote:

[...]

> This can be achieved by editing the .cabal file, and Herbert has done so
> in at least one case. It's just a matter of making the constraints
> impossible, e.g. base > 1 && < 1. It could possibly be done more
> obviously or directly, e.g. adding a dep on something impossible (though
> note that we don't currently allow adding deps).

Fwiw, I've converged to using the following construct to blacklist a
package version:

  library
    -- [reason why this package version had to be blacklisted]
    build-depends: base<0
   
    [...rest of content...]


IMHO this is somewhat obvious than any other unsatisfiable constructs

Cheers,
  hvr
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Magnus Therning
On Tue, Jan 20, 2015 at 06:22:47PM +0100, Herbert Valerio Riedel wrote:

> On 2015-01-20 at 17:43:24 +0100, Duncan Coutts wrote:
>
> [...]
>
> > This can be achieved by editing the .cabal file, and Herbert has done so
> > in at least one case. It's just a matter of making the constraints
> > impossible, e.g. base > 1 && < 1. It could possibly be done more
> > obviously or directly, e.g. adding a dep on something impossible (though
> > note that we don't currently allow adding deps).
>
> Fwiw, I've converged to using the following construct to blacklist a
> package version:
>
>   library
>     -- [reason why this package version had to be blacklisted]
>     build-depends: base<0
>    
>     [...rest of content...]
>
>
> IMHO this is somewhat obvious than any other unsatisfiable constructs
Indeed, that is a much better way to signal
blacklisting/deprecation/etc.  I first found the
unsatisfiable-dependencies solution in the random package.  It was
only due to being able to find a bug mentioning that the version in
question should be deprecated that I realised it was done on purpose.

/M

--
Magnus Therning                      OpenPGP: 0xAB4DFBA4
email: [hidden email]   jabber: [hidden email]
twitter: magthe               http://therning.org/magnus

Code as if whoever maintains your program is a violent psychopath who knows
where you live.
     -- Anonymous

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

attachment0 (188 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Kyra-3
In reply to this post by Kyra-3
On 20.01.2015 16:26, kyra wrote:
> I've wrote a small program producing old-hackage-log-like-file from
> hackage index file ("Recent additions" page is too short).
s/wrote/written

Sorry
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Hackage is flooded with old package versions reuploads

Herbert Valerio Riedel
In reply to this post by Vincent Hanquez
Hello Vincent,

On 2015-01-19 at 01:39:09 +0100, Vincent Hanquez wrote:

[...]

>> To my knowledge, the few cases where Herbert has actively done a
>> patch to the .cabal file like this without author communication is
>> because the package is in very very widespread use and the author
>> has been incommunicado for many months. As I recall, Max Bolingbroke
>> has a some packages that fit this bill.

> A simple counter example, that I noticed after the fact [1]

That was more of an oversight of mine as I got blocked on entropy being
broken[2] which transitively broke several of your packages for GHC
7.0/7.2 users (and this got fixed yesterday).  I usually did inform
upstreams in the cases I had to edit the meta-data (sometimes before,
and sometimes after the fact).

However, now that entropy got fixed and I got a clearer picture of which
failures have to be fixed in your packages, I've filed a couple of
tickets[3] against your packages to record what actions are needed to
restore some install-plans.

Moreover, as already stated elsewhere in this thread, an email
notification system is in the works to automate informing upstreams to
some extent.[4]

Does this procedure meet your expectations better?

[...]

Cheers,
  hvr

 [2] https://github.com/TomMD/entropy/issues/23#issuecomment-70705615

 [3]: https://github.com/vincenthz/hs-tls/issues/99
      https://github.com/vincenthz/hs-certificate/issues/42
      https://github.com/vincenthz/hs-cprng-aes/issues/10
      https://github.com/vincenthz/hs-crypto-numbers/issues/16
      https://github.com/vincenthz/hs-crypto-pubkey/issues/19
      https://github.com/vincenthz/hs-cryptohash/issues/33
      https://github.com/vincenthz/hs-hourglass/issues/15

 [4]  https://github.com/haskell/hackage-server/issues/230
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
12