Offer to mirror Hackage

[Dan Knapp]

With Hackage down, now seemed like a good time to push this issue
again.  It's such an important site to us that it's really rather a
shame there are no mirrors of it.  I have a personal-and-business
server in a data center in Newark, with a fair chunk of bandwidth,
which I'd like to offer for a permanent mirror.  Is there interest in
this?  Who do I need to talk to for it to happen?

Strategy-wise, I think the best approach is round-robin DNS, since
that's transparent to the end user - everything would still appear at
the URL it's at now, but behind-the-scenes magic would let things keep
working when one or the other site is down.  I haven't personally set
up such a system before but I'm willing to take on the burden of
figuring it out.

So I have a better idea of what I'm signing up for, can anyone tell me
how much disk space and how much bandwidth per month Hackage uses?  I
have a fair chunk of both, as I say, but I'd like to know in advance
to ensure that things go smoothly.

As for what I'd want in return for this, really nothing.  I wouldn't
say no to an unobtrusive mention somewhere on the site, but I'd be
happy just knowing I'd given something back to the Haskell community,
which has given a lot to me.

--
Dan Knapp
"An infallible method of conciliating a tiger is to allow oneself to
be devoured." (Konrad Adenauer)

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Jake McArthur]

I am no decision maker regarding Hackage, but I would like to echo my
support for this offer. Hackage is a vital part of my workflow, and I'm
sure I'm not the only one. Its importance to the Haskell community has
grown quickly and is continuing to do so. Each time it goes down, the
impact is larger than before. We should have a mirror in place for
situations like these.

- Jake

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Ozgur Akgun]

This is a very generous offer. However, I must say I like the following idea more:

http://www.reddit.com/r/haskell/comments/efw38/reminder_hackagehaskellorg_outage_tomorrow_due_to/c17u7nk

On 4 December 2010 16:31, Dan Knapp <[hidden email]> wrote:

With Hackage down, now seemed like a good time to push this issue
again.  It's such an important site to us that it's really rather a
shame there are no mirrors of it.  I have a personal-and-business
server in a data center in Newark, with a fair chunk of bandwidth,
which I'd like to offer for a permanent mirror.  Is there interest in
this?  Who do I need to talk to for it to happen?

Strategy-wise, I think the best approach is round-robin DNS, since
that's transparent to the end user - everything would still appear at
the URL it's at now, but behind-the-scenes magic would let things keep
working when one or the other site is down.  I haven't personally set
up such a system before but I'm willing to take on the burden of
figuring it out.

So I have a better idea of what I'm signing up for, can anyone tell me
how much disk space and how much bandwidth per month Hackage uses?  I
have a fair chunk of both, as I say, but I'd like to know in advance
to ensure that things go smoothly.

As for what I'd want in return for this, really nothing.  I wouldn't
say no to an unobtrusive mention somewhere on the site, but I'd be
happy just knowing I'd given something back to the Haskell community,
which has given a lot to me.

--
Dan Knapp
"An infallible method of conciliating a tiger is to allow oneself to
be devoured." (Konrad Adenauer)

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

--
Ozgur Akgun

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Riad S. Wahby]

Ozgur Akgun <[hidden email]> wrote:
> This is a very generous offer. However, I must say I like the following idea
> more:
>
> http://www.reddit.com/r/haskell/comments/efw38/
> reminder_hackagehaskellorg_outage_tomorrow_due_to/c17u7nk

I'd support this, but I'm strongly in favor of the use of WePay.com over
PayPal for collecting funds. The former's shady history is a matter of
public record.

Alternatively, I'd also be willing to throw some of my own bandwidth and
disk space towards a mirror.

-=rsw

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

[wren ng thornton]

On 12/4/10 11:31 AM, Dan Knapp wrote:
> With Hackage down, now seemed like a good time to push this issue
> again.  It's such an important site to us that it's really rather a
> shame there are no mirrors of it.  I have a personal-and-business
> server in a data center in Newark, with a fair chunk of bandwidth,
> which I'd like to offer for a permanent mirror.
>
> Is there interest in this?

Absolutely.

> Who do I need to talk to for it to happen?

I'd guess that'd be the haskell.org steering committee:

 
http://haskellorg.wordpress.com/2010/11/15/the-haskell-org-committee-has-formed/

--
Live well,
~wren

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

[wren ng thornton]

On 12/4/10 2:21 PM, Riad S. Wahby wrote:
> Ozgur Akgun<[hidden email]>  wrote:
>> This is a very generous offer. However, I must say I like the following idea
>> more:
>>
>> http://www.reddit.com/r/haskell/comments/efw38/
>> reminder_hackagehaskellorg_outage_tomorrow_due_to/c17u7nk

That sounds like a great idea.

The 501(c)(3) I mean. The distributed hosting is nice too, though I'd
like to see the 501(c)(3) formed before donating, just so we can get
official reports and all that. But once that's up, I'm definitely
willing to contribute.

FWIW, I've been on the board of directors for a 501(c)(3), helped write
their bylaws, and know a few people in the business (lawyers, etc). I'm
willing to offer advice, effort, and references whenever the committee
decides to do this.



> I'd support this, but I'm strongly in favor of the use of WePay.com over
> PayPal for collecting funds. The former's shady history is a matter of
> public record.

Semantic Parse Fail: did you mean "the latter" or "strongly opposed to"?


> Alternatively, I'd also be willing to throw some of my own bandwidth and
> disk space towards a mirror.

For a scalable solution I think we should be aiming for both (1) a
distributed "central" server, and (2) mirrors around the globe. The
former gives reliability to the main site, but the latter help for
locality and loadbalancing as well as failover.

--
Live well,
~wren

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Riad S. Wahby]

wren ng thornton <[hidden email]> wrote:
> Semantic Parse Fail: did you mean "the latter" or "strongly opposed to"?

s/former/latter/

:)

-=rsw

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Florian Lengyel]

Why is there even any consideration of some committee if someone wants to mirror the Hackage site? Why not mirror the site?

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Ivan Lazar Miljenovic]

On 5 December 2010 18:41, Florian Lengyel <[hidden email]> wrote:
> Why is there even any consideration of some committee if someone wants to
> mirror the Hackage site? Why not mirror the site?

Presumably to make it an official mirror, and possibly due to the
licenses of some content on there.

--
Ivan Lazar Miljenovic
[hidden email]
IvanMiljenovic.wordpress.com

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

[wren ng thornton]

On 12/4/10 10:34 PM, wren ng thornton wrote:
> FWIW, I've been on the board of directors for a 501(c)(3), helped write
> their bylaws, and know a few people in the business (lawyers, etc). I'm
> willing to offer advice, effort, and references whenever the committee
> decides to do this.

I tried cc-ing my previous to [hidden email] but got

     <[hidden email]>: host haskell.org[78.46.100.180] said:
     550 Unrouteable address (in reply to RCPT TO command)

Did I get the address wrong, or does this have to do with the electrical
downtime?

--
Live well,
~wren

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Ketil Malde-5]

Florian Lengyel <[hidden email]> writes:

> Why is there even any consideration of some committee if someone wants to
> mirror the Hackage site? Why not mirror the site?

+1

  Alright, Mr. Wiseguy," she said, "if you're so clever, you tell us
  what colour it should be."

We can either let Dan set up a mirror, and add it to the haskell.org DNS
(or just let it live at a different address), and have a mirror up in a
couple of hours -- or we can set up 501(c)(3)s (whatever they are),
decide on a payment service, write bylaws, hire lawyers etc.  All in the
name of "not-dealing-with-this-shit"¹?

In my experience, everything is a lot simpler if you can avoid dealing
directly with money.  The current problem is that hackage has sometimes
been unstable, having a mirror would fix or at least alleviate that.

-k

¹  As argued in the cited Reddit thread. Okay, in all fairness, the
"shit" being referred to is the current non-redundant DNS configuration,
not people volunteering to solve technical issues. :-)
--
If I haven't seen further, it is by standing in the footprints of giants

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Brandon S Allbery KF8NH-2]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/5/10 02:41 , Florian Lengyel wrote:
> Why is there even any consideration of some committee if someone wants to
> mirror the Hackage site? Why not mirror the site?

Because it would be nice to have a mirror run by someone (a) accountable (b)
who is unlikely to suddenly disappear due to loss of job, life becoming
hectic, etc.  (Consider that this is pretty much why *.haskell.org has been
unreliable and fixes have been slow in coming; the individual in question is
at Yale, and a good person but kinda snowed under of late.)

- --
brandon s. allbery     [linux,solaris,freebsd,perl]      [hidden email]
system administrator  [openafs,heimdal,too many hats]  [hidden email]
electrical and computer engineering, carnegie mellon university      KF8NH
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkz7wrAACgkQIn7hlCsL25VrvACZAWZq4rYMM8PARZYvyFmnt1qZ
jX4An3fgSSsuFLHR0/HsEB8hEeyj4MCO
=MI9c
-----END PGP SIGNATURE-----

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

[wren ng thornton]

On 12/5/10 11:23 AM, Ketil Malde wrote:

> Florian Lengyel<[hidden email]>  writes:
>
>> Why is there even any consideration of some committee if someone wants to
>> mirror the Hackage site? Why not mirror the site?
>
> +1
>
>    Alright, Mr. Wiseguy," she said, "if you're so clever, you tell us
>    what colour it should be."
>
> We can either let Dan set up a mirror, and add it to the haskell.org DNS
> (or just let it live at a different address), and have a mirror up in a
> couple of hours -- or we can set up 501(c)(3)s (whatever they are),
> decide on a payment service, write bylaws, hire lawyers etc.  All in the
> name of "not-dealing-with-this-shit"¹?

"Or"? There's no need to be exclusive. Fact is that haskell.org has some
money and needs to deal with that, so incorporating is just a matter of
time. But that doesn't preclude folks setting up mirrors or doing
anything non-money related. All one needs to do is convince the DNS
owners to add your IP# to their entry.

501(c)(3) is the legal term for a class of US organizations more
typically known as "non-profit organizations". By incorporating as a
501(c)(3) you get the benefits and responsibilities of being a
government-recognized organization (e.g., rights to use a company name
and prevent others from using it, certain kinds of indemnification
against legal action, ability to act as a legal entity in other ways,
rights to collect money, responsibility to pay taxes,...)

--
Live well,
~wren

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Vincent Hanquez]

  I would really like mirrors too.

But before that happens it would be nice to have signed packages on
Hackage, preventing
a mirror to distribute compromised stuff (intentionally or unintentionally).

--
Vincent

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

[wren ng thornton]

On 12/6/10 2:35 AM, Vincent Hanquez wrote:
> I would really like mirrors too.
>
> But before that happens it would be nice to have signed packages on
> Hackage, preventing
> a mirror to distribute compromised stuff (intentionally or
> unintentionally).

+1.

This should be done during sdist, before uploading, so that maintainers
can be sure that the central mirror gets the right thing too.

--
Live well,
~wren

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Dan Knapp]

Wow, this thread got long.  Good!  I'm hopeful that we can take some
action now. :)

My views on the issues that have been raised -

The Haskell steering committee is a good thing and I fully support
them.  I also support the current maintainer of the site; I don't want
to take over or anything, only to assist.  In fact, I'll go further,
please don't anybody attempt to foist any high-level responsibility on
me.  I'm a bad receptacle for it.  But I do have these technological
resources at my disposal and there's no reason the community shouldn't
benefit from them.

Re incorporation, the person who said that it has to happen was
dead-on.  So the rest of the discussion on that point is moot.  But
it's quite independent of when and how we set up mirroring.

I agree that signed packages are a good idea.  We should move with all
haste to implement them.  But I'm not sure we want to hold up
everything else while we wait for that.  That's also my take on a
peer-peer repository, as I said already.  Can somebody who understands
the technologies typically used for this suggest one, and possibly
also talk to dcoutts directly to make him aware of the discussion and
get his thoughts on how to implement it?  I've found he often makes
points that save me a lot of work. :)

I can certainly conceive of life events that could take my attention,
despite all good intentions, in much the fashion that the current
maintainer's often is.  (That's awkward to say - what's his name,
again?  I know I should know it...  It's not dcoutts, is it?)  So I
want to build something that works well with minimal manual
intervention.

I was of the impression that most of the members of the steering
committee were on this list, which is one reason I posted here.  Is
there some other way I should contact them?

I will talk to dcoutts, and see what the current status of the
distributed-operation code is and figure out how much time I can
devote to helping with that.


--
Dan Knapp
"An infallible method of conciliating a tiger is to allow oneself to
be devoured." (Konrad Adenauer)

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Ketil Malde-5]

Dan Knapp <[hidden email]> writes:

> I agree that signed packages are a good idea.  We should move with all
> haste to implement them.  But I'm not sure we want to hold up
> everything else while we wait for that.  

IMO, mirroring is orthogonal to that, too.

> That's also my take on a peer-peer repository, as I said already.  

Do you mean a two-way sync here?  I think it'd be way easier to just set
up a slave repo using rsync, and let people edit their .cabal/configs.
But I don't really know the internals, perhaps there are implementation
details of cabal or hackage that complicates this?

-k
--
If I haven't seen further, it is by standing in the footprints of giants

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Brandon S Allbery KF8NH-2]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/7/10 08:07 , Ketil Malde wrote:
> Dan Knapp <[hidden email]> writes:
>> I agree that signed packages are a good idea.  We should move with all
>> haste to implement them.  But I'm not sure we want to hold up
>> everything else while we wait for that.  
>
> IMO, mirroring is orthogonal to that, too.

Only if you consider security a minor or non-issue.  I'm tempted to say
anyone who believes that on the modern Internet is at best naïve.  (Although
admittedly security is one of my work foci.)

- --
brandon s. allbery     [linux,solaris,freebsd,perl]      [hidden email]
system administrator  [openafs,heimdal,too many hats]  [hidden email]
electrical and computer engineering, carnegie mellon university      KF8NH
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkz+k3YACgkQIn7hlCsL25W7PACdHUuh5zaPZeBTprMvN+HcLslu
VV0AoJVgmDbBZyZtcX57fGWkGeW2dT/3
=Gqlm
-----END PGP SIGNATURE-----

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Duncan Coutts-4]

On 4 December 2010 16:31, Dan Knapp <[hidden email]> wrote:
> With Hackage down, now seemed like a good time to push this issue
> again.  It's such an important site to us that it's really rather a
> shame there are no mirrors of it.  I have a personal-and-business
> server in a data center in Newark, with a fair chunk of bandwidth,
> which I'd like to offer for a permanent mirror.  Is there interest in
> this?  Who do I need to talk to for it to happen?

At the recent hackathon we were working on hackage mirroring.

By this we do not mean just using rsync to sync the current
combination of filestore + cgi programs that make up the current
hackage implementation. We want to make it easy to set up dumb or
smart package archives and to do nearly-live mirroring.

We have a pototype hackage-mirror client that can poll two servers and
copy packages from one instance to the other. This assumes the target
is a smart mirror (e.g. an instance of the new hackage-server impl).
We also need to be able to target local "dumb" mirrors that are just
passive collections of files.

> Strategy-wise, I think the best approach is round-robin DNS, since
> that's transparent to the end user - everything would still appear at
> the URL it's at now, but behind-the-scenes magic would let things keep
> working when one or the other site is down.  I haven't personally set
> up such a system before but I'm willing to take on the burden of
> figuring it out.

This is a somewhat orthogonal issue since I think you're talking about
multiple master smart servers that can accept uploads.

Duncan

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

[Ketil Malde-5]

Brandon S Allbery KF8NH <[hidden email]> writes:

>> IMO, mirroring is orthogonal to that, too.

> Only if you consider security a minor or non-issue.  

What I mean is that you can mirror a repository regardless of whether
packages are signed or not.

> I'm tempted to say anyone who believes that on the modern Internet is
> at best naïve.

It's not obvious to me that adding a mirror makes the infrastructure
more more insecure.  Any particular concerns?  (I hope I qualify as
naïve here :-)

-k
--
If I haven't seen further, it is by standing in the footprints of giants

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe