Local mirrors of hackage

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Local mirrors of hackage

Victor Miller
I have a computer that I work on which isn't directly connected to the
internet, that I want to use Haskell on.  In order to do that I want a
local version of hackage.  I was previously able to do this using
mirror_hackage.py, but since hackage 2 came around this no longer
works.  Doing this would involve running something on a computer that
*is* attached to the internet, and then transferring a bunch of files
over to the other computer.  Can someone point me to a recipe for
doing this?

Victor
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Local mirrors of hackage

Matthew Pickering
Hello Victor,

It is easily possible with a small shell script.

Here is another discussion about the same topic -
https://mail.haskell.org/pipermail/ghc-devs/2015-September/009958.html

Matt

On Wed, Feb 24, 2016 at 4:23 PM, Victor Miller <[hidden email]> wrote:

> I have a computer that I work on which isn't directly connected to the
> internet, that I want to use Haskell on.  In order to do that I want a
> local version of hackage.  I was previously able to do this using
> mirror_hackage.py, but since hackage 2 came around this no longer
> works.  Doing this would involve running something on a computer that
> *is* attached to the internet, and then transferring a bunch of files
> over to the other computer.  Can someone point me to a recipe for
> doing this?
>
> Victor
> _______________________________________________
> Haskell-Cafe mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Local mirrors of hackage

Roman Cheplyaka-2
In reply to this post by Victor Miller
On 02/24/2016 06:23 PM, Victor Miller wrote:
> I have a computer that I work on which isn't directly connected to the
> internet, that I want to use Haskell on.  In order to do that I want a
> local version of hackage.  I was previously able to do this using
> mirror_hackage.py, but since hackage 2 came around this no longer
> works.  Doing this would involve running something on a computer that
> *is* attached to the internet, and then transferring a bunch of files
> over to the other computer.  Can someone point me to a recipe for
> doing this?

I don't know how exactly you want to use Haskell on your computer, but
if you know in advance which packages you'll need, you can download only
those packages (and their versions) and place them into cabal or stack
cache directory. That will be much more efficient than mirroring whole
of hackage.

Roman

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Local mirrors of hackage

John Wiegley-2
In reply to this post by Victor Miller
>>>>> Victor Miller <[hidden email]> writes:

> I have a computer that I work on which isn't directly connected to the
> internet, that I want to use Haskell on. In order to do that I want a local
> version of hackage. I was previously able to do this using
> mirror_hackage.py, but since hackage 2 came around this no longer works.
> Doing this would involve running something on a computer that *is* attached
> to the internet, and then transferring a bunch of files over to the other
> computer. Can someone point me to a recipe for doing this?

I wrote https://github.com/jwiegley/hackage-mirror to do just this, with
relatively high performance for incremental updates.

Also, if you're ever a Nix user, you can modify the core Haskell builder to
reference whatever directory you mirror to first, before reaching out to the
Internet for tarballs. I've been using this effectively for several years now.

--
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Local mirrors of hackage

Victor Miller
John, Thanks very much.

Victor

On Wed, Feb 24, 2016 at 1:16 PM, John Wiegley <[hidden email]> wrote:

>>>>>> Victor Miller <[hidden email]> writes:
>
>> I have a computer that I work on which isn't directly connected to the
>> internet, that I want to use Haskell on. In order to do that I want a local
>> version of hackage. I was previously able to do this using
>> mirror_hackage.py, but since hackage 2 came around this no longer works.
>> Doing this would involve running something on a computer that *is* attached
>> to the internet, and then transferring a bunch of files over to the other
>> computer. Can someone point me to a recipe for doing this?
>
> I wrote https://github.com/jwiegley/hackage-mirror to do just this, with
> relatively high performance for incremental updates.
>
> Also, if you're ever a Nix user, you can modify the core Haskell builder to
> reference whatever directory you mirror to first, before reaching out to the
> Internet for tarballs. I've been using this effectively for several years now.
>
> --
> John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
> http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Local mirrors of hackage

Vo Minh Thu
Here is a script that I use. The whole mirror can also be downloaded
through rsync if that helps.

https://github.com/noteed/nginx-hackage

2016-02-24 20:37 GMT+01:00 Victor Miller <[hidden email]>:

> John, Thanks very much.
>
> Victor
>
> On Wed, Feb 24, 2016 at 1:16 PM, John Wiegley <[hidden email]> wrote:
>>>>>>> Victor Miller <[hidden email]> writes:
>>
>>> I have a computer that I work on which isn't directly connected to the
>>> internet, that I want to use Haskell on. In order to do that I want a local
>>> version of hackage. I was previously able to do this using
>>> mirror_hackage.py, but since hackage 2 came around this no longer works.
>>> Doing this would involve running something on a computer that *is* attached
>>> to the internet, and then transferring a bunch of files over to the other
>>> computer. Can someone point me to a recipe for doing this?
>>
>> I wrote https://github.com/jwiegley/hackage-mirror to do just this, with
>> relatively high performance for incremental updates.
>>
>> Also, if you're ever a Nix user, you can modify the core Haskell builder to
>> reference whatever directory you mirror to first, before reaching out to the
>> Internet for tarballs. I've been using this effectively for several years now.
>>
>> --
>> John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
>> http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2
> _______________________________________________
> Haskell-Cafe mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Local mirrors of hackage

Victor Miller
In reply to this post by John Wiegley-2
John, Thanks very much for hackage-mirror.  I'm running it now and it appears to be working as advertised.  Thanks to everyone else who responded.

Victor

On Wed, Feb 24, 2016 at 1:16 PM, John Wiegley <[hidden email]> wrote:
>>>>> Victor Miller <[hidden email]> writes:

> I have a computer that I work on which isn't directly connected to the
> internet, that I want to use Haskell on. In order to do that I want a local
> version of hackage. I was previously able to do this using
> mirror_hackage.py, but since hackage 2 came around this no longer works.
> Doing this would involve running something on a computer that *is* attached
> to the internet, and then transferring a bunch of files over to the other
> computer. Can someone point me to a recipe for doing this?

I wrote https://github.com/jwiegley/hackage-mirror to do just this, with
relatively high performance for incremental updates.

Also, if you're ever a Nix user, you can modify the core Haskell builder to
reference whatever directory you mirror to first, before reaching out to the
Internet for tarballs. I've been using this effectively for several years now.

--
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2


_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Local mirrors of hackage

John Wiegley-2
>>>>> Victor Miller <[hidden email]> writes:

> John, Thanks very much for hackage-mirror. I'm running it now and it appears
> to be working as advertised. Thanks to everyone else who responded.

I committed a fix very recently to the 'pipes' branch that you may need, since
there is one package on Hackage that is missing and thus cannot be downloaded.

--
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Local mirrors of hackage

Victor Miller
Hi John, I just got around to installing the downloaded hackage mirror, and I noticed that its structure is different than the one that I downloaded last June -- in the old one there are files

00-index.cache
00-index.tar
and a bunch of subdirectories, one per package. Within each of those is a version subdirectory which contains a cabal file and a tar.gz file of the package files.

In the new one there are files
00-checksums.bak
00-checksums.dat
00-index.tar.gz

and a directory called package

That directory contains directories called package-name-version
and a corresponding file called package-name-version.tar.gz

one for each package and version

My version of cabal 1.22 doesn't understand that.  Do I need a new version of cabal, or do I have to process the files somehow?

Victor


On Wed, Feb 24, 2016 at 4:45 PM, John Wiegley <[hidden email]> wrote:
>>>>> Victor Miller <[hidden email]> writes:

> John, Thanks very much for hackage-mirror. I'm running it now and it appears
> to be working as advertised. Thanks to everyone else who responded.

I committed a fix very recently to the 'pipes' branch that you may need, since
there is one package on Hackage that is missing and thus cannot be downloaded.

--
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2


_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Local mirrors of hackage

Victor Miller
Aha, I figured out the problem.  I needed to add the lines
remote-repo: myname:file://path-to-hackage
remote-repo-cache: cache-file

to my ~/.cabal/config

Now it works fine.  I wish that the instructions were a little less obscure.

Victor

On Thu, Mar 3, 2016 at 12:26 PM, Victor Miller <[hidden email]> wrote:
Hi John, I just got around to installing the downloaded hackage mirror, and I noticed that its structure is different than the one that I downloaded last June -- in the old one there are files

00-index.cache
00-index.tar
and a bunch of subdirectories, one per package. Within each of those is a version subdirectory which contains a cabal file and a tar.gz file of the package files.

In the new one there are files
00-checksums.bak
00-checksums.dat
00-index.tar.gz

and a directory called package

That directory contains directories called package-name-version
and a corresponding file called package-name-version.tar.gz

one for each package and version

My version of cabal 1.22 doesn't understand that.  Do I need a new version of cabal, or do I have to process the files somehow?

Victor


On Wed, Feb 24, 2016 at 4:45 PM, John Wiegley <[hidden email]> wrote:
>>>>> Victor Miller <[hidden email]> writes:

> John, Thanks very much for hackage-mirror. I'm running it now and it appears
> to be working as advertised. Thanks to everyone else who responded.

I committed a fix very recently to the 'pipes' branch that you may need, since
there is one package on Hackage that is missing and thus cannot be downloaded.

--
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2



_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe