download all of Hackage?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

download all of Hackage?

Richard Eisenberg-2
Hi devs,

Is there an easy way to download (but not compile) all of Hackage? I know of the hackager package, but that's about compiling. I just want a whole big load of Haskell code to play with. I thought I could find a link on Hackage to do this, but failed.

Thanks!
Richard
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: download all of Hackage?

Brandon Allbery
On Mon, Sep 14, 2015 at 10:43 AM, Richard Eisenberg <[hidden email]> wrote:
Is there an easy way to download (but not compile) all of Hackage? I know of the hackager package, but that's about compiling. I just want a whole big load of Haskell code to play with. I thought I could find a link on Hackage to do this, but failed.

There's hackage-mirror, but I note it says it mirrors to S3.

--
brandon s allbery kf8nh                               sine nomine associates
[hidden email]                                  [hidden email]
unix, openafs, kerberos, infrastructure, xmonad        http://sinenomine.net

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: download all of Hackage?

Herbert Valerio Riedel-3
In reply to this post by Richard Eisenberg-2
On 2015-09-14 at 16:43:44 +0200, Richard Eisenberg wrote:
> Is there an easy way to download (but not compile) all of Hackage? I
> know of the hackager package, but that's about compiling. I just want
> a whole big load of Haskell code to play with. I thought I could find
> a link on Hackage to do this, but failed.

It's quite easy, you can iterate through the list of package names and
call 'cabal get' like e.g. (untested, but I've done this already -- you
may need to protect against execution errors)

  for PKG in $(cabal list --simple | awk '{ print $1 }' | uniq); do cabal get $PKG;done

another variant is to construct the URLs based on the output;

you can also get a list of packages in JSON format via

 http://hackage.haskell.org/packages/.json

there's many ways to accomplish what you want...

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: download all of Hackage?

Alan & Kim Zimmerman

On Mon, Sep 14, 2015 at 5:17 PM, Herbert Valerio Riedel <[hidden email]> wrote:
On 2015-09-14 at 16:43:44 +0200, Richard Eisenberg wrote:
> Is there an easy way to download (but not compile) all of Hackage? I
> know of the hackager package, but that's about compiling. I just want
> a whole big load of Haskell code to play with. I thought I could find
> a link on Hackage to do this, but failed.

It's quite easy, you can iterate through the list of package names and
call 'cabal get' like e.g. (untested, but I've done this already -- you
may need to protect against execution errors)

  for PKG in $(cabal list --simple | awk '{ print $1 }' | uniq); do cabal get $PKG;done

another variant is to construct the URLs based on the output;

you can also get a list of packages in JSON format via

 http://hackage.haskell.org/packages/.json

there's many ways to accomplish what you want...

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: download all of Hackage?

Christopher Allen
It's a little out of date, but I've been using that repo I made to do surveys of Haskell code and figure out how frequently things are used.

I could really do with a Haskell-source-code-aware grep though. Being able to specify type, data, etc. would be really nice!

On Mon, Sep 14, 2015 at 10:19 AM, Alan & Kim Zimmerman <[hidden email]> wrote:

On Mon, Sep 14, 2015 at 5:17 PM, Herbert Valerio Riedel <[hidden email]> wrote:
On 2015-09-14 at 16:43:44 +0200, Richard Eisenberg wrote:
> Is there an easy way to download (but not compile) all of Hackage? I
> know of the hackager package, but that's about compiling. I just want
> a whole big load of Haskell code to play with. I thought I could find
> a link on Hackage to do this, but failed.

It's quite easy, you can iterate through the list of package names and
call 'cabal get' like e.g. (untested, but I've done this already -- you
may need to protect against execution errors)

  for PKG in $(cabal list --simple | awk '{ print $1 }' | uniq); do cabal get $PKG;done

another variant is to construct the URLs based on the output;

you can also get a list of packages in JSON format via

 http://hackage.haskell.org/packages/.json

there's many ways to accomplish what you want...

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs




--
Chris Allen
Currently working on http://haskellbook.com

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: download all of Hackage?

Richard Eisenberg-2
In reply to this post by Herbert Valerio Riedel-3
That's perfect -- thanks! It's already humming away.

On Sep 14, 2015, at 11:17 AM, Herbert Valerio Riedel <[hidden email]> wrote:

> for PKG in $(cabal list --simple | awk '{ print $1 }' | uniq); do cabal get $PKG;done

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: download all of Hackage?

John Wiegley-2
In reply to this post by Brandon Allbery
>>>>> Brandon Allbery <[hidden email]> writes:

> There's hackage-mirror, but I note it says it mirrors to S3.

It mirrors into a directory as well:

    hackage-mirror --from="http://hackage.haskell.org" --to="/some/dir"

Further, it can incrementally update very quickly.

John
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: download all of Hackage?

Mike Izbicki
Does anyone know approximately how much disk space all of hackage
takes up when compiled?  And about how long it takes to compile
everything?

On Mon, Sep 14, 2015 at 11:37 AM, John Wiegley <[hidden email]> wrote:

>>>>>> Brandon Allbery <[hidden email]> writes:
>
>> There's hackage-mirror, but I note it says it mirrors to S3.
>
> It mirrors into a directory as well:
>
>     hackage-mirror --from="http://hackage.haskell.org" --to="/some/dir"
>
> Further, it can incrementally update very quickly.
>
> John
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: download all of Hackage?

Alan & Kim Zimmerman
You will probably not be able to compile everything. If you want a compilable subset, you should probably look at stackage

Alan

On Mon, Sep 14, 2015 at 9:57 PM, Mike Izbicki <[hidden email]> wrote:
Does anyone know approximately how much disk space all of hackage
takes up when compiled?  And about how long it takes to compile
everything?

On Mon, Sep 14, 2015 at 11:37 AM, John Wiegley <[hidden email]> wrote:
>>>>>> Brandon Allbery <[hidden email]> writes:
>
>> There's hackage-mirror, but I note it says it mirrors to S3.
>
> It mirrors into a directory as well:
>
>     hackage-mirror --from="http://hackage.haskell.org" --to="/some/dir"
>
> Further, it can incrementally update very quickly.
>
> John
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: download all of Hackage?

Kostiantyn Rybnikov
In reply to this post by Mike Izbicki

Stack tool has an option of a docker image with everything compiled. I think they claim whole image is 8 or 10 GB, please check (and don't forget that's with GHC and other tools inside).

14 вер. 2015 22:57 "Mike Izbicki" <[hidden email]> пише:
Does anyone know approximately how much disk space all of hackage
takes up when compiled?  And about how long it takes to compile
everything?

On Mon, Sep 14, 2015 at 11:37 AM, John Wiegley <[hidden email]> wrote:
>>>>>> Brandon Allbery <[hidden email]> writes:
>
>> There's hackage-mirror, but I note it says it mirrors to S3.
>
> It mirrors into a directory as well:
>
>     hackage-mirror --from="http://hackage.haskell.org" --to="/some/dir"
>
> Further, it can incrementally update very quickly.
>
> John
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: download all of Hackage?

Niklas Hambüchen
In reply to this post by Alan & Kim Zimmerman
Thank you, this is exactly what I needed the other day to do a quick
survey of language extensions.

On 14/09/15 17:19, Alan & Kim Zimmerman wrote:
> You could clone https://github.com/bitemyapp/hackage-packages
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs