ANN: Overlay Hackage Package Index for GHC HEAD

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

ANN: Overlay Hackage Package Index for GHC HEAD

Herbert Valerio Riedel-3
Hi GHC devs,

A long-standing common problem when testing/dogfooding GHC HEAD is that
at some point during the development cycle more and more packages from
Hackage will run into build failures.

Obviously, constantly nagging upstream maintainers to release quickfixes
for unreleased GHC HEAD snapshots which will likely break again a few
weeks later (as things are generally in flux until freeze comes into
effect) does not scale and only introduces a latency/coordination
bottleneck, and on top of it ultimately results in spamming the primary
Hackage Package index with releases (which has non-negligible negative
impact/costs of its own on the Hackage infrastructure, and thus ought to
be minimised).

OTOH, we need the ability to easily test, debug, profile, and prototype
changes to GHC HEAD while things are still in motion, and case in point,
if you try to e.g. build `pandoc` with GHC HEAD today, you'll currently
run into a dozen or so of packages not building with GHC HEAD.

To this end, I've finally found time to work on a side-project (related
to matrix.hackage.haskell.org) which implements a scheme tailored to
`cabal new-build`, which is inspired by how Eta copes with a very
related issue (they rely on it for stable versions of the compiler);
i.e., they maintain a set of patches at

  https://github.com/typelead/eta-hackage/tree/master/patches

which fix up existing Hackage packages to work with the Eta compiler.


And this gave me the idea to use a similar scheme for GHC HEAD:

  https://github.com/hvr/head.hackage/tree/master/patches

This folder already contains several of patches (which mostly originate
from Ryan, Ben and myself) to packages which I needed to patch in order
to build popular Hackage packages & tools.

The main difference is how those patches are applied; Eta uses a
modified `cabal` which they renamed to `etlas` which is checks
availability of .patch & .cabal files in the GitHub repo linked above;

Whereas for GHC HEAD with `cabal new-build` a different scheme makes
more sense: we simply generate an add-on Hackage repo, and use the
existing `cabal` facilities (e.g. multi-repo support or the nix-style
package store which makes sure that unofficially patched packages don't
contaminate "normal" install-plans, etc.) to opt into the opt-in Hackage
repo containing fixed up packages.



I've tried to describe how to use the HEAD.hackage add-on repo in the
README at

  https://github.com/hvr/head.hackage#how-to-use


And finally, here's a practical example of how you can use it to build
e.g. the `pandoc` executable with GHC HEAD (can easily be adapted to
build your project of choice; please refer to

  http://cabal.readthedocs.io/en/latest/nix-local-build-overview.html
 
to learn more about how to describe your project via  `cabal.project`
files):


0.) This assumes you have a recent cabal 2.1 snapshot built from Git

1.) create & `cd` into a new work-folder

2.) invoke `head.hackage.sh update` to update the HEAD.hackage index cache

3.) invoke `head.hackage.sh init` to create an initial `cabal.project`
    configuration which locally activates the HEAD.hackage overlay repo

4.) If needed, edit the cabal.project file and change where GHC
    HEAD can be found (the script currently assumes GHC HEAD is
    installed from my Ubuntu PPA), e.g.
   
      with-compiler: /home/hvr/src/ghc-dev/inplace/bin/ghc-stage2

    or you can add something like `optional-packages: deps/*/*.cabal`
    to have cabal pick up package source-trees unpacked in the deps/
    folder, or you can inject ghc-options, relax upper bounds via
    `allow-newer: *:base` etc (please refer to the cabal user guide)
   
5.) Create a `dummy.cabal` file (in future we will have `cabal
    new-install` or other facilities, but for now we use this
    workaround):


--8<---------------cut here---------------start------------->8---
name:                dummy
version:             0
build-type:          Simple
cabal-version:       >=2.0
 
library
  default-language:    Haskell2010

  -- library components you want cabal to solve & build for
  -- and become accessible via .ghc.environment files and/or
  -- `cabal new-repl`
  build-depends: base, lens
 
  -- executable components you want cabal to build
  build-tool-depends: pandoc:pandoc
 
--8<---------------cut here---------------end--------------->8---

 6.) invoke `cabal new-build`

 7.) If everything works, you'll find the `pandoc:pandoc` executable
     somewhere in your ~/.cabal/store/ghc-8.3.*/ folder
     (you can use http://hackage.haskell.org/package/cabal-plan
      to conveniently list the location via `cabal-plan list-bins`)

 8.) As for libraries, you can either use `cabal new-repl`
     or you can leverage GHC's package environment files:

     `cabal new-build` will have generated a file like

       .ghc.environment.x86_64-linux-8.3.20170913

     which brings into scope all transitive dependencies of
     `build-depends: base, lens`

     Now all you need to do is simply call

       ghc-stage2 --make MyTestProg.hs

     to compile a program against those libs, or start up GHCi via

       ghc-stage2 --interactive

     and you'll be thrown into that package environment.



I hope you find this useful

Cheers,
  Herbert
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Overlay Hackage Package Index for GHC HEAD

Matthew Pickering
Something like this is definitely useful for testing.

When building GHC HEAD I override the ghcHEAD derivation on nixpkgs to
the right commit I want to use and then can similarly specify which
patches and versions of packages to use by modifying the
'configuration-ghc-head.nix' file.
This is quite a bit more flexible than just patch files as I can point
to specific commits in git repos etc.

This repo will definitely be useful for me with this workflow as well.

Thanks,

Matt

On Mon, Sep 18, 2017 at 1:04 PM, Herbert Valerio Riedel
<[hidden email]> wrote:

> Hi GHC devs,
>
> A long-standing common problem when testing/dogfooding GHC HEAD is that
> at some point during the development cycle more and more packages from
> Hackage will run into build failures.
>
> Obviously, constantly nagging upstream maintainers to release quickfixes
> for unreleased GHC HEAD snapshots which will likely break again a few
> weeks later (as things are generally in flux until freeze comes into
> effect) does not scale and only introduces a latency/coordination
> bottleneck, and on top of it ultimately results in spamming the primary
> Hackage Package index with releases (which has non-negligible negative
> impact/costs of its own on the Hackage infrastructure, and thus ought to
> be minimised).
>
> OTOH, we need the ability to easily test, debug, profile, and prototype
> changes to GHC HEAD while things are still in motion, and case in point,
> if you try to e.g. build `pandoc` with GHC HEAD today, you'll currently
> run into a dozen or so of packages not building with GHC HEAD.
>
> To this end, I've finally found time to work on a side-project (related
> to matrix.hackage.haskell.org) which implements a scheme tailored to
> `cabal new-build`, which is inspired by how Eta copes with a very
> related issue (they rely on it for stable versions of the compiler);
> i.e., they maintain a set of patches at
>
>   https://github.com/typelead/eta-hackage/tree/master/patches
>
> which fix up existing Hackage packages to work with the Eta compiler.
>
>
> And this gave me the idea to use a similar scheme for GHC HEAD:
>
>   https://github.com/hvr/head.hackage/tree/master/patches
>
> This folder already contains several of patches (which mostly originate
> from Ryan, Ben and myself) to packages which I needed to patch in order
> to build popular Hackage packages & tools.
>
> The main difference is how those patches are applied; Eta uses a
> modified `cabal` which they renamed to `etlas` which is checks
> availability of .patch & .cabal files in the GitHub repo linked above;
>
> Whereas for GHC HEAD with `cabal new-build` a different scheme makes
> more sense: we simply generate an add-on Hackage repo, and use the
> existing `cabal` facilities (e.g. multi-repo support or the nix-style
> package store which makes sure that unofficially patched packages don't
> contaminate "normal" install-plans, etc.) to opt into the opt-in Hackage
> repo containing fixed up packages.
>
>
>
> I've tried to describe how to use the HEAD.hackage add-on repo in the
> README at
>
>   https://github.com/hvr/head.hackage#how-to-use
>
>
> And finally, here's a practical example of how you can use it to build
> e.g. the `pandoc` executable with GHC HEAD (can easily be adapted to
> build your project of choice; please refer to
>
>   http://cabal.readthedocs.io/en/latest/nix-local-build-overview.html
>
> to learn more about how to describe your project via  `cabal.project`
> files):
>
>
> 0.) This assumes you have a recent cabal 2.1 snapshot built from Git
>
> 1.) create & `cd` into a new work-folder
>
> 2.) invoke `head.hackage.sh update` to update the HEAD.hackage index cache
>
> 3.) invoke `head.hackage.sh init` to create an initial `cabal.project`
>     configuration which locally activates the HEAD.hackage overlay repo
>
> 4.) If needed, edit the cabal.project file and change where GHC
>     HEAD can be found (the script currently assumes GHC HEAD is
>     installed from my Ubuntu PPA), e.g.
>
>       with-compiler: /home/hvr/src/ghc-dev/inplace/bin/ghc-stage2
>
>     or you can add something like `optional-packages: deps/*/*.cabal`
>     to have cabal pick up package source-trees unpacked in the deps/
>     folder, or you can inject ghc-options, relax upper bounds via
>     `allow-newer: *:base` etc (please refer to the cabal user guide)
>
> 5.) Create a `dummy.cabal` file (in future we will have `cabal
>     new-install` or other facilities, but for now we use this
>     workaround):
>
>
> --8<---------------cut here---------------start------------->8---
> name:                dummy
> version:             0
> build-type:          Simple
> cabal-version:       >=2.0
>
> library
>   default-language:    Haskell2010
>
>   -- library components you want cabal to solve & build for
>   -- and become accessible via .ghc.environment files and/or
>   -- `cabal new-repl`
>   build-depends: base, lens
>
>   -- executable components you want cabal to build
>   build-tool-depends: pandoc:pandoc
>
> --8<---------------cut here---------------end--------------->8---
>
>  6.) invoke `cabal new-build`
>
>  7.) If everything works, you'll find the `pandoc:pandoc` executable
>      somewhere in your ~/.cabal/store/ghc-8.3.*/ folder
>      (you can use http://hackage.haskell.org/package/cabal-plan
>       to conveniently list the location via `cabal-plan list-bins`)
>
>  8.) As for libraries, you can either use `cabal new-repl`
>      or you can leverage GHC's package environment files:
>
>      `cabal new-build` will have generated a file like
>
>        .ghc.environment.x86_64-linux-8.3.20170913
>
>      which brings into scope all transitive dependencies of
>      `build-depends: base, lens`
>
>      Now all you need to do is simply call
>
>        ghc-stage2 --make MyTestProg.hs
>
>      to compile a program against those libs, or start up GHCi via
>
>        ghc-stage2 --interactive
>
>      and you'll be thrown into that package environment.
>
>
>
> I hope you find this useful
>
> Cheers,
>   Herbert
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Overlay Hackage Package Index for GHC HEAD

Robin Palotai
Sounds amazing - could you add the procedure you described to the "stock" multi-GHC Travis example [1]?
It already has a "HEAD" env (see "env: BUILD=cabal GHCVER=head"...), but breaks [2] due to the package problems you mentioned.


2017-09-18 14:31 GMT+02:00 Matthew Pickering <[hidden email]>:
Something like this is definitely useful for testing.

When building GHC HEAD I override the ghcHEAD derivation on nixpkgs to
the right commit I want to use and then can similarly specify which
patches and versions of packages to use by modifying the
'configuration-ghc-head.nix' file.
This is quite a bit more flexible than just patch files as I can point
to specific commits in git repos etc.

This repo will definitely be useful for me with this workflow as well.

Thanks,

Matt

On Mon, Sep 18, 2017 at 1:04 PM, Herbert Valerio Riedel
<[hidden email]> wrote:
> Hi GHC devs,
>
> A long-standing common problem when testing/dogfooding GHC HEAD is that
> at some point during the development cycle more and more packages from
> Hackage will run into build failures.
>
> Obviously, constantly nagging upstream maintainers to release quickfixes
> for unreleased GHC HEAD snapshots which will likely break again a few
> weeks later (as things are generally in flux until freeze comes into
> effect) does not scale and only introduces a latency/coordination
> bottleneck, and on top of it ultimately results in spamming the primary
> Hackage Package index with releases (which has non-negligible negative
> impact/costs of its own on the Hackage infrastructure, and thus ought to
> be minimised).
>
> OTOH, we need the ability to easily test, debug, profile, and prototype
> changes to GHC HEAD while things are still in motion, and case in point,
> if you try to e.g. build `pandoc` with GHC HEAD today, you'll currently
> run into a dozen or so of packages not building with GHC HEAD.
>
> To this end, I've finally found time to work on a side-project (related
> to matrix.hackage.haskell.org) which implements a scheme tailored to
> `cabal new-build`, which is inspired by how Eta copes with a very
> related issue (they rely on it for stable versions of the compiler);
> i.e., they maintain a set of patches at
>
>   https://github.com/typelead/eta-hackage/tree/master/patches
>
> which fix up existing Hackage packages to work with the Eta compiler.
>
>
> And this gave me the idea to use a similar scheme for GHC HEAD:
>
>   https://github.com/hvr/head.hackage/tree/master/patches
>
> This folder already contains several of patches (which mostly originate
> from Ryan, Ben and myself) to packages which I needed to patch in order
> to build popular Hackage packages & tools.
>
> The main difference is how those patches are applied; Eta uses a
> modified `cabal` which they renamed to `etlas` which is checks
> availability of .patch & .cabal files in the GitHub repo linked above;
>
> Whereas for GHC HEAD with `cabal new-build` a different scheme makes
> more sense: we simply generate an add-on Hackage repo, and use the
> existing `cabal` facilities (e.g. multi-repo support or the nix-style
> package store which makes sure that unofficially patched packages don't
> contaminate "normal" install-plans, etc.) to opt into the opt-in Hackage
> repo containing fixed up packages.
>
>
>
> I've tried to describe how to use the HEAD.hackage add-on repo in the
> README at
>
>   https://github.com/hvr/head.hackage#how-to-use
>
>
> And finally, here's a practical example of how you can use it to build
> e.g. the `pandoc` executable with GHC HEAD (can easily be adapted to
> build your project of choice; please refer to
>
>   http://cabal.readthedocs.io/en/latest/nix-local-build-overview.html
>
> to learn more about how to describe your project via  `cabal.project`
> files):
>
>
> 0.) This assumes you have a recent cabal 2.1 snapshot built from Git
>
> 1.) create & `cd` into a new work-folder
>
> 2.) invoke `head.hackage.sh update` to update the HEAD.hackage index cache
>
> 3.) invoke `head.hackage.sh init` to create an initial `cabal.project`
>     configuration which locally activates the HEAD.hackage overlay repo
>
> 4.) If needed, edit the cabal.project file and change where GHC
>     HEAD can be found (the script currently assumes GHC HEAD is
>     installed from my Ubuntu PPA), e.g.
>
>       with-compiler: /home/hvr/src/ghc-dev/inplace/bin/ghc-stage2
>
>     or you can add something like `optional-packages: deps/*/*.cabal`
>     to have cabal pick up package source-trees unpacked in the deps/
>     folder, or you can inject ghc-options, relax upper bounds via
>     `allow-newer: *:base` etc (please refer to the cabal user guide)
>
> 5.) Create a `dummy.cabal` file (in future we will have `cabal
>     new-install` or other facilities, but for now we use this
>     workaround):
>
>
> --8<---------------cut here---------------start------------->8---
> name:                dummy
> version:             0
> build-type:          Simple
> cabal-version:       >=2.0
>
> library
>   default-language:    Haskell2010
>
>   -- library components you want cabal to solve & build for
>   -- and become accessible via .ghc.environment files and/or
>   -- `cabal new-repl`
>   build-depends: base, lens
>
>   -- executable components you want cabal to build
>   build-tool-depends: pandoc:pandoc
>
> --8<---------------cut here---------------end--------------->8---
>
>  6.) invoke `cabal new-build`
>
>  7.) If everything works, you'll find the `pandoc:pandoc` executable
>      somewhere in your ~/.cabal/store/ghc-8.3.*/ folder
>      (you can use http://hackage.haskell.org/package/cabal-plan
>       to conveniently list the location via `cabal-plan list-bins`)
>
>  8.) As for libraries, you can either use `cabal new-repl`
>      or you can leverage GHC's package environment files:
>
>      `cabal new-build` will have generated a file like
>
>        .ghc.environment.x86_64-linux-8.3.20170913
>
>      which brings into scope all transitive dependencies of
>      `build-depends: base, lens`
>
>      Now all you need to do is simply call
>
>        ghc-stage2 --make MyTestProg.hs
>
>      to compile a program against those libs, or start up GHCi via
>
>        ghc-stage2 --interactive
>
>      and you'll be thrown into that package environment.
>
>
>
> I hope you find this useful
>
> Cheers,
>   Herbert
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Overlay Hackage Package Index for GHC HEAD

Rahul Muttineni
In reply to this post by Herbert Valerio Riedel-3
Hi Herbert,

This sounds like a great idea :) Glad to see ideas from Eta inspiring new ideas in GHC. 

"constantly nagging upstream maintainers to release quickfixes
for unreleased GHC HEAD snapshots which will likely break again a few
weeks later"

^ This is what motivated us to start eta-hackage in the first place. In our case, we didn't know whether the patches were completely correct when porting C FFI to Java FFI, at least in the corner cases, so we needed a way to quickly deploy updates to patches without having the end user wait a significant amount of time. Once they stabilise, we do plan on pushing the patches upstream to the respective packages - we've already done that for a few of them.

Thanks,
Rahul

On Mon, Sep 18, 2017 at 8:04 AM, Herbert Valerio Riedel <[hidden email]> wrote:
Hi GHC devs,

A long-standing common problem when testing/dogfooding GHC HEAD is that
at some point during the development cycle more and more packages from
Hackage will run into build failures.

Obviously, constantly nagging upstream maintainers to release quickfixes
for unreleased GHC HEAD snapshots which will likely break again a few
weeks later (as things are generally in flux until freeze comes into
effect) does not scale and only introduces a latency/coordination
bottleneck, and on top of it ultimately results in spamming the primary
Hackage Package index with releases (which has non-negligible negative
impact/costs of its own on the Hackage infrastructure, and thus ought to
be minimised).

OTOH, we need the ability to easily test, debug, profile, and prototype
changes to GHC HEAD while things are still in motion, and case in point,
if you try to e.g. build `pandoc` with GHC HEAD today, you'll currently
run into a dozen or so of packages not building with GHC HEAD.

To this end, I've finally found time to work on a side-project (related
to matrix.hackage.haskell.org) which implements a scheme tailored to
`cabal new-build`, which is inspired by how Eta copes with a very
related issue (they rely on it for stable versions of the compiler);
i.e., they maintain a set of patches at

  https://github.com/typelead/eta-hackage/tree/master/patches

which fix up existing Hackage packages to work with the Eta compiler.


And this gave me the idea to use a similar scheme for GHC HEAD:

  https://github.com/hvr/head.hackage/tree/master/patches

This folder already contains several of patches (which mostly originate
from Ryan, Ben and myself) to packages which I needed to patch in order
to build popular Hackage packages & tools.

The main difference is how those patches are applied; Eta uses a
modified `cabal` which they renamed to `etlas` which is checks
availability of .patch & .cabal files in the GitHub repo linked above;

Whereas for GHC HEAD with `cabal new-build` a different scheme makes
more sense: we simply generate an add-on Hackage repo, and use the
existing `cabal` facilities (e.g. multi-repo support or the nix-style
package store which makes sure that unofficially patched packages don't
contaminate "normal" install-plans, etc.) to opt into the opt-in Hackage
repo containing fixed up packages.



I've tried to describe how to use the HEAD.hackage add-on repo in the
README at

  https://github.com/hvr/head.hackage#how-to-use


And finally, here's a practical example of how you can use it to build
e.g. the `pandoc` executable with GHC HEAD (can easily be adapted to
build your project of choice; please refer to

  http://cabal.readthedocs.io/en/latest/nix-local-build-overview.html

to learn more about how to describe your project via  `cabal.project`
files):


0.) This assumes you have a recent cabal 2.1 snapshot built from Git

1.) create & `cd` into a new work-folder

2.) invoke `head.hackage.sh update` to update the HEAD.hackage index cache

3.) invoke `head.hackage.sh init` to create an initial `cabal.project`
    configuration which locally activates the HEAD.hackage overlay repo

4.) If needed, edit the cabal.project file and change where GHC
    HEAD can be found (the script currently assumes GHC HEAD is
    installed from my Ubuntu PPA), e.g.

      with-compiler: /home/hvr/src/ghc-dev/inplace/bin/ghc-stage2

    or you can add something like `optional-packages: deps/*/*.cabal`
    to have cabal pick up package source-trees unpacked in the deps/
    folder, or you can inject ghc-options, relax upper bounds via
    `allow-newer: *:base` etc (please refer to the cabal user guide)

5.) Create a `dummy.cabal` file (in future we will have `cabal
    new-install` or other facilities, but for now we use this
    workaround):


--8<---------------cut here---------------start------------->8---
name:                dummy
version:             0
build-type:          Simple
cabal-version:       >=2.0

library
  default-language:    Haskell2010

  -- library components you want cabal to solve & build for
  -- and become accessible via .ghc.environment files and/or
  -- `cabal new-repl`
  build-depends: base, lens

  -- executable components you want cabal to build
  build-tool-depends: pandoc:pandoc

--8<---------------cut here---------------end--------------->8---

 6.) invoke `cabal new-build`

 7.) If everything works, you'll find the `pandoc:pandoc` executable
     somewhere in your ~/.cabal/store/ghc-8.3.*/ folder
     (you can use http://hackage.haskell.org/package/cabal-plan
      to conveniently list the location via `cabal-plan list-bins`)

 8.) As for libraries, you can either use `cabal new-repl`
     or you can leverage GHC's package environment files:

     `cabal new-build` will have generated a file like

       .ghc.environment.x86_64-linux-8.3.20170913

     which brings into scope all transitive dependencies of
     `build-depends: base, lens`

     Now all you need to do is simply call

       ghc-stage2 --make MyTestProg.hs

     to compile a program against those libs, or start up GHCi via

       ghc-stage2 --interactive

     and you'll be thrown into that package environment.



I hope you find this useful

Cheers,
  Herbert
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs



--
Rahul Muttineni

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Overlay Hackage Package Index for GHC HEAD

Sebastian Graf
Hey,

this sounds extremely related to stack's Local dependency packages. Basically, what you specify in your stack.yaml when overriding the containers version looks like this:

packages:
- location: . # The actual project
- location:
    git: [hidden email]:haskell/containers
    commit: 5879fe5cece8cd4493016b793015469046d421b2
  extra-dep: true

I find this extremely useful for playing around with stuff that's not yet published on hackage.
I think this is not so useful when stuff gets actually published on hackage, because hackage can only reference packages on hackage IIRC, but this is exactly the use case we would have for GHC.

So long,
Sebastian


On Mon, Sep 18, 2017 at 6:37 PM, Rahul Muttineni <[hidden email]> wrote:
Hi Herbert,

This sounds like a great idea :) Glad to see ideas from Eta inspiring new ideas in GHC. 

"constantly nagging upstream maintainers to release quickfixes
for unreleased GHC HEAD snapshots which will likely break again a few
weeks later"

^ This is what motivated us to start eta-hackage in the first place. In our case, we didn't know whether the patches were completely correct when porting C FFI to Java FFI, at least in the corner cases, so we needed a way to quickly deploy updates to patches without having the end user wait a significant amount of time. Once they stabilise, we do plan on pushing the patches upstream to the respective packages - we've already done that for a few of them.

Thanks,
Rahul

On Mon, Sep 18, 2017 at 8:04 AM, Herbert Valerio Riedel <[hidden email]> wrote:
Hi GHC devs,

A long-standing common problem when testing/dogfooding GHC HEAD is that
at some point during the development cycle more and more packages from
Hackage will run into build failures.

Obviously, constantly nagging upstream maintainers to release quickfixes
for unreleased GHC HEAD snapshots which will likely break again a few
weeks later (as things are generally in flux until freeze comes into
effect) does not scale and only introduces a latency/coordination
bottleneck, and on top of it ultimately results in spamming the primary
Hackage Package index with releases (which has non-negligible negative
impact/costs of its own on the Hackage infrastructure, and thus ought to
be minimised).

OTOH, we need the ability to easily test, debug, profile, and prototype
changes to GHC HEAD while things are still in motion, and case in point,
if you try to e.g. build `pandoc` with GHC HEAD today, you'll currently
run into a dozen or so of packages not building with GHC HEAD.

To this end, I've finally found time to work on a side-project (related
to matrix.hackage.haskell.org) which implements a scheme tailored to
`cabal new-build`, which is inspired by how Eta copes with a very
related issue (they rely on it for stable versions of the compiler);
i.e., they maintain a set of patches at

  https://github.com/typelead/eta-hackage/tree/master/patches

which fix up existing Hackage packages to work with the Eta compiler.


And this gave me the idea to use a similar scheme for GHC HEAD:

  https://github.com/hvr/head.hackage/tree/master/patches

This folder already contains several of patches (which mostly originate
from Ryan, Ben and myself) to packages which I needed to patch in order
to build popular Hackage packages & tools.

The main difference is how those patches are applied; Eta uses a
modified `cabal` which they renamed to `etlas` which is checks
availability of .patch & .cabal files in the GitHub repo linked above;

Whereas for GHC HEAD with `cabal new-build` a different scheme makes
more sense: we simply generate an add-on Hackage repo, and use the
existing `cabal` facilities (e.g. multi-repo support or the nix-style
package store which makes sure that unofficially patched packages don't
contaminate "normal" install-plans, etc.) to opt into the opt-in Hackage
repo containing fixed up packages.



I've tried to describe how to use the HEAD.hackage add-on repo in the
README at

  https://github.com/hvr/head.hackage#how-to-use


And finally, here's a practical example of how you can use it to build
e.g. the `pandoc` executable with GHC HEAD (can easily be adapted to
build your project of choice; please refer to

  http://cabal.readthedocs.io/en/latest/nix-local-build-overview.html

to learn more about how to describe your project via  `cabal.project`
files):


0.) This assumes you have a recent cabal 2.1 snapshot built from Git

1.) create & `cd` into a new work-folder

2.) invoke `head.hackage.sh update` to update the HEAD.hackage index cache

3.) invoke `head.hackage.sh init` to create an initial `cabal.project`
    configuration which locally activates the HEAD.hackage overlay repo

4.) If needed, edit the cabal.project file and change where GHC
    HEAD can be found (the script currently assumes GHC HEAD is
    installed from my Ubuntu PPA), e.g.

      with-compiler: /home/hvr/src/ghc-dev/inplace/bin/ghc-stage2

    or you can add something like `optional-packages: deps/*/*.cabal`
    to have cabal pick up package source-trees unpacked in the deps/
    folder, or you can inject ghc-options, relax upper bounds via
    `allow-newer: *:base` etc (please refer to the cabal user guide)

5.) Create a `dummy.cabal` file (in future we will have `cabal
    new-install` or other facilities, but for now we use this
    workaround):


--8<---------------cut here---------------start------------->8---
name:                dummy
version:             0
build-type:          Simple
cabal-version:       >=2.0

library
  default-language:    Haskell2010

  -- library components you want cabal to solve & build for
  -- and become accessible via .ghc.environment files and/or
  -- `cabal new-repl`
  build-depends: base, lens

  -- executable components you want cabal to build
  build-tool-depends: pandoc:pandoc

--8<---------------cut here---------------end--------------->8---

 6.) invoke `cabal new-build`

 7.) If everything works, you'll find the `pandoc:pandoc` executable
     somewhere in your ~/.cabal/store/ghc-8.3.*/ folder
     (you can use http://hackage.haskell.org/package/cabal-plan
      to conveniently list the location via `cabal-plan list-bins`)

 8.) As for libraries, you can either use `cabal new-repl`
     or you can leverage GHC's package environment files:

     `cabal new-build` will have generated a file like

       .ghc.environment.x86_64-linux-8.3.20170913

     which brings into scope all transitive dependencies of
     `build-depends: base, lens`

     Now all you need to do is simply call

       ghc-stage2 --make MyTestProg.hs

     to compile a program against those libs, or start up GHCi via

       ghc-stage2 --interactive

     and you'll be thrown into that package environment.



I hope you find this useful

Cheers,
  Herbert
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs



--
Rahul Muttineni

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs



_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: ANN: Overlay Hackage Package Index for GHC HEAD

Ben Gamari-2
In reply to this post by Matthew Pickering
Matthew Pickering <[hidden email]> writes:

> Something like this is definitely useful for testing.
>
> When building GHC HEAD I override the ghcHEAD derivation on nixpkgs to
> the right commit I want to use and then can similarly specify which
> patches and versions of packages to use by modifying the
> 'configuration-ghc-head.nix' file.
> This is quite a bit more flexible than just patch files as I can point
> to specific commits in git repos etc.
>
This sounds amazing. I think you may have finally sold me on Nix.


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (497 bytes) Download Attachment