Haskell CI with many repositories/packages

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Haskell CI with many repositories/packages

Jeroen Bransen-2
Hi cafe,

Does anyone know of a good setup for doing continuous integration with a
set of Haskell packages, each in its own repository? Just building
everything upon every commit is not so hard, but to speed up building
times I'd like to build and test only the minimal set of packages. In
particular, at a commit for some package A, I would like to build and
test A and all packages that depend on A.

The problem is that most CI tools use some notion of 'build artefact',
which Stack doesn't really seem to give me. Ideally building a package
results in some object file, which can then be used by the other
packages. When building failed, packages that depend on it can still use
the last succesful build. I've tried to look up some Haskell projects,
but most of them seem to use some ad hoc setup.

Some pointers are appreciated, as we are using Gitlab a gitlab-runner
specific option would be great, but I am also open to use Jenkins or
other tools. And I guess my main struggle now is on the stack/Haskell side.

Regards,
Jeroen Bransen
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Haskell CI with many repositories/packages

Adam Bergmark-2
I haven't felt the need to share build artifacts like this. Instead, e.g., if you cache your `.stack-work` and your master project does a git clone of sub projects and you put those paths in your stack.yaml then `stack build` should only rebuild the changes. You may be able to share parts of `.stack-work` as well but I haven't looked into that.

HTH,
Adam


On Tue, 24 Oct 2017 at 11:35 Jeroen Bransen <[hidden email]> wrote:
Hi cafe,

Does anyone know of a good setup for doing continuous integration with a
set of Haskell packages, each in its own repository? Just building
everything upon every commit is not so hard, but to speed up building
times I'd like to build and test only the minimal set of packages. In
particular, at a commit for some package A, I would like to build and
test A and all packages that depend on A.

The problem is that most CI tools use some notion of 'build artefact',
which Stack doesn't really seem to give me. Ideally building a package
results in some object file, which can then be used by the other
packages. When building failed, packages that depend on it can still use
the last succesful build. I've tried to look up some Haskell projects,
but most of them seem to use some ad hoc setup.

Some pointers are appreciated, as we are using Gitlab a gitlab-runner
specific option would be great, but I am also open to use Jenkins or
other tools. And I guess my main struggle now is on the stack/Haskell side.

Regards,
Jeroen Bransen
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Haskell CI with many repositories/packages

Matthew Pickering
In reply to this post by Jeroen Bransen-2
It sounds like using Hydra (the Nix based CI system) would be suitable
for your needs.

The essential work flow would be to create a package set which
contained all your packages and then create a job set which built that
package set.

What's more, if your developers are also using nix then it is trivial
to setup a build cache which serves prebuilt packages to them.

The manual provides a concise introduction - https://nixos.org/hydra/

Matt

On Tue, Oct 24, 2017 at 10:33 AM, Jeroen Bransen <[hidden email]> wrote:

> Hi cafe,
>
> Does anyone know of a good setup for doing continuous integration with a set
> of Haskell packages, each in its own repository? Just building everything
> upon every commit is not so hard, but to speed up building times I'd like to
> build and test only the minimal set of packages. In particular, at a commit
> for some package A, I would like to build and test A and all packages that
> depend on A.
>
> The problem is that most CI tools use some notion of 'build artefact', which
> Stack doesn't really seem to give me. Ideally building a package results in
> some object file, which can then be used by the other packages. When
> building failed, packages that depend on it can still use the last succesful
> build. I've tried to look up some Haskell projects, but most of them seem to
> use some ad hoc setup.
>
> Some pointers are appreciated, as we are using Gitlab a gitlab-runner
> specific option would be great, but I am also open to use Jenkins or other
> tools. And I guess my main struggle now is on the stack/Haskell side.
>
> Regards,
> Jeroen Bransen
> _______________________________________________
> Haskell-Cafe mailing list
> To (un)subscribe, modify options or view archives go to:
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
> Only members subscribed via the mailman list are allowed to post.
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Haskell CI with many repositories/packages

Alexander Kjeldaas
The following might count as a trick.  It checks that all .stack-work directories are mentioned in the .gitlab-ci.yml file.

A nicer approach than the ugly grep is to use jq-on-yaml, but this works for me.  The goal is just to fail CI if a new module is added without caching it.

# Check that all stack work directories are cached
- diff -u <( find . -name .stack-work | grep -v '.stack-work.*.stack-work' | grep -v '.stack/' | grep -v node_modules | cut -c3- | sort) <( grep .stack-work .gitlab-ci.yml | egrep -v 'find .|for a|rm -rf|^\s*#.*|grep' | tr -d '"' | awk '{print $2}' | cut -d/ -f5- | sort)

Alexander

On Tue, Oct 24, 2017 at 10:43 PM, Matthew Pickering <[hidden email]> wrote:
It sounds like using Hydra (the Nix based CI system) would be suitable
for your needs.

The essential work flow would be to create a package set which
contained all your packages and then create a job set which built that
package set.

What's more, if your developers are also using nix then it is trivial
to setup a build cache which serves prebuilt packages to them.

The manual provides a concise introduction - https://nixos.org/hydra/

Matt

On Tue, Oct 24, 2017 at 10:33 AM, Jeroen Bransen <[hidden email]> wrote:
> Hi cafe,
>
> Does anyone know of a good setup for doing continuous integration with a set
> of Haskell packages, each in its own repository? Just building everything
> upon every commit is not so hard, but to speed up building times I'd like to
> build and test only the minimal set of packages. In particular, at a commit
> for some package A, I would like to build and test A and all packages that
> depend on A.
>
> The problem is that most CI tools use some notion of 'build artefact', which
> Stack doesn't really seem to give me. Ideally building a package results in
> some object file, which can then be used by the other packages. When
> building failed, packages that depend on it can still use the last succesful
> build. I've tried to look up some Haskell projects, but most of them seem to
> use some ad hoc setup.
>
> Some pointers are appreciated, as we are using Gitlab a gitlab-runner
> specific option would be great, but I am also open to use Jenkins or other
> tools. And I guess my main struggle now is on the stack/Haskell side.
>
> Regards,
> Jeroen Bransen
> _______________________________________________
> Haskell-Cafe mailing list
> To (un)subscribe, modify options or view archives go to:
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
> Only members subscribed via the mailman list are allowed to post.
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.


_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Haskell CI with many repositories/packages

Jeroen Bransen-2
In reply to this post by Adam Bergmark-2
That is similar to what I am doing now, but I don't think this solves my problem. When I have a package A depending on B, I make a mistake in B such that it doesn't compile, then commit something correct to A, the build for A will also fail because of this error in B. I would like to let A be based on the last succesful build of B, but with shared `.stack-work` I don't think that's going to work.

Regards,
Jeroen Bransen

Op 24-10-2017 om 16:39 schreef Adam Bergmark:
I haven't felt the need to share build artifacts like this. Instead, e.g., if you cache your `.stack-work` and your master project does a git clone of sub projects and you put those paths in your stack.yaml then `stack build` should only rebuild the changes. You may be able to share parts of `.stack-work` as well but I haven't looked into that.

HTH,
Adam


On Tue, 24 Oct 2017 at 11:35 Jeroen Bransen <[hidden email]> wrote:
Hi cafe,

Does anyone know of a good setup for doing continuous integration with a
set of Haskell packages, each in its own repository? Just building
everything upon every commit is not so hard, but to speed up building
times I'd like to build and test only the minimal set of packages. In
particular, at a commit for some package A, I would like to build and
test A and all packages that depend on A.

The problem is that most CI tools use some notion of 'build artefact',
which Stack doesn't really seem to give me. Ideally building a package
results in some object file, which can then be used by the other
packages. When building failed, packages that depend on it can still use
the last succesful build. I've tried to look up some Haskell projects,
but most of them seem to use some ad hoc setup.

Some pointers are appreciated, as we are using Gitlab a gitlab-runner
specific option would be great, but I am also open to use Jenkins or
other tools. And I guess my main struggle now is on the stack/Haskell side.

Regards,
Jeroen Bransen
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.


_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Haskell CI with many repositories/packages

Jeroen Bransen-2
In reply to this post by Adam Bergmark-2
That is similar to what I am doing now, but I don't think this solves my problem. When I have a package A depending on B, I make a mistake in B such that it doesn't compile, then commit something correct to A, the build for A will also fail because of this error in B. I would like to let A be based on the last succesful build of B, but with shared `.stack-work` I don't think that's going to work.

Regards,
Jeroen Bransen

Op 24-10-2017 om 16:39 schreef Adam Bergmark:
I haven't felt the need to share build artifacts like this. Instead, e.g., if you cache your `.stack-work` and your master project does a git clone of sub projects and you put those paths in your stack.yaml then `stack build` should only rebuild the changes. You may be able to share parts of `.stack-work` as well but I haven't looked into that.

HTH,
Adam


On Tue, 24 Oct 2017 at 11:35 Jeroen Bransen <[hidden email]> wrote:
Hi cafe,

Does anyone know of a good setup for doing continuous integration with a
set of Haskell packages, each in its own repository? Just building
everything upon every commit is not so hard, but to speed up building
times I'd like to build and test only the minimal set of packages. In
particular, at a commit for some package A, I would like to build and
test A and all packages that depend on A.

The problem is that most CI tools use some notion of 'build artefact',
which Stack doesn't really seem to give me. Ideally building a package
results in some object file, which can then be used by the other
packages. When building failed, packages that depend on it can still use
the last succesful build. I've tried to look up some Haskell projects,
but most of them seem to use some ad hoc setup.

Some pointers are appreciated, as we are using Gitlab a gitlab-runner
specific option would be great, but I am also open to use Jenkins or
other tools. And I guess my main struggle now is on the stack/Haskell side.

Regards,
Jeroen Bransen
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.


_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Haskell CI with many repositories/packages

Jeroen Bransen-2
In reply to this post by Matthew Pickering
This is a very interesting suggestion. We are not using Nix, so the
whole setup would change, but it is something we can consider for the
buildserver (and maybe even other machines). If I understand correctly
the packages are then not built anymore with Stack, but rather with Nix
specific build scripts. Everything else follows then naturally because
of the nice nature of Nix.

I'll definitely look into this, but a solution using Stack would be nice
and much easier to set up currently.

Regards,
Jeroen Bransen

Op 24-10-2017 om 22:43 schreef Matthew Pickering:

> It sounds like using Hydra (the Nix based CI system) would be suitable
> for your needs.
>
> The essential work flow would be to create a package set which
> contained all your packages and then create a job set which built that
> package set.
>
> What's more, if your developers are also using nix then it is trivial
> to setup a build cache which serves prebuilt packages to them.
>
> The manual provides a concise introduction - https://nixos.org/hydra/
>
> Matt
>
> On Tue, Oct 24, 2017 at 10:33 AM, Jeroen Bransen <[hidden email]> wrote:
>> Hi cafe,
>>
>> Does anyone know of a good setup for doing continuous integration with a set
>> of Haskell packages, each in its own repository? Just building everything
>> upon every commit is not so hard, but to speed up building times I'd like to
>> build and test only the minimal set of packages. In particular, at a commit
>> for some package A, I would like to build and test A and all packages that
>> depend on A.
>>
>> The problem is that most CI tools use some notion of 'build artefact', which
>> Stack doesn't really seem to give me. Ideally building a package results in
>> some object file, which can then be used by the other packages. When
>> building failed, packages that depend on it can still use the last succesful
>> build. I've tried to look up some Haskell projects, but most of them seem to
>> use some ad hoc setup.
>>
>> Some pointers are appreciated, as we are using Gitlab a gitlab-runner
>> specific option would be great, but I am also open to use Jenkins or other
>> tools. And I guess my main struggle now is on the stack/Haskell side.
>>
>> Regards,
>> Jeroen Bransen
>> _______________________________________________
>> Haskell-Cafe mailing list
>> To (un)subscribe, modify options or view archives go to:
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
>> Only members subscribed via the mailman list are allowed to post.

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Haskell CI with many repositories/packages

Adam Bergmark-2
In reply to this post by Jeroen Bransen-2
But you could check whether sharing the .stack-work would even be a noticeable improvement. Unless your projects change often and take a lot of time to build (excluding dependencies) then I don't think it would.


On Fri, 27 Oct 2017 at 11:45 Jeroen Bransen <[hidden email]> wrote:
That is similar to what I am doing now, but I don't think this solves my problem. When I have a package A depending on B, I make a mistake in B such that it doesn't compile, then commit something correct to A, the build for A will also fail because of this error in B. I would like to let A be based on the last succesful build of B, but with shared `.stack-work` I don't think that's going to work.

Regards,
Jeroen Bransen

Op 24-10-2017 om 16:39 schreef Adam Bergmark:
I haven't felt the need to share build artifacts like this. Instead, e.g., if you cache your `.stack-work` and your master project does a git clone of sub projects and you put those paths in your stack.yaml then `stack build` should only rebuild the changes. You may be able to share parts of `.stack-work` as well but I haven't looked into that.

HTH,
Adam


On Tue, 24 Oct 2017 at 11:35 Jeroen Bransen <[hidden email]> wrote:
Hi cafe,

Does anyone know of a good setup for doing continuous integration with a
set of Haskell packages, each in its own repository? Just building
everything upon every commit is not so hard, but to speed up building
times I'd like to build and test only the minimal set of packages. In
particular, at a commit for some package A, I would like to build and
test A and all packages that depend on A.

The problem is that most CI tools use some notion of 'build artefact',
which Stack doesn't really seem to give me. Ideally building a package
results in some object file, which can then be used by the other
packages. When building failed, packages that depend on it can still use
the last succesful build. I've tried to look up some Haskell projects,
but most of them seem to use some ad hoc setup.

Some pointers are appreciated, as we are using Gitlab a gitlab-runner
specific option would be great, but I am also open to use Jenkins or
other tools. And I guess my main struggle now is on the stack/Haskell side.

Regards,
Jeroen Bransen
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.