Sharing a precompiled Haskell library between projects

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Sharing a precompiled Haskell library between projects

Alfredo Di Napoli
Hey folks,

at work we are trying to optimise our build/CI pipeline. We have one proprietary Haskell library which contains our business logic, it has more than 200 Haskell files AND
a good dose of TH which makes is very slow to build. Not only that, but the compilation itself is very memory hungry due to the TH expansion. We cannot split it into more modular
projects, for reasons I won't get into. To make things more fun, we cannot simply bump EC2 instance type, for budget constraints. We currently have 2 Jenkins servers:

* One (which I will call "A") builds each new release of the library when we merge a PR into the remote git master branch.  This CI server also uploads it to our private Hackage and can potentially
  perform other post builds operations, like creating an RPM and so on.

* One (Which I will call "B") which builds and deploy our webapp, a mixture of Haskell and Ruby. It's a modest EC2 t2.medium machine, which we cannot upgrade.

We are uniformly targeting Centos 7 as our production OS.

We are trying to find a way to get the artifacts produced by "A" and package it up into a Haskell library we could "inject" into "B" in our project's `.stack-work` directories (or globally, if that's an option),
so that "B" never has to build "the big Haskell library" from scratch.

Said that, which is our best option? Ideally we would like to "deceive" stack into thinking "the big Haskell library" can be simply copied as a precompiled package, instead of building it every time a new
version is released and we need to update our webapp Haskell microservices to align to the newly released library.

To be clear: If our Haskell microservices are at version X of the "big Haskell library", the problem is not the build time when the library doesn't change, as Jenkins will keep in the workspace the local ".stack-work"
and therefore the library won't be rebuild. The problem is when the library gets bumped to version Y and we need to cope; in that case each Haskell microservice needs to be updated, which means we need to
painfully rebuild the big Haskell library for each workspace. Not good!

Is the recently-released extensible snapshots a possible solution here?

I hope I made myself clear enough. Please share you war stories! ;)

Thanks,

Alfredo

--
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sharing a precompiled Haskell library between projects

Daniel Vigovszky
Hi Alfredo,

We have a working solution for this, but it is based on using gradle on top of stack. See https://github.com/prezi/gradle-haskell-plugin and these two small blog posts: http://vigoo.github.io/posts/2015-04-22-gradle-haskell-plugin.html and http://vigoo.github.io/posts/2015-12-22-gradle-haskell-plugin-stack.html if you are interested, and feel free to ask me about details; we are using it every day for more than a year now.

vigoo

Alfredo Di Napoli <[hidden email]> ezt írta (időpont: 2016. júl. 12., K, 9:09):
Hey folks,

at work we are trying to optimise our build/CI pipeline. We have one proprietary Haskell library which contains our business logic, it has more than 200 Haskell files AND
a good dose of TH which makes is very slow to build. Not only that, but the compilation itself is very memory hungry due to the TH expansion. We cannot split it into more modular
projects, for reasons I won't get into. To make things more fun, we cannot simply bump EC2 instance type, for budget constraints. We currently have 2 Jenkins servers:

* One (which I will call "A") builds each new release of the library when we merge a PR into the remote git master branch.  This CI server also uploads it to our private Hackage and can potentially
  perform other post builds operations, like creating an RPM and so on.

* One (Which I will call "B") which builds and deploy our webapp, a mixture of Haskell and Ruby. It's a modest EC2 t2.medium machine, which we cannot upgrade.

We are uniformly targeting Centos 7 as our production OS.

We are trying to find a way to get the artifacts produced by "A" and package it up into a Haskell library we could "inject" into "B" in our project's `.stack-work` directories (or globally, if that's an option),
so that "B" never has to build "the big Haskell library" from scratch.

Said that, which is our best option? Ideally we would like to "deceive" stack into thinking "the big Haskell library" can be simply copied as a precompiled package, instead of building it every time a new
version is released and we need to update our webapp Haskell microservices to align to the newly released library.

To be clear: If our Haskell microservices are at version X of the "big Haskell library", the problem is not the build time when the library doesn't change, as Jenkins will keep in the workspace the local ".stack-work"
and therefore the library won't be rebuild. The problem is when the library gets bumped to version Y and we need to cope; in that case each Haskell microservice needs to be updated, which means we need to
painfully rebuild the big Haskell library for each workspace. Not good!

Is the recently-released extensible snapshots a possible solution here?

I hope I made myself clear enough. Please share you war stories! ;)

Thanks,

Alfredo

--

--
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sharing a precompiled Haskell library between projects

Alfredo Di Napoli
On 12 July 2016 at 09:15, Daniel Vigovszky <[hidden email]> wrote:
Hi Alfredo,

We have a working solution for this, but it is based on using gradle on top of stack. See https://github.com/prezi/gradle-haskell-plugin and these two small blog posts: http://vigoo.github.io/posts/2015-04-22-gradle-haskell-plugin.html and http://vigoo.github.io/posts/2015-12-22-gradle-haskell-plugin-stack.html if you are interested, and feel free to ask me about details; we are using it every day for more than a year now.

vigoo


Hey Daniel,

thanks a lot for the pointers! I will be honest with you; is not that we are looking forward to introduce yet another tool in our stack (no pun intended!), but more than that I think I can surely steal ideas from your blog posts and the `gradle-haskell-plugin`. Thanks a lot!

I will deserve the right of asking stupid questions on the workflow if I have any ;)

Alfredo 


--
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Sharing a precompiled Haskell library between projects

Daniel Vigovszky
Sure, we built it on gradle because it is used by several other teams for other stacks we have to interact with. The main idea of it is completely gradle-free :) Write me if you have questions.

Alfredo Di Napoli <[hidden email]> ezt írta (időpont: 2016. júl. 12., K, 9:25):
On 12 July 2016 at 09:15, Daniel Vigovszky <[hidden email]> wrote:
Hi Alfredo,

We have a working solution for this, but it is based on using gradle on top of stack. See https://github.com/prezi/gradle-haskell-plugin and these two small blog posts: http://vigoo.github.io/posts/2015-04-22-gradle-haskell-plugin.html and http://vigoo.github.io/posts/2015-12-22-gradle-haskell-plugin-stack.html if you are interested, and feel free to ask me about details; we are using it every day for more than a year now.

vigoo


Hey Daniel,

thanks a lot for the pointers! I will be honest with you; is not that we are looking forward to introduce yet another tool in our stack (no pun intended!), but more than that I think I can surely steal ideas from your blog posts and the `gradle-haskell-plugin`. Thanks a lot!

I will deserve the right of asking stupid questions on the workflow if I have any ;)

Alfredo 

--
Loading...