Cross-compiling Template Haskell via -fexternal-interpreter and IPC

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Cross-compiling Template Haskell via -fexternal-interpreter and IPC

Alberto Valverde
Hello,

I'm trying to put together a GHC 8.0.1 cross-compiler with Template Haskell support. Initially to target Windows (32bits) from a Linux host but a similar procedure should enable to target other platforms too. I'd like to contribute the patches back so I'm asking for advice on how to implement it in order to increase the chances of them being accepted.

I've managed to get a working stage 1 cross-compiler with some patches which correctly builds all stage1 libs and GHC + stage 2 compiler and ghc-iserv.exe. However, compiling TH by using wine and ghc-iserver.exe fails because the file descriptor ids that GHC passes as arguments to "wine ghc-iserv.exe" don't make sense in the emulated windows world.

I've hacked around this to test the feasibility of the approach by using stdin/stdout instead of creating new pipes and, surprisingly, managed to cross-compile a simple Template Haskell program.

I'm considering using a socket for communicating between both processes as a more permanent solution but this would incur in a dependency on the "network" package. Would this be acceptable?

Named pipes have also crossed my mind but I'm not sure how well they're supported by wine.

Thanks for your attention.
Alberto


P.S:
Code is in the "cross-mingw-hacks" branches of these repositories and is based on the ghc-8.0.1-release tag:




_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling Template Haskell via -fexternal-interpreter and IPC

Ben Gamari-2
Explicitly CCing Simon Marlow to ensure he sees this.


Alberto Valverde <[hidden email]> writes:

> Hello,
>
> I'm trying to put together a GHC 8.0.1 cross-compiler with Template Haskell
> support. Initially to target Windows (32bits) from a Linux host but a
> similar procedure should enable to target other platforms too. I'd like to
> contribute the patches back so I'm asking for advice on how to implement it
> in order to increase the chances of them being accepted.
>
> I've managed to get a working stage 1 cross-compiler with some patches
> which correctly builds all stage1 libs and GHC + stage 2 compiler and
> ghc-iserv.exe. However, compiling TH by using wine and ghc-iserver.exe
> fails because the file descriptor ids that GHC passes as arguments to "wine
> ghc-iserv.exe" don't make sense in the emulated windows world.
>
Ahh, right. Out of curiosity what toolchain are you using to build your
stage 1 cross compiler?

> I've hacked around this to test the feasibility of the approach by using
> stdin/stdout instead of creating new pipes and, surprisingly, managed to
> cross-compile a simple Template Haskell program.
>
Well done!

> I'm considering using a socket for communicating between both processes as
> a more permanent solution but this would incur in a dependency on the
> "network" package. Would this be acceptable?
>
It would be nice if we could avoid it; GHC depending upon a library has
the very unfortunate effect that any user code also needing to link
against GHC or one of its libraries now has no choice in the version of
that library. We go to some lengths to try to keep the dependency
footprint of GHC small for this reason.

> Named pipes have also crossed my mind but I'm not sure how well they're
> supported by wine.
>
It would be great if there were some way we could make this work with
named pipes. Not only does it side-step the dependency issue, but it
feels like the right way forward.

Cheers,

- Ben

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (482 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling Template Haskell via -fexternal-interpreter and IPC

lonetiger
In reply to this post by Alberto Valverde
> However, compiling TH by using wine and ghc-iserver.exe fails because the file descriptor ids that GHC passes as arguments to "wine ghc-iserv.exe" don't make sense in the emulated windows world.

Just as a note, On Windows we don't pass FDs (As in the posix file descriptor) to the child process. It passes down Windows File Handles. This requires the Windows process inheritance/security model to be implemented in wine and I know nothing about Wine. But just wanted to clarify.

Named pipes would solve your issue, Just don't forget to set the proper ACL on the pipes.

On Wed, Jul 6, 2016 at 2:00 PM, Alberto Valverde <[hidden email]> wrote:
Hello,

I'm trying to put together a GHC 8.0.1 cross-compiler with Template Haskell support. Initially to target Windows (32bits) from a Linux host but a similar procedure should enable to target other platforms too. I'd like to contribute the patches back so I'm asking for advice on how to implement it in order to increase the chances of them being accepted.

I've managed to get a working stage 1 cross-compiler with some patches which correctly builds all stage1 libs and GHC + stage 2 compiler and ghc-iserv.exe. However, compiling TH by using wine and ghc-iserver.exe fails because the file descriptor ids that GHC passes as arguments to "wine ghc-iserv.exe" don't make sense in the emulated windows world.

I've hacked around this to test the feasibility of the approach by using stdin/stdout instead of creating new pipes and, surprisingly, managed to cross-compile a simple Template Haskell program.

I'm considering using a socket for communicating between both processes as a more permanent solution but this would incur in a dependency on the "network" package. Would this be acceptable?

Named pipes have also crossed my mind but I'm not sure how well they're supported by wine.

Thanks for your attention.
Alberto


P.S:
Code is in the "cross-mingw-hacks" branches of these repositories and is based on the ghc-8.0.1-release tag:




_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs



_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling Template Haskell via -fexternal-interpreter and IPC

Alberto Valverde
In reply to this post by Ben Gamari-2
Ben, Phyx,

On Wed, Jul 6, 2016 at 7:14 PM, Ben Gamari <[hidden email]> wrote:
(...)
Ahh, right. Out of curiosity what toolchain are you using to build your
stage 1 cross compiler?

I'm using an MXE (http://mxe.cc) environment in a Debian Jessie Docker image which has gcc-4.9.3. The build env is here: https://github.com/albertov/ghc-cross-compiler-windows-x86

On Wed, Jul 6, 2016 at 7:14 PM, Ben Gamari <[hidden email]> wrote:
It would be nice if we could avoid it; GHC depending upon a library has
the very unfortunate effect that any user code also needing to link
against GHC or one of its libraries now has no choice in the version of
that library. We go to some lengths to try to keep the dependency
footprint of GHC small for this reason.
(...)
> Named pipes have also crossed my mind but I'm not sure how well they're
> supported by wine.
>
It would be great if there were some way we could make this work with
named pipes. Not only does it side-step the dependency issue, but it
feels like the right way forward.

 On Wed, Jul 6, 2016 at 8:05 PM, Phyx <[hidden email]> wrote:
Named pipes would solve your issue, Just don't forget to set the proper ACL on the pipes.


It does indeed work! However, I've broken GHC on a Windows host until I implement the equivalent to Posix.createNamedPipe since it seems there's no binding for CreateNamedPipe that I can find in the tree. I'll look into it tomorrow.


Thanks
Alberto


Cheers,

- Ben


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling Template Haskell via -fexternal-interpreter and IPC

Moritz Angermann
Great work Alberto,

I’m in favor of adding some form of network layer, as there are scenarios
where you have to run the th compilation process on a different machine.
This would be the case for iOS for example.

When I toyed with this ~2years ago, trying to port the out of process th
solution from ghcjs, I tried to use GHC’s plugin interface, to load the
module that would allow ghc to communicate with the runner on the device.

This in principle allows to have more dependencies on the plugin and not
force them into ghc. At the same time it requires the installation of some
additional hooks into the plugin system.

I guess one could also come up with a tiny proxy that pretends to be the
iserv endpoint and would forward anything over a network layer; this again
could probably work outside of ghc.

Cheers,
 Moritz

> On Jul 7, 2016, at 8:01 AM, Alberto Valverde <[hidden email]> wrote:
>
> Ben, Phyx,
>
> On Wed, Jul 6, 2016 at 7:14 PM, Ben Gamari <[hidden email]> wrote:
> (...)
> Ahh, right. Out of curiosity what toolchain are you using to build your
> stage 1 cross compiler?
>
> I'm using an MXE (http://mxe.cc) environment in a Debian Jessie Docker image which has gcc-4.9.3. The build env is here: https://github.com/albertov/ghc-cross-compiler-windows-x86
>
> On Wed, Jul 6, 2016 at 7:14 PM, Ben Gamari <[hidden email]> wrote:
> It would be nice if we could avoid it; GHC depending upon a library has
> the very unfortunate effect that any user code also needing to link
> against GHC or one of its libraries now has no choice in the version of
> that library. We go to some lengths to try to keep the dependency
> footprint of GHC small for this reason.
> (...)
> > Named pipes have also crossed my mind but I'm not sure how well they're
> > supported by wine.
> >
> It would be great if there were some way we could make this work with
> named pipes. Not only does it side-step the dependency issue, but it
> feels like the right way forward.
>
>  On Wed, Jul 6, 2016 at 8:05 PM, Phyx <[hidden email]> wrote:
> Named pipes would solve your issue, Just don't forget to set the proper ACL on the pipes.
>
>
> It does indeed work! However, I've broken GHC on a Windows host until I implement the equivalent to Posix.createNamedPipe since it seems there's no binding for CreateNamedPipe that I can find in the tree. I'll look into it tomorrow.
>
> I've pushed the changes so far to https://github.com/albertov/ghc/tree/cross-external-interpreter.
>
> Thanks
> Alberto
>
>
> Cheers,
>
> - Ben
>
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling Template Haskell via -fexternal-interpreter and IPC

Simon Marlow-7
In reply to this post by Ben Gamari-2
On 6 July 2016 at 18:14, Ben Gamari <[hidden email]> wrote:

Alberto Valverde <[hidden email]> writes:

> I've hacked around this to test the feasibility of the approach by using
> stdin/stdout instead of creating new pipes and, surprisingly, managed to
> cross-compile a simple Template Haskell program.
>
Well done!

Indeed, nice hacking :)
 
> I'm considering using a socket for communicating between both processes as
> a more permanent solution but this would incur in a dependency on the
> "network" package. Would this be acceptable?
>
It would be nice if we could avoid it; GHC depending upon a library has
the very unfortunate effect that any user code also needing to link
against GHC or one of its libraries now has no choice in the version of
that library. We go to some lengths to try to keep the dependency
footprint of GHC small for this reason.

> Named pipes have also crossed my mind but I'm not sure how well they're
> supported by wine.
>
It would be great if there were some way we could make this work with
named pipes. Not only does it side-step the dependency issue, but it
feels like the right way forward.

I agree, named pipes are probably a better plan, perhaps a better solution overall than the way we currently pass FD numbers on the command line.  Do named pipes work work as expected through wine?  We would have to be careful to clean them up again afterwards.

Cheers,
Simon

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling Template Haskell via -fexternal-interpreter and IPC

Alberto Valverde
In reply to this post by Moritz Angermann
On Thu, Jul 7, 2016 at 2:57 AM, Moritz Angermann <[hidden email]> wrote:
Great work Alberto,

 
Thanks :)
 
I’m in favor of adding some form of network layer, as there are scenarios
where you have to run the th compilation process on a different machine.
This would be the case for iOS for example.

When I toyed with this ~2years ago, trying to port the out of process th
solution from ghcjs, I tried to use GHC’s plugin interface, to load the
module that would allow ghc to communicate with the runner on the device.

This in principle allows to have more dependencies on the plugin and not
force them into ghc. At the same time it requires the installation of some
additional hooks into the plugin system.

I guess one could also come up with a tiny proxy that pretends to be the
iserv endpoint and would forward anything over a network layer; this again
could probably work outside of ghc.

I'm pretty sure this can be done by adding networking capabilities to just iserv without adding networking capabilities to GHC itself.

However, I'm not sure if this could cause a linking problem if GHC requests iserv to load a different version of the "network" library than the one it was compiled against.

Cheers
Alberto

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling Template Haskell via -fexternal-interpreter and IPC

Alberto Valverde
In reply to this post by Simon Marlow-7
On Thu, Jul 7, 2016 at 9:30 AM, Simon Marlow <[hidden email]> wrote:
On 6 July 2016 at 18:14, Ben Gamari <[hidden email]> wrote:
It would be great if there were some way we could make this work with
named pipes. Not only does it side-step the dependency issue, but it
feels like the right way forward.

I agree, named pipes are probably a better plan, perhaps a better solution overall than the way we currently pass FD numbers on the command line.  Do named pipes work work as expected through wine?  We would have to be careful to clean them up again afterwards.

I've implemented IPC with named pipes and it appeared to work through wine but now I'm investigating an issue which causes GHC to "freeze" when talking to the external interpreter when it is not running on a TTY (ie: in a build process)

Cheers,
Alberto


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling Template Haskell via -fexternal-interpreter and IPC

Ben Gamari-3
Alberto Valverde <[hidden email]> writes:

> On Thu, Jul 7, 2016 at 9:30 AM, Simon Marlow <[hidden email]> wrote:
>
>> I agree, named pipes are probably a better plan, perhaps a better solution
>> overall than the way we currently pass FD numbers on the command line.  Do
>> named pipes work work as expected through wine?  We would have to be
>> careful to clean them up again afterwards.
>>
>
> I've implemented IPC with named pipes and it appeared to work through wine
> but now I'm investigating an issue which causes GHC to "freeze" when
> talking to the external interpreter when it is not running on a TTY (ie: in
> a build process)
Hi Alterto,

What ever happened to this? Is there any way I can be of assistance? It
would be great to get this merged.

Cheers,

- Ben

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

signature.asc (482 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling Template Haskell via -fexternal-interpreter and IPC

Alberto Valverde
Hi Ben,

I've just found this email from almost a year ago. Sorry for ignoring it until now. Given the date it was sent I think I missed it when it was fresh while I was on vacation.

Regarding the question, I have abandoned the work I was doing on getting cross-compilation for Windows to work since my employer decided a Windows build for the app we were developing could wait some time and tasked me we something else.

However, I see that Moritz is working on cross-compilation right now so I'll try get some time to see if I can get his approach to work for our use-case and give some feedback or help in any way.

Alberto

On Fri, Aug 26, 2016 at 4:56 PM, Ben Gamari <[hidden email]> wrote:
Alberto Valverde <[hidden email]> writes:

> On Thu, Jul 7, 2016 at 9:30 AM, Simon Marlow <[hidden email]> wrote:
>
>> I agree, named pipes are probably a better plan, perhaps a better solution
>> overall than the way we currently pass FD numbers on the command line.  Do
>> named pipes work work as expected through wine?  We would have to be
>> careful to clean them up again afterwards.
>>
>
> I've implemented IPC with named pipes and it appeared to work through wine
> but now I'm investigating an issue which causes GHC to "freeze" when
> talking to the external interpreter when it is not running on a TTY (ie: in
> a build process)

Hi Alterto,

What ever happened to this? Is there any way I can be of assistance? It
would be great to get this merged.

Cheers,

- Ben


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling Template Haskell via -fexternal-interpreter and IPC

Moritz Angermann
Hi Alberto,

let me know if I can be of help. As you likely saw I’ve been writing this
all up on https://medium.com/@zw3rk.  With the outstanding diffs[1], this
should hopefully just work.  As I haven’t used windows or know the linker
on windows, I’m not perfectly sure about the implications.  It could *just
work* though.  Maybe you can even use the GHCSlave’s Slave.hs[2] for the
Raspberry Pi for windows, as it doesn’t encode any Raspberry Pi specifics.

All you would need is to run GHCSlave on wine / or a windows system with
some form of network interface, so that the iserv-proxy and GHCSlave can
communicate.

Cheers,
 Moritz

[1]: you can just use the my-ghc branch from https://github.com/zw3rk/ghc
[2]: https://github.com/zw3rk/ghc-slave/blob/master/RPi/Slave.hs

> On May 27, 2017, at 4:29 PM, Alberto Valverde <[hidden email]> wrote:
>
> Hi Ben,
>
> I've just found this email from almost a year ago. Sorry for ignoring it until now. Given the date it was sent I think I missed it when it was fresh while I was on vacation.
>
> Regarding the question, I have abandoned the work I was doing on getting cross-compilation for Windows to work since my employer decided a Windows build for the app we were developing could wait some time and tasked me we something else.
>
> However, I see that Moritz is working on cross-compilation right now so I'll try get some time to see if I can get his approach to work for our use-case and give some feedback or help in any way.
>
> Alberto
>
> On Fri, Aug 26, 2016 at 4:56 PM, Ben Gamari <[hidden email]> wrote:
> Alberto Valverde <[hidden email]> writes:
>
> > On Thu, Jul 7, 2016 at 9:30 AM, Simon Marlow <[hidden email]> wrote:
> >
> >> I agree, named pipes are probably a better plan, perhaps a better solution
> >> overall than the way we currently pass FD numbers on the command line.  Do
> >> named pipes work work as expected through wine?  We would have to be
> >> careful to clean them up again afterwards.
> >>
> >
> > I've implemented IPC with named pipes and it appeared to work through wine
> > but now I'm investigating an issue which causes GHC to "freeze" when
> > talking to the external interpreter when it is not running on a TTY (ie: in
> > a build process)
>
> Hi Alterto,
>
> What ever happened to this? Is there any way I can be of assistance? It
> would be great to get this merged.
>
> Cheers,
>
> - Ben
>

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cross-compiling Template Haskell via -fexternal-interpreter and IPC

Alberto Valverde
Hi Moritz,

Sounds like it should be pretty straightforward to run GHCSlave on wine, so it might just work indeed. I'll build a nix environment for my app using your GHC branch and report back ASAP. However, what might block me won't be related to the cross-compilation per-se but to the migration to post 8.2.1 ghc since I'm still slowly making progress with all the dependencies but not there yet.

Cheers,
Alberto

On Sat, May 27, 2017 at 10:51 AM, Moritz Angermann <[hidden email]> wrote:
Hi Alberto,

let me know if I can be of help. As you likely saw I’ve been writing this
all up on https://medium.com/@zw3rk.  With the outstanding diffs[1], this
should hopefully just work.  As I haven’t used windows or know the linker
on windows, I’m not perfectly sure about the implications.  It could *just
work* though.  Maybe you can even use the GHCSlave’s Slave.hs[2] for the
Raspberry Pi for windows, as it doesn’t encode any Raspberry Pi specifics.

All you would need is to run GHCSlave on wine / or a windows system with
some form of network interface, so that the iserv-proxy and GHCSlave can
communicate.

Cheers,
 Moritz

[1]: you can just use the my-ghc branch from https://github.com/zw3rk/ghc
[2]: https://github.com/zw3rk/ghc-slave/blob/master/RPi/Slave.hs

> On May 27, 2017, at 4:29 PM, Alberto Valverde <[hidden email]> wrote:
>
> Hi Ben,
>
> I've just found this email from almost a year ago. Sorry for ignoring it until now. Given the date it was sent I think I missed it when it was fresh while I was on vacation.
>
> Regarding the question, I have abandoned the work I was doing on getting cross-compilation for Windows to work since my employer decided a Windows build for the app we were developing could wait some time and tasked me we something else.
>
> However, I see that Moritz is working on cross-compilation right now so I'll try get some time to see if I can get his approach to work for our use-case and give some feedback or help in any way.
>
> Alberto
>
> On Fri, Aug 26, 2016 at 4:56 PM, Ben Gamari <[hidden email]> wrote:
> Alberto Valverde <[hidden email]> writes:
>
> > On Thu, Jul 7, 2016 at 9:30 AM, Simon Marlow <[hidden email]> wrote:
> >
> >> I agree, named pipes are probably a better plan, perhaps a better solution
> >> overall than the way we currently pass FD numbers on the command line.  Do
> >> named pipes work work as expected through wine?  We would have to be
> >> careful to clean them up again afterwards.
> >>
> >
> > I've implemented IPC with named pipes and it appeared to work through wine
> > but now I'm investigating an issue which causes GHC to "freeze" when
> > talking to the external interpreter when it is not running on a TTY (ie: in
> > a build process)
>
> Hi Alterto,
>
> What ever happened to this? Is there any way I can be of assistance? It
> would be great to get this merged.
>
> Cheers,
>
> - Ben
>



_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Loading...