Out Of Process Template Haskell

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Out Of Process Template Haskell

Moritz Angermann
Hi,

as you may or may not know, template haskell is not available to cross compilers.  And
therefore cross compiler are devoid of an important feature of haskell.

  We are currently building ghc cross compiler as stage1 compiler with a foreign target.
This means that we use a stage0 compiler (the one that is used to build ghc), to build
the ghc that will run on the host and produce binaries for the target (e.g. running on
x86_64, building for arm).  Template Haskell and GHCi are enabled only in stage2.  To
build a stage2 compiler we use the stage1 compiler.  Hence the stage2 compiler runs
on the target of the stage1 compiler.  A motivating example could be ghc running on
an Intel based Mac OS X targeting iOS (ARM).  A *theoretical* stage2 compiler could have
GHCi and Template Haskell, but would run on the iOS Device, which is neat but probably
not a practical solution.  Hence a compiler running on a more powerful host, producing
executables for a foreign target can be a very good solution when the host is much more
powerful. (Ref. [1]). In [2] we can read the following:

> Stage 1 does not support interactive execution (GHCi) and Template Haskell. The reason
> being that when running byte code we must dynamically link the packages, and only in
> stage 2 and later can we guarantee that the packages we dynamically link are compatible
> with those that GHC was built against (because they are the very same packages).

  One of the prominent cross compilers is ghcjs, which admittedly is build a little
different than the above cross compiler would be.  ghcjs shares the same template haskell
issue though.  ghcjs--to my knowledge--followed a few different routes to solve this.  And
finally came up with the *out-of-process-template-haskell* solution.

  The Out Of Process Template Haskell approach works by having a *runner* on the target
that waits for the compiler to send the template haskell splices, and compiles those
splices on the target, whilte querying the *master* ghc for potential lookups during the
splice compilation.  The result is then shipped back to the host to integrate into
the produced binary for the target.  Hence allowing the more powerful host to do the
bulk of the compilation and using the *runner* as a *slave compiler* on the target.


     .-- host -.                                                        .- target -.
     |         |  -- ( encounters splice and sends it to runner ) --->  |          |
     |         |                                                        |          |
     |   ghc   | .<---- ( compiles and queries ghc for lookups ) ----.  |  runner  |
     |  master | '----------- ( responds to to queries ) ----------->'  |  slave   |
     |         |                                                        |          |
     |         | <- ( receives the compiled splice from the runner ) -  |          |
     '---------'                                                        '----------'

     Fig. 1: ghc - th runner communication.


  I have implemented the *out-of-process-template-haskell* approach with the help from
luite for ghc as a plugin for stage2 *non*-cross compiler[3] using dynamic libraries and
a tcp transport by adapting the code from ghcjs to run on the host.
Obviously for a regular stage2 ghc, which has GHCi and Template Haskell, this doesn't really
buy anyone anything yet.  Also a plugin is impossible to load into a stage1 compiler yet,
as the plugin code is wrapped in #ifdef GHCI sections.

  SIDENOTE: To allow the out-of-process-template-haskell, the plugin api had to be
            adapted minimally to allow plugins to install hooks.  Patches for 7.8[4]
            and 7.10[5] are readily available.
           

  Clearly the *out-of-process-template-haskell* could be integrated into the ghc tree
by hard wiring it in, where the plugin did the wiring ad-hoc previous.  This would imply
that ghc would have to integrate all dependencies which *out-of-process-template-haskell*
would bring with it.

  There are now three options discussed on #ghc so far:
a) Full integration in ghc, as described above.
b) Enabling the plugin interface in stage1 compiler [See Note 1 for details].
   This is probably only interesting for cross-compiler, where stage1 is the final
   stage on the host. [See Note 2 for an example]
c) As proposed by luite: integrate a thin layer in ghc, that uses pipes to communicate
   with an external program on the same host that ghc is running on.  That external
   program will then communicate with the *runner* on the target.

  The ultimate goal would be a multi-target compiler, that could produce a stage2 compiler
that runs on the host but can produce binaries for multiple targets of which the host
it's running on is one.

  I am in favor of extending the plugin api and making it available to stage1 compiler,
because I see much potential in the plugin api.  Which--with the noted patches [4][5]--
allows you to install hooks, and hence allow you to hook into the complete compiler
pipeline and potential other parts as well.  It also makes developing plugins easier,
because they can be developed outside of the ghc source tree.  And only be used with
cross compilers if we can load plugins in stage1 compilers.

Cheers,
 Moritz


[1]: https://ghc.haskell.org/trac/ghc/wiki/CrossCompilation
[2]: https://ghc.haskell.org/trac/ghc/wiki/Building/Architecture/Idiom/Stages
[3]: https://github.com/angerman/oopth
[4]: https://gist.github.com/angerman/7db11c24f8935c73fcf5
[5]: https://phabricator.haskell.org/D535
Note 1: where the plugin will have to be compiled with a stage0 compiler, and may even
        require a stage0 compiler that is as new as the stage1 compiler one wants to build,
        because the compiler being built has to be binary compatible with the plugin.
Note 2: An example would be building 7.10 on the host for the host with stage0=ghc7.8,
        stage1=ghc7.10(built with ghc7.8), stage2=ghc7.10(built with stage1), *and then*
        building a ghc cross compiler for arm with stage0'=stage2 and stage1'(ghc7.10 targeting
        ARM built with ghc7.10 on the host).  And allowing this newly built cross compiler
        (stage1') to accept -fplugin for plugins built with stage2 (the same that built stage1')

Reply | Threaded
Open this post in threaded view
|

Out Of Process Template Haskell

Simon Peyton Jones
Luite's "out of process TH" work is amazing.  If someone had suggested it to me I'd have said "it sounds entirely impractical", but he showed that I would have been quite wrong.

I'm open to fixing up GHC, preferably via some plugin variations, to support this.  (I would prefer not to add a whole batch of new dependencies.)

If a little sub-group would like to:
 * Articulate their preferred design (from a user point of view)
   on a wiki page
 * Sketch the implementation plan
 * Build a patch
and get consensus on all of that, let's go for it.  I guess that means at least Luite and Moritz; I'm not sure who else is keen here.

The ball is in your court.

Thank you!

Simon

|  -----Original Message-----
|  From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of
|  Moritz Angermann
|  Sent: 03 December 2014 21:24
|  To: ghc-devs
|  Subject: Out Of Process Template Haskell
|  
|  Hi,
|  
|  as you may or may not know, template haskell is not available to cross
|  compilers.  And therefore cross compiler are devoid of an important
|  feature of haskell.
|  
|    We are currently building ghc cross compiler as stage1 compiler with
|  a foreign target.
|  This means that we use a stage0 compiler (the one that is used to
|  build ghc), to build the ghc that will run on the host and produce
|  binaries for the target (e.g. running on x86_64, building for arm).
|  Template Haskell and GHCi are enabled only in stage2.  To build a
|  stage2 compiler we use the stage1 compiler.  Hence the stage2 compiler
|  runs on the target of the stage1 compiler.  A motivating example could
|  be ghc running on an Intel based Mac OS X targeting iOS (ARM).  A
|  *theoretical* stage2 compiler could have GHCi and Template Haskell,
|  but would run on the iOS Device, which is neat but probably not a
|  practical solution.  Hence a compiler running on a more powerful host,
|  producing executables for a foreign target can be a very good solution
|  when the host is much more powerful. (Ref. [1]). In [2] we can read
|  the following:
|  
|  > Stage 1 does not support interactive execution (GHCi) and Template
|  > Haskell. The reason being that when running byte code we must
|  > dynamically link the packages, and only in stage 2 and later can we
|  > guarantee that the packages we dynamically link are compatible with
|  those that GHC was built against (because they are the very same
|  packages).
|  
|    One of the prominent cross compilers is ghcjs, which admittedly is
|  build a little different than the above cross compiler would be.
|  ghcjs shares the same template haskell issue though.  ghcjs--to my
|  knowledge--followed a few different routes to solve this.  And finally
|  came up with the *out-of-process-template-haskell* solution.
|  
|    The Out Of Process Template Haskell approach works by having a
|  *runner* on the target that waits for the compiler to send the
|  template haskell splices, and compiles those splices on the target,
|  whilte querying the *master* ghc for potential lookups during the
|  splice compilation.  The result is then shipped back to the host to
|  integrate into the produced binary for the target.  Hence allowing the
|  more powerful host to do the bulk of the compilation and using the
|  *runner* as a *slave compiler* on the target.
|  
|  
|       .-- host -.
|  .- target -.
|       |         |  -- ( encounters splice and sends it to runner ) --->
|  |          |
|       |         |
|  |          |
|       |   ghc   | .<---- ( compiles and queries ghc for lookups ) ----.
|  |  runner  |
|       |  master | '----------- ( responds to to queries ) ----------->'
|  |  slave   |
|       |         |
|  |          |
|       |         | <- ( receives the compiled splice from the runner ) -
|  |          |
|       '---------'
|  '----------'
|  
|       Fig. 1: ghc - th runner communication.
|  
|  
|    I have implemented the *out-of-process-template-haskell* approach
|  with the help from luite for ghc as a plugin for stage2 *non*-cross
|  compiler[3] using dynamic libraries and a tcp transport by adapting
|  the code from ghcjs to run on the host.
|  Obviously for a regular stage2 ghc, which has GHCi and Template
|  Haskell, this doesn't really buy anyone anything yet.  Also a plugin
|  is impossible to load into a stage1 compiler yet, as the plugin code
|  is wrapped in #ifdef GHCI sections.
|  
|    SIDENOTE: To allow the out-of-process-template-haskell, the plugin
|  api had to be
|              adapted minimally to allow plugins to install hooks.
|  Patches for 7.8[4]
|              and 7.10[5] are readily available.
|  
|  
|    Clearly the *out-of-process-template-haskell* could be integrated
|  into the ghc tree by hard wiring it in, where the plugin did the
|  wiring ad-hoc previous.  This would imply that ghc would have to
|  integrate all dependencies which *out-of-process-template-haskell*
|  would bring with it.
|  
|    There are now three options discussed on #ghc so far:
|  a) Full integration in ghc, as described above.
|  b) Enabling the plugin interface in stage1 compiler [See Note 1 for
|  details].
|     This is probably only interesting for cross-compiler, where stage1
|  is the final
|     stage on the host. [See Note 2 for an example]
|  c) As proposed by luite: integrate a thin layer in ghc, that uses
|  pipes to communicate
|     with an external program on the same host that ghc is running on.
|  That external
|     program will then communicate with the *runner* on the target.
|  
|    The ultimate goal would be a multi-target compiler, that could
|  produce a stage2 compiler that runs on the host but can produce
|  binaries for multiple targets of which the host it's running on is
|  one.
|  
|    I am in favor of extending the plugin api and making it available to
|  stage1 compiler, because I see much potential in the plugin api.
|  Which--with the noted patches [4][5]-- allows you to install hooks,
|  and hence allow you to hook into the complete compiler pipeline and
|  potential other parts as well.  It also makes developing plugins
|  easier, because they can be developed outside of the ghc source tree.
|  And only be used with cross compilers if we can load plugins in stage1
|  compilers.
|  
|  Cheers,
|   Moritz
|  
|  
|  [1]: https://ghc.haskell.org/trac/ghc/wiki/CrossCompilation
|  [2]:
|  https://ghc.haskell.org/trac/ghc/wiki/Building/Architecture/Idiom/Stag
|  es
|  [3]: https://github.com/angerman/oopth
|  [4]: https://gist.github.com/angerman/7db11c24f8935c73fcf5
|  [5]: https://phabricator.haskell.org/D535
|  Note 1: where the plugin will have to be compiled with a stage0
|  compiler, and may even
|          require a stage0 compiler that is as new as the stage1
|  compiler one wants to build,
|          because the compiler being built has to be binary compatible
|  with the plugin.
|  Note 2: An example would be building 7.10 on the host for the host
|  with stage0=ghc7.8,
|          stage1=ghc7.10(built with ghc7.8), stage2=ghc7.10(built with
|  stage1), *and then*
|          building a ghc cross compiler for arm with stage0'=stage2 and
|  stage1'(ghc7.10 targeting
|          ARM built with ghc7.10 on the host).  And allowing this newly
|  built cross compiler
|          (stage1') to accept -fplugin for plugins built with stage2
|  (the same that built stage1')
|  _______________________________________________
|  ghc-devs mailing list
|  ghc-devs at haskell.org
|  http://www.haskell.org/mailman/listinfo/ghc-devs

Reply | Threaded
Open this post in threaded view
|

Out Of Process Template Haskell

Luite Stegeman
On Thu, Dec 4, 2014 at 12:07 PM, Simon Peyton Jones <simonpj at microsoft.com>
wrote:

> Luite's "out of process TH" work is amazing.  If someone had suggested it
> to me I'd have said "it sounds entirely impractical", but he showed that I
> would have been quite wrong.
>
>
I haven't had much time to actually help Moritz lately, due to other GHC
7.10 patches, the new GHCJS codegen and work to get Cabal support for GHCJS
merged.

Building/linking the code for the splices can be done in GHC without
additional dependencies, and with the recent changes in the
template-haskell package (if I haven't missed anything), serialization for
communication can also be implemented in a ligthtweight way based on
Generic or Data without additional dependencies.

I'd like to start simple and treat the Template Haskell runner as an
external build tool, possibly installable as a cabal package. GHC would
just start a runner process and communicate with it over the standard pipes
whenever it needs to run TH. The runner is then responsible for loading and
executing the code (in most cases a dynamic library for each splice) on the
target.

I intend to start by making a patch that adds the runner to the 'settings'
file, such that users can enable TH in a stage1 (cross) compiler by
pointing GHC to the correct runner program. Other than the minor changes in
SysTools (for 'settings'), the code for this would be common to all
implementation alternatives.

We can then figure out whether this approach is flexible enough and further
work out configuration options and serialization protocol, or whether
Moritz' idea of loading a plugin to have more control over TH code
generation is worth the additional complexity (a different loading
mechanism would be required for plugins in cross compilers).

luite
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20141204/22d27077/attachment.html>

Reply | Threaded
Open this post in threaded view
|

Out Of Process Template Haskell

Sean Seefried-2
I'm not a GHC dev, but I have been following the list. I recently managed
to build and run a small game written in Haskell on Android. I am also
interested in iOS development. Being able to use Template Haskell inside a
cross-compiler would allow me to use Manuel Chakravarty's
'language-c-inline' package which makes heavy use of it.

I'm very excited by your work and would be interested in testing any
solutions you came up with.

Sean

On 5 December 2014 at 01:21, Luite Stegeman <stegeman at gmail.com> wrote:

>
>
> On Thu, Dec 4, 2014 at 12:07 PM, Simon Peyton Jones <simonpj at microsoft.com
> > wrote:
>
>> Luite's "out of process TH" work is amazing.  If someone had suggested it
>> to me I'd have said "it sounds entirely impractical", but he showed that I
>> would have been quite wrong.
>>
>>
> I haven't had much time to actually help Moritz lately, due to other GHC
> 7.10 patches, the new GHCJS codegen and work to get Cabal support for GHCJS
> merged.
>
> Building/linking the code for the splices can be done in GHC without
> additional dependencies, and with the recent changes in the
> template-haskell package (if I haven't missed anything), serialization for
> communication can also be implemented in a ligthtweight way based on
> Generic or Data without additional dependencies.
>
> I'd like to start simple and treat the Template Haskell runner as an
> external build tool, possibly installable as a cabal package. GHC would
> just start a runner process and communicate with it over the standard pipes
> whenever it needs to run TH. The runner is then responsible for loading and
> executing the code (in most cases a dynamic library for each splice) on the
> target.
>
> I intend to start by making a patch that adds the runner to the 'settings'
> file, such that users can enable TH in a stage1 (cross) compiler by
> pointing GHC to the correct runner program. Other than the minor changes in
> SysTools (for 'settings'), the code for this would be common to all
> implementation alternatives.
>
> We can then figure out whether this approach is flexible enough and
> further work out configuration options and serialization protocol, or
> whether Moritz' idea of loading a plugin to have more control over TH code
> generation is worth the additional complexity (a different loading
> mechanism would be required for plugins in cross compilers).
>
> luite
>
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://www.haskell.org/mailman/listinfo/ghc-devs
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20141205/eec418e7/attachment-0001.html>