Static values language extension proposal

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Static values language extension proposal

Boespflug, Mathieu
Hi Eric,

On Wed, Jan 29, 2014 at 3:20 AM, Erik de Castro Lopo
<[hidden email]> wrote:

> Mathieu Boespflug wrote:
>
>> thank you for the good points you raise. I'll try and address each of
>> them as best I can below.
>>
>> > 0) I think you could actually implement this proposal as a userland library,
>> > at least as you've described it. Have you tried doing so?
>>
>> Indeed, this could be done without touching the compiler at all.
>
> We had this response really early on in this discussion.
>
> Quite honestly I think that should have been the end of the discussion.

The response you quote above comes in context, which includes the
sentence you also quote below. In another email, the problems we face
with a pure TH implementation are labeled as 1a), 1b), 2). We'd be
very happy if you could show us how to solve those problems using TH
alone in a way that does not impact user friendliness and static
checking of invariants in any way.

> The existing GHC release already have a huge workload getting releases
> out the door and adding to that workload without adding manpower and
> resources would be a bad idea.
>
> You really should try doing this as a library outside of GHC and if GHC
> needs a few small additional features, they can be added.
>
>> The `static e` form could as well be a piece of Template Haskell, but
>> making it a proper extension means that the compiler can enforce more
>> invariants and be a bit more helpful to the user.
>
> Once it works outside GHC and has proven useful, then it might be worthwhile
> add small specific, easily testable/maintainable features to GHC to support
> what goes on on your library.

I for one very much agree with all the principles state above. But the
wider context of the discussion is that we already have such a TH
userland solution today, implemented in packages distributed-static
and distributed-process. We already have several users, including in
the industry (to my knowledge Parallel Scientific for over a year,
Tweag I/O for a couple of months, probably others...). The proposal to
go ahead and implement an idea that was first presented in the
original Cloud Haskell paper was borne out of frustration with the
existing approach based on remote tables, which are very error prone
in practice, and the operational experience that I, Facundo, Tim and
others have had showing that making the semantics of distributed
computation depend on *all* modules across several packages being
compiled with the right incantation of compiler flags without any kind
of static checking is a problem, especially for beginners.

Is there something in the proposed extension that leads you to believe
that is neither small nor specific, or that it would not be easily
testable, or maintainable? If so, we could amend it accordingly.

Best,

Mathieu
_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Static values language extension proposal

Austin Seipp-5
In reply to this post by Facundo Domínguez-3
Hello,

I'd just like to say I haven't gone over every discussion in this
thread and had time to digest it all - I thought I would just
highlight a minor technicality.

On Fri, Jan 24, 2014 at 11:19 AM, Facundo Domínguez
<[hidden email]> wrote:

> Looking up static references
> ====================
>
> `unstatic` is implemented as a function which finds a top-level value
> from the `GlobalName`, otherwise it raises an exception. It crucially
> relies on leveraging the system’s dynamic linker, so out-of-the-box
> only works with dynamically linked binaries (but see below).
> `unstatic` proceeds as follows:
>
>   * Determines the name of the shared library from the package name
> and the package version.
>
>   * Determines the symbol of the value by Z-Encoding the package name,
> the module name and the value name.
>
>   * Uses the system’s dynamic linker interface to obtain the address
> of the symbol.
>
>   * Converts the symbol to a haskell value with `GHC.Prim.addrToAny#`
>
> In principle, only symbols in shared libraries can be found. However,
> the dynamic linker is able to find symbols in modules that are linked
> statically if GHC is fed with the option -optl-Wl,--export-dynamic. A
> future enhancement could be to have GHC warn the user when modules
> using the extension are linked statically and this option is not used
> during linking.
>
> GHC only defines symbols for exported definitions in modules. So
> unstatic won’t be able to find the private bindings of a module. For
> this sake, the implementation of static should in addition ensure that
> the bindings it gets will appear in the symbol table when they are not
> exported by their defining modules.

Regarding -optl-Wl,--export-dynamic for static builds, and all that
jazz - if I am understanding you right, note that Windows is a bit
particular here, because there is a hard limit on the number of
symbols allowed in a DLL. That means forcefully exporting *everything*
could quickly get you to the symbol limit with the dynamic linker with
large-ish applications (one exported function or data type may result
in a handful of exported symbols created.) If you want to see the pain
this has caused GHC itself, please see GHC bug #5987[1], which makes
dynamic support on windows difficult - it's currently disabled now
anyway.

Furthermore, dynamic DLLs on Windows are a bit tricky anyway as the
loader is fundamentally different from your typical ld.so (which there
are ways around[2], but a bit nasty as you have to hack the COFF
file.) Windows unfortunately isn't in an easy position here, but it's
improving and it would be unfortunate to neglect it.

This restriction does not exist with the static linker inside the RTS,
so my suggestion, I guess, is that I'm inclined to want this to work
for *both* static/dynamic configurations out of the box without
hackery, if at all possible, which would be great for Windows users
especially until the dynamic story is back up to scratch.

[1] https://ghc.haskell.org/trac/ghc/ticket/5987
[2] http://blog.omega-prime.co.uk/?p=138

> As the application depends on shared libraries, now a tool to collect
> these libraries would be required so they can be distributed together
> with the executable binary when deploying a Cloud Haskell application
> in a cluster. We won’t delve further into this problem.

And for any people interested in this - on Linux, a tool like
patchelf[3] would help immensely for moving executables+their
dependencies around in a 'bundle' style way.

[3] http://nixos.org/patchelf.html

--
Regards,

Austin Seipp, Haskell Consultant
Well-Typed LLP, http://www.well-typed.com/
_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Static values language extension proposal

Boespflug, Mathieu
Hi Austin,

this is very useful information, thanks. So it seems that the rts
linker is here to stay for a while longer still, at least because
there is no good alternative on Windows as of yet.

If I understand you correctly, on Windows dynamic linking is not an
option in part because of the number-of-exported-symbols limit, and
when linking statically one hits the same limit if stuff like
-optl-Wl,--export-dynamic is used. So at least on Windows, the only
way out is the rts linker. Supporting both linkers is certainly an
option. If I remember correctly, the issue Facundo found with the rts
linker is that to use it for looking up symbol addresses, you
apparently need to have an object file loaded twice, effectively (once
statically linked at build time, the second time through the rts
linker at runtime for doing the lookups). Maybe there's a way around
that, or that could be added? In any case for platforms with no
alternative, like Windows, I guess double loading is a tolerable price
to pay.

Best,

Mathieu

On Wed, Jan 29, 2014 at 12:11 PM, Austin Seipp <[hidden email]> wrote:

> Hello,
>
> I'd just like to say I haven't gone over every discussion in this
> thread and had time to digest it all - I thought I would just
> highlight a minor technicality.
>
> On Fri, Jan 24, 2014 at 11:19 AM, Facundo Domínguez
> <[hidden email]> wrote:
>> Looking up static references
>> ====================
>>
>> `unstatic` is implemented as a function which finds a top-level value
>> from the `GlobalName`, otherwise it raises an exception. It crucially
>> relies on leveraging the system's dynamic linker, so out-of-the-box
>> only works with dynamically linked binaries (but see below).
>> `unstatic` proceeds as follows:
>>
>>   * Determines the name of the shared library from the package name
>> and the package version.
>>
>>   * Determines the symbol of the value by Z-Encoding the package name,
>> the module name and the value name.
>>
>>   * Uses the system's dynamic linker interface to obtain the address
>> of the symbol.
>>
>>   * Converts the symbol to a haskell value with `GHC.Prim.addrToAny#`
>>
>> In principle, only symbols in shared libraries can be found. However,
>> the dynamic linker is able to find symbols in modules that are linked
>> statically if GHC is fed with the option -optl-Wl,--export-dynamic. A
>> future enhancement could be to have GHC warn the user when modules
>> using the extension are linked statically and this option is not used
>> during linking.
>>
>> GHC only defines symbols for exported definitions in modules. So
>> unstatic won't be able to find the private bindings of a module. For
>> this sake, the implementation of static should in addition ensure that
>> the bindings it gets will appear in the symbol table when they are not
>> exported by their defining modules.
>
> Regarding -optl-Wl,--export-dynamic for static builds, and all that
> jazz - if I am understanding you right, note that Windows is a bit
> particular here, because there is a hard limit on the number of
> symbols allowed in a DLL. That means forcefully exporting *everything*
> could quickly get you to the symbol limit with the dynamic linker with
> large-ish applications (one exported function or data type may result
> in a handful of exported symbols created.) If you want to see the pain
> this has caused GHC itself, please see GHC bug #5987[1], which makes
> dynamic support on windows difficult - it's currently disabled now
> anyway.
>
> Furthermore, dynamic DLLs on Windows are a bit tricky anyway as the
> loader is fundamentally different from your typical ld.so (which there
> are ways around[2], but a bit nasty as you have to hack the COFF
> file.) Windows unfortunately isn't in an easy position here, but it's
> improving and it would be unfortunate to neglect it.
>
> This restriction does not exist with the static linker inside the RTS,
> so my suggestion, I guess, is that I'm inclined to want this to work
> for *both* static/dynamic configurations out of the box without
> hackery, if at all possible, which would be great for Windows users
> especially until the dynamic story is back up to scratch.
>
> [1] https://ghc.haskell.org/trac/ghc/ticket/5987
> [2] http://blog.omega-prime.co.uk/?p=138
>
>> As the application depends on shared libraries, now a tool to collect
>> these libraries would be required so they can be distributed together
>> with the executable binary when deploying a Cloud Haskell application
>> in a cluster. We won't delve further into this problem.
>
> And for any people interested in this - on Linux, a tool like
> patchelf[3] would help immensely for moving executables+their
> dependencies around in a 'bundle' style way.
>
> [3] http://nixos.org/patchelf.html
>
> --
> Regards,
>
> Austin Seipp, Haskell Consultant
> Well-Typed LLP, http://www.well-typed.com/



--
Mathieu Boespflug
Founder at http://tweag.io.
_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Static values language extension proposal

Carter Schonwald
In reply to this post by Erik de Castro Lopo-34
indeed! Thanks erik!

On the paralllel list, edkso shares with us a single commit that adds all the requested features as a user land lib


@tweag folks, please do not write personal attacks on the issue tracker, if you find yourself frustrated, I probably am too! please keep a positive constructive tone in all future communications.  





On Tue, Jan 28, 2014 at 9:20 PM, Erik de Castro Lopo <[hidden email]> wrote:
Mathieu Boespflug wrote:

> [Sorry for the multiple reposts - couldn't quite figure out which
> email address doesn't get refused by the list..]
>
>
> Hi Carter,
>
> thank you for the good points you raise. I'll try and address each of
> them as best I can below.
>
> > 0) I think you could actually implement this proposal as a userland library,
> > at least as you've described it. Have you tried doing so?
>
> Indeed, this could be done without touching the compiler at all.

We had this response really early on in this discussion.

Quite honestly I think that should have been the end of the discussion.

The existing GHC release already have a huge workload getting releases
out the door and adding to that workload without adding manpower and
resources would be a bad idea.

You really should try doing this as a library outside of GHC and if GHC
needs a few small additional features, they can be added.

> The `static e` form could as well be a piece of Template Haskell, but
> making it a proper extension means that the compiler can enforce more
> invariants and be a bit more helpful to the user.

Once it works outside GHC and has proven useful, then it might be worthwhile
add small specific, easily testable/maintainable features to GHC to support
what goes on on your library.

Erik
--
----------------------------------------------------------------------
Erik de Castro Lopo
http://www.mega-nerd.com/
_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users


_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Reply | Threaded
Open this post in threaded view
|

Re: Static values language extension proposal

Facundo Domínguez-3
For interested fellows, discussion also continues in [1] and [2].

Best,
Facundo

[1] https://ghc.haskell.org/trac/ghc/ticket/7015
[2] https://groups.google.com/d/topic/parallel-haskell/b-x7VmjlEOw/discussion

On Thu, Jan 30, 2014 at 4:47 PM, Carter Schonwald
<[hidden email]> wrote:

> indeed! Thanks erik!
>
> On the paralllel list, edkso shares with us a single commit that adds all
> the requested features as a user land lib
>
> https://github.com/haskell-distributed/distributed-static/commit/d2bd2ebca5a96ea5df621770e98bfb7a3b745bc7
>
> @tweag folks, please do not write personal attacks on the issue tracker, if
> you find yourself frustrated, I probably am too! please keep a positive
> constructive tone in all future communications.
>
>
>
>
>
> On Tue, Jan 28, 2014 at 9:20 PM, Erik de Castro Lopo <[hidden email]>
> wrote:
>>
>> Mathieu Boespflug wrote:
>>
>> > [Sorry for the multiple reposts - couldn't quite figure out which
>> > email address doesn't get refused by the list..]
>> >
>> >
>> > Hi Carter,
>> >
>> > thank you for the good points you raise. I'll try and address each of
>> > them as best I can below.
>> >
>> > > 0) I think you could actually implement this proposal as a userland
>> > > library,
>> > > at least as you've described it. Have you tried doing so?
>> >
>> > Indeed, this could be done without touching the compiler at all.
>>
>> We had this response really early on in this discussion.
>>
>> Quite honestly I think that should have been the end of the discussion.
>>
>> The existing GHC release already have a huge workload getting releases
>> out the door and adding to that workload without adding manpower and
>> resources would be a bad idea.
>>
>> You really should try doing this as a library outside of GHC and if GHC
>> needs a few small additional features, they can be added.
>>
>> > The `static e` form could as well be a piece of Template Haskell, but
>> > making it a proper extension means that the compiler can enforce more
>> > invariants and be a bit more helpful to the user.
>>
>> Once it works outside GHC and has proven useful, then it might be
>> worthwhile
>> add small specific, easily testable/maintainable features to GHC to
>> support
>> what goes on on your library.
>>
>> Erik
>> --
>> ----------------------------------------------------------------------
>> Erik de Castro Lopo
>> http://www.mega-nerd.com/
>> _______________________________________________
>> Glasgow-haskell-users mailing list
>> [hidden email]
>> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
>
>
>
> _______________________________________________
> Glasgow-haskell-users mailing list
> [hidden email]
> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
>
_______________________________________________
Glasgow-haskell-users mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
12