Why do we prevent static archives from being loaded when DYNAMIC_GHC_PROGRAMS=YES?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Why do we prevent static archives from being loaded when DYNAMIC_GHC_PROGRAMS=YES?

Moritz Angermann
Dear friends,

when we build GHC with DYNAMIC_GHC_PROGRAMS=YES, we essentially prevent ghc/ghci
from using archives (.a).  Is there a technical reason behind this?  The only
only reasoning so far I've came across was: insist on using dynamic/shared objects,
because the user said so when building GHC.

In that case, we don't however prevent GHC from building archive (static) only
libraries.  And as a consequence when we later try to build another archive of
a different library, that depends via TH on the former library, GHC will bail
and complain that we don't have the relevant dynamic/shared object.  Of course we
don't we explicitly didn't build it.  But the linker code we have in GHC is
perfectly capable of loading archives.  So why don't we want to fall back to
archives?

Similarly, as @deech asked on twitter[1], why we prevent GHCi from loading static
libraries?

I'd like to understand the technical reason/rational for this behavior.  Can
someone help me out here?  If there is no fundamental reason for this behavior,
I'd like to go ahead and try to lift it.

Thank you!

Cheers,
 Moritz

---
[1]: https://twitter.com/deech/status/1001182709555908608
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Why do we prevent static archives from being loaded when DYNAMIC_GHC_PROGRAMS=YES?

Simon Marlow-7
There's a technical restriction. The static code would be compiled with the small memory model, so it would have 32-bit relocations for external references, assuming that those references would resolve to something in the low 2GB of the address space. But we would be trying to link it against shared libraries which could be loaded anywhere in the address space.

If the static code was compiled with -fPIC then it might be possible, but there's also the restriction that we wouldn't be able to dlopen() a shared library that depends on the statically linked code, because the system linker can't see the symbols that the RTS linker has loaded. GHC doesn't currently know about this restriction, so it would probably go ahead and try, and things would break.

Cheers
Simon


On 29 May 2018 at 04:05, Moritz Angermann <[hidden email]> wrote:
Dear friends,

when we build GHC with DYNAMIC_GHC_PROGRAMS=YES, we essentially prevent ghc/ghci
from using archives (.a).  Is there a technical reason behind this?  The only
only reasoning so far I've came across was: insist on using dynamic/shared objects,
because the user said so when building GHC.

In that case, we don't however prevent GHC from building archive (static) only
libraries.  And as a consequence when we later try to build another archive of
a different library, that depends via TH on the former library, GHC will bail
and complain that we don't have the relevant dynamic/shared object.  Of course we
don't we explicitly didn't build it.  But the linker code we have in GHC is
perfectly capable of loading archives.  So why don't we want to fall back to
archives?

Similarly, as @deech asked on twitter[1], why we prevent GHCi from loading static
libraries?

I'd like to understand the technical reason/rational for this behavior.  Can
someone help me out here?  If there is no fundamental reason for this behavior,
I'd like to go ahead and try to lift it.

Thank you!

Cheers,
 Moritz

---
[1]: https://twitter.com/deech/status/1001182709555908608
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Why do we prevent static archives from being loaded when DYNAMIC_GHC_PROGRAMS=YES?

Phyx
You could work around the dlopen issue as long as the static library is compiled with -fPIC by using  --whole-archive (assuming you permit dangling references which will need to be resolved later) and making a shared library out of the static code. But you'd have to create one shared library per static library and preserve the order so you don't end up with symbol collisions. 

And you'd likely not want to do it this on every relink. But i think the - fPIC is a much greater hurdle. Very few of the static libraries a user may want to use would have this likely.

I think it'll end up being quite a messy situation.. 


On Thu, Jun 7, 2018, 22:01 Simon Marlow <[hidden email]> wrote:
There's a technical restriction. The static code would be compiled with the small memory model, so it would have 32-bit relocations for external references, assuming that those references would resolve to something in the low 2GB of the address space. But we would be trying to link it against shared libraries which could be loaded anywhere in the address space.

If the static code was compiled with -fPIC then it might be possible, but there's also the restriction that we wouldn't be able to dlopen() a shared library that depends on the statically linked code, because the system linker can't see the symbols that the RTS linker has loaded. GHC doesn't currently know about this restriction, so it would probably go ahead and try, and things would break.

Cheers
Simon


On 29 May 2018 at 04:05, Moritz Angermann <[hidden email]> wrote:
Dear friends,

when we build GHC with DYNAMIC_GHC_PROGRAMS=YES, we essentially prevent ghc/ghci
from using archives (.a).  Is there a technical reason behind this?  The only
only reasoning so far I've came across was: insist on using dynamic/shared objects,
because the user said so when building GHC.

In that case, we don't however prevent GHC from building archive (static) only
libraries.  And as a consequence when we later try to build another archive of
a different library, that depends via TH on the former library, GHC will bail
and complain that we don't have the relevant dynamic/shared object.  Of course we
don't we explicitly didn't build it.  But the linker code we have in GHC is
perfectly capable of loading archives.  So why don't we want to fall back to
archives?

Similarly, as @deech asked on twitter[1], why we prevent GHCi from loading static
libraries?

I'd like to understand the technical reason/rational for this behavior.  Can
someone help me out here?  If there is no fundamental reason for this behavior,
I'd like to go ahead and try to lift it.

Thank you!

Cheers,
 Moritz

---
[1]: https://twitter.com/deech/status/1001182709555908608
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Why do we prevent static archives from being loaded when DYNAMIC_GHC_PROGRAMS=YES?

Moritz Angermann-2
Thank you both for the replies.

My issue with the current situation is that I can navigate myself into a situation where I’m stuck. By asking ghc to build static libraries, it will later fall over when it tries to load those.

Guess what I really want is to turn the DYNAMIC_GHC_PROGRAMS into a runtime flag.

That might help with getting out of the situation without resorting to building two ghcs.

Cheers,
 Moritz

Sent from my iPhone

On 12 Jun 2018, at 9:07 PM, Phyx <[hidden email]> wrote:

You could work around the dlopen issue as long as the static library is compiled with -fPIC by using  --whole-archive (assuming you permit dangling references which will need to be resolved later) and making a shared library out of the static code. But you'd have to create one shared library per static library and preserve the order so you don't end up with symbol collisions. 

And you'd likely not want to do it this on every relink. But i think the - fPIC is a much greater hurdle. Very few of the static libraries a user may want to use would have this likely.

I think it'll end up being quite a messy situation.. 


On Thu, Jun 7, 2018, 22:01 Simon Marlow <[hidden email]> wrote:
There's a technical restriction. The static code would be compiled with the small memory model, so it would have 32-bit relocations for external references, assuming that those references would resolve to something in the low 2GB of the address space. But we would be trying to link it against shared libraries which could be loaded anywhere in the address space.

If the static code was compiled with -fPIC then it might be possible, but there's also the restriction that we wouldn't be able to dlopen() a shared library that depends on the statically linked code, because the system linker can't see the symbols that the RTS linker has loaded. GHC doesn't currently know about this restriction, so it would probably go ahead and try, and things would break.

Cheers
Simon


On 29 May 2018 at 04:05, Moritz Angermann <[hidden email]> wrote:
Dear friends,

when we build GHC with DYNAMIC_GHC_PROGRAMS=YES, we essentially prevent ghc/ghci
from using archives (.a).  Is there a technical reason behind this?  The only
only reasoning so far I've came across was: insist on using dynamic/shared objects,
because the user said so when building GHC.

In that case, we don't however prevent GHC from building archive (static) only
libraries.  And as a consequence when we later try to build another archive of
a different library, that depends via TH on the former library, GHC will bail
and complain that we don't have the relevant dynamic/shared object.  Of course we
don't we explicitly didn't build it.  But the linker code we have in GHC is
perfectly capable of loading archives.  So why don't we want to fall back to
archives?

Similarly, as @deech asked on twitter[1], why we prevent GHCi from loading static
libraries?

I'd like to understand the technical reason/rational for this behavior.  Can
someone help me out here?  If there is no fundamental reason for this behavior,
I'd like to go ahead and try to lift it.

Thank you!

Cheers,
 Moritz

---
[1]: https://twitter.com/deech/status/1001182709555908608
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Why do we prevent static archives from being loaded when DYNAMIC_GHC_PROGRAMS=YES?

Simon Marlow-7
I'm not sure how you could make DYNAMIC_GHC_PROGRAMS into a runtime flag, since it controls whether GHC itself (and the other tools) are built as dynamic executables. If GHC is a dynamic executable, then it can only load -fPIC code to link at runtime.

Cheers
Simon

On 13 June 2018 at 02:30, Moritz Angermann <[hidden email]> wrote:
Thank you both for the replies.

My issue with the current situation is that I can navigate myself into a situation where I’m stuck. By asking ghc to build static libraries, it will later fall over when it tries to load those.

Guess what I really want is to turn the DYNAMIC_GHC_PROGRAMS into a runtime flag.

That might help with getting out of the situation without resorting to building two ghcs.

Cheers,
 Moritz

Sent from my iPhone

On 12 Jun 2018, at 9:07 PM, Phyx <[hidden email]> wrote:

You could work around the dlopen issue as long as the static library is compiled with -fPIC by using  --whole-archive (assuming you permit dangling references which will need to be resolved later) and making a shared library out of the static code. But you'd have to create one shared library per static library and preserve the order so you don't end up with symbol collisions. 

And you'd likely not want to do it this on every relink. But i think the - fPIC is a much greater hurdle. Very few of the static libraries a user may want to use would have this likely.

I think it'll end up being quite a messy situation.. 


On Thu, Jun 7, 2018, 22:01 Simon Marlow <[hidden email]> wrote:
There's a technical restriction. The static code would be compiled with the small memory model, so it would have 32-bit relocations for external references, assuming that those references would resolve to something in the low 2GB of the address space. But we would be trying to link it against shared libraries which could be loaded anywhere in the address space.

If the static code was compiled with -fPIC then it might be possible, but there's also the restriction that we wouldn't be able to dlopen() a shared library that depends on the statically linked code, because the system linker can't see the symbols that the RTS linker has loaded. GHC doesn't currently know about this restriction, so it would probably go ahead and try, and things would break.

Cheers
Simon


On 29 May 2018 at 04:05, Moritz Angermann <[hidden email]> wrote:
Dear friends,

when we build GHC with DYNAMIC_GHC_PROGRAMS=YES, we essentially prevent ghc/ghci
from using archives (.a).  Is there a technical reason behind this?  The only
only reasoning so far I've came across was: insist on using dynamic/shared objects,
because the user said so when building GHC.

In that case, we don't however prevent GHC from building archive (static) only
libraries.  And as a consequence when we later try to build another archive of
a different library, that depends via TH on the former library, GHC will bail
and complain that we don't have the relevant dynamic/shared object.  Of course we
don't we explicitly didn't build it.  But the linker code we have in GHC is
perfectly capable of loading archives.  So why don't we want to fall back to
archives?

Similarly, as @deech asked on twitter[1], why we prevent GHCi from loading static
libraries?

I'd like to understand the technical reason/rational for this behavior.  Can
someone help me out here?  If there is no fundamental reason for this behavior,
I'd like to go ahead and try to lift it.

Thank you!

Cheers,
 Moritz

---
[1]: https://twitter.com/deech/status/1001182709555908608
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs


_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs