[GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
           Reporter:  dredozubov     |             Owner:  (none)
               Type:  bug            |            Status:  new
           Priority:  normal         |         Milestone:
          Component:  Compiler       |           Version:  8.2.1
  (Linking)                          |
           Keywords:                 |  Operating System:  MacOS X
       Architecture:                 |   Type of failure:  None/Unknown
  Unknown/Multiple                   |
          Test Case:                 |        Blocked By:
           Blocking:                 |   Related Tickets:
Differential Rev(s):                 |         Wiki Page:
-------------------------------------+-------------------------------------
 I'm opening a fresh ticket as @bgamari suggested in #12479. There are few
 more related closed tickets as well: #12198 and #12588. The issue occurs
 on projects with a lot of dependencies. There are reports of that
 happening across various projects:

 [https://github.com/NixOS/nixpkgs/issues/22810]
 [https://github.com/commercialhaskell/stack/issues/2577]

 I'm still able to reproduce it with 8.2.1 and git HEAD with a work
 project:
 {{{
     ghc: panic! (the 'impossible' happened)
       (GHC version 8.2.1 for x86_64-apple-darwin):
         Loading temp shared object failed:
 dlopen(/var/folders/f8/2_rc4tgd1gj9vbgv7q9gbk4c0000gn/T/ghc94377_0/libghc_325.dylib,
 5): no suitable image found.  Did find:
 /var/folders/f8/2_rc4tgd1gj9vbgv7q9gbk4c0000gn/T/ghc94377_0/libghc_325.dylib:
 malformed mach-o: load commands size (34592) > 32768

     Please report this as a GHC bug:
 http://www.haskell.org/ghc/reportabug
 }}}

 I can't share the sources, but this is a command(generated by stack) that
 results in this error:

 `/Users/dr/.stack/setup-exe-cache/x86_64-osx/Cabal-
 simple_mPHDZzAJ_2.0.0.2_ghc-8.2.1 --builddir=.stack-
 work/dist/x86_64-osx/Cabal-2.0.0.2 build lib:projectname exe:projectname
 --ghc-options " -ddump-hi -ddump-to-file -fdiagnostics-color=always"`

 We're having a chat about this issue with @bgamari and I'll post some of
 his input:
 {{{
  bgamari: dredozubov, unfortunately this is pretty much a limitation of OS
 X's linker
  bgamari: there's no great solution other than petitioning Apple to lift
 their arbitrary size limit
  bgamari: I've been asking people to open tickets with Apple
  bgamari: As they are really the only ones that can really fix this issue
  dredozubov: bgamari, do you know if someone did this already?
  bgamari: dredozubov, No one has said they have
  dredozubov: the other issue with it, I don't how to do a repro case
  dredozubov: I can reproduce it on a project with a closed sources and
 that's it
  bgamari: essentially you just need to build a project with enough
 dependencies
  bgamari: the problem is that Apple's linker sets an artificial cap on the
 number of shared libraries that an object file can load
 }}}

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  (none)
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by bgamari):

 For the record, angerman has suggested in the past that we leverage
 recursive linking to work around this limitation. He has an example
 [[https://github.com/angerman/dylib-linking|here]]. However, I'll admit
 I'm not terribly keen on spending our complexity budget working around
 arbitrary limitations imposed by Apple.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:1>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  (none)
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by angerman):

 I'll eventually look into the recursive linking part. I've added
 aggressive name mangling to cabal for macOS a while ago, which help
 elevate this issue to a certain part. But we will eventually be unable to
 _always_ stay under the limit.

 So a temporary work around (unless you really have excessive amounts of
 dependencies), might be to use a Cabal build from HEAD.
 I'm not certain how stack handles this and haven't looked into it.

 You are basically looking for any point after
 https://github.com/haskell/cabal/pull/4656 was merged. (3da83a0 was merged
 into haskell/cabal on Aug 22 2017).

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:2>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------
Changes (by angerman):

 * cc: angerman (added)
 * owner:  (none) => angerman


--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:3>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by shlevy):

 Note that in nixpkgs we have a solution that will always let us stay under
 the limit (essentially using intermediate libraries that reexport symbols
 from all the libraries underneath them)
 https://github.com/NixOS/nixpkgs/pull/27536

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:4>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------
Changes (by chak):

 * cc: chak (added)


Comment:

 > bgamari: I've been asking people to open tickets with Apple
 > bgamari: As they are really the only ones that can really fix this issue

 I am quite sure I know what Apple will respond: "You are holding it
 wrong."

 Insider jokes aside, the core problem (as I mentioned before on one of the
 other tickets) is that GHC tries to use the macOS linker like a Linux
 linker and it was never meant to be used like that. Hence, Apple will only
 tell you to use it how it was designed to be used. (This is why I haven't
 filled a Radar —Apple speak for bug report— against this issue myself.)

 As @angerman und @shlevy explained, there are alternatives to GHC's
 current scheme, which are closer to how library paths are usually managed
 on macOS. In addition, there is a reason why library names on macOS can
 include directory paths. In combination with @loader_path and
 @executable_path that always allowed to drastically shorten GHC's load
 command size. Haskell for Mac has done this for a long time (and I also
 mentioned this on a previous ticket) — see
 https://github.com/haskellformac/GHCframework/blob/master/GHCBuild/BuildGHC.sh
 for a messy shell script that is part of how the Haskell for Mac build
 handles this. (The other parts are in that repo as well.)

 What I am trying to say here is that Apple will not change this, as they
 do not consider it a problem caused by macOS, but as a problem caused by
 GHC's abuse of the MACH-O linker format. Hence, I am afraid, we will have
 to fix this on our side.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:5>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by angerman):

 chak, thank you for putting this out here as well!

 I still plan to fix this properly in GHC. I'm not sure I will be able to
 do this for 8.4.
 The "fix" in cabal was a quick hack to make cabal work, as I had a project
 which
 needed to compile and I could not bump GHC. As such cabal needed to learn
 to
 be a bit more conservative about RPTAHs.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:6>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by dredozubov):

 FWIW nix people use symlinks to lower the command size
 https://github.com/NixOS/nixpkgs/blob/a2bebdd6cedb58819cc1738282b9f02964d34a9a/pkgs/development
 /haskell-modules/generic-builder.nix#L239

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:7>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------
Changes (by lelf):

 * cc: lelf (added)


--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:8>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by aosivitz):

 So, I've recently run into this with a large project (using stack).

 I've tried to make a small reproduce case but have not succeeded. I
 generated a stack project that builds a library along with 400 dummy
 dependencies, each of which has a long package name. This is a larger
 number of dependencies than my original project, but it doesn't trigger
 the panic. So is there something else I need to do to cause this?

 Other funny things I've seen:
 Sometimes the build fails after `slack clean`, but then builds
 successfully after running it again. The big difference I see is that in
 the first case the link command is trying to build
 '/var/folders/.../ghc4371_0/libghc_100.dylib' when it fails, and in the
 second case it builds '.stack-work/.../libHSpanictest-0.0.0
 -8KaC2kNKjFj6cJfkbgZkev-ghc8.2.2.dylib' and succeeds. Is that just a quirk
 of stack or would it be a meaningful difference? What does it mean to
 build 'libghc_X.dylib'?

 I apologize if this info is too stack specific, but it's hard for me to
 tell where this problem actually comes from. Also, if anyone could advise
 me on improving my reproduction case that would be very helpful.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:9>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by angerman):

 So I believe this was mitigated in stack one way or the other; but your
 ticket commercialhaskell/stack#2577 makes it seem like it isn't. In cabal
 it is mitigated
 by haskell/cabal#4656.

 I guess we should just go ahead and fix it properly in GHC.  I feel GHC
 should fall over
 quicker if you construct a massive dependency tree with long folder and
 library names.

 Of course now that most people using cabal, nix or stack don't hit this
 issue anymore
 as hard, the pressure to fix this is not as high anymore.

 If you as such can reproduce a sample that always fails (it might be
 sufficient to double your dependencies)that would be helpful.

 Note however: we will not be able to fix this for the case where all
 dependencies are direct, without some heavier lifting. IOW if it's the
 indirect (transitive) dependencies that cause the blow up, we might be
 able to fix that rather easy. If it is the direct dependencies that cause
 the blow up, this will pose a harder problem as we'd have to potentially
 inject pseudo aggregation libraries, which I'd rather not do.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:10>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------
Changes (by aosivitz):

 * cc: aosivitz (added)


Comment:

 Sorry for the delay- I forgot to add myself to CC and didn't see your
 message.

 So I have now successfully reproduced the problem in a clean environment
 with bitemyapp's help. The missing piece is that it fails during template
 haskell compilation, but I'm a little confused at to why. It makes sense,
 of course, that template-haskell compilation would involve some dynamic
 linking in order to run the code, but if the number of dependencies is the
 problem, doesn't that imply TH links all of its dependencies dynamically?
 What is the advantage of dynamic linking here instead of just statically
 linking for TH?

 Here's a github gist for a shell script that generates a giant stack
 project (with 150 generated dependencies) that triggers the panic:
 https://gist.github.com/asivitz/f4b983b2374a6155ac4faaf9b61aca59
 I'm not sure the best way to do the same thing without using stack, but if
 you have an idea I can do that.

 Thanks!

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:11>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by aosivitz):

 Here's another idea: instead of dynamically linking each dep separately
 for TH codegen, we could combine all the modules into one and then
 dynamically link in that large blob.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:12>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by bgamari):

 > Here's another idea: instead of dynamically linking each dep separately
 for TH codegen, we could combine all the modules into one and then
 dynamically link in that large blob.

 I fear that may increase compilation time rather significantly, however.
 Linking tends to be quite expensive.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:13>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by aosivitz):

 Hm, so- as I understand it. We create a single libghc_XX.dylib for use by
 TH, which in turn references each of its dependencies and loads them with
 'load commands' (of which there is a hard total size limit on MacOS).
 What's the reason we take this step instead of just loading each
 dependency directly? Is that also for performance? If so, could we
 generate two of these libghc_XX.dylibs? Or 1 per every 50 dependencies?

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:14>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by bgamari):

 > instead of just loading each dependency directly?

 Do you mean directly calling `dlopen` on each dependency? I suppose we
 could do this and it might even work. The problem is that it does nothing
 for the case of executables.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:15>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by aosivitz):

 Yea, dlopen for each dependency.

 Are you saying there's still a problem with executables that this wouldn't
 fix? How can I reproduce that? I was only able to reproduce the panic
 using template haskell.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:16>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by aosivitz):

 Oh, I understand. By default GHC will statically link exes, but you can
 force it to be dynamic and in that case it will fail. Well, I personally
 am less concerned about that case because I'm not trying to dynamically
 link any exes, but I see why you would want both to work equally well.

 It seems to me like the TH problem (building an extra dynamic lib
 unnecessarily which happens to cause bad behavior on Mac) is separate from
 the dynamic exe problem (how can we dynamically link a bunch of libs into
 an exe on Mac). Is there some implementation reason why those two problems
 are more tied together than I assume?

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:17>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by angerman):

 > Here's a github gist for a shell script that generates a giant stack
 project (with 150 generated dependencies) that triggers the panic:
 ​https://gist.github.com/asivitz/f4b983b2374a6155ac4faaf9b61aca59 I'm not
 sure the best way to do the same thing without using stack, but if you
 have an idea I can do that.

 This will generate 150 direct dependencies. As such any dylib that will
 reference those, will have to reference 150 libraries, in the load
 commands section. I don't think we can do much about this case. If this
 was 150 transitive dependencies spread over a few levels. we might.

 For 150 direct, we could potentially only work around this by using the
 splitting approach from nix. For 150 transitive dependencies, I still
 believe recursive linking could solve this. (See
 https://github.com/angerman/dylib-linking)

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:18>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by jship):

 We have worked around this issue by following in the Nix folks' footsteps
 and using a wrapper script around `ld` to recursively subdivide the
 dependencies into a tree of re-exporting delegate libraries.

 The script, an example using aosivitz repro, and more info about this
 issue is available here:
 https://github.com/Simspace/ld-wrapper-macos

 The example in the repo shows how the wrapper script can be used to
 successfully build a project with 750 direct dependencies.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:19>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
12