[GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
           Reporter:  dredozubov     |             Owner:  (none)
               Type:  bug            |            Status:  new
           Priority:  normal         |         Milestone:
          Component:  Compiler       |           Version:  8.2.1
  (Linking)                          |
           Keywords:                 |  Operating System:  MacOS X
       Architecture:                 |   Type of failure:  None/Unknown
  Unknown/Multiple                   |
          Test Case:                 |        Blocked By:
           Blocking:                 |   Related Tickets:
Differential Rev(s):                 |         Wiki Page:
-------------------------------------+-------------------------------------
 I'm opening a fresh ticket as @bgamari suggested in #12479. There are few
 more related closed tickets as well: #12198 and #12588. The issue occurs
 on projects with a lot of dependencies. There are reports of that
 happening across various projects:

 [https://github.com/NixOS/nixpkgs/issues/22810]
 [https://github.com/commercialhaskell/stack/issues/2577]

 I'm still able to reproduce it with 8.2.1 and git HEAD with a work
 project:
 {{{
     ghc: panic! (the 'impossible' happened)
       (GHC version 8.2.1 for x86_64-apple-darwin):
         Loading temp shared object failed:
 dlopen(/var/folders/f8/2_rc4tgd1gj9vbgv7q9gbk4c0000gn/T/ghc94377_0/libghc_325.dylib,
 5): no suitable image found.  Did find:
 /var/folders/f8/2_rc4tgd1gj9vbgv7q9gbk4c0000gn/T/ghc94377_0/libghc_325.dylib:
 malformed mach-o: load commands size (34592) > 32768

     Please report this as a GHC bug:
 http://www.haskell.org/ghc/reportabug
 }}}

 I can't share the sources, but this is a command(generated by stack) that
 results in this error:

 `/Users/dr/.stack/setup-exe-cache/x86_64-osx/Cabal-
 simple_mPHDZzAJ_2.0.0.2_ghc-8.2.1 --builddir=.stack-
 work/dist/x86_64-osx/Cabal-2.0.0.2 build lib:projectname exe:projectname
 --ghc-options " -ddump-hi -ddump-to-file -fdiagnostics-color=always"`

 We're having a chat about this issue with @bgamari and I'll post some of
 his input:
 {{{
  bgamari: dredozubov, unfortunately this is pretty much a limitation of OS
 X's linker
  bgamari: there's no great solution other than petitioning Apple to lift
 their arbitrary size limit
  bgamari: I've been asking people to open tickets with Apple
  bgamari: As they are really the only ones that can really fix this issue
  dredozubov: bgamari, do you know if someone did this already?
  bgamari: dredozubov, No one has said they have
  dredozubov: the other issue with it, I don't how to do a repro case
  dredozubov: I can reproduce it on a project with a closed sources and
 that's it
  bgamari: essentially you just need to build a project with enough
 dependencies
  bgamari: the problem is that Apple's linker sets an artificial cap on the
 number of shared libraries that an object file can load
 }}}

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  (none)
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by bgamari):

 For the record, angerman has suggested in the past that we leverage
 recursive linking to work around this limitation. He has an example
 [[https://github.com/angerman/dylib-linking|here]]. However, I'll admit
 I'm not terribly keen on spending our complexity budget working around
 arbitrary limitations imposed by Apple.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:1>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  (none)
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by angerman):

 I'll eventually look into the recursive linking part. I've added
 aggressive name mangling to cabal for macOS a while ago, which help
 elevate this issue to a certain part. But we will eventually be unable to
 _always_ stay under the limit.

 So a temporary work around (unless you really have excessive amounts of
 dependencies), might be to use a Cabal build from HEAD.
 I'm not certain how stack handles this and haven't looked into it.

 You are basically looking for any point after
 https://github.com/haskell/cabal/pull/4656 was merged. (3da83a0 was merged
 into haskell/cabal on Aug 22 2017).

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:2>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------
Changes (by angerman):

 * cc: angerman (added)
 * owner:  (none) => angerman


--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:3>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by shlevy):

 Note that in nixpkgs we have a solution that will always let us stay under
 the limit (essentially using intermediate libraries that reexport symbols
 from all the libraries underneath them)
 https://github.com/NixOS/nixpkgs/pull/27536

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:4>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------
Changes (by chak):

 * cc: chak (added)


Comment:

 > bgamari: I've been asking people to open tickets with Apple
 > bgamari: As they are really the only ones that can really fix this issue

 I am quite sure I know what Apple will respond: "You are holding it
 wrong."

 Insider jokes aside, the core problem (as I mentioned before on one of the
 other tickets) is that GHC tries to use the macOS linker like a Linux
 linker and it was never meant to be used like that. Hence, Apple will only
 tell you to use it how it was designed to be used. (This is why I haven't
 filled a Radar —Apple speak for bug report— against this issue myself.)

 As @angerman und @shlevy explained, there are alternatives to GHC's
 current scheme, which are closer to how library paths are usually managed
 on macOS. In addition, there is a reason why library names on macOS can
 include directory paths. In combination with @loader_path and
 @executable_path that always allowed to drastically shorten GHC's load
 command size. Haskell for Mac has done this for a long time (and I also
 mentioned this on a previous ticket) — see
 https://github.com/haskellformac/GHCframework/blob/master/GHCBuild/BuildGHC.sh
 for a messy shell script that is part of how the Haskell for Mac build
 handles this. (The other parts are in that repo as well.)

 What I am trying to say here is that Apple will not change this, as they
 do not consider it a problem caused by macOS, but as a problem caused by
 GHC's abuse of the MACH-O linker format. Hence, I am afraid, we will have
 to fix this on our side.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:5>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by angerman):

 chak, thank you for putting this out here as well!

 I still plan to fix this properly in GHC. I'm not sure I will be able to
 do this for 8.4.
 The "fix" in cabal was a quick hack to make cabal work, as I had a project
 which
 needed to compile and I could not bump GHC. As such cabal needed to learn
 to
 be a bit more conservative about RPTAHs.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:6>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by dredozubov):

 FWIW nix people use symlinks to lower the command size
 https://github.com/NixOS/nixpkgs/blob/a2bebdd6cedb58819cc1738282b9f02964d34a9a/pkgs/development
 /haskell-modules/generic-builder.nix#L239

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:7>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14444: Linker limit on OS X Sierra breaks builds for big projects

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14444: Linker limit on OS X Sierra breaks builds for big projects
-------------------------------------+-------------------------------------
        Reporter:  dredozubov        |                Owner:  angerman
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.2.1
  (Linking)                          |
      Resolution:                    |             Keywords:
Operating System:  MacOS X           |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------
Changes (by lelf):

 * cc: lelf (added)


--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14444#comment:8>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets