GHC exceeding command line length limit with split-objs - and a possible fix

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

GHC exceeding command line length limit with split-objs - and a possible fix

Eugene Kirpichov
Hi,

I'm building gtk2hs on a mac with -fsplit-objs, and during compilation of the largest file, which produces about 7000 split objects, the linking phase fails.
I'm assuming that's because the command line length has been exceeded, because other files compile fine, without -fsplit-objs it compiles fine too, and it compiles fine *with* -fsplit-objs on other OS - so perhaps the reason is in mac's tempfile name generation (they're longer than on other platforms) or something.

Anyway.

I think a nice fix would be to employ gcc's ability to read options from a file - gcc @file - and write overly long option strings into temp files.

It'd be fun for me to implement this as my first contribution to ghc; is this a good idea in general?

--
Eugene Kirpichov
Principal Engineer, Mirantis Inc. http://www.mirantis.com/
Editor, http://fprog.ru/

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: GHC exceeding command line length limit with split-objs - and a possible fix

Eugene Kirpichov
Oh well... looks like building ghc won't be easy, as it doesn't build with llvm-gcc and it's not easy to get a "real" gcc on Lion. But I don't stop trying :)

On Wed, Jan 11, 2012 at 11:12 AM, Eugene Kirpichov <[hidden email]> wrote:
Hi,

I'm building gtk2hs on a mac with -fsplit-objs, and during compilation of the largest file, which produces about 7000 split objects, the linking phase fails.
I'm assuming that's because the command line length has been exceeded, because other files compile fine, without -fsplit-objs it compiles fine too, and it compiles fine *with* -fsplit-objs on other OS - so perhaps the reason is in mac's tempfile name generation (they're longer than on other platforms) or something.

Anyway.

I think a nice fix would be to employ gcc's ability to read options from a file - gcc @file - and write overly long option strings into temp files.

It'd be fun for me to implement this as my first contribution to ghc; is this a good idea in general?

--
Eugene Kirpichov
Principal Engineer, Mirantis Inc. http://www.mirantis.com/
Editor, http://fprog.ru/



--
Eugene Kirpichov
Principal Engineer, Mirantis Inc. http://www.mirantis.com/
Editor, http://fprog.ru/

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: GHC exceeding command line length limit with split-objs - and a possible fix

John Lato-2
In reply to this post by Eugene Kirpichov
I used https://github.com/kennethreitz/osx-gcc-installer/downloads to
get a real gcc on Lion.  Biggish download, but it worked.  I've also
seen reports of success by self-compiling gcc, or by installing XCode
4 on top of an existing XCode 3 installation.

John L.

> From: Eugene Kirpichov <[hidden email]>
>
> Oh well... looks like building ghc won't be easy, as it doesn't build with
> llvm-gcc and it's not easy to get a "real" gcc on Lion. But I don't stop
> trying :)

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: GHC exceeding command line length limit with split-objs - and a possible fix

Hans Aberg-2
On 11 Jan 2012, at 13:38, John Lato wrote:

> I used https://github.com/kennethreitz/osx-gcc-installer/downloads to
> get a real gcc on Lion.  Biggish download, but it worked.  I've also
> seen reports of success by self-compiling gcc, or by installing XCode
> 4 on top of an existing XCode 3 installation.

GCC 4.6.2 builds on OS X 10.7 with Xcode 4.2. Build GMP 5.0.2 with clang, and then GCC with /usr/bin/gcc -> llvm-gcc-4.2. I think I saw someone guild GCC 4.7, but this is highest stable.

Hans

>> From: Eugene Kirpichov <[hidden email]>
>>
>> Oh well... looks like building ghc won't be easy, as it doesn't build with
>> llvm-gcc and it's not easy to get a "real" gcc on Lion. But I don't stop
>> trying :)



_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: GHC exceeding command line length limit with split-objs - and a possible fix

Eugene Kirpichov
Thanks, looks like I already succeeded by downloading xcode 3.

Now my original question remains - is such a change a good idea?
(I've already found the place in code where the fix has to be made; should take an hour of work at most)

On Wed, Jan 11, 2012 at 6:07 PM, Hans Aberg <[hidden email]> wrote:
On 11 Jan 2012, at 13:38, John Lato wrote:

> I used https://github.com/kennethreitz/osx-gcc-installer/downloads to
> get a real gcc on Lion.  Biggish download, but it worked.  I've also
> seen reports of success by self-compiling gcc, or by installing XCode
> 4 on top of an existing XCode 3 installation.

GCC 4.6.2 builds on OS X 10.7 with Xcode 4.2. Build GMP 5.0.2 with clang, and then GCC with /usr/bin/gcc -> llvm-gcc-4.2. I think I saw someone guild GCC 4.7, but this is highest stable.

Hans

>> From: Eugene Kirpichov <[hidden email]>
>>
>> Oh well... looks like building ghc won't be easy, as it doesn't build with
>> llvm-gcc and it's not easy to get a "real" gcc on Lion. But I don't stop
>> trying :)





--
Eugene Kirpichov
Principal Engineer, Mirantis Inc. http://www.mirantis.com/
Editor, http://fprog.ru/

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: GHC exceeding command line length limit with split-objs - and a possible fix

Ozgur Akgun
Hi Eugene, 

I think I did run into this problem before, and had to turn of split-objs temporarily to work around it. I'd appreciate a fix.

Best,
Ozgur

On 11 January 2012 14:14, Eugene Kirpichov <[hidden email]> wrote:
Now my original question remains - is such a change a good idea?
(I've already found the place in code where the fix has to be made; should take an hour of work at most)
 

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: GHC exceeding command line length limit with split-objs - and a possible fix

Brandon Allbery
In reply to this post by Eugene Kirpichov
On Wed, Jan 11, 2012 at 02:12, Eugene Kirpichov <[hidden email]> wrote:
I think a nice fix would be to employ gcc's ability to read options from a file - gcc @file - and write overly long option strings into temp files.

What immediately occurs to me is, what if the problem (or another manifestation of same) occurs in the ld step?  OS X's ld(1) doesn't have such an option /per se/, and binutils ld(1) does not reliably create valid Mach-O objects.

I would consider "batching" split-objs files into static archives (see ar(1) and ranlib(1)).  This also has the advantages of being portable (other platforms other have length limits; I believe it's the main reason split-objs is disabled by default on e.g. Solaris) and that with many linkers it's faster than individual objects because it can use the archive table of contents to speed up searching for files and symbols.

--
brandon s allbery                                      [hidden email]
wandering unix systems administrator (available)     (412) 475-9364 vm/sms


_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: GHC exceeding command line length limit with split-objs - and a possible fix

Eugene Kirpichov
Hi Brandon,

Thanks - looks like this would be a modification of the linking stage, splitting it into two parts: batching objects and then actually linking them.

Do you think that what I originally proposed is still a good thing to have before implementing your solution? (it definitely would be for myself, as it's easier to do and I'd then quicker become able to build my application with split-objs and shrink its executable size from 11 to 2Mb)

On Wed, Jan 11, 2012 at 9:48 PM, Brandon Allbery <[hidden email]> wrote:
On Wed, Jan 11, 2012 at 02:12, Eugene Kirpichov <[hidden email]> wrote:
I think a nice fix would be to employ gcc's ability to read options from a file - gcc @file - and write overly long option strings into temp files.

What immediately occurs to me is, what if the problem (or another manifestation of same) occurs in the ld step?  OS X's ld(1) doesn't have such an option /per se/, and binutils ld(1) does not reliably create valid Mach-O objects.

I would consider "batching" split-objs files into static archives (see ar(1) and ranlib(1)).  This also has the advantages of being portable (other platforms other have length limits; I believe it's the main reason split-objs is disabled by default on e.g. Solaris) and that with many linkers it's faster than individual objects because it can use the archive table of contents to speed up searching for files and symbols.

--
brandon s allbery                                      [hidden email]
wandering unix systems administrator (available)     <a href="tel:%28412%29%20475-9364" value="+14124759364" target="_blank">(412) 475-9364 vm/sms




--
Eugene Kirpichov
Principal Engineer, Mirantis Inc. http://www.mirantis.com/
Editor, http://fprog.ru/

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: GHC exceeding command line length limit with split-objs - and a possible fix

Brandon Allbery
On Wed, Jan 11, 2012 at 14:50, Eugene Kirpichov <[hidden email]> wrote:
Do you think that what I originally proposed is still a good thing to have before implementing your solution? (it definitely would be for myself, as it's easier to do and I'd then quicker become able to build my application with split-objs and shrink its executable size from 11 to 2Mb)

Only if it works; my expectation is that all you will accomplish is to move the failure point, as the command line for ld is usually larger than the one for gcc.  It is possible that gcc will do something to deal with it itself, but I have a suspicion that only works (if it does) with binutils ld, which is not usable on OS X.

BTW, I'm actually suggesting that split-objs create an archive, where non-split-objs creates an object; from there I suspect you wouldn't actually need to handle anything any differently between the two cases (aside from specifying the .a instead of the .o as needed).

-- 
brandon s allbery                                      [hidden email]
wandering unix systems administrator (available)     (412) 475-9364 vm/sms


_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe