ghc leaking memory?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

ghc leaking memory?

Joachim Breitner
Hi,

in Debian, we are having some problems building large libraries (such as
agda, highlighting-kate, xmonad-contrib) on weaker architectures (sparc,
armel, s390) which compile with -fvia-C, especially when building the
profiling libraries.

In the case of highlighting-kate, there were really issues with the
code: A huge list of words which ghc seemed to try to inline caused it
consuming a lot of memory.

But in the case of agda, it even fails with relatively small modules:
https://buildd.debian.org/fetch.cgi?pkg=agda&arch=sparc&ver=2.2.6-3&stamp=1267334936&file=log&as=raw

Now, looking at the memory gauge on my computer while compiling such a
program with Cabal (which compiles all modules in one ghc call), it
looks like it is steadily increasing. I’d expect ghc to free a lot of
memory when it is done with a module, but this does not seem to be the
case.

Is this a bug (i.e. memory leak) in ghc or expected behavior?

Is there a way to make Cabal build each module on its own?

Are there any fixes or work-arounds?

Greetings,
Joachim

--
Joachim "nomeata" Breitner
Debian Developer
  [hidden email] | ICQ# 74513189 | GPG-Keyid: 4743206C
  JID: [hidden email] | http://people.debian.org/~nomeata

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: ghc leaking memory?

Antoine Latter-2
Including ghc-users.

On Sat, Mar 6, 2010 at 11:57 AM, Joachim Breitner <[hidden email]> wrote:

> Hi,
>
> in Debian, we are having some problems building large libraries (such as
> agda, highlighting-kate, xmonad-contrib) on weaker architectures (sparc,
> armel, s390) which compile with -fvia-C, especially when building the
> profiling libraries.
>
> In the case of highlighting-kate, there were really issues with the
> code: A huge list of words which ghc seemed to try to inline caused it
> consuming a lot of memory.
>
> But in the case of agda, it even fails with relatively small modules:
> https://buildd.debian.org/fetch.cgi?pkg=agda&arch=sparc&ver=2.2.6-3&stamp=1267334936&file=log&as=raw
>
> Now, looking at the memory gauge on my computer while compiling such a
> program with Cabal (which compiles all modules in one ghc call), it
> looks like it is steadily increasing. I’d expect ghc to free a lot of
> memory when it is done with a module, but this does not seem to be the
> case.
>
> Is this a bug (i.e. memory leak) in ghc or expected behavior?
>
> Is there a way to make Cabal build each module on its own?
>
> Are there any fixes or work-arounds?
>
> Greetings,
> Joachim
>
> --
> Joachim "nomeata" Breitner
> Debian Developer
>  [hidden email] | ICQ# 74513189 | GPG-Keyid: 4743206C
>  JID: [hidden email] | http://people.debian.org/~nomeata
>
> _______________________________________________
> Haskell-Cafe mailing list
> [hidden email]
> http://www.haskell.org/mailman/listinfo/haskell-cafe
>
>
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: ghc leaking memory?

Don Stewart-2
Sounds like the known issue of ghc --make (the compilation manager)
retaining meta data (like .hi file info) when compiling many modules.
This isn't the case with one-shot compiltation.

I'm not sure there's anything you can do about it.

aslatter:
> Including ghc-users.
>
> On Sat, Mar 6, 2010 at 11:57 AM, Joachim Breitner <[hidden email]> wrote:
> > Hi,
> >
> > in Debian, we are having some problems building large libraries (such as
> > agda, highlighting-kate, xmonad-contrib) on weaker architectures (sparc,
> > armel, s390) which compile with -fvia-C, especially when building the
> > profiling libraries.
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: ghc leaking memory?

Ivan Lazar Miljenovic
In reply to this post by Joachim Breitner
Joachim Breitner <[hidden email]> writes:
> In the case of highlighting-kate, there were really issues with the
> code: A huge list of words which ghc seemed to try to inline caused it
> consuming a lot of memory.

This is a known problem; so much so that John is writing a new library
illuminate [1] to replace it.

[1] http://github.com/jgm/illuminate

--
Ivan Lazar Miljenovic
[hidden email]
IvanMiljenovic.wordpress.com
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: ghc leaking memory?

Simon Marlow-7
In reply to this post by Antoine Latter-2
On 06/03/2010 18:43, Antoine Latter wrote:

> Including ghc-users.
>
> On Sat, Mar 6, 2010 at 11:57 AM, Joachim Breitner<[hidden email]>  wrote:
>> Hi,
>>
>> in Debian, we are having some problems building large libraries (such as
>> agda, highlighting-kate, xmonad-contrib) on weaker architectures (sparc,
>> armel, s390) which compile with -fvia-C, especially when building the
>> profiling libraries.
>>
>> In the case of highlighting-kate, there were really issues with the
>> code: A huge list of words which ghc seemed to try to inline caused it
>> consuming a lot of memory.
>>
>> But in the case of agda, it even fails with relatively small modules:
>> https://buildd.debian.org/fetch.cgi?pkg=agda&arch=sparc&ver=2.2.6-3&stamp=1267334936&file=log&as=raw
>>
>> Now, looking at the memory gauge on my computer while compiling such a
>> program with Cabal (which compiles all modules in one ghc call), it
>> looks like it is steadily increasing. I’d expect ghc to free a lot of
>> memory when it is done with a module, but this does not seem to be the
>> case.
>>
>> Is this a bug (i.e. memory leak) in ghc or expected behavior?

Can you verify that it is GHC leaking memory, and not gcc?  I recommend
turning off -fvia-C in particular unless you're sure that you need it.

It is expected that a --make compilation grows in memory over time,
since the compiler accumulates information about modules that it has
loaded.  Having said that, it should not be retaining the entire
compiled code of each module, only its interface.  It might be hard to
tell - in any case if you have an example where --make appears to be
retaining too much memory then you should submit it as a bug.

>> Is there a way to make Cabal build each module on its own?

not at the moment, sorry.

 >> Are there any fixes or work-arounds?

You can try using GC settings, e.g. +RTS -M<size>.  It might be a good
idea to set this using an environment variable, e.g. GHCRTS=-M<size>
where <size> is roughly equal to the amount of real memory on your machine.

Cheers,
        Simon
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: ghc leaking memory?

Joachim Breitner
Hi Simon,

Am Montag, den 08.03.2010, 11:07 +0000 schrieb Simon Marlow:

> > On Sat, Mar 6, 2010 at 11:57 AM, Joachim Breitner<[hidden email]>  wrote:
> >> Hi,
> >>
> >> in Debian, we are having some problems building large libraries (such as
> >> agda, highlighting-kate, xmonad-contrib) on weaker architectures (sparc,
> >> armel, s390) which compile with -fvia-C, especially when building the
> >> profiling libraries.
> >>
> >> In the case of highlighting-kate, there were really issues with the
> >> code: A huge list of words which ghc seemed to try to inline caused it
> >> consuming a lot of memory.
> >>
> >> But in the case of agda, it even fails with relatively small modules:
> >> https://buildd.debian.org/fetch.cgi?pkg=agda&arch=sparc&ver=2.2.6-3&stamp=1267334936&file=log&as=raw
> >>
> >> Now, looking at the memory gauge on my computer while compiling such a
> >> program with Cabal (which compiles all modules in one ghc call), it
> >> looks like it is steadily increasing. I’d expect ghc to free a lot of
> >> memory when it is done with a module, but this does not seem to be the
> >> case.
> >>
> >> Is this a bug (i.e. memory leak) in ghc or expected behavior?
>
> Can you verify that it is GHC leaking memory, and not gcc?  I recommend
> turning off -fvia-C in particular unless you're sure that you need it.
Sorry if I was not clear: The issues arise on “strange” architectures
(armel, s390, sparc, mips) where, as far as I know, -fvia-C is the only
option and thus the default. I was not setting this flag myself.

Isn’t GHC calling gcc once for each module? In that case, that can not
leak across modules, can it?

Greetings,
Joachim

--
Joachim "nomeata" Breitner
Debian Developer
  [hidden email] | ICQ# 74513189 | GPG-Keyid: 4743206C
  JID: [hidden email] | http://people.debian.org/~nomeata

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: ghc leaking memory?

Simon Marlow-7
On 08/03/2010 13:30, Joachim Breitner wrote:

> Hi Simon,
>
> Am Montag, den 08.03.2010, 11:07 +0000 schrieb Simon Marlow:
>>> On Sat, Mar 6, 2010 at 11:57 AM, Joachim Breitner<[hidden email]>   wrote:
>>>> Hi,
>>>>
>>>> in Debian, we are having some problems building large libraries (such as
>>>> agda, highlighting-kate, xmonad-contrib) on weaker architectures (sparc,
>>>> armel, s390) which compile with -fvia-C, especially when building the
>>>> profiling libraries.
>>>>
>>>> In the case of highlighting-kate, there were really issues with the
>>>> code: A huge list of words which ghc seemed to try to inline caused it
>>>> consuming a lot of memory.
>>>>
>>>> But in the case of agda, it even fails with relatively small modules:
>>>> https://buildd.debian.org/fetch.cgi?pkg=agda&arch=sparc&ver=2.2.6-3&stamp=1267334936&file=log&as=raw
>>>>
>>>> Now, looking at the memory gauge on my computer while compiling such a
>>>> program with Cabal (which compiles all modules in one ghc call), it
>>>> looks like it is steadily increasing. I’d expect ghc to free a lot of
>>>> memory when it is done with a module, but this does not seem to be the
>>>> case.
>>>>
>>>> Is this a bug (i.e. memory leak) in ghc or expected behavior?
>>
>> Can you verify that it is GHC leaking memory, and not gcc?  I recommend
>> turning off -fvia-C in particular unless you're sure that you need it.
>
> Sorry if I was not clear: The issues arise on “strange” architectures
> (armel, s390, sparc, mips) where, as far as I know, -fvia-C is the only
> option and thus the default. I was not setting this flag myself.

Ah, I see.

> Isn’t GHC calling gcc once for each module? In that case, that can not
> leak across modules, can it?

Right, gcc is only being invoked once per module, but the files being
sent to gcc can get quite large, particularly when profiling is enabled,
and even a single run of gcc can grow very large.  This has been more of
a problem with the newer gcc versions since they changed their memory
manager and started doing whole-module optimisations (though we do try
to disable this with -fno-toplevel-reorder, I'm not sure if that
disbales module-at-a-time compilation completely).

Cheers,
        Simon
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe