"Out of memory" if compiled with -O2, why?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

"Out of memory" if compiled with -O2, why?

Dmitry Kulagin
Hi,

I have problems with memory leaks and can't find out how to avoid them.
I tried to reduce sample to demonstrate the following problems:
1) when compiled without -O2 option, it iconsumes 1582MB (!) total memory
2) when compiled with -O2 option it terminates with "out of memory"

Actually I don't understand the reasons, particulary why  GC can't
collect already processed objects g,...,n (see code below)?

I would appreciate very much any help with this situation.
Thanks!


module Main where

import qualified Data.Map as M

len = 15*1024*1024
lst from = take len $ zip [from..] [0..]

g = M.size $ M.fromList $ lst 0
h = M.size $ M.fromList $ lst 0
i = M.size $ M.fromList $ lst 0
j = M.size $ M.fromList $ lst 0
k = M.size $ M.fromList $ lst 0
l = M.size $ M.fromList $ lst 0
m = M.size $ M.fromList $ lst 0
n = M.size $ M.fromList $ lst 0

main = do
    mapM_ print [g,h,i,j,k,l,m,n]

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: "Out of memory" if compiled with -O2, why?

Bugzilla from prokhorenkov@gmail.com
Hi, Dmitry

I recently had the same problem:
http://www.haskell.org/pipermail/haskell-cafe/2010-November/086450.html

Memory is taken by the list returned by your lst function wich is
being shared across g,h,i,j,k,l,m,n.
Apparently there is no safe and easy way to overcome this yet :(

--
Regards,
Petr



On Wed, Dec 1, 2010 at 5:23 PM, Dmitry Kulagin <[hidden email]> wrote:

> Hi,
>
> I have problems with memory leaks and can't find out how to avoid them.
> I tried to reduce sample to demonstrate the following problems:
> 1) when compiled without -O2 option, it iconsumes 1582MB (!) total memory
> 2) when compiled with -O2 option it terminates with "out of memory"
>
> Actually I don't understand the reasons, particulary why  GC can't
> collect already processed objects g,...,n (see code below)?
>
> I would appreciate very much any help with this situation.
> Thanks!
>
>
> module Main where
>
> import qualified Data.Map as M
>
> len = 15*1024*1024
> lst from = take len $ zip [from..] [0..]
>
> g = M.size $ M.fromList $ lst 0
> h = M.size $ M.fromList $ lst 0
> i = M.size $ M.fromList $ lst 0
> j = M.size $ M.fromList $ lst 0
> k = M.size $ M.fromList $ lst 0
> l = M.size $ M.fromList $ lst 0
> m = M.size $ M.fromList $ lst 0
> n = M.size $ M.fromList $ lst 0
>
> main = do
>    mapM_ print [g,h,i,j,k,l,m,n]
>
> _______________________________________________
> Haskell-Cafe mailing list
> [hidden email]
> http://www.haskell.org/mailman/listinfo/haskell-cafe
>

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: "Out of memory" if compiled with -O2, why?

Jason Dagit-2
In reply to this post by Dmitry Kulagin


On Wed, Dec 1, 2010 at 6:23 AM, Dmitry Kulagin <[hidden email]> wrote:
Hi,

I have problems with memory leaks and can't find out how to avoid them.
I tried to reduce sample to demonstrate the following problems:
1) when compiled without -O2 option, it iconsumes 1582MB (!) total memory
2) when compiled with -O2 option it terminates with "out of memory"

Actually I don't understand the reasons, particulary why  GC can't
collect already processed objects g,...,n (see code below)?

I would appreciate very much any help with this situation.
Thanks!


module Main where

import qualified Data.Map as M

len = 15*1024*1024
lst from = take len $ zip [from..] [0..]

g = M.size $ M.fromList $ lst 0
h = M.size $ M.fromList $ lst 0
i = M.size $ M.fromList $ lst 0
j = M.size $ M.fromList $ lst 0
k = M.size $ M.fromList $ lst 0
l = M.size $ M.fromList $ lst 0
m = M.size $ M.fromList $ lst 0
n = M.size $ M.fromList $ lst 0

main = do
   mapM_ print [g,h,i,j,k,l,m,n]

I'm using ghc7 here.  If I run your program with -O2, it takes 1943 MB of memory max.

If I comment out everything except g and h then with -O2 it takes 1521 MB.

If I comment out everything except g then with -O2 it takes 1521 MB.

I'm not sure where the extra 400 MB of memory are going.

When I compile with: -fno-cse -fno-full-laziness and -O2, the memory usage (with g though n) is 1585 MB.

Jason

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: "Out of memory" if compiled with -O2, why?

Dmitry Kulagin
In reply to this post by Bugzilla from prokhorenkov@gmail.com
Thank you, it is indeed very similar problem.
Nevertheless it seems that the lst function is not the direct reason, because:
1) if I inline lst (by hands), the problem is still there
2) size of the list is actially not so large - just 15 millions elements

I am almost sure that the reason is Map.fromList - result of the
function perhaps somehow memoized and not released by GC.

Dmitry.

On Wed, Dec 1, 2010 at 8:15 PM, Petr Prokhorenkov
<[hidden email]> wrote:

> Hi, Dmitry
>
> I recently had the same problem:
> http://www.haskell.org/pipermail/haskell-cafe/2010-November/086450.html
>
> Memory is taken by the list returned by your lst function wich is
> being shared across g,h,i,j,k,l,m,n.
> Apparently there is no safe and easy way to overcome this yet :(
>
> --
> Regards,
> Petr
>
>
>
> On Wed, Dec 1, 2010 at 5:23 PM, Dmitry Kulagin <[hidden email]> wrote:
>> Hi,
>>
>> I have problems with memory leaks and can't find out how to avoid them.
>> I tried to reduce sample to demonstrate the following problems:
>> 1) when compiled without -O2 option, it iconsumes 1582MB (!) total memory
>> 2) when compiled with -O2 option it terminates with "out of memory"
>>
>> Actually I don't understand the reasons, particulary why  GC can't
>> collect already processed objects g,...,n (see code below)?
>>
>> I would appreciate very much any help with this situation.
>> Thanks!
>>
>>
>> module Main where
>>
>> import qualified Data.Map as M
>>
>> len = 15*1024*1024
>> lst from = take len $ zip [from..] [0..]
>>
>> g = M.size $ M.fromList $ lst 0
>> h = M.size $ M.fromList $ lst 0
>> i = M.size $ M.fromList $ lst 0
>> j = M.size $ M.fromList $ lst 0
>> k = M.size $ M.fromList $ lst 0
>> l = M.size $ M.fromList $ lst 0
>> m = M.size $ M.fromList $ lst 0
>> n = M.size $ M.fromList $ lst 0
>>
>> main = do
>>    mapM_ print [g,h,i,j,k,l,m,n]
>>
>> _______________________________________________
>> Haskell-Cafe mailing list
>> [hidden email]
>> http://www.haskell.org/mailman/listinfo/haskell-cafe
>>
>

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: "Out of memory" if compiled with -O2, why?

Bugzilla from prokhorenkov@gmail.com
I've tried it out with and without -O2 and with and without manual
inlining. I also used len=1024*1024.
Just in case, the system is Linux x86-64, GHC 6.12.3.
You can find output of memory profiler attached.

It seems that list of pairs is shared only when lst is not manually
inlined and -O2 is enabled.
And the GC certainly cannot collect any of map nodes until M.size has returned.
So, at the same time list [(Integer, Integer)] of length len and Map
Integer Integer of size len are in memory, which
means that an (Integer, Integer) and s single node of this map are
taking together 100 bytes of memory, which I find plausible.

Bottom line  --- I see nothing strange about it, except for
unnecessary sharing of lst and M.Map being quite memory-costy.

--
Regards,
Petr



On Thu, Dec 2, 2010 at 1:27 PM, Dmitry Kulagin <[hidden email]> wrote:

> Thank you, it is indeed very similar problem.
> Nevertheless it seems that the lst function is not the direct reason, because:
> 1) if I inline lst (by hands), the problem is still there
> 2) size of the list is actially not so large - just 15 millions elements
>
> I am almost sure that the reason is Map.fromList - result of the
> function perhaps somehow memoized and not released by GC.
>
> Dmitry.
>
> On Wed, Dec 1, 2010 at 8:15 PM, Petr Prokhorenkov
> <[hidden email]> wrote:
>> Hi, Dmitry
>>
>> I recently had the same problem:
>> http://www.haskell.org/pipermail/haskell-cafe/2010-November/086450.html
>>
>> Memory is taken by the list returned by your lst function wich is
>> being shared across g,h,i,j,k,l,m,n.
>> Apparently there is no safe and easy way to overcome this yet :(
>>
>> --
>> Regards,
>> Petr
>>
>>
>>
>> On Wed, Dec 1, 2010 at 5:23 PM, Dmitry Kulagin <[hidden email]> wrote:
>>> Hi,
>>>
>>> I have problems with memory leaks and can't find out how to avoid them.
>>> I tried to reduce sample to demonstrate the following problems:
>>> 1) when compiled without -O2 option, it iconsumes 1582MB (!) total memory
>>> 2) when compiled with -O2 option it terminates with "out of memory"
>>>
>>> Actually I don't understand the reasons, particulary why  GC can't
>>> collect already processed objects g,...,n (see code below)?
>>>
>>> I would appreciate very much any help with this situation.
>>> Thanks!
>>>
>>>
>>> module Main where
>>>
>>> import qualified Data.Map as M
>>>
>>> len = 15*1024*1024
>>> lst from = take len $ zip [from..] [0..]
>>>
>>> g = M.size $ M.fromList $ lst 0
>>> h = M.size $ M.fromList $ lst 0
>>> i = M.size $ M.fromList $ lst 0
>>> j = M.size $ M.fromList $ lst 0
>>> k = M.size $ M.fromList $ lst 0
>>> l = M.size $ M.fromList $ lst 0
>>> m = M.size $ M.fromList $ lst 0
>>> n = M.size $ M.fromList $ lst 0
>>>
>>> main = do
>>>    mapM_ print [g,h,i,j,k,l,m,n]
>>>
>>> _______________________________________________
>>> Haskell-Cafe mailing list
>>> [hidden email]
>>> http://www.haskell.org/mailman/listinfo/haskell-cafe
>>>
>>
>

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

bloat_O0.png (47K) Download Attachment
bloat_O0_inline.png (48K) Download Attachment
bloat_O2.png (51K) Download Attachment
bloat_O2_inline.png (48K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: "Out of memory" if compiled with -O2, why?

Bulat Ziganshin-2
In reply to this post by Jason Dagit-2
Hello Jason,

Wednesday, December 1, 2010, 8:54:58 PM, you wrote:


> I'm using ghc7 here.  If I run your program with -O2, it takes 1943 MB of memory max.
> If I comment out everything except g then with -O2 it takes 1521 MB.

> I'm not sure where the extra 400 MB of memory are going.

i think, it's because memory isn't collected immediately, so in first
case you just have more garbage hanging around. if you need to measure
real workset of your program, you should apply very aggressive (and
slow) garbage collection settings

--
Best regards,
 Bulat                            mailto:[hidden email]


_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe