help understanding zlib space leak

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

help understanding zlib space leak

Diego Souza-5
Dear haskellers,

I'd like assistance figuring out a strange space leak using zlib
package.

To make it easier to reproduce I've come up with the following snippet
that pretty much resumes up the problem I'm trying to solve:

> -- omitting imports and function signatures
> encode = compress . L.fromChunks
>  
> main = do { hSetBinaryMode stdout True
>           ; hSetBinaryMode stdin True
>           ; hSetBuffering stdin NoBuffering
>           ; hSetBuffering stdout NoBuffering
>           ; loop []
>           }
>  where loop buff
>          | length buff == 64 = L.hPut stdout (encode buff) >> loop []
>          | otherwise         = do { eof <- hIsEOF stdin
>                                   ; when (not eof) (fmap (: buff) (B.hGetSome stdin 512) >>= loop)
>                                   }

N.B.: Removing the `compress' function from the above code also removes the
      space leak.

Now, feeding the above program with ~8GB worth of binary data:

> $ uname -a
Linux mephisto.localhost.localdomain 3.8.7-1-ARCH #1 SMP PREEMPT Sat Apr 13 09:01:47 CEST 2013 x86_64 GNU/Linux
> $ ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.6.3
> $ ghc -W -Wall -rtsopts --make -O2 test.hs
> $ sudo dd if=/dev/sda bs=4K count=2048K | ./test +RTS -M1M -s >/dev/null
...
8589934592 bytes (8.6 GB) copied, 277.263 s, 31.0 MB/s
...
               2 MB total memory in use (0 MB lost due to fragmentation)
...

Which is fine. However, monitoring the RSS size is a different story:

> $ while pidof test; do ps -o rss= -p $(pidof test); done | tail
16967
25620
16967
25628
16967
25628
16967
25628
16967  # ~16M
    0

I know the RSS usually overestimates the memory consumption but the
problem is that it is forever growing.

The following I found very intriguing:

  * `+RTS -hc` gives me no hint about whats wrong [at least I couldn't see one];
  * `+RTS -M1M` doesn't produce an error;
  * removing the `compress' functions makes the problem disappear;

I couldn't figure these out and I don't think this is matter of
strictness, though. Has anyone seen this before?

Thanks in advance,
~dsouza

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: help understanding zlib space leak

Diego Souza-5

Sorry, I should've removed the pid number from the output. The
following should be correct:

> $ sudo dd if=/dev/sda bs=4K count=2048K | ./test +RTS -M1M -s >/dev/null                                                  
...
8589934592 bytes (8.6 GB) copied, 243.525 s, 35.3 MB/s
  41,942,119,192 bytes allocated in the heap
     228,827,904 bytes copied during GC
         104,048 bytes maximum residency (6 sample(s))
          24,408 bytes maximum slop
               2 MB total memory in use (0 MB lost due to fragmentation)
...

> $ while pidof test >/dev/null; do ps -o rss= -p $(pidof test); sleep 1; done | tail
32056
32408
32832
33264
33684
34100
34560
34900
35384
35816 # ~ 35MB

Thanks!
~dsouza

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe