Help with profiling

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Help with profiling

Uri Braun
Help with profiling

Hi Café,

I'm attempting to profile my rather large (~18K LOC) Haskell program that primarily manipulates graph structures based on Data.IntMap. The program is chewing up _lots_ of memory (approx. 4G for 2.5K nodes & 2.5K edges) and I have profiled it in an attempt to figure out why. It appears that the .prof and .hp (which I visualize with hp2ps -c) files do not agree on the source of the problem. The hp results are located at: http://www.eecs.harvard.edu/~uribraun/sybil.ps and the top of the .prof file looks like this:

COST CENTRE                   MODULE                  %time %alloc

compare                       Policy.Edge              21.7   50.0

eqPartition.edges             Policy.EdgeQuad           6.5    3.7

toConstructedSet              Types.TaggedIntMultimap   6.1    3.6

elems                         Types.TaggedIntMap        6.0    6.8

elems                         Types.TaggedIntSet        5.7    3.8

toPairs                       Types.TaggedIntMultimap   3.9    5.2

assocs                        Types.TaggedIntMap        3.2    4.1

eqIsValid                     Policy.EdgeQuad           2.9    0.5

rIsValid                      Policy.Relations          2.2    0.2

assocs.\                      Types.TaggedIntMap        1.8    0.0

keysSet                       Types.TaggedIntMap        1.5    1.7

mergeIdenticalNodesT.exactMap Minimize.MergeIdentical   1.4    1.2

fromList                      Types.TaggedIntSet        1.3    1.6

tePartitionSrc                Policy.TaggedEdge         1.2    0.9

null                          Types.TaggedIntSet        1.2    0.0

scc.comps                     Policy.Scc                1.2    0.9

tePartitionTgt                Policy.TaggedEdge         1.1    0.8

member                        Types.TaggedIntSet        1.1    0.8

alter                         Types.TaggedIntMap        1.0    1.1

Also, the line in the .prof that corresponds to the anormUnion.name (what the hprof image claims is taking up space) looks like (some spaces removed):

anormUnion.name Anormalize.Anormalize                2914          60    0.0    0.0     0.0    0.0

My main questions at this point are:

1.      What are the chances the hprof image is just wrong?

2.      What would you suggest I try next in debugging my space issue?

This is my first big Haskell program, please do inform me of any big war tools I don’t know about.

Many thanks in advance!

+Uri


_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Help with profiling

Carter Schonwald
Hey Uri,
could you explain what those numbers represent? many of use the profiling tools, but we may not remember what columns / numbers mean what. What number seems wrong and what does the number mean? Also its very hard to help debug if you don't link to the code

If you're hitting space leak problems, one culprit could be your data structure or the operations acting on it is too lazy! Have you tried making it stricter?

cheers
-Carter


On Thu, Dec 5, 2013 at 8:13 PM, Uri Braun <[hidden email]> wrote:

Hi Café,

I'm attempting to profile my rather large (~18K LOC) Haskell program that primarily manipulates graph structures based on Data.IntMap. The program is chewing up _lots_ of memory (approx. 4G for 2.5K nodes & 2.5K edges) and I have profiled it in an attempt to figure out why. It appears that the .prof and .hp (which I visualize with hp2ps -c) files do not agree on the source of the problem. The hp results are located at: http://www.eecs.harvard.edu/~uribraun/sybil.ps and the top of the .prof file looks like this:

COST CENTRE                   MODULE                  %time %alloc

compare                       Policy.Edge              21.7   50.0

eqPartition.edges             Policy.EdgeQuad           6.5    3.7

toConstructedSet              Types.TaggedIntMultimap   6.1    3.6

elems                         Types.TaggedIntMap        6.0    6.8

elems                         Types.TaggedIntSet        5.7    3.8

toPairs                       Types.TaggedIntMultimap   3.9    5.2

assocs                        Types.TaggedIntMap        3.2    4.1

eqIsValid                     Policy.EdgeQuad           2.9    0.5

rIsValid                      Policy.Relations          2.2    0.2

assocs.\                      Types.TaggedIntMap        1.8    0.0

keysSet                       Types.TaggedIntMap        1.5    1.7

mergeIdenticalNodesT.exactMap Minimize.MergeIdentical   1.4    1.2

fromList                      Types.TaggedIntSet        1.3    1.6

tePartitionSrc                Policy.TaggedEdge         1.2    0.9

null                          Types.TaggedIntSet        1.2    0.0

scc.comps                     Policy.Scc                1.2    0.9

tePartitionTgt                Policy.TaggedEdge         1.1    0.8

member                        Types.TaggedIntSet        1.1    0.8

alter                         Types.TaggedIntMap        1.0    1.1

Also, the line in the .prof that corresponds to the anormUnion.name (what the hprof image claims is taking up space) looks like (some spaces removed):

anormUnion.name Anormalize.Anormalize                2914          60    0.0    0.0     0.0    0.0

My main questions at this point are:

1.      What are the chances the hprof image is just wrong?

2.      What would you suggest I try next in debugging my space issue?

This is my first big Haskell program, please do inform me of any big war tools I don’t know about.

Many thanks in advance!

+Uri


_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe



_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Help with profiling

Carter Schonwald
you may want to try out using the Strict variant of IntMap 

I've definitely had some scary space usage explosions with the lazy one on occasions! (one embarrassing instance nearly two years ago was a 40gb space blowup that turned into 100mb usage once i switched to a strict map)


On Thu, Dec 5, 2013 at 9:00 PM, Carter Schonwald <[hidden email]> wrote:
Hey Uri,
could you explain what those numbers represent? many of use the profiling tools, but we may not remember what columns / numbers mean what. What number seems wrong and what does the number mean? Also its very hard to help debug if you don't link to the code

If you're hitting space leak problems, one culprit could be your data structure or the operations acting on it is too lazy! Have you tried making it stricter?

cheers
-Carter


On Thu, Dec 5, 2013 at 8:13 PM, Uri Braun <[hidden email]> wrote:

Hi Café,

I'm attempting to profile my rather large (~18K LOC) Haskell program that primarily manipulates graph structures based on Data.IntMap. The program is chewing up _lots_ of memory (approx. 4G for 2.5K nodes & 2.5K edges) and I have profiled it in an attempt to figure out why. It appears that the .prof and .hp (which I visualize with hp2ps -c) files do not agree on the source of the problem. The hp results are located at: http://www.eecs.harvard.edu/~uribraun/sybil.ps and the top of the .prof file looks like this:

COST CENTRE                   MODULE                  %time %alloc

compare                       Policy.Edge              21.7   50.0

eqPartition.edges             Policy.EdgeQuad           6.5    3.7

toConstructedSet              Types.TaggedIntMultimap   6.1    3.6

elems                         Types.TaggedIntMap        6.0    6.8

elems                         Types.TaggedIntSet        5.7    3.8

toPairs                       Types.TaggedIntMultimap   3.9    5.2

assocs                        Types.TaggedIntMap        3.2    4.1

eqIsValid                     Policy.EdgeQuad           2.9    0.5

rIsValid                      Policy.Relations          2.2    0.2

assocs.\                      Types.TaggedIntMap        1.8    0.0

keysSet                       Types.TaggedIntMap        1.5    1.7

mergeIdenticalNodesT.exactMap Minimize.MergeIdentical   1.4    1.2

fromList                      Types.TaggedIntSet        1.3    1.6

tePartitionSrc                Policy.TaggedEdge         1.2    0.9

null                          Types.TaggedIntSet        1.2    0.0

scc.comps                     Policy.Scc                1.2    0.9

tePartitionTgt                Policy.TaggedEdge         1.1    0.8

member                        Types.TaggedIntSet        1.1    0.8

alter                         Types.TaggedIntMap        1.0    1.1

Also, the line in the .prof that corresponds to the anormUnion.name (what the hprof image claims is taking up space) looks like (some spaces removed):

anormUnion.name Anormalize.Anormalize                2914          60    0.0    0.0     0.0    0.0

My main questions at this point are:

1.      What are the chances the hprof image is just wrong?

2.      What would you suggest I try next in debugging my space issue?

This is my first big Haskell program, please do inform me of any big war tools I don’t know about.

Many thanks in advance!

+Uri


_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe




_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe