[GHC] #13564: Why does memory usage increase so much during CoreTidy?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

[GHC] #13564: Why does memory usage increase so much during CoreTidy?

GHC - devs mailing list
#13564: Why does memory usage increase so much during CoreTidy?
-------------------------------------+-------------------------------------
           Reporter:  rwbarton       |             Owner:  (none)
               Type:  task           |            Status:  new
           Priority:  normal         |         Milestone:
          Component:  Compiler       |           Version:  8.1
           Keywords:                 |  Operating System:  Unknown/Multiple
       Architecture:                 |   Type of failure:  Compile-time
  Unknown/Multiple                   |  performance bug
          Test Case:                 |        Blocked By:
           Blocking:                 |   Related Tickets:
Differential Rev(s):                 |         Wiki Page:
-------------------------------------+-------------------------------------
 I made GHC display the live data size before and after each pass and while
 compiling DynFlags, the live data size increased by almost 50% during the
 CoreTidy pass, even though the Core program size hardly changed:
 {{{
 ...
 Result size of Simplifier
   = {terms: 104,494,
      types: 264,683,
      coercions: 15,760,
      joins: 58/680}
 !!! Simplifier [DynFlags]: 19.69 221248736: finished in 5337.48
 milliseconds, allocated 4416.624 megabytes
 *** Demand analysis [DynFlags]: 19.69 221248152:
 Result size of Demand analysis
   = {terms: 104,494,
      types: 264,683,
      coercions: 15,760,
      joins: 58/680}
 !!! Demand analysis [DynFlags]: 20.77 248604416: finished in 1349.45
 milliseconds, allocated 2298.950 megabytes
 *** CoreTidy [DynFlags]: 20.77 248603432:
 Result size of Tidy Core
   = {terms: 104,440,
      types: 264,565,
      coercions: 15,700,
      joins: 58/679}
 !!! CoreTidy [DynFlags]: 21.43 342594336: finished in 1124.37
 milliseconds, allocated 736.209 megabytes
 }}}
 I find the amount of this increase surprising considering what CoreTidy
 does.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13564>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13564: Why does memory usage increase so much during CoreTidy?

GHC - devs mailing list
#13564: Why does memory usage increase so much during CoreTidy?
-------------------------------------+-------------------------------------
        Reporter:  rwbarton          |                Owner:  rwbarton
            Type:  task              |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
 Type of failure:  Compile-time      |  Unknown/Multiple
  performance bug                    |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------
Changes (by rwbarton):

 * owner:  (none) => rwbarton


--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13564#comment:1>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13564: Why does memory usage increase so much during CoreTidy?

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13564: Why does memory usage increase so much during CoreTidy?
-------------------------------------+-------------------------------------
        Reporter:  rwbarton          |                Owner:  rwbarton
            Type:  task              |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
 Type of failure:  Compile-time      |  Unknown/Multiple
  performance bug                    |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by rwbarton):

 Happily the cause turns out to be simple. In `tidyUnfolding` we tidy the
 RHS of a stable `CoreUnfolding`. But `CoreUnfolding` also has fields
 `uf_is_value`, `uf_is_conlike`, etc., which are cached computations on the
 original RHS. Apparently it's common for these to never be used at any
 point (the iface will only record `uf_guidance`); so we should evaluate
 these in `tidyUnfolding`, or just throw away the ones that won't ever be
 used.

 I'm in the process of verifying this fix still has the expected result
 with HEAD.

 There might be further improvements to be made here, but indications are
 that this fix at least decreases memory usage during code generation to
 the point where it is no longer larger than the peak usage during
 simplification. In other words, max memory usage can't be reduced any more
 from this side.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13564#comment:2>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13564: Why does memory usage increase so much during CoreTidy?

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13564: Why does memory usage increase so much during CoreTidy?
-------------------------------------+-------------------------------------
        Reporter:  rwbarton          |                Owner:  rwbarton
            Type:  task              |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
 Type of failure:  Compile-time      |  Unknown/Multiple
  performance bug                    |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------
Changes (by rwbarton):

 * Attachment "ghc-stage2.pdf" added.

 heap profile with HEAD

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13564>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13564: Why does memory usage increase so much during CoreTidy?

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13564: Why does memory usage increase so much during CoreTidy?
-------------------------------------+-------------------------------------
        Reporter:  rwbarton          |                Owner:  rwbarton
            Type:  task              |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
 Type of failure:  Compile-time      |  Unknown/Multiple
  performance bug                    |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------
Changes (by rwbarton):

 * Attachment "ghc-stage2.2.pdf" added.

 heap profile with seqUnfolding in tidyUnfolding

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13564>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13564: Why does memory usage increase so much during CoreTidy?

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13564: Why does memory usage increase so much during CoreTidy?
-------------------------------------+-------------------------------------
        Reporter:  rwbarton          |                Owner:  rwbarton
            Type:  task              |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
 Type of failure:  Compile-time      |  Unknown/Multiple
  performance bug                    |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by rwbarton):

 After adding `seqUnfolding` there is still a tall but narrow spike in the
 heap profile. I found that this spike went away with `-v`, so I added a
 `seqBinds` which also eliminates the spike. Trac won't let me upload
 another heap profile PDF though.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13564#comment:3>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13564: Why does memory usage increase so much during CoreTidy?

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13564: Why does memory usage increase so much during CoreTidy?
-------------------------------------+-------------------------------------
        Reporter:  rwbarton          |                Owner:  rwbarton
            Type:  task              |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
 Type of failure:  Compile-time      |  Unknown/Multiple
  performance bug                    |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------
Changes (by rwbarton):

 * Attachment "ghc-stage2.3.pdf" added.

 heap profile with seqBinds in tidyTopBinds as well

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13564>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13564: Why does memory usage increase so much during CoreTidy?

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13564: Why does memory usage increase so much during CoreTidy?
-------------------------------------+-------------------------------------
        Reporter:  rwbarton          |                Owner:  rwbarton
            Type:  task              |               Status:  patch
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
 Type of failure:  Compile-time      |  Unknown/Multiple
  performance bug                    |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):  Phab:D3516
       Wiki Page:                    |
-------------------------------------+-------------------------------------
Changes (by rwbarton):

 * status:  new => patch
 * differential:   => Phab:D3516


--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13564#comment:4>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13564: Why does memory usage increase so much during CoreTidy?

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13564: Why does memory usage increase so much during CoreTidy?
-------------------------------------+-------------------------------------
        Reporter:  rwbarton          |                Owner:  rwbarton
            Type:  task              |               Status:  patch
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
 Type of failure:  Compile-time      |  Unknown/Multiple
  performance bug                    |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):  Phab:D3516
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by Ben Gamari <ben@…>):

 In [changeset:"5c602d2228d28530621cc6c94fbb736b13f474fb/ghc"
 5c602d22/ghc]:
 {{{
 #!CommitTicketReference repository="ghc"
 revision="5c602d2228d28530621cc6c94fbb736b13f474fb"
 Avoid excessive space usage from unfoldings in CoreTidy

 Test Plan: validate

 Reviewers: austin, bgamari

 Reviewed By: bgamari

 Subscribers: thomie

 GHC Trac Issues: #13564

 Differential Revision: https://phabricator.haskell.org/D3516
 }}}

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13564#comment:5>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13564: Why does memory usage increase so much during CoreTidy?

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13564: Why does memory usage increase so much during CoreTidy?
-------------------------------------+-------------------------------------
        Reporter:  rwbarton          |                Owner:  rwbarton
            Type:  task              |               Status:  closed
        Priority:  normal            |            Milestone:  8.2.1
       Component:  Compiler          |              Version:  8.1
      Resolution:  fixed             |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
 Type of failure:  Compile-time      |  Unknown/Multiple
  performance bug                    |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):  Phab:D3516
       Wiki Page:                    |
-------------------------------------+-------------------------------------
Changes (by bgamari):

 * status:  patch => closed
 * resolution:   => fixed
 * milestone:   => 8.2.1


Comment:

 Merged with fc2236e8db7550287e764d90410a002b02c55180.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13564#comment:6>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13564: Why does memory usage increase so much during CoreTidy?

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13564: Why does memory usage increase so much during CoreTidy?
-------------------------------------+-------------------------------------
        Reporter:  rwbarton          |                Owner:  (none)
            Type:  task              |               Status:  new
        Priority:  normal            |            Milestone:  8.2.1
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
 Type of failure:  Compile-time      |  Unknown/Multiple
  performance bug                    |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):  Phab:D3516
       Wiki Page:                    |
-------------------------------------+-------------------------------------
Changes (by simonpj):

 * status:  closed => new
 * owner:  rwbarton => (none)
 * resolution:  fixed =>


Comment:

 > I found that this spike went away with -v, so I added a seqBinds which
 also eliminates the spike

 This is ok, but it's an un-satisfying Big Hammer. It forces us to make yet
 another pass over the entire program, when there is probably a more
 refined solution to hand.

 You might well be able to eliminate the other change in `tidyUnfolding`
 when you use the Big Hammer.  And maybe we should eliminate it, since we
 are now seq'ing the unfolding twice.

 But it would be nicer to find the source of the spike in the first place.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13564#comment:7>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13564: Why does memory usage increase so much during CoreTidy?

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13564: Why does memory usage increase so much during CoreTidy?
-------------------------------------+-------------------------------------
        Reporter:  rwbarton          |                Owner:  (none)
            Type:  task              |               Status:  new
        Priority:  normal            |            Milestone:  8.2.1
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
 Type of failure:  Compile-time      |  Unknown/Multiple
  performance bug                    |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):  Phab:D3516,
       Wiki Page:                    |  Phab:D3524
-------------------------------------+-------------------------------------
Changes (by bgamari):

 * differential:  Phab:D3516 => Phab:D3516, Phab:D3524


--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13564#comment:8>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13564: Why does memory usage increase so much during CoreTidy?

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13564: Why does memory usage increase so much during CoreTidy?
-------------------------------------+-------------------------------------
        Reporter:  rwbarton          |                Owner:  (none)
            Type:  task              |               Status:  new
        Priority:  normal            |            Milestone:  8.2.1
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
 Type of failure:  Compile-time      |  Unknown/Multiple
  performance bug                    |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):  Phab:D3516,
       Wiki Page:                    |  Phab:D3524
-------------------------------------+-------------------------------------

Comment (by Ben Gamari <ben@…>):

 In [changeset:"b3da6a6c3546562d5c5e83b8af5d3fd04c07e0c1/ghc"
 b3da6a6c/ghc]:
 {{{
 #!CommitTicketReference repository="ghc"
 revision="b3da6a6c3546562d5c5e83b8af5d3fd04c07e0c1"
 CoreTidy: Don't seq unfoldings

 Previously we would force uf_is_value and friends to ensure that we didn't
 retain a reference to the pre-tidying template, resulting in a space leak.
 Instead, we now just reinitialize these fields (despite the fact that they
 should not have changed). This may result in a bit more computation, but
 most of
 the time we won't ever evaluate them anyways, so the damage shouldn't be
 so bad.

 See #13564.
 }}}

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13564#comment:9>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13564: Why does memory usage increase so much during CoreTidy?

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13564: Why does memory usage increase so much during CoreTidy?
-------------------------------------+-------------------------------------
        Reporter:  rwbarton          |                Owner:  (none)
            Type:  task              |               Status:  new
        Priority:  normal            |            Milestone:  8.2.2
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
 Type of failure:  Compile-time      |  Unknown/Multiple
  performance bug                    |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):  Phab:D3516,
       Wiki Page:                    |  Phab:D3524
-------------------------------------+-------------------------------------
Changes (by bgamari):

 * milestone:  8.2.1 => 8.2.2


--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13564#comment:10>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13564: Why does memory usage increase so much during CoreTidy?

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13564: Why does memory usage increase so much during CoreTidy?
-------------------------------------+-------------------------------------
        Reporter:  rwbarton          |                Owner:  (none)
            Type:  task              |               Status:  new
        Priority:  normal            |            Milestone:  8.4.1
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
 Type of failure:  Compile-time      |  Unknown/Multiple
  performance bug                    |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):  Phab:D3516,
       Wiki Page:                    |  Phab:D3524
-------------------------------------+-------------------------------------
Changes (by bgamari):

 * milestone:  8.2.2 => 8.4.1


Comment:

 Bumping this off to 8.4.1.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13564#comment:11>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets