[GHC] #14697: Redundant computation in fingerprintDynFlags when compiling many modules

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[GHC] #14697: Redundant computation in fingerprintDynFlags when compiling many modules

GHC - devs mailing list
#14697: Redundant computation in fingerprintDynFlags when compiling many modules
-------------------------------------+-------------------------------------
           Reporter:  niteria        |             Owner:  (none)
               Type:  bug            |            Status:  new
           Priority:  normal         |         Milestone:
          Component:  Compiler       |           Version:
           Keywords:                 |  Operating System:  Unknown/Multiple
       Architecture:                 |   Type of failure:  Compile-time
  Unknown/Multiple                   |  performance bug
          Test Case:                 |        Blocked By:
           Blocking:                 |   Related Tickets:
Differential Rev(s):                 |         Wiki Page:
-------------------------------------+-------------------------------------
 I profiled a build of a production code base with thousands of modules and
 computing `fingerprintDynFlags` is `7%` of time and `14%` of allocations.

 Here's a synthetic test case inspired by what I observed:
 {{{
 SIZE=1000

 for i in $(seq -w 1 $SIZE); do
   echo "module A$i where" > A$i.hs
   echo "data A$i = A$i" >> A$i.hs
 done
 }}}

 This generates a 1000 modules each with one datatype.
 Compiling them with:
 {{{
 inplace/bin/ghc-stage2 A*.hs -optP-D__F{1..10000}__
 }}}
 results in `fingerprintDynFlags` being the top cost centre in the profile.
 AFAICT there's only one module dependent piece that goes into computing
 `fingerprintDynFlags` and the rest is the same between those 1000 modules.

 Now, why would I have so many preprocessor flags?
 This is how the Buck build system currently works. If a Haskell library
 depends on a C++ library then the GHC invocation gets the C++ library's
 directory as include path (`-optP-I -optP-I some/library/path`). This can
 grow quite big.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14697>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14697: Redundant computation in fingerprintDynFlags when compiling many modules

GHC - devs mailing list
#14697: Redundant computation in fingerprintDynFlags when compiling many modules
-------------------------------------+-------------------------------------
        Reporter:  niteria           |                Owner:  (none)
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
 Type of failure:  Compile-time      |  Unknown/Multiple
  performance bug                    |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------
Changes (by niteria):

 * cc: simonmar (added)


Old description:

> I profiled a build of a production code base with thousands of modules
> and computing `fingerprintDynFlags` is `7%` of time and `14%` of
> allocations.
>
> Here's a synthetic test case inspired by what I observed:
> {{{
> SIZE=1000
>
> for i in $(seq -w 1 $SIZE); do
>   echo "module A$i where" > A$i.hs
>   echo "data A$i = A$i" >> A$i.hs
> done
> }}}
>
> This generates a 1000 modules each with one datatype.
> Compiling them with:
> {{{
> inplace/bin/ghc-stage2 A*.hs -optP-D__F{1..10000}__
> }}}
> results in `fingerprintDynFlags` being the top cost centre in the
> profile.
> AFAICT there's only one module dependent piece that goes into computing
> `fingerprintDynFlags` and the rest is the same between those 1000
> modules.
>
> Now, why would I have so many preprocessor flags?
> This is how the Buck build system currently works. If a Haskell library
> depends on a C++ library then the GHC invocation gets the C++ library's
> directory as include path (`-optP-I -optP-I some/library/path`). This can
> grow quite big.
New description:

 I profiled a build of a production code base with thousands of modules and
 computing `fingerprintDynFlags` is `7%` of time and `14%` of allocations.

 Here's a synthetic test case inspired by what I observed:
 {{{
 SIZE=1000

 for i in $(seq -w 1 $SIZE); do
   echo "module A$i where" > A$i.hs
   echo "data A$i = A$i" >> A$i.hs
 done
 }}}

 This generates a 1000 modules each with one datatype.
 Compiling them with:
 {{{
 inplace/bin/ghc-stage2 A*.hs -optP-D__F{1..10000}__
 }}}
 results in `fingerprintDynFlags` being the top cost centre in the profile.
 AFAICT there's only one module dependent piece that goes into computing
 `fingerprintDynFlags` and the rest is the same between those 1000 modules.

 Now, why would I have so many preprocessor flags?
 This is how the Buck build system currently works. If a Haskell library
 depends on a C++ library then the GHC invocation gets the C++ library's
 directory as include path (`-optP -I -optP some/library/path`). This can
 grow quite big.

--

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14697#comment:1>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14697: Redundant computation in fingerprintDynFlags when compiling many modules

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14697: Redundant computation in fingerprintDynFlags when compiling many modules
-------------------------------------+-------------------------------------
        Reporter:  niteria           |                Owner:  (none)
            Type:  bug               |               Status:  patch
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
 Type of failure:  Compile-time      |  Unknown/Multiple
  performance bug                    |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):  phab:D4445
       Wiki Page:                    |
-------------------------------------+-------------------------------------
Changes (by niteria):

 * status:  new => patch
 * differential:   => phab:D4445


--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14697#comment:2>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14697: Redundant computation in fingerprintDynFlags when compiling many modules

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14697: Redundant computation in fingerprintDynFlags when compiling many modules
-------------------------------------+-------------------------------------
        Reporter:  niteria           |                Owner:  (none)
            Type:  bug               |               Status:  patch
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
 Type of failure:  Compile-time      |  Unknown/Multiple
  performance bug                    |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):  phab:D4445
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by Bartosz Nitka <niteria@…>):

 In [changeset:"b8f03bbe16af7a09b494a33fbbe523ecd82f1a50/ghc"
 b8f03bbe/ghc]:
 {{{
 #!CommitTicketReference repository="ghc"
 revision="b8f03bbe16af7a09b494a33fbbe523ecd82f1a50"
 Cache the fingerprint of sOpt_P

 Before this change we would compute a hash of
 all the command line -optP flags once per file.
 With a lot of files and many -optP flags, that's a lot
 of repeated work.

 I added a new Note that explains the approach and rationale.

 Test Plan: new test

 Reviewers: simonmar, simonpj, bgamari

 Reviewed By: simonpj

 Subscribers: rwbarton, thomie, carter

 GHC Trac Issues: #14697

 Differential Revision: https://phabricator.haskell.org/D4445
 }}}

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14697#comment:3>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14697: Redundant computation in fingerprintDynFlags when compiling many modules

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14697: Redundant computation in fingerprintDynFlags when compiling many modules
-------------------------------------+-------------------------------------
        Reporter:  niteria           |                Owner:  (none)
            Type:  bug               |               Status:  closed
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:
      Resolution:  fixed             |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
 Type of failure:  Compile-time      |  Unknown/Multiple
  performance bug                    |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):  phab:D4445
       Wiki Page:                    |
-------------------------------------+-------------------------------------
Changes (by niteria):

 * status:  patch => closed
 * resolution:   => fixed


--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14697#comment:4>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #14697: Redundant computation in fingerprintDynFlags when compiling many modules

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#14697: Redundant computation in fingerprintDynFlags when compiling many modules
-------------------------------------+-------------------------------------
        Reporter:  niteria           |                Owner:  (none)
            Type:  bug               |               Status:  closed
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:
      Resolution:  fixed             |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
 Type of failure:  Compile-time      |  Unknown/Multiple
  performance bug                    |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):  phab:D4445
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by Ben Gamari <ben@…>):

 In [changeset:"af986f9dfb49db5544e4d670cde9a9a70a9a1223/ghc"
 af986f9d/ghc]:
 {{{
 #!CommitTicketReference repository="ghc"
 revision="af986f9dfb49db5544e4d670cde9a9a70a9a1223"
 testsuite: Disable T14697 on Windows

 Test Plan: Validate on Windows

 Subscribers: thomie, carter

 GHC Trac Issues: #14697, #15072

 Differential Revision: https://phabricator.haskell.org/D4619
 }}}

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/14697#comment:5>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets