[GHC] #13527: ghc has a stack space leak when prints warnings

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

[GHC] #13527: ghc has a stack space leak when prints warnings

GHC - devs mailing list
#13527: ghc has a stack space leak when prints warnings
-------------------------------------+-------------------------------------
           Reporter:  slyfox         |             Owner:  (none)
               Type:  bug            |            Status:  new
           Priority:  normal         |         Milestone:
          Component:  Compiler       |           Version:  8.1
           Keywords:                 |  Operating System:  Unknown/Multiple
       Architecture:                 |   Type of failure:  None/Unknown
  Unknown/Multiple                   |
          Test Case:                 |        Blocked By:
           Blocking:                 |   Related Tickets:
Differential Rev(s):                 |         Wiki Page:
-------------------------------------+-------------------------------------
 Found on a crash when ghc tried to print a warning in a huge assembly
 file.

 I've simplified it to autogenerated C file with an error:
 {{{
 # generate 10 million lines file
 # and put an error into the very end of file
 $ { for i in `seq 1 $((10 * 1000 * 1000))`; do echo "// a comment number
 $i"; done; echo 'static int foo = ;'; } > a.c
 }}}

 {{{
 $ LANG=C powerpc64le-unknown-linux-gnu-ghc -c a.c +RTS -K128M

 stack overflow: use +RTS -K<size> to increase it
 }}}

 C compiler tried to output something like the following to stderr:
 {{{
 a.c:10000001:0: error:
      error: expected expression before ';' token
      static int foo = ;
 }}}

 GHC finds 'a.c:10000001' location and tries to pretty-print the line.
 We get a space leak somewhere.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13527>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13527: ghc has a stack space leak when prints warnings

GHC - devs mailing list
#13527: ghc has a stack space leak when prints warnings
-------------------------------------+-------------------------------------
        Reporter:  slyfox            |                Owner:  (none)
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by slyfox):

 I've built ghc-prof and found out GHC is most stressed in
 'ErrUtils.getCaretDiagnostic' where
 source file is read from disk and then chunked by newlines. lexemeToString
 generates a lot of
 heap traffic as it decodes whole file into String.

 This example is enough to take more that 100M of stack:

 {{{#!hs
 {-# LANGUAGE PackageImports #-}
 module Main (main) where

 import qualified StringBuffer as SB

 main = do
     b <- SB.hGetStringBuffer "a.c"
     let s = SB.lexemeToString b (SB.len b)
     print (length $ lines s)
 }}}

 {{{
 $ ghc --make a.hs -o a -package=ghc -debug -rtsopts
 $ ./a +RTS -K100M
 a: Stack space overflow: current size 33624 bytes.
 a: Use `+RTS -Ksize -RTS' to increase it.
 }}}

 Looks like there is 2 bugs here:
 - '''lexemeToString''' takes a lot of stack to decode StringBuffer into
 String
 - '''getCaretDiagnostic ''' could use more efficient mechanism to split
 file into lines before converting everything to String.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13527#comment:1>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13527: ghc has a stack space leak when prints warnings

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13527: ghc has a stack space leak when prints warnings
-------------------------------------+-------------------------------------
        Reporter:  slyfox            |                Owner:  (none)
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------
Changes (by RyanGlScott):

 * cc: Rufflewind (added)


--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13527#comment:2>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13527: ghc has a stack space leak when prints warnings

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13527: ghc has a stack space leak when prints warnings
-------------------------------------+-------------------------------------
        Reporter:  slyfox            |                Owner:  (none)
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by slyfox):

 Slightly more context. '''utf8DecodeString''' overflows stack:
 http://git.haskell.org/ghc.git/blob/HEAD:/compiler/utils/Encoding.hs#l118

 {{{#!hs
 utf8DecodeString :: Ptr Word8 -> Int -> IO [Char]
 utf8DecodeString ptr len
   = unpack ptr
    where
      !end = ptr `plusPtr` len

      unpack p
          | p >= end = return []
          | otherwise  =
          case utf8DecodeChar# (unPtr p) of
             (# c#, nBytes# #) -> do
                  chs <- unpack (p `plusPtr#` nBytes#) -- deep recursion
                  return (C# c# : chs)
 }}}

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13527#comment:3>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13527: ghc has a stack space leak when prints warnings

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13527: ghc has a stack space leak when prints warnings
-------------------------------------+-------------------------------------
        Reporter:  slyfox            |                Owner:  (none)
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by Rufflewind):

 Strange that `utfDecodeString` returns `IO` but doesn’t call any `IO`
 functions besides `return`.

 As for `getCaretDiagnostic`, could perhaps revive this diff?
 https://phabricator.haskell.org/D2875

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13527#comment:4>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13527: ghc has a stack space leak when prints warnings

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13527: ghc has a stack space leak when prints warnings
-------------------------------------+-------------------------------------
        Reporter:  slyfox            |                Owner:  (none)
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by simonpj):

 Looks like a good diagnosis, thank you. Both problems are surely fixiable
 if someone cares to try.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13527#comment:5>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13527: ghc has a stack space leak when prints warnings

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13527: ghc has a stack space leak when prints warnings
-------------------------------------+-------------------------------------
        Reporter:  slyfox            |                Owner:  (none)
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by bgamari):

 The issue with `utf8DecodeString` is that we need to decode all `len`
 characters before we can return; we cannot do the decoding lazily as the
 list is demanded since the `Ptr Word8` may be freed. Phab:D3442 adds such
 a variant..

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13527#comment:6>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13527: ghc has a stack space leak when prints warnings

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13527: ghc has a stack space leak when prints warnings
-------------------------------------+-------------------------------------
        Reporter:  slyfox            |                Owner:  (none)
            Type:  bug               |               Status:  new
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by Ben Gamari <ben@…>):

 In [changeset:"1cc82d38759c7a5f527ccc6cb514b8ba576cc3d1/ghc" 1cc82d3/ghc]:
 {{{
 #!CommitTicketReference repository="ghc"
 revision="1cc82d38759c7a5f527ccc6cb514b8ba576cc3d1"
 utils: Lazily decode UTF8 strings

 Reviewers: austin, hvr

 Subscribers: rwbarton, thomie

 GHC Trac Issues: #13527

 Differential Revision: https://phabricator.haskell.org/D3442
 }}}

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13527#comment:7>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13527: ghc has a stack space leak when prints warnings

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13527: ghc has a stack space leak when prints warnings
-------------------------------------+-------------------------------------
        Reporter:  slyfox            |                Owner:  (none)
            Type:  bug               |               Status:  merge
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------
Changes (by RyanGlScott):

 * status:  new => merge


Comment:

 I'd say this is worth merging to 8.2, no?

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13527#comment:8>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13527: ghc has a stack space leak when prints warnings

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13527: ghc has a stack space leak when prints warnings
-------------------------------------+-------------------------------------
        Reporter:  slyfox            |                Owner:  (none)
            Type:  bug               |               Status:  merge
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by bgamari):

 I suppose; I've been trying to be a bit more strict about merging to
 stable, but given that this is a new feature we should probably make it
 work correctly out of the box.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13527#comment:9>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13527: ghc has a stack space leak when prints warnings

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13527: ghc has a stack space leak when prints warnings
-------------------------------------+-------------------------------------
        Reporter:  slyfox            |                Owner:  (none)
            Type:  bug               |               Status:  merge
        Priority:  normal            |            Milestone:
       Component:  Compiler          |              Version:  8.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by bgamari):

 comment:7 addresses comment:6. With this patch and
 065be6e9eb5114c5f0e3a20626ec93042ce47f13 the test in comment:1 runs to
 completion without a stack overflow.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13527#comment:10>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #13527: ghc has a stack space leak when prints warnings

GHC - devs mailing list
In reply to this post by GHC - devs mailing list
#13527: ghc has a stack space leak when prints warnings
-------------------------------------+-------------------------------------
        Reporter:  slyfox            |                Owner:  (none)
            Type:  bug               |               Status:  closed
        Priority:  normal            |            Milestone:  8.2.1
       Component:  Compiler          |              Version:  8.1
      Resolution:  fixed             |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------
Changes (by bgamari):

 * status:  merge => closed
 * resolution:   => fixed
 * milestone:   => 8.2.1


Comment:

 Merged as ec5a49fe34180da0adcd7956ad60a9f8ba04c775 and
 0eb5004ae3d58032bb48d77a19bed556af7c4f72.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13527#comment:11>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets