[GHC] #15136: High CPU when asynchronous exception and unblocking retry on TVar raced

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[GHC] #15136: High CPU when asynchronous exception and unblocking retry on TVar raced

GHC - devs mailing list
#15136: High CPU when asynchronous exception and unblocking retry on TVar raced
-------------------------------------+-------------------------------------
           Reporter:  nshimaza       |             Owner:  (none)
               Type:  bug            |            Status:  new
           Priority:  highest        |         Milestone:  8.6.1
          Component:  Runtime        |           Version:  8.4.2
  System                             |
           Keywords:                 |  Operating System:  Unknown/Multiple
       Architecture:                 |   Type of failure:  Runtime crash
  Unknown/Multiple                   |
          Test Case:                 |        Blocked By:
           Blocking:                 |   Related Tickets:
Differential Rev(s):                 |         Wiki Page:
-------------------------------------+-------------------------------------
 Detail:  https://github.com/nshimaza/race-tmvar-async-exception

 Runtime falls into high CPU under racing condition between async exception
 and unblocking retry on TVar.

 * Reproduces with +RTS -Nx where x > 1
 * Does NOT reproduce with +RTS -N1
 * Program stalls at `killThread`
 * High CPU based on given -Nx
     * CPU won't be 100% if you gave x smaller than available hardware
 threads of your platform.
 * Does NOT reproduce if TVar/retry is replaced by MVar
 * Reproduced with GHC 8.4.2 (macOS High Sierra (10.13.4))
 * Reproduced with GHC 8.4.2 (Docker for Mac Version 18.03.1-ce-mac65)
 * Reproduced with ghc-8.5.20180506 (Docker for Mac Version 18.03.1-ce-
 mac65)


 Minimal reproducing code here.  (You can find more verbose code on the
 above github repo.)
 {{{#!hs
 main :: IO ()
 main = do
     let volume = 1000
     forM_ [1..1000] $ \i -> do
         putStrFlush $ show i ++ " "

         -- Spawn massive number of threads.
         threads <- replicateM volume $ do
             trigger <- newTVarIO False
             tid <- forkIO $ void $ atomically $ do
                 t <- readTVar trigger
                 if t then pure t else retry
             pure (trigger, tid)

         -- Make sure all threads are spawned.
         threadDelay 30000

         -- Let threads start to exit normally.
         forkIO $ forM_ threads $ \(trigger, _) -> threadDelay 1 *>
 atomically (writeTVar trigger True)

         -- Concurrently kill threads in order to create race.
         -- TMVar operation and asynchronous exception can hit same thread
 simultaneously.
         -- Adjust threadDelay if you don't reproduce very well.
         threadDelay 1000
         forM_ threads $ \(_, tid) -> do
             putCharFlush 'A'
             killThread tid      -- When the issue reproduced, this
 killThread doesn't return.
             putCharFlush '\b'
 }}}

 This program intentionally creates race condition between asynchronous
 exception
 and unblocking operation of `retry` on TVar.  From one side, a `writeTVar
 trigger True` is attempted from external thread while target thread is
 blocking
 at `retry` on the same `TVar`.  On the other side, an asynchronous
 exception
 `ThreadKilled` is thrown by yet another external thread to the same target
 thread.

 In other word, it attempts to kill a thread about to unblock.

 I guess when the above two operation hit the same thread at the same time
 in
 parallel in SMP environment, GHC runtime falls into high CPU.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15136>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets
Reply | Threaded
Open this post in threaded view
|

Re: [GHC] #15136: High CPU when asynchronous exception and unblocking retry on TVar raced

GHC - devs mailing list
#15136: High CPU when asynchronous exception and unblocking retry on TVar raced
-------------------------------------+-------------------------------------
        Reporter:  nshimaza          |                Owner:  (none)
            Type:  bug               |               Status:  new
        Priority:  highest           |            Milestone:  8.6.1
       Component:  Runtime System    |              Version:  8.4.2
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  Runtime crash     |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by bgamari):

 Thanks for the small repro! Indeed I am able to easily reproduce this.

 The racing threads look like this,
 {{{

 Thread 8 (LWP 25960):
 #0  0x000000000049c0df in stmCommitTransaction ()
 #1  0x00000000004a9de4 in stg_atomically_frame_info ()
 #2  0x0000000000000000 in ?? ()

 Thread 1 (LWP 25948):
 #0  0x000000000049b290 in lock_tvar ()
 #1  0x000000000049b559 in remove_watch_queue_entries_for_trec ()
 #2  0x000000000049b95b in stmAbortTransaction ()
 #3  0x000000000049984c in raiseAsync ()
 #4  0x0000000000499ba3 in throwToMsg ()
 #5  0x0000000000499e82 in throwTo ()
 #6  0x00000000004a6aac in stg_killThreadzh ()
 #7  0x0000000000000000 in ?? ()
 }}}

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15136#comment:1>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler

_______________________________________________
ghc-tickets mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-tickets