Blocking a task indefinitely in the RTS

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Blocking a task indefinitely in the RTS

Phyx
Hi All,

I'm looking for a way to block a task indefinitely until it is woken up by an external event in both the threaded and non-threaded RTS and returns a value that was stored/passed. MVar works great for the threaded RTS, but for the non-threaded there's a bunch of deadlock detection in the scheduler that would forcibly free the lock and resume the task with an opaque exception. This means that MVar and anything derived from it is not usable.

STMs are more expensive but also have the same deadlock code. So again no go. The reason it looks like a deadlock to the RTS is that the "Wake-up" call in the non-threaded rts will come from C code running inside the RTS.  The RTS essentially just sees all tasks blocked on it's main capability and (usually rightly so) assumes a deadlock occurred.

You have other states like BlockedOnForeign etc but those are not usable as a primitive. Previous iterations of I/O managers have each invented primitives for this such as asyncRead#, but they are not general and can't be re-used, and requires a different solution for threaded and non-threaded.

I have started making a new primitive IOPort for this, based on the MVar code, but this is not trivial... (currently I'm getting a segfault *somewhere* in the primitive's cmm code). The reason is that the semantics are decidedly different from what MVars guarantee. I should also mention that this is meant to be internal to base (i.e no exported).

So before I continue down this path and some painful debugging..., does anyone know of a way to block a task, unblock it later and pass a value back? It does not need to support anything complicated such as multiple take/put requests etc.

Cheers,
Tamar

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Blocking a task indefinitely in the RTS

Phil Ruffwind
What if you wrap the MVar in a foreign closure?

    import Control.Concurrent.MVar (newEmptyMVar, putMVar, takeMVar)
    import Control.Exception (bracket)
    import Foreign.Ptr (FunPtr, freeHaskellFunPtr)

    foreign import ccall "wrapper" wrapAwaken :: IO () -> IO (FunPtr (IO ()))

    main = do
      mvar <- newEmptyMVar
      bracket (wrapAwaken (putMVar mvar ())) freeHaskellFunPtr $ \ awaken -> do
        -- giveToExternalCode awaken
        takeMVar mvar

On Sun, Jan 6, 2019, at 10:37, Phyx wrote:

> Hi All,
>
> I'm looking for a way to block a task indefinitely until it is woken up by
> an external event in both the threaded and non-threaded RTS and returns a
> value that was stored/passed. MVar works great for the threaded RTS, but
> for the non-threaded there's a bunch of deadlock detection in the scheduler
> that would forcibly free the lock and resume the task with an opaque
> exception. This means that MVar and anything derived from it is not usable.
>
> STMs are more expensive but also have the same deadlock code. So again no
> go. The reason it looks like a deadlock to the RTS is that the "Wake-up"
> call in the non-threaded rts will come from C code running inside the RTS.
> The RTS essentially just sees all tasks blocked on it's main capability and
> (usually rightly so) assumes a deadlock occurred.
>
> You have other states like BlockedOnForeign etc but those are not usable as
> a primitive. Previous iterations of I/O managers have each invented
> primitives for this such as asyncRead#, but they are not general and can't
> be re-used, and requires a different solution for threaded and non-threaded.
>
> I have started making a new primitive IOPort for this, based on the MVar
> code, but this is not trivial... (currently I'm getting a segfault
> *somewhere* in the primitive's cmm code). The reason is that the semantics
> are decidedly different from what MVars guarantee. I should also mention
> that this is meant to be internal to base (i.e no exported).
>
> So before I continue down this path and some painful debugging..., does
> anyone know of a way to block a task, unblock it later and pass a value
> back? It does not need to support anything complicated such as multiple
> take/put requests etc.
>
> Cheers,
> Tamar
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Blocking a task indefinitely in the RTS

John Lato-2
In reply to this post by Phyx
Can you use an os-level structure? E.g. block on a file descriptor, socket, or something like that?

On Sun, Jan 6, 2019, 10:37 Phyx <[hidden email] wrote:
Hi All,

I'm looking for a way to block a task indefinitely until it is woken up by an external event in both the threaded and non-threaded RTS and returns a value that was stored/passed. MVar works great for the threaded RTS, but for the non-threaded there's a bunch of deadlock detection in the scheduler that would forcibly free the lock and resume the task with an opaque exception. This means that MVar and anything derived from it is not usable.

STMs are more expensive but also have the same deadlock code. So again no go. The reason it looks like a deadlock to the RTS is that the "Wake-up" call in the non-threaded rts will come from C code running inside the RTS.  The RTS essentially just sees all tasks blocked on it's main capability and (usually rightly so) assumes a deadlock occurred.

You have other states like BlockedOnForeign etc but those are not usable as a primitive. Previous iterations of I/O managers have each invented primitives for this such as asyncRead#, but they are not general and can't be re-used, and requires a different solution for threaded and non-threaded.

I have started making a new primitive IOPort for this, based on the MVar code, but this is not trivial... (currently I'm getting a segfault *somewhere* in the primitive's cmm code). The reason is that the semantics are decidedly different from what MVars guarantee. I should also mention that this is meant to be internal to base (i.e no exported).

So before I continue down this path and some painful debugging..., does anyone know of a way to block a task, unblock it later and pass a value back? It does not need to support anything complicated such as multiple take/put requests etc.

Cheers,
Tamar
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Blocking a task indefinitely in the RTS

Phyx
In reply to this post by Phil Ruffwind
Hi Phil,

Thanks for the reply, however that just gives me a forced deadlock removal as before.

new bound thread (1)
cap 0: schedule()
cap 0: running thread 1 (ThreadRunGHC)
cap 0: thread 1 stopped (blocked on an MVar)
        thread    1 @ 0000000003205388 is blocked on an MVar @ 00000000032040c8 (TSO_DIRTY)
deadlocked, forcing major GC...
all threads:
threads on capability 0:
other threads:
        thread    1 @ 0000000003205388 is blocked on an MVar @ 00000000032040c8 (TSO_DIRTY)
cap 0: starting GC

I don't believe any solution involving MVars will work for the non-threaded RTS. Though I'd love to be wrong here...

Regards,
Tamar

On Sun, Jan 6, 2019 at 9:38 PM Phil Ruffwind <[hidden email]> wrote:
What if you wrap the MVar in a foreign closure?

    import Control.Concurrent.MVar (newEmptyMVar, putMVar, takeMVar)
    import Control.Exception (bracket)
    import Foreign.Ptr (FunPtr, freeHaskellFunPtr)

    foreign import ccall "wrapper" wrapAwaken :: IO () -> IO (FunPtr (IO ()))

    main = do
      mvar <- newEmptyMVar
      bracket (wrapAwaken (putMVar mvar ())) freeHaskellFunPtr $ \ awaken -> do
        -- giveToExternalCode awaken
        takeMVar mvar

On Sun, Jan 6, 2019, at 10:37, Phyx wrote:
> Hi All,
>
> I'm looking for a way to block a task indefinitely until it is woken up by
> an external event in both the threaded and non-threaded RTS and returns a
> value that was stored/passed. MVar works great for the threaded RTS, but
> for the non-threaded there's a bunch of deadlock detection in the scheduler
> that would forcibly free the lock and resume the task with an opaque
> exception. This means that MVar and anything derived from it is not usable.
>
> STMs are more expensive but also have the same deadlock code. So again no
> go. The reason it looks like a deadlock to the RTS is that the "Wake-up"
> call in the non-threaded rts will come from C code running inside the RTS.
> The RTS essentially just sees all tasks blocked on it's main capability and
> (usually rightly so) assumes a deadlock occurred.
>
> You have other states like BlockedOnForeign etc but those are not usable as
> a primitive. Previous iterations of I/O managers have each invented
> primitives for this such as asyncRead#, but they are not general and can't
> be re-used, and requires a different solution for threaded and non-threaded.
>
> I have started making a new primitive IOPort for this, based on the MVar
> code, but this is not trivial... (currently I'm getting a segfault
> *somewhere* in the primitive's cmm code). The reason is that the semantics
> are decidedly different from what MVars guarantee. I should also mention
> that this is meant to be internal to base (i.e no exported).
>
> So before I continue down this path and some painful debugging..., does
> anyone know of a way to block a task, unblock it later and pass a value
> back? It does not need to support anything complicated such as multiple
> take/put requests etc.
>
> Cheers,
> Tamar
> _______________________________________________
> ghc-devs mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Blocking a task indefinitely in the RTS

Phyx
In reply to this post by John Lato-2
Hi John,

I can, but the only reason that would work is because a blocking foreign call is except from the
deadlock detection code.  I can't block on the Handle itself as all the I/O calls are non-blocking for performance reasons.

Instead what I want to do is just pause the current task, which means I don't have to allocate OS level locks for each task
and have a mutable state for each task, which is essentially what an MVar does.  My other concern is that using OS primitives
would hit performance a bit, which increases as the load increases. So I'd much rather do it purely in the Haskell world.

Kind regards,
Tamar

On Mon, Jan 7, 2019 at 5:44 AM John Lato <[hidden email]> wrote:
Can you use an os-level structure? E.g. block on a file descriptor, socket, or something like that?

On Sun, Jan 6, 2019, 10:37 Phyx <[hidden email] wrote:
Hi All,

I'm looking for a way to block a task indefinitely until it is woken up by an external event in both the threaded and non-threaded RTS and returns a value that was stored/passed. MVar works great for the threaded RTS, but for the non-threaded there's a bunch of deadlock detection in the scheduler that would forcibly free the lock and resume the task with an opaque exception. This means that MVar and anything derived from it is not usable.

STMs are more expensive but also have the same deadlock code. So again no go. The reason it looks like a deadlock to the RTS is that the "Wake-up" call in the non-threaded rts will come from C code running inside the RTS.  The RTS essentially just sees all tasks blocked on it's main capability and (usually rightly so) assumes a deadlock occurred.

You have other states like BlockedOnForeign etc but those are not usable as a primitive. Previous iterations of I/O managers have each invented primitives for this such as asyncRead#, but they are not general and can't be re-used, and requires a different solution for threaded and non-threaded.

I have started making a new primitive IOPort for this, based on the MVar code, but this is not trivial... (currently I'm getting a segfault *somewhere* in the primitive's cmm code). The reason is that the semantics are decidedly different from what MVars guarantee. I should also mention that this is meant to be internal to base (i.e no exported).

So before I continue down this path and some painful debugging..., does anyone know of a way to block a task, unblock it later and pass a value back? It does not need to support anything complicated such as multiple take/put requests etc.

Cheers,
Tamar
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Blocking a task indefinitely in the RTS

Phil Ruffwind
In reply to this post by Phyx
Strange, how could the scheduler assume a deadlock if the MVar could be called from a closure that is still alive?  Can you show the code that you're testing?  

On Mon, Jan 7, 2019, at 14:09, Phyx wrote:

> Hi Phil,
>
> Thanks for the reply, however that just gives me a forced deadlock removal
> as before.
>
> new bound thread (1)
> cap 0: schedule()
> cap 0: running thread 1 (ThreadRunGHC)
> cap 0: thread 1 stopped (blocked on an MVar)
>         thread    1 @ 0000000003205388 is blocked on an MVar @
> 00000000032040c8 (TSO_DIRTY)
> deadlocked, forcing major GC...
> all threads:
> threads on capability 0:
> other threads:
>         thread    1 @ 0000000003205388 is blocked on an MVar @
> 00000000032040c8 (TSO_DIRTY)
> cap 0: starting GC
>
> I don't believe any solution involving MVars will work for the non-threaded
> RTS. Though I'd love to be wrong here...
>
> Regards,
> Tamar
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Blocking a task indefinitely in the RTS

Phyx
I have simply copy pasted the code you provided. Note that my actual code code doesn't pass anything to a foreign interface. It stores everything in a Haskell mutable object.
The RTS on completion of actions simply schedules a task which inspects this objects and wakes up as many blocked tasks as possible.

> Strange, how could the scheduler assume a deadlock if the MVar could be called from a closure that is still alive?  Can you show the code that you're testing?

Because as far as I can tell, it doesn't care. When it comes to MVars and STMs the scheduler assumes a deadlock when all tasks on all capabilities are blocked.
For the threaded runtime it has an early exit condition from this code in the case where there has been any activity in a complete timeslice or when you're blocked on
specific calls such as I/O.

On the non-threaded runtime the timeslice case doesn't apply and you only have one capability, it will force a GC to try to revive some tasks, and if at the end of
this the tasks are still blocked it will release one in order to attempt to proceed. In short, as far as I can tell I don't think it considers reach-ability at all, not for MVars.

This is why things e.g. doing a takeMVar will also process pending puts etc, so that if you actually enter a blocked state, you know you had no other choice.

On Mon, Jan 7, 2019 at 11:24 PM Phil Ruffwind <[hidden email]> wrote:
Strange, how could the scheduler assume a deadlock if the MVar could be called from a closure that is still alive?  Can you show the code that you're testing? 

On Mon, Jan 7, 2019, at 14:09, Phyx wrote:
> Hi Phil,
>
> Thanks for the reply, however that just gives me a forced deadlock removal
> as before.
>
> new bound thread (1)
> cap 0: schedule()
> cap 0: running thread 1 (ThreadRunGHC)
> cap 0: thread 1 stopped (blocked on an MVar)
>         thread    1 @ 0000000003205388 is blocked on an MVar @
> 00000000032040c8 (TSO_DIRTY)
> deadlocked, forcing major GC...
> all threads:
> threads on capability 0:
> other threads:
>         thread    1 @ 0000000003205388 is blocked on an MVar @
> 00000000032040c8 (TSO_DIRTY)
> cap 0: starting GC
>
> I don't believe any solution involving MVars will work for the non-threaded
> RTS. Though I'd love to be wrong here...
>
> Regards,
> Tamar

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Blocking a task indefinitely in the RTS

Phil Ruffwind
Okay, I skimmed rts/Schedule.c and now see the problem you mentioned :(

> On the non-threaded runtime the timeslice case doesn't apply and you only
> have one capability, it will force a GC to try to revive some tasks, and if
> at the end of
> this the tasks are still blocked it will release one in order to attempt to
> proceed. In short, as far as I can tell I don't think it considers
> reach-ability at all, not for MVars.

Maybe that should be considered a false positive (bug) for the deadlock checker?  Just because the Haskell runtime has a single thread, that doesn't imply the whole program is necessarily single-threaded (in the presence of foreign things).  I'd think this is a legitimate use case for MVars.
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Blocking a task indefinitely in the RTS

Phyx
> Maybe that should be considered a false positive (bug) for the deadlock checker?  Just because the Haskell runtime has a single thread, that doesn't imply the whole program is necessarily single-threaded (in the presence of foreign things).  I'd think this is a legitimate use case for MVars.

Perhaps, I can see it both ways really :(. The papers and docs for MVars put a strong emphasis on this deadlock guarantee (though I've really only skimmed the paper), taking foreign calls into consideration does somewhat weaken this of course. I do agree with you that this wasn't the behavior I expected from MVars and STMs, particularly because they are such low level building blocks, for instance QSem is constructed with MVar, the docs make no mention of this but I suspect QSem and the rest all don't do what you'd expect on the non-threaded RTS.

I did try removing this check to see, but it really didn't like that. It caused GC to be triggered over and over again as the RTS tried desperately to find something to do, doesn't seem to consider "do nothing" as a valid state.

On Tue, Jan 8, 2019 at 6:59 AM Phil Ruffwind <[hidden email]> wrote:
Okay, I skimmed rts/Schedule.c and now see the problem you mentioned :(

> On the non-threaded runtime the timeslice case doesn't apply and you only
> have one capability, it will force a GC to try to revive some tasks, and if
> at the end of
> this the tasks are still blocked it will release one in order to attempt to
> proceed. In short, as far as I can tell I don't think it considers
> reach-ability at all, not for MVars.

Maybe that should be considered a false positive (bug) for the deadlock checker?  Just because the Haskell runtime has a single thread, that doesn't imply the whole program is necessarily single-threaded (in the presence of foreign things).  I'd think this is a legitimate use case for MVars.

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Blocking a task indefinitely in the RTS

Phil Ruffwind
> I did try removing this check to see, but it really didn't like that. It
> caused GC to be triggered over and over again as the RTS tried desperately
> to find something to do, doesn't seem to consider "do nothing" as a valid
> state.

Oh, I see :(  I guess it's not that easy of a fix then.  Perhaps the RTS could use a new intrinsic for blocking on foreign state.
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Blocking a task indefinitely in the RTS

Phyx
> Oh, I see :(  I guess it's not that easy of a fix then.  Perhaps the RTS could use a new intrinsic for blocking on foreign state

Yeah, that's what I was/am currently working on, "IOPort" has much of the same property of MVar but doesn't have this deadlock guarantee and only supports a single put/take at a time.
But debugging CMM is... not fun :( so I was wondering if I was just missing something with the existing mechanisms.


On Tue, Jan 8, 2019 at 8:23 AM Phil Ruffwind <[hidden email]> wrote:
> I did try removing this check to see, but it really didn't like that. It
> caused GC to be triggered over and over again as the RTS tried desperately
> to find something to do, doesn't seem to consider "do nothing" as a valid
> state.

Oh, I see :(  I guess it's not that easy of a fix then.  Perhaps the RTS could use a new intrinsic for blocking on foreign state.

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Blocking a task indefinitely in the RTS

Carter Schonwald
What’s the underlying problem you’re trying to model?

On Tue, Jan 8, 2019 at 3:56 AM Phyx <[hidden email]> wrote:
> Oh, I see :(  I guess it's not that easy of a fix then.  Perhaps the RTS could use a new intrinsic for blocking on foreign state

Yeah, that's what I was/am currently working on, "IOPort" has much of the same property of MVar but doesn't have this deadlock guarantee and only supports a single put/take at a time.
But debugging CMM is... not fun :( so I was wondering if I was just missing something with the existing mechanisms.


On Tue, Jan 8, 2019 at 8:23 AM Phil Ruffwind <[hidden email]> wrote:
> I did try removing this check to see, but it really didn't like that. It
> caused GC to be triggered over and over again as the RTS tried desperately
> to find something to do, doesn't seem to consider "do nothing" as a valid
> state.

Oh, I see :(  I guess it's not that easy of a fix then.  Perhaps the RTS could use a new intrinsic for blocking on foreign state.
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Reply | Threaded
Open this post in threaded view
|

Re: Blocking a task indefinitely in the RTS

Phyx
I'm trying to model an Asynchronous I/O call/interface against a synchronous Haskell call.
I want the behavior of an unsafe blocking foreign call, without blocking in the foreign call itself.



On Tue, Jan 8, 2019 at 6:24 PM Carter Schonwald <[hidden email]> wrote:
What’s the underlying problem you’re trying to model?

On Tue, Jan 8, 2019 at 3:56 AM Phyx <[hidden email]> wrote:
> Oh, I see :(  I guess it's not that easy of a fix then.  Perhaps the RTS could use a new intrinsic for blocking on foreign state

Yeah, that's what I was/am currently working on, "IOPort" has much of the same property of MVar but doesn't have this deadlock guarantee and only supports a single put/take at a time.
But debugging CMM is... not fun :( so I was wondering if I was just missing something with the existing mechanisms.


On Tue, Jan 8, 2019 at 8:23 AM Phil Ruffwind <[hidden email]> wrote:
> I did try removing this check to see, but it really didn't like that. It
> caused GC to be triggered over and over again as the RTS tried desperately
> to find something to do, doesn't seem to consider "do nothing" as a valid
> state.

Oh, I see :(  I guess it's not that easy of a fix then.  Perhaps the RTS could use a new intrinsic for blocking on foreign state.
_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

_______________________________________________
ghc-devs mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs