Proposal: Control.Concurrent.Async

[Simon Marlow-7]

I'd like to add a higher-level concurrency API to the base package.
This API has grown while I've been working on examples for my book, and
I'm convinced that we should have something like this in the standard
libraries.  Here's the API I propose:

http://community.haskell.org/~simonmar/async-stm/Control-Concurrent-Async.html

In fact it already exists in at least two packages on Hackage: 'async'
and 'threads'.  My version is a superset of both of these, except for a
few small differences (which are up for discussion, of course).

One good reason to have this package is that it avoids some common
pitfalls, such as forgetting to handle exceptions in your child threads.
  In the Async API, you get to choose whether to (a) receive the result
as 'Either SomeException a', or (b) have the exception re-thrown in the
waiting thread.  You don't get to ignore it altogether.

Another common pitfall avoided by this library is accidentally leaving
threads running in the background.  This is handled by withAsync:

   withAsync :: IO a -> (Async a -> IO b) -> IO b

the child thread is always cancelled (i.e. killed) when the second
argument returns or throws an exception, if it hasn't finished already.


I'm opening this up for discussion so that we can tidy up any
inconsistent details and add any functions that people feel are missing.
  Naming is obviously up for discussion too.

Cheers,
        Simon



_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Michael Snoyman]

On Jun 8, 2012 11:37 AM, "Simon Marlow" <[hidden email]> wrote:
>
> I'd like to add a higher-level concurrency API to the base package. This API has grown while I've been working on examples for my book, and I'm convinced that we should have something like this in the standard libraries.  Here's the API I propose:
>
> http://community.haskell.org/~simonmar/async-stm/Control-Concurrent-Async.html
>
> In fact it already exists in at least two packages on Hackage: 'async' and 'threads'.  My version is a superset of both of these, except for a few small differences (which are up for discussion, of course).
>
> One good reason to have this package is that it avoids some common pitfalls, such as forgetting to handle exceptions in your child threads.  In the Async API, you get to choose whether to (a) receive the result as 'Either SomeException a', or (b) have the exception re-thrown in the waiting thread.  You don't get to ignore it altogether.
>
> Another common pitfall avoided by this library is accidentally leaving threads running in the background.  This is handled by withAsync:
>
>  withAsync :: IO a -> (Async a -> IO b) -> IO b
>
> the child thread is always cancelled (i.e. killed) when the second argument returns or throws an exception, if it hasn't finished already.
>
>
> I'm opening this up for discussion so that we can tidy up any inconsistent details and add any functions that people feel are missing.  Naming is obviously up for discussion too.
>
> Cheers,
>        Simon
>
>
>
> _______________________________________________
> Libraries mailing list
> [hidden email]
> http://www.haskell.org/mailman/listinfo/libraries

What's the advantage to putting this in base instead of a separate package? If it goes in base, it will make it more difficult to upgrade, and take longer for this module to be adopted at all. If possible, I'd opt for a standalone package.

Michael

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Ian Lynagh]

On Fri, Jun 08, 2012 at 12:41:29PM +0300, Michael Snoyman wrote:
> On Jun 8, 2012 11:37 AM, "Simon Marlow" <[hidden email]> wrote:
> >
> > I'd like to add a higher-level concurrency API to the base package.
>
> What's the advantage to putting this in base instead of a separate package?
> If it goes in base, it will make it more difficult to upgrade, and take
> longer for this module to be adopted at all. If possible, I'd opt for a
> standalone package.

I agree. This doesn't sound like it needs to be a package shipped with
GHC, let alone in the base package.


Thanks
Ian


_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Simon Marlow-7]

On 08/06/2012 10:41, Michael Snoyman wrote:

> What's the advantage to putting this in base instead of a separate
> package? If it goes in base, it will make it more difficult to upgrade,
> and take longer for this module to be adopted at all. If possible, I'd
> opt for a standalone package.

Uploading it to Hackage is certainly an option, and there are arguments
in both directions.  My thinking was:

  - it's a bit small for a package by itself. There's a lot of
    overhead for a package (github repo, Haskell Platform proposal,
    issue tracker, blah blah)

  - To avoid further fragmentation, I would like this package to
    be more visible, especially if we go to the trouble as a community
    of building some consensus around it.

  - The obvious package name 'async' is already taken


Cheers,
        Simon

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Michael Snoyman]

On Jun 8, 2012 1:07 PM, "Simon Marlow" <[hidden email]> wrote:
>
> On 08/06/2012 10:41, Michael Snoyman wrote:
>
>> What's the advantage to putting this in base instead of a separate
>> package? If it goes in base, it will make it more difficult to upgrade,
>> and take longer for this module to be adopted at all. If possible, I'd
>> opt for a standalone package.
>
>
> Uploading it to Hackage is certainly an option, and there are arguments in both directions.  My thinking was:
>
>  - it's a bit small for a package by itself. There's a lot of
>   overhead for a package (github repo, Haskell Platform proposal,
>   issue tracker, blah blah)
>
>  - To avoid further fragmentation, I would like this package to
>   be more visible, especially if we go to the trouble as a community
>   of building some consensus around it.
>
>  - The obvious package name 'async' is already taken
>
>
> Cheers,
>        Simon

Maybe we could address some of those issues: perhaps Peter Simons would be willing to forfeit the package name in favor of this updated one. That would provide more consolidation, and let the community-driven package use the obvious name.

Michael

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Simon Marlow-7]

On 08/06/2012 11:29, Michael Snoyman wrote:

>
> On Jun 8, 2012 1:07 PM, "Simon Marlow" <[hidden email]
> <mailto:[hidden email]>> wrote:
>  >
>  > On 08/06/2012 10:41, Michael Snoyman wrote:
>  >
>  >> What's the advantage to putting this in base instead of a separate
>  >> package? If it goes in base, it will make it more difficult to upgrade,
>  >> and take longer for this module to be adopted at all. If possible, I'd
>  >> opt for a standalone package.
>  >
>  >
>  > Uploading it to Hackage is certainly an option, and there are
> arguments in both directions.  My thinking was:
>  >
>  >  - it's a bit small for a package by itself. There's a lot of
>  >   overhead for a package (github repo, Haskell Platform proposal,
>  >   issue tracker, blah blah)
>  >
>  >  - To avoid further fragmentation, I would like this package to
>  >   be more visible, especially if we go to the trouble as a community
>  >   of building some consensus around it.
>  >
>  >  - The obvious package name 'async' is already taken
>  >
>  >
>  > Cheers,
>  >        Simon
>
> Maybe we could address some of those issues: perhaps Peter Simons would
> be willing to forfeit the package name in favor of this updated one.
> That would provide more consolidation, and let the community-driven
> package use the obvious name.

Peter let me know that he's happy for me to reuse the 'async' name.
I've uploaded the code to github so people can play around with it and
send me patches easily:

https://github.com/simonmar/async

Cheers,
        Simon

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Ryan Newton]

+1

 

What's the advantage to putting this in base instead of a separate
package? If it goes in base, it will make it more difficult to upgrade,
and take longer for this module to be adopted at all. If possible, I'd
opt for a standalone package.

I thought the idea was that we would try to point people towards using these as a best practice as opposed to using forkIO?

I find that a helper function to fork a child thread with error handling is something I'm always pasting into various projects [1], which I take as a sign it should be easily available.  I never really thought about looking for better threads packages on hackage, because I didn't want to add an extra dependency (or deal with something big and complex).  But if it were standard I would have picked it up!

  -Ryan

[1] P.S.  Non-sequitor: The other little helper function I'm ALWAYS pasting into files is a function to print large numbers with interspersed commas.  I'd like to see THAT in the standard ;-), but I realize there are internationalization issues.

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Thomas Schilling-2]

On 8 June 2012 11:07, Simon Marlow <[hidden email]> wrote:

> On 08/06/2012 10:41, Michael Snoyman wrote:
>
>> What's the advantage to putting this in base instead of a separate
>> package? If it goes in base, it will make it more difficult to upgrade,
>> and take longer for this module to be adopted at all. If possible, I'd
>> opt for a standalone package.
>
>
> Uploading it to Hackage is certainly an option, and there are arguments in
> both directions.  My thinking was:
>
>  - it's a bit small for a package by itself. There's a lot of
>   overhead for a package (github repo, Haskell Platform proposal,
>   issue tracker, blah blah)
>
>  - To avoid further fragmentation, I would like this package to
>   be more visible, especially if we go to the trouble as a community
>   of building some consensus around it.
>
>  - The obvious package name 'async' is already taken

I am very much against putting things into base unless there is a
*very* good argument to do this.  If it is supposed to be *the*
standard package, then it belongs into the platform, not base.  If the
platform process is too heavy-weight to discourage this path, then we
need to do something about that rather than sneak things in through
the base library.  It already is a pain to work around issues in the
base library (and many other core packages) since it's (a) takes a
long time for an update to be released, and (b) cannot be upgraded
independently from GHC (pulling in all the changes to other libraries
as well).

I'm glad that the name issues is resolved.  If it is a small library a
platform addition proposal shouldn't be too difficult either.  We just
have to beware of too much bike shedding.

(BTW, have we had another platform proposal since the text package?)

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Thomas Schilling-2]

I fixed the docs a tiny bit in: https://github.com/simonmar/async/pull/1

Other comments:

  - Module introduction is missing.

  - There are no tests whatsoever.  I would like to have a few stress
tests that throw lots of asynchronous exceptions to make sure that
exception masking etc behaves correctly.

  - Why is there no waitAnyCancel?
  - Same with waitEitherCancel

On 8 June 2012 09:37, Simon Marlow <[hidden email]> wrote:

> I'd like to add a higher-level concurrency API to the base package. This API
> has grown while I've been working on examples for my book, and I'm convinced
> that we should have something like this in the standard libraries.  Here's
> the API I propose:
>
> http://community.haskell.org/~simonmar/async-stm/Control-Concurrent-Async.html
>
> In fact it already exists in at least two packages on Hackage: 'async' and
> 'threads'.  My version is a superset of both of these, except for a few
> small differences (which are up for discussion, of course).
>
> One good reason to have this package is that it avoids some common pitfalls,
> such as forgetting to handle exceptions in your child threads.  In the Async
> API, you get to choose whether to (a) receive the result as 'Either
> SomeException a', or (b) have the exception re-thrown in the waiting thread.
>  You don't get to ignore it altogether.
>
> Another common pitfall avoided by this library is accidentally leaving
> threads running in the background.  This is handled by withAsync:
>
>  withAsync :: IO a -> (Async a -> IO b) -> IO b
>
> the child thread is always cancelled (i.e. killed) when the second argument
> returns or throws an exception, if it hasn't finished already.
>
>
> I'm opening this up for discussion so that we can tidy up any inconsistent
> details and add any functions that people feel are missing.  Naming is
> obviously up for discussion too.
>
> Cheers,
>        Simon
>
>
>
> _______________________________________________
> Libraries mailing list
> [hidden email]
> http://www.haskell.org/mailman/listinfo/libraries

--
Push the envelope. Watch it bend.

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Heinrich Apfelmus]

Thomas Schilling wrote:
> On 8 June 2012 11:07, Simon Marlow <[hidden email]> wrote:
>>
>> Uploading it to Hackage is certainly an option, and there are arguments in
>> both directions.  My thinking was:
>> [..]
>>  - it's a bit small for a package by itself. There's a lot of
>>   overhead for a package (github repo, Haskell Platform proposal,
>>   issue tracker, blah blah)

Github provides much of the infrastructure by default, like an issue
tracker.

> I am very much against putting things into base unless there is a
> *very* good argument to do this.  If it is supposed to be *the*
> standard package, then it belongs into the platform, not base.  If the
> platform process is too heavy-weight to discourage this path, then we
> need to do something about that rather than sneak things in through
> the base library.  It already is a pain to work around issues in the
> base library (and many other core packages) since it's (a) takes a
> long time for an update to be released, and (b) cannot be upgraded
> independently from GHC (pulling in all the changes to other libraries
> as well).
>
> I'm glad that the name issues is resolved.  If it is a small library a
> platform addition proposal shouldn't be too difficult either.  We just
> have to beware of too much bike shedding.
>
> (BTW, have we had another platform proposal since the text package?)

+1 for a separate package that is included in the Haskell Platform.

If merging a package into the base library is easier than the process of
including it in the Haskell Platform, then the latter needs to be fixed.


Best regards,
Heinrich Apfelmus

--
http://apfelmus.nfshost.com


_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Herbert Valerio Riedel]

Simon Marlow <[hidden email]> writes:

>  - To avoid further fragmentation, I would like this package to
>    be more visible, especially if we go to the trouble as a community
>    of building some consensus around it.

...would it help to advertise the new `async` package in the appropriate
places in the Haddoc documentation for `base` to make it more visible?

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[haskell-2]

Your superset API is excellent.  I have one important suggestions: non-blocking
killThread.

By default, when your combinators like 'withAsync' and 'concurrently' use
'cancel' or 'killThread' they should use it in a non-blocking way.  The exported
'cancel' & 'cancelWith' should be unchanged and their documention should note
that they are blocking.

Consider 'concurrently'.  Perhaps there need to to be two 'concurrently'
versions?  One is blocking-killThread and guarantees that both are done or dead
when it returns and the other is non-blocking-killThread and returns ASAP.
Neither can be turned into the other by the user.

Rather than API duplication the 'concurrently' operation could get a new
parameter that selects blocking vs non-blocking killThread.

Hmmm.... the root problem is some Async tasks that, probably due to foreign
calls, are uninterpretable.  Maybe the user could label which Async tasks must
be killed blocking or non-blocking.  The default is non-blocking to avoid
infecting other threads with uninterpretable calls.  When the user needs the
extra property of guaranteed kills the user can change the label.  I have no
immediate idea whether a mutable or immutable label would be better.

--
Chris

On 2012/06/08 09:37, Simon Marlow wrote:

> I'd like to add a higher-level concurrency API to the base package. This API has
> grown while I've been working on examples for my book, and I'm convinced that we
> should have something like this in the standard libraries.  Here's the API I
> propose:
>
> http://community.haskell.org/~simonmar/async-stm/Control-Concurrent-Async.html
>
> In fact it already exists in at least two packages on Hackage: 'async' and
> 'threads'.  My version is a superset of both of these, except for a few small
> differences (which are up for discussion, of course).
>
> One good reason to have this package is that it avoids some common pitfalls,
> such as forgetting to handle exceptions in your child threads.  In the Async
> API, you get to choose whether to (a) receive the result as 'Either
> SomeException a', or (b) have the exception re-thrown in the waiting thread.
> You don't get to ignore it altogether.
>
> Another common pitfall avoided by this library is accidentally leaving threads
> running in the background.  This is handled by withAsync:
>
>   withAsync :: IO a -> (Async a -> IO b) -> IO b
>
> the child thread is always cancelled (i.e. killed) when the second argument
> returns or throws an exception, if it hasn't finished already.
>
>
> I'm opening this up for discussion so that we can tidy up any inconsistent
> details and add any functions that people feel are missing.  Naming is obviously
> up for discussion too.
>
> Cheers,
>     Simon
>
>
>
> _______________________________________________
> Libraries mailing list
> [hidden email]
> http://www.haskell.org/mailman/listinfo/libraries

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Iavor Diatchki]

Hello,

A nice library, I've had to re-implement similar functionality, so this would be useful!

Some questions about the waiting combinators:

1. I was expecting that `waitAny` and friends, would also return the result of the first computation that completed.  Is there a reason not to do so?

2. Is `waitAny` waiting for one of the Asyncs to complete _successfully_ or is throwing an exception considered as completion?  I am guessing the former, with `waitAnyThrow` providing the latter functionality?

3. As far as I understand, `waitAny` does not stop the other asyncs (it'd be nice to add this note to its documentation, explicitly).  If that's the case, perhaps it's type should be:  `waitAny :: [Async a] -> IO (a,[Async a])`, so that we can keep on waiting for more asyncs, if needed?

-Iavor

On Fri, Jun 8, 2012 at 1:37 AM, Simon Marlow <[hidden email]> wrote:

I'd like to add a higher-level concurrency API to the base package. This API has grown while I've been working on examples for my book, and I'm convinced that we should have something like this in the standard libraries.  Here's the API I propose:

http://community.haskell.org/~simonmar/async-stm/Control-Concurrent-Async.html

In fact it already exists in at least two packages on Hackage: 'async' and 'threads'.  My version is a superset of both of these, except for a few small differences (which are up for discussion, of course).

One good reason to have this package is that it avoids some common pitfalls, such as forgetting to handle exceptions in your child threads.  In the Async API, you get to choose whether to (a) receive the result as 'Either SomeException a', or (b) have the exception re-thrown in the waiting thread.  You don't get to ignore it altogether.

Another common pitfall avoided by this library is accidentally leaving threads running in the background.  This is handled by withAsync:

 withAsync :: IO a -> (Async a -> IO b) -> IO b

the child thread is always cancelled (i.e. killed) when the second argument returns or throws an exception, if it hasn't finished already.


I'm opening this up for discussion so that we can tidy up any inconsistent details and add any functions that people feel are missing.  Naming is obviously up for discussion too.

Cheers,
       Simon



_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Simon Marlow-7]

On 09/06/12 17:50, Iavor Diatchki wrote:

> Hello,
> A nice library, I've had to re-implement similar functionality, so this
> would be useful!
>
> Some questions about the waiting combinators:
>
> 1. I was expecting that `waitAny` and friends, would also return the
> result of the first computation that completed.  Is there a reason not
> to do so?
> 2. Is `waitAny` waiting for one of the Asyncs to complete _successfully_
> or is throwing an exception considered as completion?  I am guessing the
> former, with `waitAnyThrow` providing the latter functionality?
> 3. As far as I understand, `waitAny` does not stop the other asyncs
> (it'd be nice to add this note to its documentation, explicitly).  If
> that's the case, perhaps it's type should be:  `waitAny :: [Async a] ->
> IO (a,[Async a])`, so that we can keep on waiting for more asyncs, if
> needed?

All good points.  I don't have many use cases for the waitAny* family
yet so I'd been holding off refining the API until I had a clearer
picture of what would be useful for clients.  There are lots of
possibilities here.  But in light of your comments I have changed the
type of waitAny to this:

  waitAny :: [Async a] -> IO (Async a, Either SomeException a)

so that it now returns both the Async that completed and its result.
Since Async supports Eq, the caller can remove the completed one from
the list and call waitAny again if necessary.

Similarly waitAnyThrow now has this type:

  waitAnyThrow :: [Async a] -> IO (Async a, a)

I've also added a non-blocking wait:

  tryWait :: Async a -> IO (Maybe (Either SomeException a))

so that after a waitAny the caller can check all of the Asyncs for
completion and remove multiple completed Asyncs from the list.

Latest Haddocks here:

http://community.haskell.org/~simonmar/async/

Cheers,
        Simon

> -Iavor
>
>
>
>
> On Fri, Jun 8, 2012 at 1:37 AM, Simon Marlow <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     I'd like to add a higher-level concurrency API to the base package.
>     This API has grown while I've been working on examples for my book,
>     and I'm convinced that we should have something like this in the
>     standard libraries.  Here's the API I propose:
>
>     http://community.haskell.org/~__simonmar/async-stm/Control-__Concurrent-Async.html
>     <http://community.haskell.org/~simonmar/async-stm/Control-Concurrent-Async.html>
>
>     In fact it already exists in at least two packages on Hackage:
>     'async' and 'threads'.  My version is a superset of both of these,
>     except for a few small differences (which are up for discussion, of
>     course).
>
>     One good reason to have this package is that it avoids some common
>     pitfalls, such as forgetting to handle exceptions in your child
>     threads.  In the Async API, you get to choose whether to (a) receive
>     the result as 'Either SomeException a', or (b) have the exception
>     re-thrown in the waiting thread.  You don't get to ignore it altogether.
>
>     Another common pitfall avoided by this library is accidentally
>     leaving threads running in the background.  This is handled by
>     withAsync:
>
>       withAsync :: IO a -> (Async a -> IO b) -> IO b
>
>     the child thread is always cancelled (i.e. killed) when the second
>     argument returns or throws an exception, if it hasn't finished already.
>
>
>     I'm opening this up for discussion so that we can tidy up any
>     inconsistent details and add any functions that people feel are
>     missing.  Naming is obviously up for discussion too.
>
>     Cheers,
>             Simon
>
>
>
>     _________________________________________________
>     Libraries mailing list
>     [hidden email] <mailto:[hidden email]>
>     http://www.haskell.org/__mailman/listinfo/libraries
>     <http://www.haskell.org/mailman/listinfo/libraries>
>
>

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[wren ng thornton]

On 6/9/12 4:32 AM, Herbert Valerio Riedel wrote:
> Simon Marlow<[hidden email]>  writes:
>
>>   - To avoid further fragmentation, I would like this package to
>>     be more visible, especially if we go to the trouble as a community
>>     of building some consensus around it.
>
> ...would it help to advertise the new `async` package in the appropriate
> places in the Haddoc documentation for `base` to make it more visible?

I wonder if it'd be worthwhile to have a "metabase package" which
includes all the modules which are, used to be, or 'should' be in base.
What I mean is not to have an actual package (in the ghc/hackage sense),
but just to have a single unified Haddock site that covers and
advertises all of:

* base

* the things broken out from base-3.0 (array, bytestring, containers,
directory, old-locale, old-time, packedstring, pretty, process,
random)--- modulo things like packedstring which have been deprecated.

* things broken out from base-4.0 (syb,...)

* things which 'should' be part of base (text? vector? async?)

* but not including all of the HP

Perhaps this already exists in an official incarnation somewhere, but if
so then it's poorly advertised. The url should be listed in the
description of the base package on Hackage, at the very least.

--
Live well,
~wren

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[wren ng thornton]

On 6/9/12 5:28 PM, Simon Marlow wrote:
> All good points. I don't have many use cases for the waitAny* family yet
> so I'd been holding off refining the API until I had a clearer picture
> of what would be useful for clients. There are lots of possibilities
> here.

The way I typically use it is as follows: The main thread spawns off a
bunch of heterogeneous workers--- the kind that interact with one
another in an event-loop like manner (not of the fire-and-forget nor the
web-server kind). But, the workers may fail/terminate for various
reasons--- including thrown exceptions, but also more benign things like
expected failure modes (encoded in normal ADTs or passed along via
TChans/TVars/etc), and temporary failures that just require some
'manual' help to get unstuck. After firing them off to do the work, the
main thread's job is to monitor the workers in order to (a) handle any
of the failure modes it can, (b) ensure clean and safe shutdown in the
event of irrecoverable errors.

A related but somewhat different style of using waitAny* parallelism is
for things like grid managers. Again, there are a bunch of heterogeneous
worker threads. But this time, rather than focusing on errorful
termination as above, the focus here is on proper termination. In
particular, once some job A has finished, we can now start jobs B, C,
and D which depend on the output of A. In the small scale, you can just
have A finish by firing off the other jobs; but in the large scale that
doesn't work. You need to (a) be able to specify the job dependencies
independently of the work that job is doing, (b) be able to handle
failures and restarts as above, (c) perform load balancing[1], and so on.



[1] For a proper grid manager this is "real" load balancing by deciding
which machines a particular job should be run on. But even within a
single OS process, we can know that certain tasks require more or less
memory or other resources, and so can alter how aggressively to fire off
jobs in order to keep from overloading the machine. It's this extreme
heterogeneity of jobs which is a core part of being a "grid manager"
IMO; when the jobs are relatively homogeneous (even if doing different
things) then a lot of these complications go away.

--
Live well,
~wren

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Simon Marlow-7]

On 09/06/2012 09:50, Chris Kuklewicz wrote:

> Your superset API is excellent.  I have one important suggestions: non-blocking
> killThread.
>
> By default, when your combinators like 'withAsync' and 'concurrently' use
> 'cancel' or 'killThread' they should use it in a non-blocking way.  The exported
> 'cancel'&  'cancelWith' should be unchanged and their documention should note
> that they are blocking.
>
> Consider 'concurrently'.  Perhaps there need to to be two 'concurrently'
> versions?  One is blocking-killThread and guarantees that both are done or dead
> when it returns and the other is non-blocking-killThread and returns ASAP.
> Neither can be turned into the other by the user.
>
> Rather than API duplication the 'concurrently' operation could get a new
> parameter that selects blocking vs non-blocking killThread.
>
> Hmmm.... the root problem is some Async tasks that, probably due to foreign
> calls, are uninterpretable.  Maybe the user could label which Async tasks must
> be killed blocking or non-blocking.  The default is non-blocking to avoid
> infecting other threads with uninterpretable calls.  When the user needs the
> extra property of guaranteed kills the user can change the label.  I have no
> immediate idea whether a mutable or immutable label would be better.

I appreciate that sometimes cancel may block and that you may want a
non-blocking cancel, but I don't think that should be the default.

Firstly, the user might find it surprising if when withAsync has
returned, the Async still performs some more operations (that are not
inside an exception handler).  I think it's an important guarantee that
this doesn't happen - otherwise, what can you say with any certainty
about the behaviour of withAsync?

Secondly, asynchronous cancel is easily implemented by the user, as

   asyncCancel = async . cancel

what a lovely definition :-)  We even get to decide later on to wait for
the cancel to happen, by waiting on the result of asyncCancel, and we
can cancel the asyncCancel itself.

Thirdly, asynchronous cancel is less efficient than synchronous cancel,
because it involves forking a new thread.


I will document carefully that we are using the synchronous cancel and
the implications of that, and I'll add asyncCancel.

Cheers,
        Simon

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Simon Marlow-7]

On 08/06/2012 17:31, Thomas Schilling wrote:
> I fixed the docs a tiny bit in: https://github.com/simonmar/async/pull/1
>
> Other comments:
>
>    - Module introduction is missing.

Now added.

>    - There are no tests whatsoever.  I would like to have a few stress
> tests that throw lots of asynchronous exceptions to make sure that
> exception masking etc behaves correctly.

Also added, including a couple of stress tests.  (but the test coverage
isn't great, I'll try to improve it in due course)

>    - Why is there no waitAnyCancel?
>    - Same with waitEitherCancel

Both added.

I think that now addresses all the comments that people had.

There's one more change I'm thinking about: perhaps instead of

   wait      :: Async a -> IO (Either SomeException a)
   waitThrow :: Async a -> IO a

we should reverse the naming scheme, rename waitThrow to wait and wait
to something else (waitCatch?).  Rationale: waitThrow seems to be the
version we need most often, and it's simpler to use.

   wait      :: Async a -> IO a
   waitCatch :: Async a -> IO (Either SomeException a)

Thoughts? Better names for waitCatch?

Latest Haddocks:

   http://community.haskell.org/~simonmar/async/

Repo:

   https://github.com/simonmar/async

Cheers,
        Simon

> On 8 June 2012 09:37, Simon Marlow<[hidden email]>  wrote:
>> I'd like to add a higher-level concurrency API to the base package. This API
>> has grown while I've been working on examples for my book, and I'm convinced
>> that we should have something like this in the standard libraries.  Here's
>> the API I propose:
>>
>> http://community.haskell.org/~simonmar/async-stm/Control-Concurrent-Async.html
>>
>> In fact it already exists in at least two packages on Hackage: 'async' and
>> 'threads'.  My version is a superset of both of these, except for a few
>> small differences (which are up for discussion, of course).
>>
>> One good reason to have this package is that it avoids some common pitfalls,
>> such as forgetting to handle exceptions in your child threads.  In the Async
>> API, you get to choose whether to (a) receive the result as 'Either
>> SomeException a', or (b) have the exception re-thrown in the waiting thread.
>>   You don't get to ignore it altogether.
>>
>> Another common pitfall avoided by this library is accidentally leaving
>> threads running in the background.  This is handled by withAsync:
>>
>>   withAsync :: IO a ->  (Async a ->  IO b) ->  IO b
>>
>> the child thread is always cancelled (i.e. killed) when the second argument
>> returns or throws an exception, if it hasn't finished already.
>>
>>
>> I'm opening this up for discussion so that we can tidy up any inconsistent
>> details and add any functions that people feel are missing.  Naming is
>> obviously up for discussion too.
>>
>> Cheers,
>>         Simon
>>
>>
>>
>> _______________________________________________
>> Libraries mailing list
>> [hidden email]
>> http://www.haskell.org/mailman/listinfo/libraries
>
>
>

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Gábor Lehel]

On Tue, Jun 12, 2012 at 12:58 PM, Simon Marlow <[hidden email]> wrote:

> I think that now addresses all the comments that people had.
>
> There's one more change I'm thinking about: perhaps instead of
>
>  wait      :: Async a -> IO (Either SomeException a)
>  waitThrow :: Async a -> IO a
>
> we should reverse the naming scheme, rename waitThrow to wait and wait to
> something else (waitCatch?).  Rationale: waitThrow seems to be the version
> we need most often, and it's simpler to use.
>
>  wait      :: Async a -> IO a
>  waitCatch :: Async a -> IO (Either SomeException a)
>
> Thoughts? Better names for waitCatch?
>
> Latest Haddocks:
>
>  http://community.haskell.org/~simonmar/async/

Two small ones:

- In the documentation for cancel: "Has no effect if the Async has
already", it seems like something might have been cut off.

- Can we find a more accurate name for tryWait? It doesn't try and
possibly fail to wait, it avoids waiting altogether. Maybe don'tWait,
although that doesn't say what it does do. The only other idea I have
is 'check' which is not very descriptive...

>
> Repo:
>
>  https://github.com/simonmar/async
>
> Cheers,
>        Simon
>
>
>
>
>> On 8 June 2012 09:37, Simon Marlow<[hidden email]>  wrote:
>>>
>>> I'd like to add a higher-level concurrency API to the base package. This
>>> API
>>> has grown while I've been working on examples for my book, and I'm
>>> convinced
>>> that we should have something like this in the standard libraries.
>>>  Here's
>>> the API I propose:
>>>
>>>
>>> http://community.haskell.org/~simonmar/async-stm/Control-Concurrent-Async.html
>>>
>>> In fact it already exists in at least two packages on Hackage: 'async'
>>> and
>>> 'threads'.  My version is a superset of both of these, except for a few
>>> small differences (which are up for discussion, of course).
>>>
>>> One good reason to have this package is that it avoids some common
>>> pitfalls,
>>> such as forgetting to handle exceptions in your child threads.  In the
>>> Async
>>> API, you get to choose whether to (a) receive the result as 'Either
>>> SomeException a', or (b) have the exception re-thrown in the waiting
>>> thread.
>>>  You don't get to ignore it altogether.
>>>
>>> Another common pitfall avoided by this library is accidentally leaving
>>> threads running in the background.  This is handled by withAsync:
>>>
>>>  withAsync :: IO a ->  (Async a ->  IO b) ->  IO b
>>>
>>> the child thread is always cancelled (i.e. killed) when the second
>>> argument
>>> returns or throws an exception, if it hasn't finished already.
>>>
>>>
>>> I'm opening this up for discussion so that we can tidy up any
>>> inconsistent
>>> details and add any functions that people feel are missing.  Naming is
>>> obviously up for discussion too.
>>>
>>> Cheers,
>>>        Simon
>>>
>>>
>>>
>>> _______________________________________________
>>> Libraries mailing list
>>> [hidden email]
>>> http://www.haskell.org/mailman/listinfo/libraries
>>
>>
>>
>>
>
>
> _______________________________________________
> Libraries mailing list
> [hidden email]
> http://www.haskell.org/mailman/listinfo/libraries

--
Work is punishment for failing to procrastinate effectively.

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Simon Marlow-7]

On 12/06/2012 12:38, Gábor Lehel wrote:

> On Tue, Jun 12, 2012 at 12:58 PM, Simon Marlow<[hidden email]>  wrote:
>> I think that now addresses all the comments that people had.
>>
>> There's one more change I'm thinking about: perhaps instead of
>>
>>   wait      :: Async a ->  IO (Either SomeException a)
>>   waitThrow :: Async a ->  IO a
>>
>> we should reverse the naming scheme, rename waitThrow to wait and wait to
>> something else (waitCatch?).  Rationale: waitThrow seems to be the version
>> we need most often, and it's simpler to use.
>>
>>   wait      :: Async a ->  IO a
>>   waitCatch :: Async a ->  IO (Either SomeException a)
>>
>> Thoughts? Better names for waitCatch?
>>
>> Latest Haddocks:
>>
>>   http://community.haskell.org/~simonmar/async/
>
> Two small ones:
>
> - In the documentation for cancel: "Has no effect if the Async has
> already", it seems like something might have been cut off.
>
> - Can we find a more accurate name for tryWait? It doesn't try and
> possibly fail to wait, it avoids waiting altogether. Maybe don'tWait,
> although that doesn't say what it does do. The only other idea I have
> is 'check' which is not very descriptive...

Good point.  How about "poll"?

Cheers,
        Simon

>
>>
>> Repo:
>>
>>   https://github.com/simonmar/async
>>
>> Cheers,
>>         Simon
>>
>>
>>
>>
>>> On 8 June 2012 09:37, Simon Marlow<[hidden email]>    wrote:
>>>>
>>>> I'd like to add a higher-level concurrency API to the base package. This
>>>> API
>>>> has grown while I've been working on examples for my book, and I'm
>>>> convinced
>>>> that we should have something like this in the standard libraries.
>>>>   Here's
>>>> the API I propose:
>>>>
>>>>
>>>> http://community.haskell.org/~simonmar/async-stm/Control-Concurrent-Async.html
>>>>
>>>> In fact it already exists in at least two packages on Hackage: 'async'
>>>> and
>>>> 'threads'.  My version is a superset of both of these, except for a few
>>>> small differences (which are up for discussion, of course).
>>>>
>>>> One good reason to have this package is that it avoids some common
>>>> pitfalls,
>>>> such as forgetting to handle exceptions in your child threads.  In the
>>>> Async
>>>> API, you get to choose whether to (a) receive the result as 'Either
>>>> SomeException a', or (b) have the exception re-thrown in the waiting
>>>> thread.
>>>>   You don't get to ignore it altogether.
>>>>
>>>> Another common pitfall avoided by this library is accidentally leaving
>>>> threads running in the background.  This is handled by withAsync:
>>>>
>>>>   withAsync :: IO a ->    (Async a ->    IO b) ->    IO b
>>>>
>>>> the child thread is always cancelled (i.e. killed) when the second
>>>> argument
>>>> returns or throws an exception, if it hasn't finished already.
>>>>
>>>>
>>>> I'm opening this up for discussion so that we can tidy up any
>>>> inconsistent
>>>> details and add any functions that people feel are missing.  Naming is
>>>> obviously up for discussion too.
>>>>
>>>> Cheers,
>>>>         Simon
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Libraries mailing list
>>>> [hidden email]
>>>> http://www.haskell.org/mailman/listinfo/libraries
>>>
>>>
>>>
>>>
>>
>>
>> _______________________________________________
>> Libraries mailing list
>> [hidden email]
>> http://www.haskell.org/mailman/listinfo/libraries
>
>
>

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries