Re: conduit/pipes/streaming and prompt cleanup - using a decorator

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: conduit/pipes/streaming and prompt cleanup - using a decorator

Daniel Díaz Carrete
I tried to tackle this problem in my "streaming-bracketed" library http://hackage.haskell.org/package/streaming-bracketed, by using a "decorator" that wraps regular streams, as opposed to having a resource-aware base monad.

When lifting take-like functions to the decorator, new deallocation actions are inserted.

On Tue, Jun 4, 2019 at 2:53 PM <[hidden email]> wrote:
 
I'm using the 'streaming' library and realized it doesn't close files in a
timely way for the way I'm using it, and in fact can't, due to how the library
works.  I know conduit has put a lot of thought into closing resources in a
timely way, so I did an experiment to see what it does, but as far as I can
tell, conduit has the same problem.  Maybe I'm doing it wrong?

The situation is that I'm opening multiple files and mixing their output.
I want to close the inputs as soon as I'm done with them.  But I can get
done with them earlier than the end of the file.  Since all of these libraries
are based on pulling from downstream, if you don't pull all the way to the end,
the close at the end doesn't happen, and has to wait until runResourceT
returns, which is too late.  I remember long ago reading Oleg's original
iteratee paper, and it seems like he called out this problem with pull-based
iterators, that the iterator doesn't know when its caller is done with it, so
it can't close files on time.

Here's a conduit version that I think illustrates the situation:

    import qualified Conduit as C
    import           Conduit ((.|))
    import qualified Control.Monad.Trans as Trans
    import qualified System.IO as IO

    main :: IO ()
    main = C.runResourceT $ C.runConduit pipe

    pipe :: C.ConduitM a c (C.ResourceT IO) ()
    pipe = fileLines "TODO" .| (C.takeC 3 >> C.yield "***")
        .| C.mapM_C (Trans.liftIO . putStrLn)

    fileLines :: C.MonadResource m => FilePath -> C.ConduitT i String m ()
    fileLines fname = C.bracketP (IO.openFile fname IO.ReadMode) close
handleLines

    handleLines :: Trans.MonadIO m => IO.Handle -> C.ConduitT i String m ()
    handleLines hdl = loop
        where
        loop = do
            eof <- Trans.liftIO $ IO.hIsEOF hdl
            if eof then return () else do
                line <- Trans.liftIO $ IO.hGetLine hdl
                C.yield line
                loop

    close :: IO.Handle -> IO ()
    close hdl = IO.hClose hdl >> putStrLn "=== close"

This prints the first three lines of TOOD, then ***, and then "=== close",
where the close should go before the ***s.

As far as I can see, conduit can't do this any more than 'streaming' can,
because 'C.takeC' is just some awaits and yields, with no indication that
the final await is more special than any other await.

I think what would be necessary to solve this is that combinators like 'take'
have to be able to tell the stream to close, and that has to propagate back up
to each producer that has registered a cleanup.  Of course this renders the
stream invalid, so it's not safe to have any other references to the stream
around, but I guess streams are stateful in general so that's always true.
Maybe I could accumulate the finalizers in the stream data type and have
combinators like 'take' call it as soon as they've taken their last.  What
I actually wound up doing was make a 'takeClose' that also takes a 'close'
action to run when its done with the stream.  It's not exactly general but I'm
not writing a library so I don't need general.

Is there some kind of standard or built-in solution for this situation?
I know others have given a lot more thought to streaming than I have,
so surely this issue has come up.

I know there is a lot of talk about "prompt finalisation" and streams vs.
pipes vs. conduit, and talk about brackets and whatnot, but despite reading
various documents (https://hackage.haskell.org/package/streaming-with,
http://www.haskellforall.com/2013/01/pipes-safe-10-resource-management-and.html,
etc.) I still don't really understand what they're talking about.
It seems like they're really about reliable cleanup when there are
exceptions, not really about prompt cleanup.  Certainly pipes-safe doesn't do
prompt cleanup, at least not the kind I'm talking about.



_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: conduit/pipes/streaming and prompt cleanup - using a decorator

Evan Laforge
On Wed, Jun 5, 2019 at 1:30 PM Daniel Díaz <[hidden email]> wrote:
>
> I tried to tackle this problem in my "streaming-bracketed" library http://hackage.haskell.org/package/streaming-bracketed, by using a "decorator" that wraps regular streams, as opposed to having a resource-aware base monad.
>
> When lifting take-like functions to the decorator, new deallocation actions are inserted.

Oops, I didn't notice this response, sorry!

I did look at streaming-bracketed, but I think it doesn't apply in my
case, because the file lifespans are overlapping, not nested.  If I
understand correctly, that's the case where you need something like
resourcet, and brackets won't do.

In the end, I just gave 'take' a finalizer argument.  Not exactly
general purpose but it solved my problem.
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.