Conduits and large ConduitResult chunks

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Conduits and large ConduitResult chunks

Nathan Howell-2
I'm porting lzma-enumerator over to conduits and I've run into a snag.
The output chunks from lzma can be quite large, so I'd like to stream
the results out in smaller chunks instead of tens (or hundreds) of
megabytes at a time. It seems as though this should be possible, as it
is with enumeratees, but I can't see how. Is there a way to get this
working without unsafeInterleaveIO?

https://github.com/alphaHeavy/lzma-conduit/blob/2b1451f5596b2db5a23df9a0fb128c7444e112b2/src/Data/Conduit/Lzma.hs#L165

thanks,
-n

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Conduits and large ConduitResult chunks

Michael Snoyman
On Thu, Mar 1, 2012 at 11:50 PM, Nathan Howell
<[hidden email]> wrote:

> I'm porting lzma-enumerator over to conduits and I've run into a snag.
> The output chunks from lzma can be quite large, so I'd like to stream
> the results out in smaller chunks instead of tens (or hundreds) of
> megabytes at a time. It seems as though this should be possible, as it
> is with enumeratees, but I can't see how. Is there a way to get this
> working without unsafeInterleaveIO?
>
> https://github.com/alphaHeavy/lzma-conduit/blob/2b1451f5596b2db5a23df9a0fb128c7444e112b2/src/Data/Conduit/Lzma.hs#L165
>
> thanks,
> -n
>
> _______________________________________________
> Haskell-Cafe mailing list
> [hidden email]
> http://www.haskell.org/mailman/listinfo/haskell-cafe

I just wrote a blog post on this topic:
http://www.yesodweb.com/blog/2012/03/more-powerful-conduit

Feedback requested!

Michael

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe