Hi all, I've hit a problem that feels like it has a straight-forward answer.--
I have a large XML file that I'd like to split up into subfiles of roughly equal size. My first pass looks like:
My intent is to stream groups of 1000 XML elements (and their children) to separate files. Luckily the XML in question is only ever one layer deep, like:
So this `foo` group here would count as 1 written element, not 5.
What stands out right away is the type signature of `element`. The `[T.Text]` feels very unidiomatic, but I couldn't think of another way to group all the parent and child nodes together in such a way that `osm` would know it had processed 1000 of such groups.
I read the `pipes-group` tutorial, but it wasn't immediately clear to me if that's what I needed. I do know that at maximum any given `<foo>` parent can only have a few hundred `<bar>` children, but that still breaks the output streaming as I wait for the List to populate.
Question: How can I structure things such that `osm` knows when to start writing to a new file?
PS. `osm` defined as it currently is probably also ends up in an infinite loop.
Yes, the `pipes-group` library is what you need. I recommend reading this if you haven't done so already:
... so the type of `element` would become:element :: Monad m => Producer Text m r -> FreeT (Producer Text m) m r
The result is a "stream of streams" that preserves the lazy streaming behavior of each sub-stream so that you don't have to wait to collect all children before processing them.
There is also another library you should check out which is more specialized to this particular use case which is the `streaming` library:
On 01/06/2017 05:00 PM, Colin Woodbury wrote:
|Free forum by Nabble||Edit this page|