Pipes vs Foldl: where to end one and start the other

Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
Report Content as Inappropriate

Pipes vs Foldl: where to end one and start the other

Chris Pollard

I am rewriting some old code, and I have a design question regarding the "right" way to interface pipes (or generic streams) and folds that may themselves be quite complex. Currently my setup goes something like this (this is pseudocode--haven't checked that it compiles/runs):

- I have one pipe that produces Events, where a simplified event might look like

data Event a b = Event { myA :: a, myBs :: [b] }

- I want to loop over events and store statistics of type r; I do this by implementing Fold(M)s:

foldA :: Fold a r

foldB :: Fold b r

- I combine the statistics using the Applicative or Monoid instances of Fold(M):

foldEvent :: Fold (Event a b) [r]
foldEvent = sequenceA
  [ premap myA foldA
  , premap myBs $ handles folded foldB

- I then connect these stats-gathering folds to my Event Producer, which works quite nicely.

My problem with this setup is one of complexity. In fact my Events are not so simple, and I want to categorize them in arbitrarily complicated ways and then feed them into the r statistics for each category. Currently all the complexity lives in the Fold side of my code, but it seems like Pipes are much better suited to handling non-trivial flows of information. I wonder if it's possible to have a complicated set of Pipes that connect (at the last second) to a bunch of Folds, which I then combine applicatively. There's a lot of Pipes infrastructure that could make the flow easier to understand, and I think the Pipes Monad instance may help clean up a lot of hard-to-read Fold code. In my silly example, Pipes.Prelude.map and Pipes.Prelude.mapFoldable are much easier to reason about (in my mind) than Foldl.premap and "handles folded".

Unfortunately, I can't see how to implement something as simple as my foldEvent using Pipes and still get the correct streaming properties. Is there a solution that I am missing, or should I focus on building up more composable primitives on the Fold side of my program?

Sorry if this is too vague: I don't want to dump a huge codebase on you, but I can provide more examples. Questions and comments welcome!