Say I have a `Pipes a b m r` called 'p'.--
How do I convert it to a `Pipes (ZipList a) (ZipList b) m r`?
That is, I want to apply the pipe 'p' and apply zip-wise to an input ziplist, without losing the state of each pipe in the ziplist.
The reason I want this, is that I want to run multiple attoparsec parsers in parallel in one pass of a file.
My approach for this is to use Control.Foldl to combine the attoparsec parsers into a `Control.Foldl.Fold ByteString [PartialParseResult]`, then convert it (using Control.Foldl.purely Pipes.Prelude.scan) to a `Pipe ByteString [PartialParseResult]`
My other requirement is that I want to look at intermediate parser output (eg. each line of a file as it's parsed), in order to summarise each line, in a stateful way as I'm keeping track of the current line number.
My approach for this is to use Control.Foldl.Fold for the summary logic, and then convert it to a pipe. I am able to create a `Pipe PartialParseResult AnalysisResult m r` for this.
However, how do I connect a `Pipe ByteString [PartialParseResult] m r` to a `Pipe PartialParseResult AnalysisResult m r`?
One approach I can think of is the use Pipes.Concurrent to manually split the ZipList and then combine then results. Is there a nicer way?--
On Tuesday, 23 August 2016 11:35:50 UTC+10, Louis Pan wrote:
I would do what you're already doing with `Control.Foldl.Fold`. That's the approach I usually recommend for this.
Sorry, could you please elaborate? I'm a bit slow.--
How do I use Control.Foldl.Fold to convert `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r` ?
On Tuesday, 23 August 2016 13:20:24 UTC+10, Gabriel Gonzalez wrote:
I managed to convert between `P.Producer (ZipList a) m r` and `ZipList (P.Producer a m ()` using the StateT trick used in Pipes.Parse--
Does the following code make sense? Am I breaking any laws?
import qualified Pipes as P
import qualified Pipes.Lift as PL
-- ZipList Traversable is only in 126.96.36.199
sequenceAZipList :: Applicative f => ZipList (f a) -> f (ZipList a)
sequenceAZipList xs = ZipList <$> sequenceA (getZipList xs)
-- | Similar to Pipes.Parse.Parser, except it stores a ZipList of Producers.
type ZipParser a m r = forall x . StateT (ZipList (P.Producer a m x)) m r
-- | Draw one element from each underlying Producer, returning 'Nothing' if any of the producers are empty
drawZ :: Monad m => ZipParser a m (Maybe (ZipList a))
drawZ = do
ps <- get
rs <- lift (sequenceAZipList (P.next <$> ps))
case sequenceAZipList rs of
Left _ -> pure Nothing
Right rs' -> do
put $ snd <$> rs'
pure . Just $ fst <$> rs'
-- | Push back a Ziplist element onto the underlying ZipList of Producers
unDrawZ :: Monad m => ZipList a -> ZipParser a m ()
unDrawZ as = modify (\ps -> appendA <$> ps <*> as)
appendA p a = do
r <- p
toZipList :: Monad m => P.Producer (ZipList a) m r -> m (ZipList (P.Producer a m ()))
toZipList p = execStateT (toZipParser p) (pure (pure ()))
toZipParser :: Monad m => P.Producer (ZipList a) m r -> ZipParser a m r
toZipParser p' = do
r <- lift $ P.next p'
On Tuesday, 23 August 2016 15:01:54 UTC+10, Louis Pan wrote:
My code doesn't stream properly - it forces consumption of the original producer before outputting anything. :(--
On Wednesday, 24 August 2016 04:24:03 UTC+10, Louis Pan wrote:
I think I understand what you mean by using Control.Fold.Foldl.--
It's not possible to "applicatively" lift a Pipe from "Pipe a b m r" to "Applicative f => Pipe (f a) (f a) m r", but it is possible to lift a Control.Foldl.Fold from "Fold a b" to "Applicative f => Fold a b -> Fold (f a) (f b)" (see below).
This means it's better in my case to code the consumer as a Fold (and then convert it to a Pipes Parse), as opposed to a Pipe Consumer.
-- requires a function to force the applicative to prevent space leaks.
liftFold :: Applicative f => (forall x. f x -> f x) -> L.Fold a b -> L.Fold (f a) (f b)
liftFold f (L.Fold step begin done) = L.Fold step' begin' done'
step' xs as = f (step <$> xs <*> as)
begin' = pure begin
done' xs = done <$> xs
forceZipList :: ZipList a -> ZipList a
forceZipList (ZipList xs) = ZipList (forceFoldable xs)
forceFoldable :: Foldable t => t a -> t a
forceFoldable xs = case foldr seq () xs of
() -> xs
liftZipList :: L.Fold a b -> L.Fold (ZipList a) (ZipList b)
liftZipList = liftFold forceZipList
I've confirmed this prevents space leaks by profiling with "+RTS -h -p" and using hp2ps.
However, I'm a novice with the significance of using seq and $!.
Am I doing anything dangerously? Is there something I should be careful of?
On Wednesday, 24 August 2016 20:16:29 UTC+10, Louis Pan wrote:
Yeah, what you wrote looks correct. The general idea is that you compose `Fold`s and then convert to a `Pipe` at the last minute.
There's another related type which also has nice properties for this sort of thing, which is basically a mealy machine. I usually encode it like this:
Then you can use `focus traverse` to lift a `Scan` to work on anything that is `Traversable` (like `ZipList`):
Hi Gabriel, thank you for taking the time to answer my questions.--
Is your Scan encoding available anywhere? The closest thing I can find is ekmett's "strict Mealy Machine" at https://hackage.haskell.org/package/folds-0.7.1/docs/Data-Fold-L1'.html but it looks quite different.
On Saturday, 27 August 2016 02:15:25 UTC+10, Gabriel Gonzalez wrote:
The closest thing is the mealy machine type from this library, which is equivalent, although possibly less efficient (and missing the `focus` utility):
This is something I've been meaning to write up in a small library at some point, but I keep forgetting.
|Free forum by Nabble||Edit this page|