Is it possible to applicatively lift `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r`?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Is it possible to applicatively lift `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r`?

Louis Pan
Say I have a `Pipes a b m r` called 'p'.

How do I convert it to a `Pipes (ZipList a) (ZipList b) m r`?
That is, I want to apply the pipe 'p' and apply zip-wise to an input ziplist, without losing the state of each pipe in the ziplist.

The reason I want this, is that I want to run multiple attoparsec parsers in parallel in one pass of a file.

My approach for this is to use Control.Foldl  to combine the attoparsec parsers into a `Control.Foldl.Fold ByteString [PartialParseResult]`, then convert it (using Control.Foldl.purely Pipes.Prelude.scan) to a `Pipe ByteString [PartialParseResult]`

My other requirement is that I want to look at intermediate parser output (eg. each line of a file as it's parsed), in order to summarise each line, in a stateful way as I'm keeping track of the current line number.

My approach for this is to use Control.Foldl.Fold for the summary logic, and then convert it to a pipe. I am able to create a `Pipe PartialParseResult AnalysisResult m r` for this.

However, how do I connect a `Pipe ByteString [PartialParseResult] m r` to a `Pipe PartialParseResult AnalysisResult m r`?

Regards,

Louis

--
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to applicatively lift `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r`?

Louis Pan
One approach I can think of is the use Pipes.Concurrent to manually split the ZipList and then combine then results. Is there a nicer way?

On Tuesday, 23 August 2016 11:35:50 UTC+10, Louis Pan wrote:
Say I have a `Pipes a b m r` called 'p'.

How do I convert it to a `Pipes (ZipList a) (ZipList b) m r`?
That is, I want to apply the pipe 'p' and apply zip-wise to an input ziplist, without losing the state of each pipe in the ziplist.

The reason I want this, is that I want to run multiple attoparsec parsers in parallel in one pass of a file.

My approach for this is to use Control.Foldl  to combine the attoparsec parsers into a `Control.Foldl.Fold ByteString [PartialParseResult]`, then convert it (using Control.Foldl.purely Pipes.Prelude.scan) to a `Pipe ByteString [PartialParseResult]`

My other requirement is that I want to look at intermediate parser output (eg. each line of a file as it's parsed), in order to summarise each line, in a stateful way as I'm keeping track of the current line number.

My approach for this is to use Control.Foldl.Fold for the summary logic, and then convert it to a pipe. I am able to create a `Pipe PartialParseResult AnalysisResult m r` for this.

However, how do I connect a `Pipe ByteString [PartialParseResult] m r` to a `Pipe PartialParseResult AnalysisResult m r`?

Regards,

Louis

--
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to applicatively lift `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r`?

Gabriel Gonzalez
I would do what you're already doing with `Control.Foldl.Fold`.  That's the approach I usually recommend for this.

On Aug 22, 2016, at 6:45 PM, Louis Pan <[hidden email]> wrote:

One approach I can think of is the use Pipes.Concurrent to manually split the ZipList and then combine then results. Is there a nicer way?

On Tuesday, 23 August 2016 11:35:50 UTC+10, Louis Pan wrote:
Say I have a `Pipes a b m r` called 'p'.

How do I convert it to a `Pipes (ZipList a) (ZipList b) m r`?
That is, I want to apply the pipe 'p' and apply zip-wise to an input ziplist, without losing the state of each pipe in the ziplist.

The reason I want this, is that I want to run multiple attoparsec parsers in parallel in one pass of a file.

My approach for this is to use Control.Foldl  to combine the attoparsec parsers into a `Control.Foldl.Fold ByteString [PartialParseResult]`, then convert it (using Control.Foldl.purely Pipes.Prelude.scan) to a `Pipe ByteString [PartialParseResult]`

My other requirement is that I want to look at intermediate parser output (eg. each line of a file as it's parsed), in order to summarise each line, in a stateful way as I'm keeping track of the current line number.

My approach for this is to use Control.Foldl.Fold for the summary logic, and then convert it to a pipe. I am able to create a `Pipe PartialParseResult AnalysisResult m r` for this.

However, how do I connect a `Pipe ByteString [PartialParseResult] m r` to a `Pipe PartialParseResult AnalysisResult m r`?

Regards,

Louis


--

--
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to applicatively lift `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r`?

Louis Pan
Sorry, could you please elaborate? I'm a bit slow.

How do I use Control.Foldl.Fold to convert `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r` ?


On Tuesday, 23 August 2016 13:20:24 UTC+10, Gabriel Gonzalez wrote:
I would do what you're already doing with `Control.Foldl.Fold`.  That's the approach I usually recommend for this.

On Aug 22, 2016, at 6:45 PM, Louis Pan <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="Xad38a33BAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">lo...@...> wrote:

One approach I can think of is the use Pipes.Concurrent to manually split the ZipList and then combine then results. Is there a nicer way?

On Tuesday, 23 August 2016 11:35:50 UTC+10, Louis Pan wrote:
Say I have a `Pipes a b m r` called 'p'.

How do I convert it to a `Pipes (ZipList a) (ZipList b) m r`?
That is, I want to apply the pipe 'p' and apply zip-wise to an input ziplist, without losing the state of each pipe in the ziplist.

The reason I want this, is that I want to run multiple attoparsec parsers in parallel in one pass of a file.

My approach for this is to use Control.Foldl  to combine the attoparsec parsers into a `Control.Foldl.Fold ByteString [PartialParseResult]`, then convert it (using Control.Foldl.purely Pipes.Prelude.scan) to a `Pipe ByteString [PartialParseResult]`

My other requirement is that I want to look at intermediate parser output (eg. each line of a file as it's parsed), in order to summarise each line, in a stateful way as I'm keeping track of the current line number.

My approach for this is to use Control.Foldl.Fold for the summary logic, and then convert it to a pipe. I am able to create a `Pipe PartialParseResult AnalysisResult m r` for this.

However, how do I connect a `Pipe ByteString [PartialParseResult] m r` to a `Pipe PartialParseResult AnalysisResult m r`?

Regards,

Louis


--

--
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to applicatively lift `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r`?

Louis Pan
I managed to convert between `P.Producer (ZipList a) m r` and `ZipList (P.Producer a m ()` using the StateT trick used in Pipes.Parse

Does the following code make sense? Am I breaking any laws?

import Control.Applicative
import Control.Monad.Trans.Class
import Control.Monad.Trans.State.Strict
import qualified Pipes as P
import qualified Pipes.Lift as PL

-- ZipList Traversable is only in 4.9.0.0
sequenceAZipList :: Applicative f => ZipList (f a) -> f (ZipList a)
sequenceAZipList xs = ZipList <$> sequenceA (getZipList xs)

-- | Similar to Pipes.Parse.Parser, except it stores a ZipList of Producers.
type ZipParser a m r = forall x . StateT (ZipList (P.Producer a m x)) m r

-- | Draw one element from each underlying Producer, returning 'Nothing' if any of the producers are empty
drawZ :: Monad m => ZipParser a m (Maybe (ZipList a))
drawZ = do
  ps <- get
  rs <- lift (sequenceAZipList (P.next <$> ps))
  case sequenceAZipList rs of
    Left _ -> pure Nothing
    Right rs' -> do
      put $ snd <$> rs'
      pure . Just $ fst <$> rs'

-- | Push back a Ziplist element onto the underlying ZipList of Producers
unDrawZ :: Monad m => ZipList a -> ZipParser a m ()
unDrawZ as = modify (\ps -> appendA <$> ps <*> as)
 where
  appendA p a = do
    r <- p
    P.yield a
    pure r

toZipList :: Monad m => P.Producer (ZipList a) m r -> m (ZipList (P.Producer a m ()))
toZipList p = execStateT (toZipParser p) (pure (pure ()))
 where
  toZipParser :: Monad m => P.Producer (ZipList a) m r -> ZipParser a m r
  toZipParser p' = do
    r <- lift $ P.next p'


On Tuesday, 23 August 2016 15:01:54 UTC+10, Louis Pan wrote:
Sorry, could you please elaborate? I'm a bit slow.

How do I use Control.Foldl.Fold to convert `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r` ?


On Tuesday, 23 August 2016 13:20:24 UTC+10, Gabriel Gonzalez wrote:
I would do what you're already doing with `Control.Foldl.Fold`.  That's the approach I usually recommend for this.

On Aug 22, 2016, at 6:45 PM, Louis Pan <[hidden email]> wrote:

One approach I can think of is the use Pipes.Concurrent to manually split the ZipList and then combine then results. Is there a nicer way?

On Tuesday, 23 August 2016 11:35:50 UTC+10, Louis Pan wrote:
Say I have a `Pipes a b m r` called 'p'.

How do I convert it to a `Pipes (ZipList a) (ZipList b) m r`?
That is, I want to apply the pipe 'p' and apply zip-wise to an input ziplist, without losing the state of each pipe in the ziplist.

The reason I want this, is that I want to run multiple attoparsec parsers in parallel in one pass of a file.

My approach for this is to use Control.Foldl  to combine the attoparsec parsers into a `Control.Foldl.Fold ByteString [PartialParseResult]`, then convert it (using Control.Foldl.purely Pipes.Prelude.scan) to a `Pipe ByteString [PartialParseResult]`

My other requirement is that I want to look at intermediate parser output (eg. each line of a file as it's parsed), in order to summarise each line, in a stateful way as I'm keeping track of the current line number.

My approach for this is to use Control.Foldl.Fold for the summary logic, and then convert it to a pipe. I am able to create a `Pipe PartialParseResult AnalysisResult m r` for this.

However, how do I connect a `Pipe ByteString [PartialParseResult] m r` to a `Pipe PartialParseResult AnalysisResult m r`?

Regards,

Louis


--

--
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to applicatively lift `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r`?

Louis Pan
My code doesn't stream properly - it forces consumption of the original producer before outputting anything. :(

On Wednesday, 24 August 2016 04:24:03 UTC+10, Louis Pan wrote:
I managed to convert between `P.Producer (ZipList a) m r` and `ZipList (P.Producer a m ()` using the StateT trick used in Pipes.Parse

Does the following code make sense? Am I breaking any laws?

import Control.Applicative
import Control.Monad.Trans.Class
import Control.Monad.Trans.State.Strict
import qualified Pipes as P
import qualified Pipes.Lift as PL

-- ZipList Traversable is only in 4.9.0.0
sequenceAZipList :: Applicative f => ZipList (f a) -> f (ZipList a)
sequenceAZipList xs = ZipList <$> sequenceA (getZipList xs)

-- | Similar to Pipes.Parse.Parser, except it stores a ZipList of Producers.
type ZipParser a m r = forall x . StateT (ZipList (P.Producer a m x)) m r

-- | Draw one element from each underlying Producer, returning 'Nothing' if any of the producers are empty
drawZ :: Monad m => ZipParser a m (Maybe (ZipList a))
drawZ = do
  ps <- get
  rs <- lift (sequenceAZipList (P.next <$> ps))
  case sequenceAZipList rs of
    Left _ -> pure Nothing
    Right rs' -> do
      put $ snd <$> rs'
      pure . Just $ fst <$> rs'

-- | Push back a Ziplist element onto the underlying ZipList of Producers
unDrawZ :: Monad m => ZipList a -> ZipParser a m ()
unDrawZ as = modify (\ps -> appendA <$> ps <*> as)
 where
  appendA p a = do
    r <- p
    P.yield a
    pure r

toZipList :: Monad m => P.Producer (ZipList a) m r -> m (ZipList (P.Producer a m ()))
toZipList p = execStateT (toZipParser p) (pure (pure ()))
 where
  toZipParser :: Monad m => P.Producer (ZipList a) m r -> ZipParser a m r
  toZipParser p' = do
    r <- lift $ P.next p'


On Tuesday, 23 August 2016 15:01:54 UTC+10, Louis Pan wrote:
Sorry, could you please elaborate? I'm a bit slow.

How do I use Control.Foldl.Fold to convert `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r` ?


On Tuesday, 23 August 2016 13:20:24 UTC+10, Gabriel Gonzalez wrote:
I would do what you're already doing with `Control.Foldl.Fold`.  That's the approach I usually recommend for this.

On Aug 22, 2016, at 6:45 PM, Louis Pan <[hidden email]> wrote:

One approach I can think of is the use Pipes.Concurrent to manually split the ZipList and then combine then results. Is there a nicer way?

On Tuesday, 23 August 2016 11:35:50 UTC+10, Louis Pan wrote:
Say I have a `Pipes a b m r` called 'p'.

How do I convert it to a `Pipes (ZipList a) (ZipList b) m r`?
That is, I want to apply the pipe 'p' and apply zip-wise to an input ziplist, without losing the state of each pipe in the ziplist.

The reason I want this, is that I want to run multiple attoparsec parsers in parallel in one pass of a file.

My approach for this is to use Control.Foldl  to combine the attoparsec parsers into a `Control.Foldl.Fold ByteString [PartialParseResult]`, then convert it (using Control.Foldl.purely Pipes.Prelude.scan) to a `Pipe ByteString [PartialParseResult]`

My other requirement is that I want to look at intermediate parser output (eg. each line of a file as it's parsed), in order to summarise each line, in a stateful way as I'm keeping track of the current line number.

My approach for this is to use Control.Foldl.Fold for the summary logic, and then convert it to a pipe. I am able to create a `Pipe PartialParseResult AnalysisResult m r` for this.

However, how do I connect a `Pipe ByteString [PartialParseResult] m r` to a `Pipe PartialParseResult AnalysisResult m r`?

Regards,

Louis


--

--
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to applicatively lift `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r`?

Louis Pan
I think I understand what you mean by using Control.Fold.Foldl.

It's not possible to "applicatively" lift a Pipe from "Pipe a b m r" to "Applicative f => Pipe (f a) (f a) m r", but it is possible to lift a Control.Foldl.Fold from "Fold a b" to "Applicative f => Fold a b -> Fold (f a) (f b)" (see below).

This means it's better in my case to code the consumer as a Fold (and then convert it to a Pipes Parse), as opposed to a Pipe Consumer.

-- requires a function to force the applicative to prevent space leaks.
liftFold :: Applicative f => (forall x. f x -> f x) -> L.Fold a b -> L.Fold (f a) (f b)
liftFold f (L.Fold step begin done) = L.Fold step' begin' done'
 where
  step' xs as = f (step <$> xs <*> as)
  begin' = pure begin
  done' xs = done <$> xs

forceZipList :: ZipList a -> ZipList a
forceZipList (ZipList xs) = ZipList (forceFoldable xs)

-- https://ro-che.info/articles/2015-05-28-force-list
forceFoldable :: Foldable t => t a -> t a
forceFoldable xs = case foldr seq () xs of
      () -> xs

liftZipList ::  L.Fold a b -> L.Fold (ZipList a) (ZipList b)
liftZipList = liftFold forceZipList

I've confirmed this prevents space leaks by profiling with "+RTS -h -p" and using hp2ps.

However, I'm a novice with the significance of using seq and $!.

Am I doing anything dangerously? Is there something I should be careful of?


On Wednesday, 24 August 2016 20:16:29 UTC+10, Louis Pan wrote:
My code doesn't stream properly - it forces consumption of the original producer before outputting anything. :(

On Wednesday, 24 August 2016 04:24:03 UTC+10, Louis Pan wrote:
I managed to convert between `P.Producer (ZipList a) m r` and `ZipList (P.Producer a m ()` using the StateT trick used in Pipes.Parse

Does the following code make sense? Am I breaking any laws?

import Control.Applicative
import Control.Monad.Trans.Class
import Control.Monad.Trans.State.Strict
import qualified Pipes as P
import qualified Pipes.Lift as PL

-- ZipList Traversable is only in 4.9.0.0
sequenceAZipList :: Applicative f => ZipList (f a) -> f (ZipList a)
sequenceAZipList xs = ZipList <$> sequenceA (getZipList xs)

-- | Similar to Pipes.Parse.Parser, except it stores a ZipList of Producers.
type ZipParser a m r = forall x . StateT (ZipList (P.Producer a m x)) m r

-- | Draw one element from each underlying Producer, returning 'Nothing' if any of the producers are empty
drawZ :: Monad m => ZipParser a m (Maybe (ZipList a))
drawZ = do
  ps <- get
  rs <- lift (sequenceAZipList (P.next <$> ps))
  case sequenceAZipList rs of
    Left _ -> pure Nothing
    Right rs' -> do
      put $ snd <$> rs'
      pure . Just $ fst <$> rs'

-- | Push back a Ziplist element onto the underlying ZipList of Producers
unDrawZ :: Monad m => ZipList a -> ZipParser a m ()
unDrawZ as = modify (\ps -> appendA <$> ps <*> as)
 where
  appendA p a = do
    r <- p
    P.yield a
    pure r

toZipList :: Monad m => P.Producer (ZipList a) m r -> m (ZipList (P.Producer a m ()))
toZipList p = execStateT (toZipParser p) (pure (pure ()))
 where
  toZipParser :: Monad m => P.Producer (ZipList a) m r -> ZipParser a m r
  toZipParser p' = do
    r <- lift $ P.next p'


On Tuesday, 23 August 2016 15:01:54 UTC+10, Louis Pan wrote:
Sorry, could you please elaborate? I'm a bit slow.

How do I use Control.Foldl.Fold to convert `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r` ?


On Tuesday, 23 August 2016 13:20:24 UTC+10, Gabriel Gonzalez wrote:
I would do what you're already doing with `Control.Foldl.Fold`.  That's the approach I usually recommend for this.

On Aug 22, 2016, at 6:45 PM, Louis Pan <[hidden email]> wrote:

One approach I can think of is the use Pipes.Concurrent to manually split the ZipList and then combine then results. Is there a nicer way?

On Tuesday, 23 August 2016 11:35:50 UTC+10, Louis Pan wrote:
Say I have a `Pipes a b m r` called 'p'.

How do I convert it to a `Pipes (ZipList a) (ZipList b) m r`?
That is, I want to apply the pipe 'p' and apply zip-wise to an input ziplist, without losing the state of each pipe in the ziplist.

The reason I want this, is that I want to run multiple attoparsec parsers in parallel in one pass of a file.

My approach for this is to use Control.Foldl  to combine the attoparsec parsers into a `Control.Foldl.Fold ByteString [PartialParseResult]`, then convert it (using Control.Foldl.purely Pipes.Prelude.scan) to a `Pipe ByteString [PartialParseResult]`

My other requirement is that I want to look at intermediate parser output (eg. each line of a file as it's parsed), in order to summarise each line, in a stateful way as I'm keeping track of the current line number.

My approach for this is to use Control.Foldl.Fold for the summary logic, and then convert it to a pipe. I am able to create a `Pipe PartialParseResult AnalysisResult m r` for this.

However, how do I connect a `Pipe ByteString [PartialParseResult] m r` to a `Pipe PartialParseResult AnalysisResult m r`?

Regards,

Louis


--

--
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to applicatively lift `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r`?

Gabriel Gonzalez
Yeah, what you wrote looks correct.  The general idea is that you compose `Fold`s and then convert to a `Pipe` at the last minute.

There's another related type which also has nice properties for this sort of thing, which is basically a mealy machine.  I usually encode it like this:

{-# LANGUAGE ExistentialQuantification #-}

data Scan a b = forall m . Monoid m => Scan { begin :: s, step :: s -> a -> (s, b) }

instance Category Scan where ...
instance Arrow Scan where ...
instance Functor (Scan a) where ...
instance Applicative (Scan a) where ...

focus
    :: (forall x. LensLike (State x) s t a b)
    -> Scan a b
    -> Scan s t
focus k (Scan s f) = Scan s (Control.Lens.mapAccumLOf k f)

Then you can use `focus traverse` to lift a `Scan` to work on anything that is `Traversable` (like `ZipList`):

focus traverse :: Traversable t => Scan a b -> Scan (t a) (t b)

On Aug 26, 2016, at 3:37 AM, Louis Pan <[hidden email]> wrote:

I think I understand what you mean by using Control.Fold.Foldl.

It's not possible to "applicatively" lift a Pipe from "Pipe a b m r" to "Applicative f => Pipe (f a) (f a) m r", but it is possible to lift a Control.Foldl.Fold from "Fold a b" to "Applicative f => Fold a b -> Fold (f a) (f b)" (see below).

This means it's better in my case to code the consumer as a Fold (and then convert it to a Pipes Parse), as opposed to a Pipe Consumer.

-- requires a function to force the applicative to prevent space leaks.
liftFold :: Applicative f => (forall x. f x -> f x) -> L.Fold a b -> L.Fold (f a) (f b)
liftFold f (L.Fold step begin done) = L.Fold step' begin' done'
 where
  step' xs as = f (step <$> xs <*> as)
  begin' = pure begin
  done' xs = done <$> xs

forceZipList :: ZipList a -> ZipList a
forceZipList (ZipList xs) = ZipList (forceFoldable xs)

forceFoldable :: Foldable t => t a -> t a
forceFoldable xs = case foldr seq () xs of
      () -> xs

liftZipList ::  L.Fold a b -> L.Fold (ZipList a) (ZipList b)
liftZipList = liftFold forceZipList

I've confirmed this prevents space leaks by profiling with "+RTS -h -p" and using hp2ps.

However, I'm a novice with the significance of using seq and $!.

Am I doing anything dangerously? Is there something I should be careful of?


On Wednesday, 24 August 2016 20:16:29 UTC+10, Louis Pan wrote:
My code doesn't stream properly - it forces consumption of the original producer before outputting anything. :(

On Wednesday, 24 August 2016 04:24:03 UTC+10, Louis Pan wrote:
I managed to convert between `P.Producer (ZipList a) m r` and `ZipList (P.Producer a m ()` using the StateT trick used in Pipes.Parse

Does the following code make sense? Am I breaking any laws?

import Control.Applicative
import Control.Monad.Trans.Class
import Control.Monad.Trans.State.Strict
import qualified Pipes as P
import qualified Pipes.Lift as PL

-- ZipList Traversable is only in 4.9.0.0
sequenceAZipList :: Applicative f => ZipList (f a) -> f (ZipList a)
sequenceAZipList xs = ZipList <$> sequenceA (getZipList xs)

-- | Similar to Pipes.Parse.Parser, except it stores a ZipList of Producers.
type ZipParser a m r = forall x . StateT (ZipList (P.Producer a m x)) m r

-- | Draw one element from each underlying Producer, returning 'Nothing' if any of the producers are empty
drawZ :: Monad m => ZipParser a m (Maybe (ZipList a))
drawZ = do
  ps <- get
  rs <- lift (sequenceAZipList (P.next <$> ps))
  case sequenceAZipList rs of
    Left _ -> pure Nothing
    Right rs' -> do
      put $ snd <$> rs'
      pure . Just $ fst <$> rs'

-- | Push back a Ziplist element onto the underlying ZipList of Producers
unDrawZ :: Monad m => ZipList a -> ZipParser a m ()
unDrawZ as = modify (\ps -> appendA <$> ps <*> as)
 where
  appendA p a = do
    r <- p
    P.yield a
    pure r

toZipList :: Monad m => P.Producer (ZipList a) m r -> m (ZipList (P.Producer a m ()))
toZipList p = execStateT (toZipParser p) (pure (pure ()))
 where
  toZipParser :: Monad m => P.Producer (ZipList a) m r -> ZipParser a m r
  toZipParser p' = do
    r <- lift $ P.next p'


On Tuesday, 23 August 2016 15:01:54 UTC+10, Louis Pan wrote:
Sorry, could you please elaborate? I'm a bit slow.

How do I use Control.Foldl.Fold to convert `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r` ?


On Tuesday, 23 August 2016 13:20:24 UTC+10, Gabriel Gonzalez wrote:
I would do what you're already doing with `Control.Foldl.Fold`.  That's the approach I usually recommend for this.

On Aug 22, 2016, at 6:45 PM, Louis Pan <[hidden email]> wrote:

One approach I can think of is the use Pipes.Concurrent to manually split the ZipList and then combine then results. Is there a nicer way?

On Tuesday, 23 August 2016 11:35:50 UTC+10, Louis Pan wrote:
Say I have a `Pipes a b m r` called 'p'.

How do I convert it to a `Pipes (ZipList a) (ZipList b) m r`?
That is, I want to apply the pipe 'p' and apply zip-wise to an input ziplist, without losing the state of each pipe in the ziplist.

The reason I want this, is that I want to run multiple attoparsec parsers in parallel in one pass of a file.

My approach for this is to use Control.Foldl  to combine the attoparsec parsers into a `Control.Foldl.Fold ByteString [PartialParseResult]`, then convert it (using Control.Foldl.purely Pipes.Prelude.scan) to a `Pipe ByteString [PartialParseResult]`

My other requirement is that I want to look at intermediate parser output (eg. each line of a file as it's parsed), in order to summarise each line, in a stateful way as I'm keeping track of the current line number.

My approach for this is to use Control.Foldl.Fold for the summary logic, and then convert it to a pipe. I am able to create a `Pipe PartialParseResult AnalysisResult m r` for this.

However, how do I connect a `Pipe ByteString [PartialParseResult] m r` to a `Pipe PartialParseResult AnalysisResult m r`?

Regards,

Louis


--


--

--
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to applicatively lift `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r`?

Louis Pan
Hi Gabriel, thank you for taking the time to answer my questions.

Is your Scan encoding available anywhere? The closest thing I can find is ekmett's "strict Mealy Machine" at https://hackage.haskell.org/package/folds-0.7.1/docs/Data-Fold-L1'.html but it looks quite different.

On Saturday, 27 August 2016 02:15:25 UTC+10, Gabriel Gonzalez wrote:
Yeah, what you wrote looks correct.  The general idea is that you compose `Fold`s and then convert to a `Pipe` at the last minute.

There's another related type which also has nice properties for this sort of thing, which is basically a mealy machine.  I usually encode it like this:

{-# LANGUAGE ExistentialQuantification #-}

data Scan a b = forall m . Monoid m => Scan { begin :: s, step :: s -> a -> (s, b) }

instance Category Scan where ...
instance Arrow Scan where ...
instance Functor (Scan a) where ...
instance Applicative (Scan a) where ...

focus
    :: (forall x. LensLike (State x) s t a b)
    -> Scan a b
    -> Scan s t
focus k (Scan s f) = Scan s (Control.Lens.mapAccumLOf k f)

Then you can use `focus traverse` to lift a `Scan` to work on anything that is `Traversable` (like `ZipList`):

focus traverse :: Traversable t => Scan a b -> Scan (t a) (t b)

On Aug 26, 2016, at 3:37 AM, Louis Pan <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="EvpSdrYNBgAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">lo...@...> wrote:

I think I understand what you mean by using Control.Fold.Foldl.

It's not possible to "applicatively" lift a Pipe from "Pipe a b m r" to "Applicative f => Pipe (f a) (f a) m r", but it is possible to lift a Control.Foldl.Fold from "Fold a b" to "Applicative f => Fold a b -> Fold (f a) (f b)" (see below).

This means it's better in my case to code the consumer as a Fold (and then convert it to a Pipes Parse), as opposed to a Pipe Consumer.

-- requires a function to force the applicative to prevent space leaks.
liftFold :: Applicative f => (forall x. f x -> f x) -> L.Fold a b -> L.Fold (f a) (f b)
liftFold f (L.Fold step begin done) = L.Fold step' begin' done'
 where
  step' xs as = f (step <$> xs <*> as)
  begin' = pure begin
  done' xs = done <$> xs

forceZipList :: ZipList a -> ZipList a
forceZipList (ZipList xs) = ZipList (forceFoldable xs)

-- <a href="https://ro-che.info/articles/2015-05-28-force-list" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fro-che.info%2Farticles%2F2015-05-28-force-list\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFvGIDSw8gCr8Gtq02lUdftq_xt_Q&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fro-che.info%2Farticles%2F2015-05-28-force-list\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFvGIDSw8gCr8Gtq02lUdftq_xt_Q&#39;;return true;">https://ro-che.info/articles/2015-05-28-force-list
forceFoldable :: Foldable t => t a -> t a
forceFoldable xs = case foldr seq () xs of
      () -> xs

liftZipList ::  L.Fold a b -> L.Fold (ZipList a) (ZipList b)
liftZipList = liftFold forceZipList

I've confirmed this prevents space leaks by profiling with "+RTS -h -p" and using hp2ps.

However, I'm a novice with the significance of using seq and $!.

Am I doing anything dangerously? Is there something I should be careful of?


On Wednesday, 24 August 2016 20:16:29 UTC+10, Louis Pan wrote:
My code doesn't stream properly - it forces consumption of the original producer before outputting anything. :(

On Wednesday, 24 August 2016 04:24:03 UTC+10, Louis Pan wrote:
I managed to convert between `P.Producer (ZipList a) m r` and `ZipList (P.Producer a m ()` using the StateT trick used in Pipes.Parse

Does the following code make sense? Am I breaking any laws?

import Control.Applicative
import Control.Monad.Trans.Class
import Control.Monad.Trans.State.Strict
import qualified Pipes as P
import qualified Pipes.Lift as PL

-- ZipList Traversable is only in 4.9.0.0
sequenceAZipList :: Applicative f => ZipList (f a) -> f (ZipList a)
sequenceAZipList xs = ZipList <$> sequenceA (getZipList xs)

-- | Similar to Pipes.Parse.Parser, except it stores a ZipList of Producers.
type ZipParser a m r = forall x . StateT (ZipList (P.Producer a m x)) m r

-- | Draw one element from each underlying Producer, returning 'Nothing' if any of the producers are empty
drawZ :: Monad m => ZipParser a m (Maybe (ZipList a))
drawZ = do
  ps <- get
  rs <- lift (sequenceAZipList (P.next <$> ps))
  case sequenceAZipList rs of
    Left _ -> pure Nothing
    Right rs' -> do
      put $ snd <$> rs'
      pure . Just $ fst <$> rs'

-- | Push back a Ziplist element onto the underlying ZipList of Producers
unDrawZ :: Monad m => ZipList a -> ZipParser a m ()
unDrawZ as = modify (\ps -> appendA <$> ps <*> as)
 where
  appendA p a = do
    r <- p
    P.yield a
    pure r

toZipList :: Monad m => P.Producer (ZipList a) m r -> m (ZipList (P.Producer a m ()))
toZipList p = execStateT (toZipParser p) (pure (pure ()))
 where
  toZipParser :: Monad m => P.Producer (ZipList a) m r -> ZipParser a m r
  toZipParser p' = do
    r <- lift $ P.next p'


On Tuesday, 23 August 2016 15:01:54 UTC+10, Louis Pan wrote:
Sorry, could you please elaborate? I'm a bit slow.

How do I use Control.Foldl.Fold to convert `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r` ?


On Tuesday, 23 August 2016 13:20:24 UTC+10, Gabriel Gonzalez wrote:
I would do what you're already doing with `Control.Foldl.Fold`.  That's the approach I usually recommend for this.

On Aug 22, 2016, at 6:45 PM, Louis Pan <[hidden email]> wrote:

One approach I can think of is the use Pipes.Concurrent to manually split the ZipList and then combine then results. Is there a nicer way?

On Tuesday, 23 August 2016 11:35:50 UTC+10, Louis Pan wrote:
Say I have a `Pipes a b m r` called 'p'.

How do I convert it to a `Pipes (ZipList a) (ZipList b) m r`?
That is, I want to apply the pipe 'p' and apply zip-wise to an input ziplist, without losing the state of each pipe in the ziplist.

The reason I want this, is that I want to run multiple attoparsec parsers in parallel in one pass of a file.

My approach for this is to use Control.Foldl  to combine the attoparsec parsers into a `Control.Foldl.Fold ByteString [PartialParseResult]`, then convert it (using Control.Foldl.purely Pipes.Prelude.scan) to a `Pipe ByteString [PartialParseResult]`

My other requirement is that I want to look at intermediate parser output (eg. each line of a file as it's parsed), in order to summarise each line, in a stateful way as I'm keeping track of the current line number.

My approach for this is to use Control.Foldl.Fold for the summary logic, and then convert it to a pipe. I am able to create a `Pipe PartialParseResult AnalysisResult m r` for this.

However, how do I connect a `Pipe ByteString [PartialParseResult] m r` to a `Pipe PartialParseResult AnalysisResult m r`?

Regards,

Louis


--


--

--
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to applicatively lift `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r`?

Gabriel Gonzalez
The closest thing is the mealy machine type from this library, which is equivalent, although possibly less efficient (and missing the `focus` utility):

https://hackage.haskell.org/package/machines-0.6.1/docs/Data-Machine-Mealy.html

This is something I've been meaning to write up in a small library at some point, but I keep forgetting.

On Aug 29, 2016, at 4:49 PM, Louis Pan <[hidden email]> wrote:

Hi Gabriel, thank you for taking the time to answer my questions.

Is your Scan encoding available anywhere? The closest thing I can find is ekmett's "strict Mealy Machine" at https://hackage.haskell.org/package/folds-0.7.1/docs/Data-Fold-L1'.html but it looks quite different.

On Saturday, 27 August 2016 02:15:25 UTC+10, Gabriel Gonzalez wrote:
Yeah, what you wrote looks correct.  The general idea is that you compose `Fold`s and then convert to a `Pipe` at the last minute.

There's another related type which also has nice properties for this sort of thing, which is basically a mealy machine.  I usually encode it like this:

{-# LANGUAGE ExistentialQuantification #-}

data Scan a b = forall m . Monoid m => Scan { begin :: s, step :: s -> a -> (s, b) }

instance Category Scan where ...
instance Arrow Scan where ...
instance Functor (Scan a) where ...
instance Applicative (Scan a) where ...

focus
    :: (forall x. LensLike (State x) s t a b)
    -> Scan a b
    -> Scan s t
focus k (Scan s f) = Scan s (Control.Lens.mapAccumLOf k f)

Then you can use `focus traverse` to lift a `Scan` to work on anything that is `Traversable` (like `ZipList`):

focus traverse :: Traversable t => Scan a b -> Scan (t a) (t b)

On Aug 26, 2016, at 3:37 AM, Louis Pan <lo...@pan.me> wrote:

I think I understand what you mean by using Control.Fold.Foldl.

It's not possible to "applicatively" lift a Pipe from "Pipe a b m r" to "Applicative f => Pipe (f a) (f a) m r", but it is possible to lift a Control.Foldl.Fold from "Fold a b" to "Applicative f => Fold a b -> Fold (f a) (f b)" (see below).

This means it's better in my case to code the consumer as a Fold (and then convert it to a Pipes Parse), as opposed to a Pipe Consumer.

-- requires a function to force the applicative to prevent space leaks.
liftFold :: Applicative f => (forall x. f x -> f x) -> L.Fold a b -> L.Fold (f a) (f b)
liftFold f (L.Fold step begin done) = L.Fold step' begin' done'
 where
  step' xs as = f (step <$> xs <*> as)
  begin' = pure begin
  done' xs = done <$> xs

forceZipList :: ZipList a -> ZipList a
forceZipList (ZipList xs) = ZipList (forceFoldable xs)

forceFoldable :: Foldable t => t a -> t a
forceFoldable xs = case foldr seq () xs of
      () -> xs

liftZipList ::  L.Fold a b -> L.Fold (ZipList a) (ZipList b)
liftZipList = liftFold forceZipList

I've confirmed this prevents space leaks by profiling with "+RTS -h -p" and using hp2ps.

However, I'm a novice with the significance of using seq and $!.

Am I doing anything dangerously? Is there something I should be careful of?


On Wednesday, 24 August 2016 20:16:29 UTC+10, Louis Pan wrote:
My code doesn't stream properly - it forces consumption of the original producer before outputting anything. :(

On Wednesday, 24 August 2016 04:24:03 UTC+10, Louis Pan wrote:
I managed to convert between `P.Producer (ZipList a) m r` and `ZipList (P.Producer a m ()` using the StateT trick used in Pipes.Parse

Does the following code make sense? Am I breaking any laws?

import Control.Applicative
import Control.Monad.Trans.Class
import Control.Monad.Trans.State.Strict
import qualified Pipes as P
import qualified Pipes.Lift as PL

-- ZipList Traversable is only in 4.9.0.0
sequenceAZipList :: Applicative f => ZipList (f a) -> f (ZipList a)
sequenceAZipList xs = ZipList <$> sequenceA (getZipList xs)

-- | Similar to Pipes.Parse.Parser, except it stores a ZipList of Producers.
type ZipParser a m r = forall x . StateT (ZipList (P.Producer a m x)) m r

-- | Draw one element from each underlying Producer, returning 'Nothing' if any of the producers are empty
drawZ :: Monad m => ZipParser a m (Maybe (ZipList a))
drawZ = do
  ps <- get
  rs <- lift (sequenceAZipList (P.next <$> ps))
  case sequenceAZipList rs of
    Left _ -> pure Nothing
    Right rs' -> do
      put $ snd <$> rs'
      pure . Just $ fst <$> rs'

-- | Push back a Ziplist element onto the underlying ZipList of Producers
unDrawZ :: Monad m => ZipList a -> ZipParser a m ()
unDrawZ as = modify (\ps -> appendA <$> ps <*> as)
 where
  appendA p a = do
    r <- p
    P.yield a
    pure r

toZipList :: Monad m => P.Producer (ZipList a) m r -> m (ZipList (P.Producer a m ()))
toZipList p = execStateT (toZipParser p) (pure (pure ()))
 where
  toZipParser :: Monad m => P.Producer (ZipList a) m r -> ZipParser a m r
  toZipParser p' = do
    r <- lift $ P.next p'


On Tuesday, 23 August 2016 15:01:54 UTC+10, Louis Pan wrote:
Sorry, could you please elaborate? I'm a bit slow.

How do I use Control.Foldl.Fold to convert `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r` ?


On Tuesday, 23 August 2016 13:20:24 UTC+10, Gabriel Gonzalez wrote:
I would do what you're already doing with `Control.Foldl.Fold`.  That's the approach I usually recommend for this.

On Aug 22, 2016, at 6:45 PM, Louis Pan <[hidden email]> wrote:

One approach I can think of is the use Pipes.Concurrent to manually split the ZipList and then combine then results. Is there a nicer way?

On Tuesday, 23 August 2016 11:35:50 UTC+10, Louis Pan wrote:
Say I have a `Pipes a b m r` called 'p'.

How do I convert it to a `Pipes (ZipList a) (ZipList b) m r`?
That is, I want to apply the pipe 'p' and apply zip-wise to an input ziplist, without losing the state of each pipe in the ziplist.

The reason I want this, is that I want to run multiple attoparsec parsers in parallel in one pass of a file.

My approach for this is to use Control.Foldl  to combine the attoparsec parsers into a `Control.Foldl.Fold ByteString [PartialParseResult]`, then convert it (using Control.Foldl.purely Pipes.Prelude.scan) to a `Pipe ByteString [PartialParseResult]`

My other requirement is that I want to look at intermediate parser output (eg. each line of a file as it's parsed), in order to summarise each line, in a stateful way as I'm keeping track of the current line number.

My approach for this is to use Control.Foldl.Fold for the summary logic, and then convert it to a pipe. I am able to create a `Pipe PartialParseResult AnalysisResult m r` for this.

However, how do I connect a `Pipe ByteString [PartialParseResult] m r` to a `Pipe PartialParseResult AnalysisResult m r`?

Regards,

Louis


-- 

--
Reply | Threaded
Open this post in threaded view
|

Re: Is it possible to applicatively lift `Pipes a b m r` into `Pipes (ZipList a) (ZipList b) m r`?

Mitchell Rosen
This library would be awesome to have!

--