Long-running request/response protocol server using enumerator/iterator/iterIO/pipes/conduits/...

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Long-running request/response protocol server using enumerator/iterator/iterIO/pipes/conduits/...

Nicolas Trangez
Hello Cafe,

Some time ago I tried to implement a network service using iteratee (or
enumerator, can't remember), but gave up in the end. More recently I
wanted to create something similar (a similar protocol), but failed
again.

So I'm looking for some example code or something similar (Google only
helped slightly).

First of all, I don't care which API/library to use, I guess for my
purpose all of enumerator, iteratee, iterIO, pipes, conduits,... are OK,
so all feedback is welcome.

Here's the catch. Most examples out there implement some server which
accepts a single client request, interprets it, creates a response,
returns this, and closes the connection (or something alike, think
HTTP).

The protocol I'd like to implement is different: it's long-running using
repeated requests & responses on a single client connection. Basically,
a client connects and sends some data to the server (where the length of
this data is encoded in the header). Now the server reads & parses this
(binary) data, sets up some initial state for this client connection
(e.g. opening a file handle), and returns a reply. Now the client can
send another request, server parses/interprets it using the connection
state, sends a reply, and so on.

Might sound easy (and actually it's pretty easy in most other languages
I know, including an OCaml implementation), yet I fail to figure out how
to get this done using some enumerator-style library.

Thanks for any help, I'll most likely write up something if I get things
working for future reference.

Nicolas


_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Long-running request/response protocol server using enumerator/iterator/iterIO/pipes/conduits/...

Christopher Done
On 26 June 2012 21:22, Nicolas Trangez <[hidden email]> wrote:
> Might sound easy (and actually it's pretty easy in most other languages
> I know, including an OCaml implementation), yet I fail to figure out how
> to get this done using some enumerator-style library.

Well, it's easy in Haskell, too. Just use the standard libraries.

If you want to mess around with these still-in-research iteratees and
eumerators for the composability then go for it, but when it's hard or
weird, you can't really blame that on the language. :-)

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Long-running request/response protocol server using enumerator/iterator/iterIO/pipes/conduits/...

Michael Snoyman
In reply to this post by Nicolas Trangez
On Tue, Jun 26, 2012 at 10:22 PM, Nicolas Trangez <[hidden email]> wrote:

> Hello Cafe,
>
> Some time ago I tried to implement a network service using iteratee (or
> enumerator, can't remember), but gave up in the end. More recently I
> wanted to create something similar (a similar protocol), but failed
> again.
>
> So I'm looking for some example code or something similar (Google only
> helped slightly).
>
> First of all, I don't care which API/library to use, I guess for my
> purpose all of enumerator, iteratee, iterIO, pipes, conduits,... are OK,
> so all feedback is welcome.
>
> Here's the catch. Most examples out there implement some server which
> accepts a single client request, interprets it, creates a response,
> returns this, and closes the connection (or something alike, think
> HTTP).
>
> The protocol I'd like to implement is different: it's long-running using
> repeated requests & responses on a single client connection. Basically,
> a client connects and sends some data to the server (where the length of
> this data is encoded in the header). Now the server reads & parses this
> (binary) data, sets up some initial state for this client connection
> (e.g. opening a file handle), and returns a reply. Now the client can
> send another request, server parses/interprets it using the connection
> state, sends a reply, and so on.
>
> Might sound easy (and actually it's pretty easy in most other languages
> I know, including an OCaml implementation), yet I fail to figure out how
> to get this done using some enumerator-style library.
>
> Thanks for any help, I'll most likely write up something if I get things
> working for future reference.
>
> Nicolas
>
>
> _______________________________________________
> Haskell-Cafe mailing list
> [hidden email]
> http://www.haskell.org/mailman/listinfo/haskell-cafe

I've run into those kinds of problems in the past as well. In general,
interleaving of data streams can be difficult with enumerator. That's
the reason I added connect-and-resume to conduit. I use the technique
in warp[1], which in fact *does* support multiple request/response
pairs due to connection keep-alive. But the code base isn't the
easiest introduction to the technique. If there's interest, I'll try
to put together a blog post on using connect-and-resume to solve this
kind of problem.

Michael

[1] https://github.com/yesodweb/wai/blob/beta/warp/Network/Wai/Handler/Warp.hs#L296

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Long-running request/response protocol server using enumerator/iterator/iterIO/pipes/conduits/...

Nicolas Trangez
In reply to this post by Christopher Done
On Tue, 2012-06-26 at 21:32 +0200, Christopher Done wrote:
> On 26 June 2012 21:22, Nicolas Trangez <[hidden email]> wrote:
> > Might sound easy (and actually it's pretty easy in most other languages
> > I know, including an OCaml implementation), yet I fail to figure out how
> > to get this done using some enumerator-style library.
>
> Well, it's easy in Haskell, too. Just use the standard libraries.

Sure, that could work.

> If you want to mess around with these still-in-research iteratees and
> eumerators for the composability then go for it, but when it's hard or
> weird, you can't really blame that on the language. :-)

Make no mistake, I'm not blaming anything except my own inability to
figure this out ;-)

Thanks,

Nicolas


_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Long-running request/response protocol server using enumerator/iterator/iterIO/pipes/conduits/...

Paolo Capriotti
In reply to this post by Nicolas Trangez
On Tue, Jun 26, 2012 at 8:22 PM, Nicolas Trangez <[hidden email]> wrote:

> Hello Cafe,
>
> Some time ago I tried to implement a network service using iteratee (or
> enumerator, can't remember), but gave up in the end. More recently I
> wanted to create something similar (a similar protocol), but failed
> again.
>
> So I'm looking for some example code or something similar (Google only
> helped slightly).
>
> First of all, I don't care which API/library to use, I guess for my
> purpose all of enumerator, iteratee, iterIO, pipes, conduits,... are OK,
> so all feedback is welcome.
>
> Here's the catch. Most examples out there implement some server which
> accepts a single client request, interprets it, creates a response,
> returns this, and closes the connection (or something alike, think
> HTTP).
>
> The protocol I'd like to implement is different: it's long-running using
> repeated requests & responses on a single client connection. Basically,
> a client connects and sends some data to the server (where the length of
> this data is encoded in the header). Now the server reads & parses this
> (binary) data, sets up some initial state for this client connection
> (e.g. opening a file handle), and returns a reply. Now the client can
> send another request, server parses/interprets it using the connection
> state, sends a reply, and so on.
>
> Might sound easy (and actually it's pretty easy in most other languages
> I know, including an OCaml implementation), yet I fail to figure out how
> to get this done using some enumerator-style library.

With the current development version of pipes-core
(https://github.com/pcapriotti/pipes-core/tree/devel) I would write something
like the following (completely untested) code:

    import qualified Control.Pipe.Binary as B
    ...

    request :: PipeL IO ByteString ByteString u ()
    request = do
        h <- header
        let n = hdrSize h
        B.take n

    -- I assume a fixed-size header for simplicity
    header :: PipeL IO ByteString b u Header
    header = do
        h <- B.take headerSize >+> fold (<>) ByteString.empty
        return $ parseHeader h -- the function doing the actual parsing

    handler :: Pipe IO ByteString ByteString u ()
    handler = do
        -- server logic here
        -- just echo the input data as an example
        void idP

    server hInput hOutput
           -- read from the socket
         = B.handleReader hInput
           -- process all requests
       >+> forever (withUnawait $ request >+> handler)
           -- write to socket
       >+> B.handleWriter hOutput

Requests are handled sequentially by the `forever` loop. This whole pipeline
works with chunks of data represented as `ByteString`s, and `PipeL` (a new
feature of pipes-core 0.2.0) is used to pass leftover data along.

In a real implementation, you would also probably need to wrap `handler` in
something like:

    catch handler $ \e ->
        liftIO $ logException e
    discard

so that failures (or early termination) in `handler` don't bring the whole
pipeline down.

Sorry for the not very practical reply, involving experimental
unreleased code. :)

BR,
Paolo

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Long-running request/response protocol server using enumerator/iterator/iterIO/pipes/conduits/...

Nicolas Trangez
In reply to this post by Michael Snoyman
On Tue, 2012-06-26 at 22:39 +0300, Michael Snoyman wrote:
> I've run into those kinds of problems in the past as well. In general,
> interleaving of data streams can be difficult with enumerator. That's
> the reason I added connect-and-resume to conduit. I use the technique
> in warp[1], which in fact *does* support multiple request/response
> pairs due to connection keep-alive. But the code base isn't the
> easiest introduction to the technique. If there's interest, I'll try
> to put together a blog post on using connect-and-resume to solve this
> kind of problem.

Thank you, Michael. I thought about HTTP keep-alive as well, but felt
reluctant to start by looking at a 'large' codebase like warp... Anyway,
what you point to seems reasonable to interpret, I should be able to
write something similar based on this (even though I never used
Conduits/ResourceT before).

Thanks!

Nicolas


_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Long-running request/response protocol server using enumerator/iterator/iterIO/pipes/conduits/...

Erik de Castro Lopo-34
In reply to this post by Michael Snoyman
Michael Snoyman wrote:

> That's
> the reason I added connect-and-resume to conduit. I use the technique
> in warp[1], which in fact *does* support multiple request/response
> pairs due to connection keep-alive. But the code base isn't the
> easiest introduction to the technique. If there's interest, I'll try
> to put together a blog post on using connect-and-resume to solve this
> kind of problem.

+1

Erik
--
----------------------------------------------------------------------
Erik de Castro Lopo
http://www.mega-nerd.com/

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Long-running request/response protocol server using enumerator/iterator/iterIO/pipes/conduits/...

Michael Snoyman
On Wed, Jun 27, 2012 at 10:41 AM, Erik de Castro Lopo
<[hidden email]> wrote:

> Michael Snoyman wrote:
>
>> That's
>> the reason I added connect-and-resume to conduit. I use the technique
>> in warp[1], which in fact *does* support multiple request/response
>> pairs due to connection keep-alive. But the code base isn't the
>> easiest introduction to the technique. If there's interest, I'll try
>> to put together a blog post on using connect-and-resume to solve this
>> kind of problem.
>
> +1
>
> Erik
> --
> ----------------------------------------------------------------------
> Erik de Castro Lopo
> http://www.mega-nerd.com/
>
> _______________________________________________
> Haskell-Cafe mailing list
> [hidden email]
> http://www.haskell.org/mailman/listinfo/haskell-cafe

I've written up the blog post, but it will only be going public once
conduit is released (hopefully pretty soon). For the curious, it's
available on Github at:

https://github.com/yesodweb/yesodweb.com-content/blob/master/blog/2012/6/conduit-0-5.md

Michael

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe