ANN: conduit-network-stream, A base layer for network protocols with Conduits

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

ANN: conduit-network-stream, A base layer for network protocols with Conduits

Nils Schweinsberg-2
Hello everyone,

I just uploaded my "conduit-network-stream" package to hackage. It's a
base layer for network protocols based on the "Conduit" package by
Michael Snoyman which makes it possible to send multiple messages over a
continuous network connection in a convenient way.

I wrote more about it at:

https://github.com/mcmaniac/conduit-network-stream/blob/master/README.md

It is available on hackage and github:

http://hackage.haskell.org/package/conduit-network-stream
https://github.com/mcmaniac/conduit-network-stream

Until the documentation on hackage is generated, I also host the haddock
documentation at:

http://hs.nils.cc/conduit-network-stream/index.html

For any questions/problems/requests, please ask!


- Nils

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: ANN: conduit-network-stream, A base layer for network protocols with Conduits

Alexander Vershilov
Hello, Nils.

Can you describe if there any difference with latest conduit API  (yield, await) that can be
used to write functions in a very similar style, but without using exeternal packages:

>  runTCPServer settings app = appSource ap $$ go =$= CL.mapM_ encode =$ appSink app
>     where
>         go = forever $ do
>            bp <- decode <$> await {- decode is inlined here assuming it can return different types -}
>            {- logic here-}
>            yield $ result
>            mapM (yeild) [result1,result2...]

more over you don't need to write appSource $$ ... appSink on the top.
Here is an example of a client that authorizes and then reads commands
from TBMChan and requests server using json format.

> runTCPClient settings ad go
>   where
>       go = do
>          source $$ await
>          is <- authorize
>          when is loop'
>       authorize = do
>            yield u >> yield (S8.pack "\n") $$ sink -- send authorization token
>            mx <- source $$ sinkParser json -- get responce
>            ...
>        loop' = do
>            minfo <- atomically $ readTBMChan ch
>            case minfo of
>              Nothing -> return ()
>              Just (message, respBox) -> do
>                yield message $$ C.concatMap (SL.toChunks . encode) =$ sink
>                resp <- source $$ sinkParser json
>                atomically $ putTMVar respBox resp
>                loop'
>       source = ppSource ad
>       sink = appSink ad



On 24 February 2013 22:23, Nils <[hidden email]> wrote:
Hello everyone,

I just uploaded my "conduit-network-stream" package to hackage. It's a base layer for network protocols based on the "Conduit" package by Michael Snoyman which makes it possible to send multiple messages over a continuous network connection in a convenient way.

I wrote more about it at:

https://github.com/mcmaniac/conduit-network-stream/blob/master/README.md

It is available on hackage and github:

http://hackage.haskell.org/package/conduit-network-stream
https://github.com/mcmaniac/conduit-network-stream

Until the documentation on hackage is generated, I also host the haddock documentation at:

http://hs.nils.cc/conduit-network-stream/index.html

For any questions/problems/requests, please ask!


- Nils

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe



--
Alexander

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: ANN: conduit-network-stream, A base layer for network protocols with Conduits

Nils Schweinsberg-2
Hi Alexander,

Am 25.02.2013 06:52, schrieb Alexander V Vershilov:
> Can you describe if there any difference with latest conduit API
> (yield, await) that can be
> used to write functions in a very similar style, but without using
> exeternal packages:

I have indeed written this library for the 0.5.6 API, so things might
have changed a bit in the 1.0 API, but I'd be surprised if the
fundamental flaws of this approach would have been fixed.

>  >  runTCPServer settings app = appSource ap $$ go =$= CL.mapM_ encode
> =$ appSink app
>  >     where
>  >         go = forever $ do
>  >            bp <- decode <$> await

`await` works on a conduit of strict `ByteString` chunks. The
size/length of each `ByteString` is dependent on your network
connection, so if you have a fast internet connection and your
bytestrings are sufficiently separated because you're waiting for a
response between each message your program might actually work as you
expect it to do (with a little bit of luck). But consider a simple
server/client application where the messages are not seperated by a
small delay:

     server = runTCPServer (..) $ \ad ->
         appSource ad $$ Data.Conduit.List.mapM_ (liftIO . print)
     client = runTCPClient (..) $ \ad ->
         (yield "msg1" >> yield "msg2") $$ appSink ad

The server will simply print out "msg1msg2" as one message, not as two
separate messages. Even worse, if your network connection is bad or your
chunks are getting too big for buffering, you might end up with
something like:

     "msg1ms"
     "g2"

`await` is not reliable in that regard because the network source is not
consistent and non-deterministic. My libraray makes sure that every
"yield" from the client corresponds to exactly one (not more or less)
"await" at the server.

There are more benefits when using my library. For example consider a
client which first sends an authorization message, then a couple of
hashes from different files and then maybe some timestamps on some other
files. Writing that server is straight forward:

     client = runTCPClient (..) $ \ad ->
         send1    ad $$ yield authenticationMsg
         sendList ad $$ mapM_ yield [file1hash, file2hash, file3hash]
         sendList ad $$ mapM_ yield [timestamp1, timestamp2]

     server = runTCPServer (..) $ \ad ->
         (next,[authmsg])  <- receive ad   $$ Data.Conduit.List.consume
         (next,hashes)     <- receive next $$ Data.Conduit.List.consume
         (next,timestamps) <- receive next $$ Data.Conduit.List.consume
         close next

Each `receive` corresponds to exactly one `send`. Without this library
you have to manually check/verify/associate each message by hand in one
big loop, whereas this library allows you to split your application into
logical conduit "groups" which are straight forward to use.


- Nils

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: ANN: conduit-network-stream, A base layer for network protocols with Conduits

Alexander Vershilov
Thanks for explanation.

However I'm still confused as if I'll use one of protocols (protobuf, json, sereal message, binary message) all of them has
iteration interface i.e. function of type (ByteString -> Result a) that returns
data Result = Error Text ByteString | NeedMore (ByteString -> Result a) | HaveResult a
all of them can be used in conduits using sinkParser or same approach and so there will be
no problem in sending and receiving messages even if they are split, and that will be done in a logical block.

I see that there will be problems if I'll use raw data and I see that your library solves it, am I right?



On 25 February 2013 15:53, Nils <[hidden email]> wrote:
Hi Alexander,

Am 25.02.2013 06:52, schrieb Alexander V Vershilov:

Can you describe if there any difference with latest conduit API
(yield, await) that can be
used to write functions in a very similar style, but without using
exeternal packages:

I have indeed written this library for the 0.5.6 API, so things might have changed a bit in the 1.0 API, but I'd be surprised if the fundamental flaws of this approach would have been fixed.


 >  runTCPServer settings app = appSource ap $$ go =$= CL.mapM_ encode
=$ appSink app
 >     where
 >         go = forever $ do
 >            bp <- decode <$> await

`await` works on a conduit of strict `ByteString` chunks. The size/length of each `ByteString` is dependent on your network connection, so if you have a fast internet connection and your bytestrings are sufficiently separated because you're waiting for a response between each message your program might actually work as you expect it to do (with a little bit of luck). But consider a simple server/client application where the messages are not seperated by a small delay:

    server = runTCPServer (..) $ \ad ->
        appSource ad $$ Data.Conduit.List.mapM_ (liftIO . print)
    client = runTCPClient (..) $ \ad ->
        (yield "msg1" >> yield "msg2") $$ appSink ad

The server will simply print out "msg1msg2" as one message, not as two separate messages. Even worse, if your network connection is bad or your chunks are getting too big for buffering, you might end up with something like:

    "msg1ms"
    "g2"

`await` is not reliable in that regard because the network source is not consistent and non-deterministic. My libraray makes sure that every "yield" from the client corresponds to exactly one (not more or less) "await" at the server.

There are more benefits when using my library. For example consider a client which first sends an authorization message, then a couple of hashes from different files and then maybe some timestamps on some other files. Writing that server is straight forward:

    client = runTCPClient (..) $ \ad ->
        send1    ad $$ yield authenticationMsg
        sendList ad $$ mapM_ yield [file1hash, file2hash, file3hash]
        sendList ad $$ mapM_ yield [timestamp1, timestamp2]

    server = runTCPServer (..) $ \ad ->
        (next,[authmsg])  <- receive ad   $$ Data.Conduit.List.consume
        (next,hashes)     <- receive next $$ Data.Conduit.List.consume
        (next,timestamps) <- receive next $$ Data.Conduit.List.consume
        close next

Each `receive` corresponds to exactly one `send`. Without this library you have to manually check/verify/associate each message by hand in one big loop, whereas this library allows you to split your application into logical conduit "groups" which are straight forward to use.


- Nils



--
Alexander

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: ANN: conduit-network-stream, A base layer for network protocols with Conduits

Nils Schweinsberg-2
Of course you don't have to use my library if your library already
solves the same problem.

Am 25.02.2013 13:10, schrieb Alexander V Vershilov:
> data Result = Error Text ByteString | NeedMore (ByteString -> Result a)
> | HaveResult a

I still see an issue with this if you're receiving multiple messages in
one `ByteString` chunk. A type like `HaveResult a ByteString` which also
returns the leftover `ByteString` after the first result has been found
could solve this, but you'd still have to re-yield this bytestring by
hand and you'll possible end up with more calls to "decode" than necessary.

Of course, with json you could avoid this problem by wrapping all
messages into a json-list and sending this list as one big message. But
that way you lose almost all advantages of a conduit - you cannot map
over those list iems, you cannot process each item one by one, you
cannot fold or filter them. When using this approach I don't really see
the advantage over using the plain old "connectTo/hGetContents" method?

Anyway, my library is independent of the "bytestring encoding" you want
to use with it. I think it's the most general solution and avoids having
to re-implement the same solution for each and every "protocol design"
over and over again.

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: ANN: conduit-network-stream, A base layer for network protocols with Conduits

ozataman
One question - and sorry that I didn't get a chance to try some examples myself - but can bi-directional conversations be carried between server/client in interleaving fashion using your library? Something like (simplified):

myClient = receive >>= send . compute

I have previously accomplished this using TBMChan and cereal-conduit to achieve the message delineation, but your library would certainly remove a layer of complexity if interleaved bi-directional talk is possible.

- Ozgun

On Monday, February 25, 2013 at 8:09 AM, Nils wrote:

Of course you don't have to use my library if your library already
solves the same problem.

Am 25.02.2013 13:10, schrieb Alexander V Vershilov:
data Result = Error Text ByteString | NeedMore (ByteString -> Result a)
| HaveResult a

I still see an issue with this if you're receiving multiple messages in
one `ByteString` chunk. A type like `HaveResult a ByteString` which also
returns the leftover `ByteString` after the first result has been found
could solve this, but you'd still have to re-yield this bytestring by
hand and you'll possible end up with more calls to "decode" than necessary.

Of course, with json you could avoid this problem by wrapping all
messages into a json-list and sending this list as one big message. But
that way you lose almost all advantages of a conduit - you cannot map
over those list iems, you cannot process each item one by one, you
cannot fold or filter them. When using this approach I don't really see
the advantage over using the plain old "connectTo/hGetContents" method?

Anyway, my library is independent of the "bytestring encoding" you want
to use with it. I think it's the most general solution and avoids having
to re-implement the same solution for each and every "protocol design"
over and over again.

_______________________________________________
Haskell-Cafe mailing list


_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe