Network client - reading and writing to a socket

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Network client - reading and writing to a socket

Manfred Lotz
Hi there,
I'm trying to write a network client which connects to an IMAP server
thus reading from and writing to socket 143.

I have an initial solution which works except that laziness bites me.
Perhaps the whole solution using connectTo and handle is the wrong
approach.

Here is the code:


module Main where

import System.IO
import qualified Data.ByteString.Char8 as B
import Network


server :: String
server = "127.0.0.1"
port :: PortID
port = PortNumber 143


sendMsg :: Handle -> String -> IO ()
sendMsg h m = do
  putStrLn ("Sending command: " ++ m)
  B.hPutStrLn h (B.pack m) >> hFlush h


recvMsg :: Handle -> IO B.ByteString
recvMsg h = do
  c <- B.hGet h 1
  c' <- B.hGetNonBlocking h 40000
  return $ B.concat [c,c']

 
 
main :: IO ()
main = withSocketsDo $ do
  h <- connectTo server port
  hSetBuffering h LineBuffering
  imapDialog h


imapDialog :: Handle -> IO ()
imapDialog h = do
  greet <- recvMsg h
  B.putStrLn greet

  sendMsg h "1 LOGIN manfred \"password\""
  resp <- recvMsg h
  B.putStrLn resp

  sendMsg h "2 SELECT HAM-learn"
  resp1 <- recvMsg h
  B.putStrLn resp1

  sendMsg h "3 FETCH 1:* (uid flags internaldate body[header.fields
  (Message-Id)])"
  resp2 <- recvMsg h
  B.putStrLn resp2

  sendMsg h "4 FETCH 1 (rfc822)"
  resp3 <- recvMsg h
  B.putStrLn resp3


The problem is that the message itself is some 30K big and I only
get some 16K of the message.

How could I force to get the whole message?

 

--
Manfred




Reply | Threaded
Open this post in threaded view
|

Network client - reading and writing to a socket

David Place

On Jul 31, 2011, at 11:39 AM, Manfred Lotz wrote:

> How could I force to get the whole message?

Did you try hFlush?



Reply | Threaded
Open this post in threaded view
|

Network client - reading and writing to a socket

Manfred Lotz
On Sun, 31 Jul 2011 12:02:16 -0400
David Place <d at vidplace.com> wrote:

>
> On Jul 31, 2011, at 11:39 AM, Manfred Lotz wrote:
>
> > How could I force to get the whole message?
>
> Did you try hFlush?
>
>

I tried it like this

recvMsg :: Handle -> IO B.ByteString
recvMsg h = do
  c <- B.hGet h 1
  hFlush h
  c' <- B.hGetNonBlocking h 40000
  return $ B.concat [c,c']


and it didn't help. Still an incomplete message.


--
Manfred




Reply | Threaded
Open this post in threaded view
|

Network client - reading and writing to a socket

David Place

____________________
David Place  
Owner, Panpipes Ho! LLC
http://panpipesho.com
d at vidplace.com



On Jul 31, 2011, at 12:17 PM, Manfred Lotz wrote:

>  c <- B.hGet h 1
>  hFlush h
>  c' <- B.hGetNonBlocking h 40000

Try putting one here too.  Can't hurt right?

>  return $ B.concat [c,c']



Reply | Threaded
Open this post in threaded view
|

Network client - reading and writing to a socket

Manfred Lotz
On Sun, 31 Jul 2011 12:20:17 -0400
David Place <d at vidplace.com> wrote:

>
> ____________________
> David Place  
> Owner, Panpipes Ho! LLC
> http://panpipesho.com
> d at vidplace.com
>
>
>
> On Jul 31, 2011, at 12:17 PM, Manfred Lotz wrote:
>
> >  c <- B.hGet h 1
> >  hFlush h
> >  c' <- B.hGetNonBlocking h 40000
>
> Try putting one here too.  Can't hurt right?
>
> >  return $ B.concat [c,c']
>

Now I have  

recvMsg h = do
  c <- B.hGet h 1
  hFlush h
  c' <- B.hGetNonBlocking h 40000
  hFlush h
  return $ B.concat [c,c']


Doesn't help either.




--
Manfred




Reply | Threaded
Open this post in threaded view
|

Network client - reading and writing to a socket

David Place
Shouldn't be laziness that is biting you.  You are using strict ByteStrings.  I'm stumped!  Did you try artificially inserting something like this to force c and c'?

On Jul 31, 2011, at 12:25 PM, Manfred Lotz wrote:

> recvMsg h = do
>  c <- B.hGet h 1
>  hFlush h

print c

>  c' <- B.hGetNonBlocking h 40000

print c'

>  hFlush h
>  return $ B.concat [c,c']



Reply | Threaded
Open this post in threaded view
|

Network client - reading and writing to a socket

Manfred Lotz
On Sun, 31 Jul 2011 12:42:45 -0400
David Place <d at vidplace.com> wrote:

> Shouldn't be laziness that is biting you.  You are using strict
> ByteStrings.  I'm stumped!  Did you try artificially inserting
> something like this to force c and c'?
>
> On Jul 31, 2011, at 12:25 PM, Manfred Lotz wrote:
>
> > recvMsg h = do
> >  c <- B.hGet h 1
> >  hFlush h
>
> print c
>
> >  c' <- B.hGetNonBlocking h 40000
>
> print c'
>
> >  hFlush h
> >  return $ B.concat [c,c']
>
>

Didn't help either.


If I add another

  resp4 <- recvMsg h
  B.putStrLn resp4

at the end of imapDialog then I get the rest of the message.





--
Manfred




Reply | Threaded
Open this post in threaded view
|

Network client - reading and writing to a socket

David Place
On Jul 31, 2011, at 1:20 PM, Manfred Lotz wrote:

> Didn't help either.
>
>
> If I add another
>
>  resp4 <- recvMsg h
>  B.putStrLn resp4
>
> at the end of imapDialog then I get the rest of the message.

Sorry, I'm just shooting in the dark here.  Did you  try this?

hSetBuffering h NoBuffering


Reply | Threaded
Open this post in threaded view
|

Network client - reading and writing to a socket

Manfred Lotz
On Sun, 31 Jul 2011 13:46:56 -0400
David Place <d at vidplace.com> wrote:

> On Jul 31, 2011, at 1:20 PM, Manfred Lotz wrote:
>
> > Didn't help either.
> >
> >
> > If I add another
> >
> >  resp4 <- recvMsg h
> >  B.putStrLn resp4
> >
> > at the end of imapDialog then I get the rest of the message.
>
> Sorry, I'm just shooting in the dark here.  Did you  try this?
>

No sorry. I'm happy you try to help me.


> hSetBuffering h NoBuffering
>

Didn't make a difference.


--
Manfred




Reply | Threaded
Open this post in threaded view
|

Network client - reading and writing to a socket

Patrick LeBoutillier
In reply to this post by Manfred Lotz
Manfred,

> The problem is that the message itself is some 30K big and I only
> get some 16K of the message.
>
> How could I force to get the whole message?

My guess is that you can't. This call:

c' <- B.hGetNonBlocking h 40000

tries to read as much as it can (up to 40000 bytes) but it won't block
to wait for data. Perhaps the rest of your message is in a different
TCP packet or delayed or whatever, but I think you have to keep on
reading (and maybe block) until you know you have read the entire
message. The IMAP specs will tell you how to identify the "end of the
message".

BTW: This issue is not Haskell specific. If you implement the same
code in C, Perl or Java you will have to deal with the same problem.
When you read from a socket, there is no general way of knowing that
the other side has sent everything.


Patrick

>
> --
> Manfred
>
>
>
> _______________________________________________
> Beginners mailing list
> Beginners at haskell.org
> http://www.haskell.org/mailman/listinfo/beginners
>



--
=====================
Patrick LeBoutillier
Rosem?re, Qu?bec, Canada


Reply | Threaded
Open this post in threaded view
|

Network client - reading and writing to a socket

David McBride-7
It seems you are expected to fetch until you have the entire thing.
You can't just fetch 40000 and hope you get it all.  If you do it
blocking, it will be almost guaranteed to block unless there were
40000 to fetch.  If you do it non blocking, you have to keep fetching
until you are sure you have the entire message.

If you are doing this as simply as possible, that means you have to
fetch, and then repeatedly check the string until you are sure you
have the whole thing.  You'll do that by fetching once, checking the
beginning of the string for the {342} in the following example from
the rfc:

C:    a004 fetch 12 body[header]
S:    * 12 FETCH (BODY[HEADER] {342}
S:    Date: Wed, 17 Jul 1996 02:23:25 -0700 (PDT)
S:    From: Terry Gray <gray at cac.washington.edu>
S:    Subject: IMAP4rev1 WG mtg summary and minutes
S:    To: imap at cac.washington.edu
S:    cc: minutes at CNRI.Reston.VA.US, John Klensin <KLENSIN at MIT.EDU>
S:    Message-Id: <B27397-0100000 at cac.washington.edu>
S:    MIME-Version: 1.0
S:    Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
S:
S:    )

And then fetching until you have the next 342 characters.  This is
pretty tedious in any language, not just haskell, but that's how this
protocol is.

If you want to do this the haskell way, which I highly recommend, I
would pull out the network-attoparsec library.  You should be able to
use parseWith and pass it a recv function that returns bytestrings.
Write a simple parser which will look for "* <num> FETCH
([a-zA-Z]+\[[a-zA-Z]+\] {[0-9]+} and then use that last number to take
(take is an attoparsec function) that last byte count.  If you wire
this up to your network socket, it will parse and slurp up the
response and it will pull the exact number of bytes needed and your
code will be very elegant.  Attoparsec is a little hard to use at
first, but it is well worth if if you have a mind to parse internet
protocols.

On Sun, Jul 31, 2011 at 3:14 PM, Patrick LeBoutillier
<patrick.leboutillier at gmail.com> wrote:

> Manfred,
>
>> The problem is that the message itself is some 30K big and I only
>> get some 16K of the message.
>>
>> How could I force to get the whole message?
>
> My guess is that you can't. This call:
>
> c' <- B.hGetNonBlocking h 40000
>
> tries to read as much as it can (up to 40000 bytes) but it won't block
> to wait for data. Perhaps the rest of your message is in a different
> TCP packet or delayed or whatever, but I think you have to keep on
> reading (and maybe block) until you know you have read the entire
> message. The IMAP specs will tell you how to identify the "end of the
> message".
>
> BTW: This issue is not Haskell specific. If you implement the same
> code in C, Perl or Java you will have to deal with the same problem.
> When you read from a socket, there is no general way of knowing that
> the other side has sent everything.
>
>
> Patrick
>
>>
>> --
>> Manfred
>>
>>
>>
>> _______________________________________________
>> Beginners mailing list
>> Beginners at haskell.org
>> http://www.haskell.org/mailman/listinfo/beginners
>>
>
>
>
> --
> =====================
> Patrick LeBoutillier
> Rosem?re, Qu?bec, Canada
>
> _______________________________________________
> Beginners mailing list
> Beginners at haskell.org
> http://www.haskell.org/mailman/listinfo/beginners
>


Reply | Threaded
Open this post in threaded view
|

Network client - reading and writing to a socket

David McBride-7
Sorry I said network-attoparsec, the attoparsec library is all you
need.  I'm not sure where I got network-attoparsec from.

On Sun, Jul 31, 2011 at 6:21 PM, David McBride <dmcbride at neondsl.com> wrote:

> It seems you are expected to fetch until you have the entire thing.
> You can't just fetch 40000 and hope you get it all. ?If you do it
> blocking, it will be almost guaranteed to block unless there were
> 40000 to fetch. ?If you do it non blocking, you have to keep fetching
> until you are sure you have the entire message.
>
> If you are doing this as simply as possible, that means you have to
> fetch, and then repeatedly check the string until you are sure you
> have the whole thing. ?You'll do that by fetching once, checking the
> beginning of the string for the {342} in the following example from
> the rfc:
>
> C: ? ?a004 fetch 12 body[header]
> S: ? ?* 12 FETCH (BODY[HEADER] {342}
> S: ? ?Date: Wed, 17 Jul 1996 02:23:25 -0700 (PDT)
> S: ? ?From: Terry Gray <gray at cac.washington.edu>
> S: ? ?Subject: IMAP4rev1 WG mtg summary and minutes
> S: ? ?To: imap at cac.washington.edu
> S: ? ?cc: minutes at CNRI.Reston.VA.US, John Klensin <KLENSIN at MIT.EDU>
> S: ? ?Message-Id: <B27397-0100000 at cac.washington.edu>
> S: ? ?MIME-Version: 1.0
> S: ? ?Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
> S:
> S: ? ?)
>
> And then fetching until you have the next 342 characters. ?This is
> pretty tedious in any language, not just haskell, but that's how this
> protocol is.
>
> If you want to do this the haskell way, which I highly recommend, I
> would pull out the network-attoparsec library. ?You should be able to
> use parseWith and pass it a recv function that returns bytestrings.
> Write a simple parser which will look for "* <num> FETCH
> ([a-zA-Z]+\[[a-zA-Z]+\] {[0-9]+} and then use that last number to take
> (take is an attoparsec function) that last byte count. ?If you wire
> this up to your network socket, it will parse and slurp up the
> response and it will pull the exact number of bytes needed and your
> code will be very elegant. ?Attoparsec is a little hard to use at
> first, but it is well worth if if you have a mind to parse internet
> protocols.
>
> On Sun, Jul 31, 2011 at 3:14 PM, Patrick LeBoutillier
> <patrick.leboutillier at gmail.com> wrote:
>> Manfred,
>>
>>> The problem is that the message itself is some 30K big and I only
>>> get some 16K of the message.
>>>
>>> How could I force to get the whole message?
>>
>> My guess is that you can't. This call:
>>
>> c' <- B.hGetNonBlocking h 40000
>>
>> tries to read as much as it can (up to 40000 bytes) but it won't block
>> to wait for data. Perhaps the rest of your message is in a different
>> TCP packet or delayed or whatever, but I think you have to keep on
>> reading (and maybe block) until you know you have read the entire
>> message. The IMAP specs will tell you how to identify the "end of the
>> message".
>>
>> BTW: This issue is not Haskell specific. If you implement the same
>> code in C, Perl or Java you will have to deal with the same problem.
>> When you read from a socket, there is no general way of knowing that
>> the other side has sent everything.
>>
>>
>> Patrick
>>
>>>
>>> --
>>> Manfred
>>>
>>>
>>>
>>> _______________________________________________
>>> Beginners mailing list
>>> Beginners at haskell.org
>>> http://www.haskell.org/mailman/listinfo/beginners
>>>
>>
>>
>>
>> --
>> =====================
>> Patrick LeBoutillier
>> Rosem?re, Qu?bec, Canada
>>
>> _______________________________________________
>> Beginners mailing list
>> Beginners at haskell.org
>> http://www.haskell.org/mailman/listinfo/beginners
>>
>


Reply | Threaded
Open this post in threaded view
|

Network client - reading and writing to a socket

Manfred Lotz
In reply to this post by Patrick LeBoutillier
On Sun, 31 Jul 2011 15:14:23 -0400
Patrick LeBoutillier <patrick.leboutillier at gmail.com> wrote:

> Manfred,
>
> > The problem is that the message itself is some 30K big and I only
> > get some 16K of the message.
> >
> > How could I force to get the whole message?
>
> My guess is that you can't. This call:
>
> c' <- B.hGetNonBlocking h 40000
>
> tries to read as much as it can (up to 40000 bytes) but it won't block
> to wait for data. Perhaps the rest of your message is in a different
> TCP packet or delayed or whatever, but I think you have to keep on
> reading (and maybe block) until you know you have read the entire
> message. The IMAP specs will tell you how to identify the "end of the
> message".
>
> BTW: This issue is not Haskell specific. If you implement the same
> code in C, Perl or Java you will have to deal with the same problem.
> When you read from a socket, there is no general way of knowing that
> the other side has sent everything.
>

Hmm. I'm not quite sure you are fully right. On the one hand I believe
that this could be an issue which arises in python/perl etc. as well.
On the other hand I believe it should be possible to receive from a
socket what is available at a certain point of time.

I found this link http://sequence.complete.org/node/257, and when I run
the code I get the full message from the imap server even if the
message is a couple of megabytes big.

I have to figure out how to use the code for my need as I do not get
the input from the keyboard. As a haskell beginner things like this are
not always trivial.


--
Thanks,
Manfred



 


>
> Patrick
>
> >
> > --
> > Manfred
> >
> >
> >
> > _______________________________________________
> > Beginners mailing list
> > Beginners at haskell.org
> > http://www.haskell.org/mailman/listinfo/beginners
> >
>
>
>



--
Manfred




Reply | Threaded
Open this post in threaded view
|

Network client - reading and writing to a socket

Chaddaï Fouché
On Mon, Aug 1, 2011 at 9:16 PM, Manfred Lotz <manfred.lotz at arcor.de> wrote:

> On Sun, 31 Jul 2011 15:14:23 -0400
> Patrick LeBoutillier <patrick.leboutillier at gmail.com> wrote:
>
>> Manfred,
>>
>> > The problem is that the message itself is some 30K big and I only
>> > get some 16K of the message.
>> >
>> > How could I force to get the whole message?
>>
>> My guess is that you can't. This call:
>>
>> c' <- B.hGetNonBlocking h 40000
>>
>> tries to read as much as it can (up to 40000 bytes) but it won't block
>> to wait for data. Perhaps the rest of your message is in a different
>> TCP packet or delayed or whatever, but I think you have to keep on
>> reading (and maybe block) until you know you have read the entire
>> message. The IMAP specs will tell you how to identify the "end of the
>> message".
>>
>> BTW: This issue is not Haskell specific. If you implement the same
>> code in C, Perl or Java you will have to deal with the same problem.
>> When you read from a socket, there is no general way of knowing that
>> the other side has sent everything.
>>
>
> Hmm. I'm not quite sure you are fully right. On the one hand I believe
> that this could be an issue which arises in python/perl etc. as well.
> On the other hand I believe it should be possible to receive from a
> socket what is available at a certain point of time.

Yes and that's exactly what you have done : your hGetLine blocks until
there is something to read on the socket and then your hGetNonBlocking
gets *everything* there is to read on this socket at this exact
moment... Except as had been said by others that your message has been
split into several packets and they're not all there when you hGet.

What you want is not "what is available at a certain point of time",
you want to read a whole message, except that this "message" notion is
only in your head (and in a specific protocol, IMAP here) it has no
direct relevance to how things happens on the network.

> I found this link http://sequence.complete.org/node/257, and when I run
> the code I get the full message from the imap server even if the
> message is a couple of megabytes big.

Of course you get the whole message ! This code try to read (with
blocking calls) forever what's on your socket, it reads it, send it on
a TChan and then retry reading it, bit by bit it gets your whole
message, of course it has no idea that it got your whole message and
if nothing is done it will continue to wait on your socket for all
eternity...
The key point is this function :

> listenLoop :: IO a -> TChan a -> IO ()
> listenLoop act chan =
>       sequence_ (repeat (act >>= atomically . writeTChan chan))

This does not stop short of an exception. Two of those loops are
started each in their own thread (so that they don't block the rest of
the program) to read stdin and a socket respectively.

> I have to figure out how to use the code for my need as I do not get
> the input from the keyboard.

This code is probably not doing what you want, this is a toy example
where most of the complexity comes from handling two source of input
simultaneously and collating their answer on the same TChan. Its main
problem is that it don't even try to read messages, it just read
everything and doesn't know when it's finished. You need to listen to
David's advice and read a bit on the protocol you're trying to handle
and how it encode the notion of message, how it signals that the
message has ended.

--
Jeda?


Reply | Threaded
Open this post in threaded view
|

Network client - reading and writing to a socket

Manfred Lotz
On Tue, 2 Aug 2011 02:59:44 +0200
Chadda? Fouch? <chaddai.fouche at gmail.com> wrote:

> On Mon, Aug 1, 2011 at 9:16 PM, Manfred Lotz <manfred.lotz at arcor.de>
> wrote:
> > On Sun, 31 Jul 2011 15:14:23 -0400
> > Patrick LeBoutillier <patrick.leboutillier at gmail.com> wrote:
> >
> >> Manfred,
> >>
> >> > The problem is that the message itself is some 30K big and I only
> >> > get some 16K of the message.
> >> >
> >> > How could I force to get the whole message?
> >>
> >> My guess is that you can't. This call:
> >>
> >> c' <- B.hGetNonBlocking h 40000
> >>
> >> tries to read as much as it can (up to 40000 bytes) but it won't
> >> block to wait for data. Perhaps the rest of your message is in a
> >> different TCP packet or delayed or whatever, but I think you have
> >> to keep on reading (and maybe block) until you know you have read
> >> the entire message. The IMAP specs will tell you how to identify
> >> the "end of the message".
> >>
> >> BTW: This issue is not Haskell specific. If you implement the same
> >> code in C, Perl or Java you will have to deal with the same
> >> problem. When you read from a socket, there is no general way of
> >> knowing that the other side has sent everything.
> >>
> >
> > Hmm. I'm not quite sure you are fully right. On the one hand I
> > believe that this could be an issue which arises in python/perl
> > etc. as well. On the other hand I believe it should be possible to
> > receive from a socket what is available at a certain point of time.
>
> Yes and that's exactly what you have done : your hGetLine blocks until
> there is something to read on the socket and then your hGetNonBlocking
> gets *everything* there is to read on this socket at this exact
> moment... Except as had been said by others that your message has been
> split into several packets and they're not all there when you hGet.
>
> What you want is not "what is available at a certain point of time",
> you want to read a whole message, except that this "message" notion is
> only in your head (and in a specific protocol, IMAP here) it has no
> direct relevance to how things happens on the network.
>
> > I found this link http://sequence.complete.org/node/257, and when I
> > run the code I get the full message from the imap server even if the
> > message is a couple of megabytes big.
>
> Of course you get the whole message ! This code try to read (with
> blocking calls) forever what's on your socket, it reads it, send it on
> a TChan and then retry reading it, bit by bit it gets your whole
> message, of course it has no idea that it got your whole message and
> if nothing is done it will continue to wait on your socket for all
> eternity...
> The key point is this function :
>
> > listenLoop :: IO a -> TChan a -> IO ()
> > listenLoop act chan =
> >       sequence_ (repeat (act >>= atomically . writeTChan chan))
>
> This does not stop short of an exception. Two of those loops are
> started each in their own thread (so that they don't block the rest of
> the program) to read stdin and a socket respectively.
>
> > I have to figure out how to use the code for my need as I do not get
> > the input from the keyboard.
>
> This code is probably not doing what you want, this is a toy example
> where most of the complexity comes from handling two source of input
> simultaneously and collating their answer on the same TChan. Its main

The code works pretty well, and nevertheless you are right. Because it
is operating in its own thread it doesn't matter if it blocks.


I guess I understand now. I have to know what to expect back from a
certain command (so that I can adjust the receive accordingly) I did
send to the imap server.

Thanks to you and the others for explaining.


--
Manfred