binary IO

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
42 messages Options
123
Reply | Threaded
Open this post in threaded view
|

binary IO

J. Pablo Fernández
It seems I have found a hole in Haskell... :(
Before I start to develop a library with functions such us those on
http://haskell.org/hawiki/BinaryIo (hGetWord8, hGetWord16le, hGetWord16be,
etc), is there some reliable library that can help me ?
I basically need a set of functions to read binary data out of a Handle (a
higher lever of hGetBuf and hPutBuf). What I am doing is implementing a DNS
server, so, my typical need is:
First two bytes are an integer.
The next bit is a boolean.
So are the following three bits.
Then I have 4 bits which is also an integer.
Etc.
Any help is greatly appreciated.
Thank you.
--
Pupeno <[hidden email]> (http://pupeno.com)

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

attachment0 (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: binary IO

Donald Bruce Stewart
pupeno:

> It seems I have found a hole in Haskell... :(
> Before I start to develop a library with functions such us those on
> http://haskell.org/hawiki/BinaryIo (hGetWord8, hGetWord16le, hGetWord16be,
> etc), is there some reliable library that can help me ?
> I basically need a set of functions to read binary data out of a Handle (a
> higher lever of hGetBuf and hPutBuf). What I am doing is implementing a DNS
> server, so, my typical need is:
> First two bytes are an integer.
> The next bit is a boolean.
> So are the following three bits.
> Then I have 4 bits which is also an integer.
> Etc.
> Any help is greatly appreciated.
> Thank you.

Use NewBinary, as suggested by that wiki page :)
This is the standard binary lib, used in ghc, nhc and lots of other
projects (even lambdabot!). With this, and Data.Bits, you should be able
to do whatever you need, I think.

-- Don
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: binary IO

Jeremy Shaw
On Dec 26, 2005 08:55 PM, Donald Bruce Stewart <[hidden email]>
wrote:

> pupeno:
> > It seems I have found a hole in Haskell... :(
> > Before I start to develop a library with functions such us those on
> > http://haskell.org/hawiki/BinaryIo (hGetWord8, hGetWord16le,
> > hGetWord16be,
> > etc), is there some reliable library that can help me ?

> Use NewBinary, as suggested by that wiki page :)

Some caveats. The NewBinary library contains two things:

 (1) A uniform interface for reading and writing binary files and
     binary memory, including bitwise reading/writing.

 (2) A Binary class for serialization/deserialization

While it is tempting to use the functions in the Binary class (put/get),
it is important to realize that they do not care about big endian vs
little endian. Nor do they attempt to be compatible with equivalent
C data types. For example, Char is stored as 4 bytes.

However, while the Binary class in NewBinary may not be appropriate,
the uniform interface to binary files/memory could be a good foundation
for building hGetWord16le, hGetWord16be, etc.

If you submitted a module that added these functions, I would be glad
to update the archive.

j.

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: binary IO

Tomasz Zielonka
In reply to this post by J. Pablo Fernández
On Tue, Dec 27, 2005 at 01:10:24AM -0300, Pupeno wrote:
> It seems I have found a hole in Haskell... :(

I don't think so. Even if there was no way to do this, you could fill
the hole yourself.

> Before I start to develop a library with functions such us those on
> http://haskell.org/hawiki/BinaryIo (hGetWord8, hGetWord16le, hGetWord16be,
> etc), is there some reliable library that can help me ?
> I basically need a set of functions to read binary data out of a Handle (a
> higher lever of hGetBuf and hPutBuf). What I am doing is implementing a DNS
> server, so, my typical need is:

I think in the case of DNS, where the packets are quite small, it's best
to separate reading the Handle from interpreting the data. For example,
you can read from socket to a FastPackedString, and have a purely
functional deserialization code work on it.

Some time ago I was playing with DNS too. I have a library that can
construct and interpret DNS packets, but it's a bit incomplete right
now. It reads packets as Strings, but it should be quite straightforward
to make it read and interpret FastPackedStrings.

    http://www.uncurry.com/repos/TzDNS

I've just glanced at the code, and yes, it is a bit of a mess. I'll see
if I can tidy it up. Perhaps you could help me?

Best regards
Tomasz

--
I am searching for a programmer who is good at least in some of
[Haskell, ML, C++, Linux, FreeBSD, math] for work in Warsaw, Poland
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: binary IO

Brian McQueen
How is this different from the (un)pickle  process that has been
discussed here recently?  Recently I've seen the Binary discussions,
and the pickeling discussions, and I noticed they seemed to be talking
about the same process.

Brian McQueen

On 12/26/05, Tomasz Zielonka <[hidden email]> wrote:

> On Tue, Dec 27, 2005 at 01:10:24AM -0300, Pupeno wrote:
> > It seems I have found a hole in Haskell... :(
>
> I don't think so. Even if there was no way to do this, you could fill
> the hole yourself.
>
> > Before I start to develop a library with functions such us those on
> > http://haskell.org/hawiki/BinaryIo (hGetWord8, hGetWord16le, hGetWord16be,
> > etc), is there some reliable library that can help me ?
> > I basically need a set of functions to read binary data out of a Handle (a
> > higher lever of hGetBuf and hPutBuf). What I am doing is implementing a DNS
> > server, so, my typical need is:
>
> I think in the case of DNS, where the packets are quite small, it's best
> to separate reading the Handle from interpreting the data. For example,
> you can read from socket to a FastPackedString, and have a purely
> functional deserialization code work on it.
>
> Some time ago I was playing with DNS too. I have a library that can
> construct and interpret DNS packets, but it's a bit incomplete right
> now. It reads packets as Strings, but it should be quite straightforward
> to make it read and interpret FastPackedStrings.
>
>     http://www.uncurry.com/repos/TzDNS
>
> I've just glanced at the code, and yes, it is a bit of a mess. I'll see
> if I can tidy it up. Perhaps you could help me?
>
> Best regards
> Tomasz
>
> --
> I am searching for a programmer who is good at least in some of
> [Haskell, ML, C++, Linux, FreeBSD, math] for work in Warsaw, Poland
> _______________________________________________
> Haskell-Cafe mailing list
> [hidden email]
> http://www.haskell.org/mailman/listinfo/haskell-cafe
>
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: binary IO

Donald Bruce Stewart
mcqueenorama:
> How is this different from the (un)pickle  process that has been
> discussed here recently?  Recently I've seen the Binary discussions,
> and the pickeling discussions, and I noticed they seemed to be talking
> about the same process.

Yep, same thing.

-- Don
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: binary IO

Tomasz Zielonka
In reply to this post by Brian McQueen
On Mon, Dec 26, 2005 at 10:20:15PM -0800, Brian McQueen wrote:
> How is this different from the (un)pickle  process that has been
> discussed here recently?  Recently I've seen the Binary discussions,
> and the pickeling discussions, and I noticed they seemed to be talking
> about the same process.

Not much different I guess, but I haven't followed the discussion
closely. Anyway, the only value of my DNS code is that it already
handles DNS packets in pure Haskell.

BTW, if I abstract the deserialization monad using the BinaryParser type
class, I can use different concrete backends, for example Parsec,
low-level UArray based parser, FastPackedString based parser, NewBinary,
etc. Can be handy for debugging - Parsec can give nice error messages,
the only problem is that it insists in line/column positions.

Best regards
Tomasz

--
I am searching for a programmer who is good at least in some of
[Haskell, ML, C++, Linux, FreeBSD, math] for work in Warsaw, Poland
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: binary IO

Tomasz Zielonka
In reply to this post by Tomasz Zielonka
On Tue, Dec 27, 2005 at 07:00:12AM +0100, Tomasz Zielonka wrote:
> I've just glanced at the code, and yes, it is a bit of a mess. I'll see
> if I can tidy it up. Perhaps you could help me?

I've just made the parser more generic, which helped to reduce code
duplication. Maybe I will finally manage to do something with this
code.

Best regards
Tomasz

--
I am searching for a programmer who is good at least in some of
[Haskell, ML, C++, Linux, FreeBSD, math] for work in Warsaw, Poland
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: binary IO

Jeremy Shaw
In reply to this post by Brian McQueen
On Dec 26, 2005 10:20 PM, Brian McQueen <[hidden email]> wrote:

> How is this different from the (un)pickle  process that has been
> discussed here recently?

There is one important difference in my mind. Most of the pickling code
is concerned with turning haskell data into a binary stream that can
later be turned back into haskell data. There is no interoperability
required -- the thing that writes it is the thing that will read it.

For parsing arbitrary binary data formats a few other important
issues arise that you can sometimes forget about with pickling such
as:

 ~ byte-alignment
 ~ big-endian vs little endian
 ~ size of data types (for example is Char one byte or four)
 ~ bitwise parsing

Both pickling and binary format parsing can be built on the same
underlying low-level binary access methods.

I have also wondered if they might be able to share the same
high level interface because they don't really seem that far
removed from one another in many respects. But I have not had
time to think about it much.

SerTH can be used to derive pickling code for arbitrary haskell
data types. And I am pretty sure I once saw code that could
derive a binary parser from a C struct. So, there really does
seem to be a lot of overlap.

j.

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: binary IO

Joel Reymont
This is what I spent the past 3 months on. Pickling code that  
interoperates with a C++ server that sends things to me little-
endian. And sends other wierd data like unicode strings that are zero-
terminated.

SerTH does not handle things like that since it cannot divine the  
wire format for you. I used the pickler combinators approach which  
worked great in theory since I could specify the wire format and have  
a single spec for pickling and unpickling. In practice this turned  
out to be too slow and I still don't know why.

I'm still waiting to see how this should really be done.

        Joel

On Dec 27, 2005, at 7:35 AM, Jeremy Shaw wrote:

> There is one important difference in my mind. Most of the pickling  
> code
> is concerned with turning haskell data into a binary stream that can
> later be turned back into haskell data. There is no interoperability
> required -- the thing that writes it is the thing that will read it.
>
> For parsing arbitrary binary data formats a few other important
> issues arise that you can sometimes forget about with pickling such
> as:
>
>  ~ byte-alignment
>  ~ big-endian vs little endian
>  ~ size of data types (for example is Char one byte or four)
>  ~ bitwise parsing
>
> Both pickling and binary format parsing can be built on the same
> underlying low-level binary access methods.

--
http://wagerlabs.com/





_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: binary IO

Tomasz Zielonka
On Tue, Dec 27, 2005 at 08:37:27AM +0000, Joel Reymont wrote:

> This is what I spent the past 3 months on. Pickling code that  
> interoperates with a C++ server that sends things to me little-
> endian. And sends other wierd data like unicode strings that are zero-
> terminated.
>
> SerTH does not handle things like that since it cannot divine the  
> wire format for you. I used the pickler combinators approach which  
> worked great in theory since I could specify the wire format and have  
> a single spec for pickling and unpickling. In practice this turned  
> out to be too slow and I still don't know why.

What do you mean by too slow? What is the desired input/output
throughput?

Maybe I'll find some time to improve my BinaryParser/BinaryUnparser
code.

Best regards
Tomasz

--
I am searching for a programmer who is good at least in some of
[Haskell, ML, C++, Linux, FreeBSD, math] for work in Warsaw, Poland
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: binary IO

Joel Reymont
Tomasz,

Try http://wagerlabs.com/timeleak.tgz. See the "Killer pickler  
combinators" thread as well.

My desired goal is to have 4k bots (threads?) running at the same  
time. At, say, 1k/s per bot I figure something like 4Mb/s round-trip.  
Each bot cannot spend more than a couple of seconds on pickling/
unpickling. I'm not even close to hitting that goal even reading from  
a file.

I'm getting delays of 4s+ with just 100-200 bots reading from a file  
and even less than that in a networked environment. The more bots I  
run the higher the delays, to the point of frequent delays of 10s+.  
The kicker is that some packets come in at 4k compressed with Zlib  
but become something like 50k uncompressed and then expand to a list  
of 500+ records, etc.

Each bot is given 5, 15 or 35 seconds to respond by the poker server  
and this is proving to be too little for my Haskell implementation.

All the data comes in little-endian and there are a lot of zero-
terminated unicode strings (each short needs to be swapped). My  
customer is running FreeBSD/Intel so swapping should not affect them  
but their performance is that much different than mine. It could well  
be that the scheduler is playing a role in this also.

I spent 3 hard months (barely any weekends, no holidays) polishing my  
Haskell code. I started the Erlang rewrite at about 3pm this saturday  
and about 1.5 workdays later I have 80% of what I need. I expect to  
finish the rest today and will have a good yardstick to measure  
Haskell against.

It took me ~2 hours of adapting the Pickler Combinators paper to  
Erlang. See http://wagerlabs.com/erlang/pickle.erl. Scroll to the  
bottom for examples. I spent the rest of the time converting the 150+  
records to the new format.

Once I deliver my rewritten project I would like to setup a Haskell  
vs. Erlang pickling contest to see if Haskell can match up and figure  
out what is preventing it from doing so if it can't. Then I lend a  
hand to add whatever is missing.

My only requirement is that there be a _single_ spec for pickling and  
unpickling, i.e. no separate methods. The following is not acceptable  
to me ;-).

puTableInfo :: PU TableInfo
puTableInfo = PU fn1 fn2 fn3
     where fn1 ptr ptr ti =
               do ptr <- appP endianW64 ptr $ ti_1 ti
                  ptr <- appP endianW16 ptr $ ti_2 ti
                  ptr <- appP endianW16 ptr $ ti_3 ti
                  ptr <- appP w8 ptr $ ti_4 ti
                  ptr <- appP wstring ptr $ ti_5 ti
                  ptr <- appP endianW32 ptr $ ti_6 ti
                  ptr <- appP (enum endianW16 :: PU GameType) ptr $  
ti_7 ti
                  ptr <- appP endianW16 ptr $ ti_8 ti
                  ptr <- appP bool ptr $ ti_9 ti
                  ptr <- appP endianW64 ptr $ ti_10 ti
                  ptr <- appP endianW64 ptr $ ti_11 ti
                  ptr <- appP endianW64 ptr $ ti_12 ti
                  ptr <- appP endianW64 ptr $ ti_13 ti
                  ptr <- appP endianW16 ptr $ ti_14 ti
                  ptr <- appP (enum w8) ptr $ ti_15 ti
                  ptr <- appP endianW32 ptr $ ti_16 ti
                  ptr <- appP (enum w8) ptr $ ti_17 ti
                  ptr <- appP endianW32 ptr $ ti_18 ti
                  ptr <- appP (list endianW32 w8) ptr $ ti_19 ti
                  ptr <- appP endianW32 ptr $ ti_20 ti
                  return $! ptr
           fn2 ptr ptr =
               do (ti1, ptr) <- appU endianW64 ptr
                  (ti2, ptr) <- appU endianW16 ptr
                  (ti3, ptr) <- appU endianW16 ptr
                  (ti4, ptr) <- appU w8 ptr
                  (ti5, ptr) <- appU wstring ptr
                  (ti6, ptr) <- appU endianW32 ptr
                  (ti7, ptr) <- appU (enum endianW16 :: PU GameType) ptr
                  (ti8, ptr) <- appU endianW16 ptr
                  (ti9, ptr) <- appU bool ptr
                  (ti10, ptr) <- appU endianW64 ptr
                  (ti11, ptr) <- appU endianW64 ptr
                  (ti12, ptr) <- appU endianW64 ptr
                  (ti13, ptr) <- appU endianW64 ptr
                  (ti14, ptr) <- appU endianW16 ptr
                  (ti15, ptr) <- appU (enum w8) ptr
                  (ti16, ptr) <- appU endianW32 ptr
                  (ti17, ptr) <- appU (enum w8) ptr
                  (ti18, ptr) <- appU endianW32 ptr
                  (ti19, ptr) <- appU (list endianW32 w8) ptr
                  (ti20, ptr) <- appU endianW32 ptr
                  return $!
                         (TableInfo ti1 ti2 ti3 ti4 ti5 ti6 ti7 ti8 ti9
                                    ti10 ti11 ti12 ti13 ti14 ti15 ti16
                                    ti17 ti18 ti19 ti20, ptr)

        Joel


On Dec 27, 2005, at 8:44 AM, Tomasz Zielonka wrote:

> What do you mean by too slow? What is the desired input/output
> throughput?
>
> Maybe I'll find some time to improve my BinaryParser/BinaryUnparser
> code.

--
http://wagerlabs.com/





_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: binary IO

Bulat Ziganshin
In reply to this post by J. Pablo Fernández
Hello Pupeno,

Tuesday, December 27, 2005, 7:10:24 AM, you wrote:

P> It seems I have found a hole in Haskell... :(

lazy language can't contain holes, they are just yet unevaluated thunks :)

P> I basically need a set of functions to read binary data out of a Handle (a
P> higher lever of hGetBuf and hPutBuf). What I am doing is implementing a DNS
P> server, so, my typical need is:
P> First two bytes are an integer.
P> The next bit is a boolean.
P> So are the following three bits.
P> Then I have 4 bits which is also an integer.

it's my day :)

main = do h <- openBinaryFile "test" WriteMode
          bh <- openBitAligned h
          putWord16 bh   (101::Int)
          put_      bh   True
          putBits   bh 3 (2::Int)
          flushBits bh
          hClose h
         
          h <- openBinaryFile "test" ReadMode
          bh <- openBitAligned h
          a <- getWord16 bh  :: IO Int
          b <- get bh        :: IO Boolean
          c <- getBits bh 3  :: IO Int
          print (a,b,c)
          hClose h

http://freearc.narod.ru/Binary.tar.gz

but the better way is defining instance of Binary class:

data DNS = DNS Int Bool Int   deriving Show

instance Binary DNS where
  put_ bh (DNS a b c) = do putWord16 bh a
                           put_ bh b
                           putBits bh 3 c
  get bh = do a <- getWord16 bh
              b <- get bh
              c <- getBits bh 3
              return (DNS a b c)

main = do h <- openBinaryFile "test" WriteMode
          bh <- openBitAligned h
          put_ bh (DNS 37 True 3)
          flushBits bh
          hClose h
         
          h <- openBinaryFile "test" ReadMode
          bh <- openBitAligned h
          dns <- get bh :: IO DNS
          print dns
          hClose h

you must use `openBitAligned` to open bit-aligned stream over Handle,
and use `flushBits` at the end of writing. there is much more
features, ask me about what you need

disclaimer: the library was written in last 3 weeks, and you will be
(if you want) its first user :)

--
Best regards,
 Bulat                            mailto:[hidden email]



_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: binary IO

Branimir Maksimovic-2
In reply to this post by Joel Reymont



>From: Joel Reymont <[hidden email]>
>To: Tomasz Zielonka <[hidden email]>
>CC: Jeremy Shaw <[hidden email]>,[hidden email]
>Subject: Re: [Haskell-cafe] binary IO
>Date: Tue, 27 Dec 2005 09:18:54 +0000
>
>Tomasz,
>
>Try http://wagerlabs.com/timeleak.tgz. See the "Killer pickler  
>combinators" thread as well.
>
>My desired goal is to have 4k bots (threads?) running at the same  time.
>At, say, 1k/s per bot I figure something like 4Mb/s round-trip.  Each bot
>cannot spend more than a couple of seconds on pickling/ unpickling. I'm not
>even close to hitting that goal even reading from  a file.
>
>I'm getting delays of 4s+ with just 100-200 bots reading from a file  and
>even less than that in a networked environment. The more bots I  run the
>higher the delays, to the point of frequent delays of 10s+.  The kicker is
>that some packets come in at 4k compressed with Zlib  but become something
>like 50k uncompressed and then expand to a list  of 500+ records, etc.
>

I have C++ concurrent server that performs 2600 reqs/sec on about 500
connections and dual Xeon 2.8Ghz, but no pickling /unpickling, just short
text.
Has sepparate IO threads that divide descriptor sets (num descs / IO thread)
and worker threads as number of CPU's * 2, no locking of shared queue.
So with 4k connections I guess that would be maximum 2k requests on *dual*
box
per second, without pickling / unpickling, just short textual protocol and
simple services.
I think that you will get hard time even with C to achieve your goal.

Greetings, Bane.

_________________________________________________________________
Don't just search. Find. Check out the new MSN Search!
http://search.msn.com/

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: binary IO

Joel Reymont
We'll see, Erlang is built for this type of stuff. I might have  
results from the "timeleak" test today and will probably have first  
networking results tomorrow.

But I wish I could achieve even a fraction of that with Haskell.

On Dec 27, 2005, at 9:51 AM, Branimir Maksimovic wrote:

> I have C++ concurrent server that performs 2600 reqs/sec on about 500
> connections and dual Xeon 2.8Ghz, but no pickling /unpickling, just  
> short text.
> Has sepparate IO threads that divide descriptor sets (num descs /  
> IO thread)
> and worker threads as number of CPU's * 2, no locking of shared queue.
> So with 4k connections I guess that would be maximum 2k requests on  
> *dual* box
> per second, without pickling / unpickling, just short textual  
> protocol and
> simple services.
> I think that you will get hard time even with C to achieve your goal.

--
http://wagerlabs.com/





_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: binary IO

Einar Karttunen
In reply to this post by Tomasz Zielonka
On 27.12 07:00, Tomasz Zielonka wrote:
> Some time ago I was playing with DNS too. I have a library that can
> construct and interpret DNS packets, but it's a bit incomplete right
> now. It reads packets as Strings, but it should be quite straightforward
> to make it read and interpret FastPackedStrings.
>
>     http://www.uncurry.com/repos/TzDNS


Nice, here is my shot at DNS -
http://cs.helsinki.fi/u/ekarttun/haskell/hdnsd-20051227.tar.bz2
feel free to take bits if you are interested. The serialization/deserialization
uses Ptrs.

- Einar Karttunen
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re[2]: binary IO

Bulat Ziganshin
In reply to this post by Joel Reymont
Hello Joel,

Tuesday, December 27, 2005, 12:18:54 PM, you wrote:

JR> My desired goal is to have 4k bots (threads?) running at the same
JR> time. At, say, 1k/s per bot I figure something like 4Mb/s round-trip.

no problem. my library handle about 10-15mb/s, and i think that speed can
be doubled by using unboxed ints

JR> I'm getting delays of 4s+ with just 100-200 bots reading from a file

divide and conquer! calc speed of networking, unzipping, unpickling
separately. compare these speeds with overall program througput to
calc multithreading "expenses"

JR> and even less than that in a networked environment. The more bots I  
JR> run the higher the delays, to the point of frequent delays of 10s+.

these delays says nothing about speed. you are mixing two things -
your end goal is to make delays short, but your instrument - speeds of
different processes and to decide what you need to optimize you must
calc these speeds separately. without it, your "optimization" is just
random game

JR> Each bot is given 5, 15 or 35 seconds to respond by the poker server

so you don't need to create complex timer thread machinery, just use 3
threads which proceed requests in FIFO order

JR> I spent 3 hard months (barely any weekends, no holidays) polishing my
JR> Haskell code.

... wasting your time to speed up code that is slow by design. don't
forget that this article was published as FUNCTIONAL pearl, not
damn-fast pearl

JR> I started the Erlang rewrite at about 3pm this saturday  
JR> and about 1.5 workdays later I have 80% of what I need. I expect to  
JR> finish the rest today and will have a good yardstick to measure  
JR> Haskell against.

i think that you will get just the same problems as with Haskell and
forced to switch back because it's easier to low-level optimize in
Haskell than in Erlang

JR> My only requirement is that there be a _single_ spec for pickling and
JR> unpickling, i.e. no separate methods. The following is not acceptable  
JR> to me ;-).

say exactly WHY you need this single spec. i know at least 4
solutions, what is the best depends on your exact needs

for example, one of these solutions looks like this:

instance Binary TableInfo where
  put_ bh (TableInfo a b c d) = put_ bh (a,b,c,d)
  get bh = do (a,b,c,d) <- get bh; return (TableInfo a b c d)

instance Binary GameType where
  put_ bh = putWord16 bh . fromEnum
  get = liftM toEnum . getWord16

....

--
Best regards,
 Bulat                            mailto:[hidden email]



_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: binary IO

J. Pablo Fernández
In reply to this post by Jeremy Shaw
On Tuesday 27 December 2005 02:10, Jeremy Shaw wrote:
> they do not care about big endian vs little endian.
Does it mean that it just reads the data in whatever endianess the computer is
in, right ?

> However, while the Binary class in NewBinary may not be appropriate,
> the uniform interface to binary files/memory could be a good foundation
> for building hGetWord16le, hGetWord16be, etc.
>
> If you submitted a module that added these functions, I would be glad
> to update the archive.
I've been reading NewBinary's code, it is a bit intimidating for me[1]. It
seems easier to define my own binary functions[2]. Could you point me a bit
what is the unifor interface so I might give it another chance (to work with
NewBinary) ?
Thanks.
--
Pupeno <[hidden email]> (http://pupeno.com)

PS: Is anything wrong with [2] ?

[1] I've been with Haskell for a little more than a couple of weeks.

[2] Doing some experimentation I wrote:

-- | Read a Word8 from a Ptr of any type.
ptrToWord8 p = do
  let np :: Ptr Word8
      np = castPtr p
  r <- (peek np)
  return r

-- | Read a Word16 Big Endian from a Ptr of any type.
ptrToWord16BE p = do
  b1 <- ptrToWord8 p
  b2 <- ptrToWord8 (plusPtr p 1)
  let nb1 :: Word16
      nb1 = fromIntegral b1
      nb2 :: Word16
      nb2 = fromIntegral b2
  return ((shift nb1 8) .|. nb2)

-- | Read a Word16 Little Endian from a Ptr of any type.
ptrToWord16LE p = do
  b1 <- ptrToWord8 p
  b2 <- ptrToWord8 (plusPtr p 1)
  let nb1 :: Word16
      nb1 = fromIntegral b1
      nb2 :: Word16
      nb2 = fromIntegral b2
  return ((shift nb2 8) .|. nb1)

 which I used to read Word16s successfully.

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

attachment0 (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re[2]: binary IO

Joel Reymont
In reply to this post by Bulat Ziganshin
Bulat,

On Dec 27, 2005, at 1:58 PM, Bulat Ziganshin wrote:

> no problem. my library handle about 10-15mb/s, and i think that  
> speed can
> be doubled by using unboxed ints

Would you like to present your version of the timeleak code plus  
statistics from a test run?

This will demonstrate the technical superiority of your library. I'm  
sure it can talk the talk but can it walk the walk?

Please make sure that you have bots reading from the file all at the  
_at the same time_, not sequentially.

> these delays says nothing about speed. you are mixing two things -
> your end goal is to make delays short, but your instrument - speeds of
> different processes and to decide what you need to optimize you must
> calc these speeds separately. without it, your "optimization" is just
> random game

I need to reduce the overall delay. Individual delays do me no good.  
Show me a 1-2s delay in the "timeleak" unstuff code then I will be  
convinced.

> JR> Each bot is given 5, 15 or 35 seconds to respond by the poker  
> server
>
> so you don't need to create complex timer thread machinery, just use 3
> threads which proceed requests in FIFO order

Socket read is a blocking operation. An easy way to handle a blocking  
read is to launch a thread that reads from a socket and posts packets  
somewhere once they are retrieved. I cannot handle large numbers of  
bots if I block on a socket read.

> JR> I spent 3 hard months (barely any weekends, no holidays)  
> polishing my
> JR> Haskell code.
>
> ... wasting your time to speed up code that is slow by design. don't
> forget that this article was published as FUNCTIONAL pearl, not
> damn-fast pearl

Not quite. I ended up with the pickler code but I did not start with it.

> i think that you will get just the same problems as with Haskell and
> forced to switch back because it's easier to low-level optimize in
> Haskell than in Erlang

Absolutely not. I wrote a poker server in Erlang (see the openpoker/
erlang sections of http://wagerlabs.com) so I know what I'm talking  
about. There's usually no need to optimize anything low-level with  
Erlang, it's fast enough as it is for network operations. It's also  
optimized for pickling as well. See http://erlang.se/doc/doc-5.4.12/ 
doc/programming_examples/bit_syntax.html#4, for example.

> JR> My only requirement is that there be a _single_ spec for  
> pickling and
> JR> unpickling, i.e. no separate methods. The following is not  
> acceptable
> JR> to me ;-).
>
> say exactly WHY you need this single spec. i know at least 4
> solutions, what is the best depends on your exact needs

Because I have 150 records with a lot of fields and writing separate  
code for pickling and unpickling is a nightmare?

> for example, one of these solutions looks like this:
>
> instance Binary TableInfo where
>   put_ bh (TableInfo a b c d) = put_ bh (a,b,c,d)
>   get bh = do (a,b,c,d) <- get bh; return (TableInfo a b c d)
>
> instance Binary GameType where
>   put_ bh = putWord16 bh . fromEnum
>   get = liftM toEnum . getWord16

This gets tedious very quickly.

        Thanks, Joel

--
http://wagerlabs.com/





_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re[2]: binary IO

Bulat Ziganshin
In reply to this post by Bulat Ziganshin
Hello Pupeno,

Tuesday, December 27, 2005, 6:03:50 PM, you wrote:

>> you must use `openBitAligned` to open bit-aligned stream over Handle,
>> and use `flushBits` at the end of writing. there is much more
>> features, ask me about what you need
P> Well, it seems I am seeing a bit of light here :)
P> I have a couple of questions:
P> - Is it possible to serialize from and to a Ptr ? Because TCP gives me a
P> Handle, but UDP just gives me a Ptr with the packet and I'll turn the TCP
P> into a Ptr as well.

reading:

h <- openMemory ptr size
bh <- openBitAligned h
a <- get bh
b <- get bh
....
-- you don't need to close/flush/free resources at the end

writing:

h <- createMemoryStream 64  -- replace 64 with initial buffer size you need
                            -- the buffer will be extended if more data
                            -- will be written
bh <- openBitAligned h
put_ bh a
put_ bh b
...
flushBits bh

but currently there is no way to get data from this buffer :)  just
see structure of SimpleMemoryStream (created by createMemoryStream)
and extract these data yourself - something like:

copyMemToHandle mem handle = do
  let (Mem buf' pos' end') = mem
  buf <- readPtr buf'
  pos <- readPtr pos'
  hPutBuf handle buf (pos-:buf)

P> - Is it possible to serialize from and to a Ptr ? Because TCP gives me a
P> Handle, but UDP just gives me a Ptr with the packet and I'll turn the TCP
P> into a Ptr as well.

btw, you don't need to "convert" anything, my library works with
Handle and memory buffers in the same way. so just parse data directly
from the original source

P> - Is it possible to specify the endiannes ? I need to read everything as
P> big-endian.

my library works in network format which is what you used in
ptrToWord16BE. if you need another format - you can either define and
use new low-level functions:

  getWord16LE h = do w1 <- vGetByte h
                     w2 <- vGetByte h
                     return $! ((w2 `shiftL` 8) .|. w1)

or define these functions and use them in defining instances of Binary
class:

instance Binary Word16 where
    put_ = putWord16LE
    get  = getWord16LE

instance Binary Word32 where
    put_ = putWord32LE
    get  = getWord32LE
....

and then just read Word16/Word32/... types. the second solution needs
to edit Binary.hs


--
Best regards,
 Bulat                            mailto:[hidden email]



_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
123