Streams: the extensible I/O library

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Streams: the extensible I/O library

Bulat Ziganshin
Hello

I have developed a new I/O library that IMHO is so sharp that it can
eventually replace the current I/O facilities based on using Handles.
The main advantage of the new library is its strong modular design
using typeclasses. The library consists of small independent modules,
each implementing one type of stream (file, memory buffer, pipe) or
one part of common stream functionality (buffering, Char encoding,
locking). 3rd-party libs can easily add new stream types and new
common functionality. Other benefits of the new library include
support for streams functioning in any monad, Hugs and GHC
compatibility, high speed and an easy migration path from the existing
I/O library.

You can find further information about the library at the page
http://haskell.org/haskellwiki/Library/Streams and download it as
http://freearc.narod.ru/Streams.tar.gz

I thank Jean Philippe Bernardy, Jared Updike and Henk Jan Van Tuyl,
who are corrected library documentation

ps: the library also includes two more layers - binary I/O and
serialization - on top of Streams. now i'm hardly working on
documenting these modules


--
Best regards,
 Bulat                          mailto:[hidden email]



_______________________________________________
Haskell mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell
Reply | Threaded
Open this post in threaded view
|

Re: Streams: the extensible I/O library

Aaron Denney
On 2006-02-06, Bulat Ziganshin <[hidden email]> wrote:

> Hello
>
> I have developed a new I/O library that IMHO is so sharp that it can
> eventually replace the current I/O facilities based on using Handles.
> The main advantage of the new library is its strong modular design
> using typeclasses. The library consists of small independent modules,
> each implementing one type of stream (file, memory buffer, pipe) or
> one part of common stream functionality (buffering, Char encoding,
> locking). 3rd-party libs can easily add new stream types and new
> common functionality. Other benefits of the new library include
> support for streams functioning in any monad, Hugs and GHC
> compatibility, high speed and an easy migration path from the existing
> I/O library.
>
> You can find further information about the library at the page
> http://haskell.org/haskellwiki/Library/Streams and download it as
> http://freearc.narod.ru/Streams.tar.gz
>
> I thank Jean Philippe Bernardy, Jared Updike and Henk Jan Van Tuyl,
> who are corrected library documentation
>
> ps: the library also includes two more layers - binary I/O and
> serialization - on top of Streams. now i'm hardly working on
> documenting these modules

Disclaimer: I haven't looked at the code yet.

Having binary I/O on top seems backwards.  Clearly text should be
implemented in terms of binary, rather than the reverse.

--
Aaron Denney
-><-

_______________________________________________
Haskell mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell
Reply | Threaded
Open this post in threaded view
|

Re: Re: Streams: the extensible I/O library

Bulat Ziganshin
Hello Aaron,

Monday, February 06, 2006, 9:46:56 PM, you wrote:

>> ps: the library also includes two more layers - binary I/O and
>> serialization - on top of Streams. now i'm hardly working on
>> documenting these modules

AD> Disclaimer: I haven't looked at the code yet.

AD> Having binary I/O on top seems backwards.  Clearly text should be
AD> implemented in terms of binary, rather than the reverse.

the hierarchy is (i wrote the class names in parentheses):

buffer i/o        (BlockStream)
byte i/o          (ByteStream)
bit and word i/o  (BinaryStream)
value i/o         (Binary)

text i/o is sitting on top of byte i/o and included in the same
ByteStream class (in order to reduce implementation complexity)

so, i can say that there is just two branches, both based on the
vGetByte/vPutByte

--
Best regards,
 Bulat                            mailto:[hidden email]



_______________________________________________
Haskell mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell
Reply | Threaded
Open this post in threaded view
|

Re: Streams: the extensible I/O library

Peter Simons
In reply to this post by Bulat Ziganshin
Bulat Ziganshin writes:

 > You can find further information about the library at the
 > page http://haskell.org/haskellwiki/Library/Streams and
 > download it as http://freearc.narod.ru/Streams.tar.gz

Is there any chance of running this code on a non-Windows
system? I tried to compile the example programs, but failed
for lack of a System.Win32 module. I might be able to port
the code, but I figured it would be wise to ask first.

Peter

_______________________________________________
Haskell mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell
Reply | Threaded
Open this post in threaded view
|

Re[2]: Streams: the extensible I/O library

Bulat Ziganshin
In reply to this post by Bulat Ziganshin
Hello Andrew,

Wednesday, February 08, 2006, 8:24:59 PM, you wrote:

>> AP> Bulat, it wouldn't hurt to include a motivation section at the top.  As
>> AP> I understand, it's ultimately all about speed, right?  Otherwise, we
>> AP> would all be happy with lists (and unsafeInterleave*).  So maybe a
>> AP> comparison between Stream and [] should be given.
>>
>> you guessed wrong :)  this library is ultimately about replacing
>> System.IO library (i.e. Handles)

AP> Let me rephrase my question:  Why not just reimplement the Handles API
AP> (with some extensions like binary IO)?  Is there really a need to use a
AP> handle-like API for more than real IO?  If so, what is the need,
AP> expressivity or performance (or both)?  Maybe a use case showing what
AP> you can do with your library, and how you would have to do it otherwise?

now i understood you. actually, my presentation was meant only for a
few people who are already know about limitations of current library,
who are already requested additional features but don't got it. here i
need to give some history:

when a System.IO interface was developed, it implements much less
features than now, and its implementation was enough simple and
straightforward. as time goes, the more and more features was added to
this library: complex buffering scheme, several async i/o
implementations, locking, networking. And at current moment, GHC's
System.IO implementation consists of about 3000 lines with a rather
monolithic structure. you can't easily add new feature or make using
of some "heavyweight" feature optional because it needs to make
changes through the entire library. As the result, GHC users can't get
improvements in this library, required for his work. Some of them even
develop his own libraries what implements just that they need. for
example, Einar Karttunen developed networking library with advanced
async i/o and support for i/o of fast packed strings. But such
solutions is not universal - his library can't be used for file i/o,
fo example, although the code is essentially the same.

what i done? the main merit of Streams library is not implementation
of any particular feature, but birth of framework for the I/O
sub-libraries. and my library essentially is just a collection of such
sublibs. first, for example, implements file i/o, second implements
buffering, third - utf-8 encoding, and so on. the most important
property of all these sublibs is that no one of them is greater than
300 lines. that means that it is far easier to understand, modify or
even replace any of them. and that will have no impact to other part
of library because all these sublibs binded together not via data
fields, but with well defined interfaces

now, implementing any new I/O feature or new I/O source means only
implementing Stream class-comforming interface - all other features,
including locking, buffering, encoding, compression, serialization,
binary and text i/o, async i/o, will become available automatically.
the same for transformers - once implemented gzip compression or
UTF-16 encoding support will become automatically available for all
the I/O sources, present and future. is not that great? :)  moreover,
user apllication or third party lib can easily implement new stream
types or transformers without bothering the original library.

so, this lib in some aspect is meta-meta-instrument, whose capital
will be automatically increasing as new sublibs will appear. just at
this moment its advantages over System.IO is in the following areas:

faster i/o
support for optional utf-8 encoding
binary i/o and serialization
user-controlled locking

if you will look inside the archive, you will find directory Examples,
which demonstrates usage of all these features. as i said in docs, i
also plan to implement other user requests to the System.IO library

another consequence of emerging this library is that all these
features will become available on Hugs and other Haskell compilers,
that never had enough man resources to develop such great library as
GHC's System.IO.

and about using streams in monads other than IO. i really don't know,
whether it will be used or not. at least, for seriazliation it looks
promising. for example, there is functions "encode" and "decode" that
is like show/read pair, but implements binary encoding according to
the instances of Binary class. and of course, it is implemented
through the StringBuffer instance of Stream class, working in the ST
monad.

comparing to the Handles, library provides essentially the same
interface. again, you can find information about swithcing from
Handles to Streams in doc. i plan to provide in future "legacy layer"
which will emulate 100% of System.IO interface, but use the streams
internally. It will be essential for old apps, especially if Streams
will become official and sole GHC i/o library

about internal organization - Streams is somewhat like that if Simon
himself splitted up Handles library to the small independent parts and
then replaced part of them with simpler/faster implementations.
nothing more, except for common Stream class interface, developed by
John Goerzen. my work was mainly to bring the best ideas together :)

--
Best regards,
 Bulat                            mailto:[hidden email]



_______________________________________________
Haskell mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell