Memory-mapped arrays?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory-mapped arrays?

Joel Reymont
Is there such a thing as memory-mapped arrays in GHC?

I'm looking for something that would let me memory-map a file of  
floats and access it as an array.

        Thanks, Joel

--
http://wagerlabs.com





_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Memory-mapped arrays?

Don Stewart-2
joelr1:
> Is there such a thing as memory-mapped arrays in GHC?
>
> I'm looking for something that would let me memory-map a file of  
> floats and access it as an array.
>

There's a commented out mmapFile for ByteString in Data.ByteString's
source. Use that, and then extract the ForeignPtr from the resulting
ByteString, and castPtr it to a Ptr CFloat, then you're in business.

-- Don
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Memory-mapped arrays? (IArray interfaces, slices, and so on)

Jules Bean
In reply to this post by Joel Reymont
Joel Reymont wrote:
> Is there such a thing as memory-mapped arrays in GHC?

In principle, there could be an IArray instance to memory-mapped files.

(There could also be a mutable version, but just the IArray version
would be useful).

I noticed just the other day that there are some 'obvious' IArray
constructors missing. It ought, for example, be possible to build a new
IArray from an old from a subset of the elements; a dimensional slice
going from an (Int,Int,Int) indexed array to (Int,Int), or a stride
taking 'one element in three' along each axis, etc.

Annoyingly, it doesn't seem to be straightforward to make your own
instances of IArray, since the important methods aren't exported.

I think there is real scope for some expansion here.


Jules
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Memory-mapped arrays? (IArray interfaces, slices, and so on)

Stefan O'Rear
On Wed, Nov 07, 2007 at 10:10:16PM +0000, Jules Bean wrote:

> Joel Reymont wrote:
>> Is there such a thing as memory-mapped arrays in GHC?
>
> In principle, there could be an IArray instance to memory-mapped files.
>
> (There could also be a mutable version, but just the IArray version would
> be useful).
>
> I noticed just the other day that there are some 'obvious' IArray
> constructors missing. It ought, for example, be possible to build a new
> IArray from an old from a subset of the elements; a dimensional slice going
> from an (Int,Int,Int) indexed array to (Int,Int), or a stride taking 'one
> element in three' along each axis, etc.
>
> Annoyingly, it doesn't seem to be straightforward to make your own
> instances of IArray, since the important methods aren't exported.
They are, from the undocumented module Data.Array.Base.

Stefan

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Memory-mapped arrays? (IArray interfaces, slices, and so on)

David Roundy-2
In reply to this post by Jules Bean
On Wed, Nov 07, 2007 at 10:10:16PM +0000, Jules Bean wrote:
> Joel Reymont wrote:
> >Is there such a thing as memory-mapped arrays in GHC?
>
> In principle, there could be an IArray instance to memory-mapped files.
>
> (There could also be a mutable version, but just the IArray version
> would be useful).

The IArray instance would be unsafe, however, because the contents of the
file could change after you opened it, breaking referential transparency.
I don't know what all is possible with file open modes, but I don't think
you can guarantee that once you've opened a file it won't change (unless
you unlink it, and know that noone else has an opened file handle to it).
It may be that by opening it in write mode you could ensure that noone else
modifies it (although I don't think this would work e.g. on nfs), but then
you eliminate the very useful possibility of mmapping read-only files as
IArrays (e.g. to access /usr/share/dict/words).

So it seems reasonable that the mutable version would necesarily be
primary, with the IArray version accessible only by an unsafe operation.
--
David Roundy
Department of Physics
Oregon State University
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Memory-mapped arrays? (IArray interfaces, slices, and so on)

Aaron Denney
On 2007-11-08, David Roundy <[hidden email]> wrote:

> On Wed, Nov 07, 2007 at 10:10:16PM +0000, Jules Bean wrote:
>> Joel Reymont wrote:
>> >Is there such a thing as memory-mapped arrays in GHC?
>>
>> In principle, there could be an IArray instance to memory-mapped files.
>>
>> (There could also be a mutable version, but just the IArray version
>> would be useful).
>
> The IArray instance would be unsafe, however, because the contents of the
> file could change after you opened it, breaking referential transparency.

Or even crashing, if the size becomes smaller than the mapped area.

> I don't know what all is possible with file open modes, but I don't think
> you can guarantee that once you've opened a file it won't change (unless
> you unlink it, and know that noone else has an opened file handle to it).

File open modes won't do it, and I don't think any thing else will do
it using just POSIX behavior, either.  Linux's mmap() used to support a
DENY_WRITE flag, but it enabled DoS attacks, so it's gone.

> It may be that by opening it in write mode you could ensure that noone else
> modifies it (although I don't think this would work e.g. on nfs),

It doesn't even work locally.

--
Aaron Denney
-><-

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Re: Memory-mapped arrays? (IArray interfaces, slices, and so on)

Jules Bean
Aaron Denney wrote:
>> It may be that by opening it in write mode you could ensure that noone else
>> modifies it (although I don't think this would work e.g. on nfs),
>
> It doesn't even work locally.
>

Right. But mmap is only sensible to use (even in C) when you know about
all the other processes who can modify the file; bearing this in mind,
it's still useful! Obviously the haskell RTS can't insulate us from
certain things, if the OS-level primitives let it down, but we can
document what you need to do to be safe.

Jules
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe