Proposal: Add 'fillBytes' to Foreign.Marshal.Utils

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Proposal: Add 'fillBytes' to Foreign.Marshal.Utils

Alex Petrov
Hi everyone,

Currently the memory allocated with (Foreign.Marshal.Alloc) malloc may potentially be dirty. In C this problem is usually solved by using memset.

This would be extremely useful for FFI / C interop, when a data structure is allocated within Haskell code. With memset, you can do something like

customMem <- malloc
_         <- memset (castPtr customMem) 0 #{size custom_t}

This will fill a block of allocated memory with zeroes.

For example, I've been working on FFI bindings for collectd, and when I was allocating memory, previously used within the process, it was dirty, so my CString contained a value of

"custom name7e0700490, test_plugin_LTX_module_register): symbol nd"

Instead of just

"custom name"

After using memset and filling everything with zeroes, problem never appeared again.

This can be implemented in user applications, too, although it would be really nice to have it by default.

This is a common practice in C world, and this function is not only useful for cases when the memory was just allocated from Haskell process, but also when one receives a dirty memory buffer from C code. 


The patch for proposal was implemented and is available on Phabricator: https://phabricator.haskell.org/D465



Please share your thoughts.



_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Add 'fillBytes' to Foreign.Marshal.Utils

David Feuer
 I don't know which systems do and don't guarantee what is already cleared; are you careful not to repeat someone else's work?

On Wed, Nov 12, 2014 at 9:21 AM, Alex Petrov <[hidden email]> wrote:
Hi everyone,

Currently the memory allocated with (Foreign.Marshal.Alloc) malloc may potentially be dirty. In C this problem is usually solved by using memset.

This would be extremely useful for FFI / C interop, when a data structure is allocated within Haskell code. With memset, you can do something like

customMem <- malloc
_         <- memset (castPtr customMem) 0 #{size custom_t}

This will fill a block of allocated memory with zeroes.

For example, I've been working on FFI bindings for collectd, and when I was allocating memory, previously used within the process, it was dirty, so my CString contained a value of

"custom name7e0700490, test_plugin_LTX_module_register): symbol nd"

Instead of just

"custom name"

After using memset and filling everything with zeroes, problem never appeared again.

This can be implemented in user applications, too, although it would be really nice to have it by default.

This is a common practice in C world, and this function is not only useful for cases when the memory was just allocated from Haskell process, but also when one receives a dirty memory buffer from C code. 


The patch for proposal was implemented and is available on Phabricator: https://phabricator.haskell.org/D465



Please share your thoughts.



_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries



_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Add 'fillBytes' to Foreign.Marshal.Utils

Edward Kmett-2
David, traditionally malloc doesn't clear and calloc does. (We don't currently provide a "calloc" -like API), but we don't have a member equivalent to fill the gap.

Of course, we just pretend these things are the c analogues, the work differently.


On Nov 12, 2014, at 10:34 AM, David Feuer <[hidden email]> wrote:

 I don't know which systems do and don't guarantee what is already cleared; are you careful not to repeat someone else's work?

On Wed, Nov 12, 2014 at 9:21 AM, Alex Petrov <[hidden email]> wrote:
Hi everyone,

Currently the memory allocated with (Foreign.Marshal.Alloc) malloc may potentially be dirty. In C this problem is usually solved by using memset.

This would be extremely useful for FFI / C interop, when a data structure is allocated within Haskell code. With memset, you can do something like

customMem <- malloc
_         <- memset (castPtr customMem) 0 #{size custom_t}

This will fill a block of allocated memory with zeroes.

For example, I've been working on FFI bindings for collectd, and when I was allocating memory, previously used within the process, it was dirty, so my CString contained a value of

"custom name7e0700490, test_plugin_LTX_module_register): symbol nd"

Instead of just

"custom name"

After using memset and filling everything with zeroes, problem never appeared again.

This can be implemented in user applications, too, although it would be really nice to have it by default.

This is a common practice in C world, and this function is not only useful for cases when the memory was just allocated from Haskell process, but also when one receives a dirty memory buffer from C code. 


The patch for proposal was implemented and is available on Phabricator: https://phabricator.haskell.org/D465



Please share your thoughts.



_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries


_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Add 'fillBytes' to Foreign.Marshal.Utils

Austin Seipp-5
FWIW, since nobody has followed up on this, I'm +1 on the proposal and
patch, and think it fits in nicely with the other utilities in
Foreign.Marshal.Utils (like copyBytes and moveBytes).

On Wed, Nov 12, 2014 at 4:43 PM, Edward Kmett <[hidden email]> wrote:

> David, traditionally malloc doesn't clear and calloc does. (We don't
> currently provide a "calloc" -like API), but we don't have a member
> equivalent to fill the gap.
>
> Of course, we just pretend these things are the c analogues, the work
> differently.
>
>
> On Nov 12, 2014, at 10:34 AM, David Feuer <[hidden email]> wrote:
>
>  I don't know which systems do and don't guarantee what is already cleared;
> are you careful not to repeat someone else's work?
>
> On Wed, Nov 12, 2014 at 9:21 AM, Alex Petrov <[hidden email]>
> wrote:
>>
>> Hi everyone,
>>
>> Currently the memory allocated with (Foreign.Marshal.Alloc) malloc may
>> potentially be dirty. In C this problem is usually solved by using memset.
>>
>> This would be extremely useful for FFI / C interop, when a data structure
>> is allocated within Haskell code. With memset, you can do something like
>>
>> customMem <- malloc
>> _         <- memset (castPtr customMem) 0 #{size custom_t}
>>
>> This will fill a block of allocated memory with zeroes.
>>
>> For example, I've been working on FFI bindings for collectd, and when I
>> was allocating memory, previously used within the process, it was dirty, so
>> my CString contained a value of
>>
>> "custom name7e0700490, test_plugin_LTX_module_register): symbol nd"
>>
>> Instead of just
>>
>> "custom name"
>>
>> After using memset and filling everything with zeroes, problem never
>> appeared again.
>>
>> This can be implemented in user applications, too, although it would be
>> really nice to have it by default.
>>
>> This is a common practice in C world, and this function is not only useful
>> for cases when the memory was just allocated from Haskell process, but also
>> when one receives a dirty memory buffer from C code.
>>
>>
>> The patch for proposal was implemented and is available on Phabricator:
>> https://phabricator.haskell.org/D465
>>
>>
>>
>> Please share your thoughts.
>>
>>
>>
>> Alex
>>
>> https://twitter.com/ifesdjeen
>>
>> http://clojurewerkz.org/
>>
>>
>> _______________________________________________
>> Libraries mailing list
>> [hidden email]
>> http://www.haskell.org/mailman/listinfo/libraries
>>
>
> _______________________________________________
> Libraries mailing list
> [hidden email]
> http://www.haskell.org/mailman/listinfo/libraries
>
>
> _______________________________________________
> Libraries mailing list
> [hidden email]
> http://www.haskell.org/mailman/listinfo/libraries
>



--
Regards,

Austin Seipp, Haskell Consultant
Well-Typed LLP, http://www.well-typed.com/
_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Add 'fillBytes' to Foreign.Marshal.Utils

Ben Millwood
In reply to this post by Alex Petrov
From what I can see on phabricator, this proposal has been implemented
and closed. If that's true it would have been useful to mention that on
this thread, I think.

Anyway, I wonder if fillBytes is ever likely to be used with a non-zero
byte. Wouldn't zeroBytes have been just as good?

On Wed, Nov 12, 2014 at 03:21:17PM +0100, Alex Petrov wrote:

>Hi everyone,
>
>Currently the memory allocated with (Foreign.Marshal.Alloc) malloc may potentially be dirty. In C this problem is usually solved by using memset.
>
>This would be extremely useful for FFI / C interop, when a data structure is allocated within Haskell code. With memset, you can do something like
>
>customMem <- malloc
>_         <- memset (castPtr customMem) 0 #{size custom_t}
>This will fill a block of allocated memory with zeroes.
>
>For example, I've been working on FFI bindings for collectd, and when I was allocating memory, previously used within the process, it was dirty, so my CString contained a value of
>
>"custom name7e0700490, test_plugin_LTX_module_register): symbol nd"
>Instead of just
>
>"custom name"
>After using memset and filling everything with zeroes, problem never appeared again.
>
>This can be implemented in user applications, too, although it would be really nice to have it by default.
>This is a common practice in C world, and this function is not only useful for cases when the memory was just allocated from Haskell process, but also when one receives a dirty memory buffer from C code. 
>
>The patch for proposal was implemented and is available on Phabricator: https://phabricator.haskell.org/D465
>
>
>Please share your thoughts.
>
>
>Alex
>https://twitter.com/ifesdjeen
>http://clojurewerkz.org/

>_______________________________________________
>Libraries mailing list
>[hidden email]
>http://www.haskell.org/mailman/listinfo/libraries

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Proposal: Add 'fillBytes' to Foreign.Marshal.Utils

Edward Kmett-2
People find all sorts of weird usecases for memset once they have it in my experience, so nothing is hurt by gaining generality here.

-Edward

On Sun, Nov 23, 2014 at 10:17 AM, Ben Millwood <[hidden email]> wrote:
From what I can see on phabricator, this proposal has been implemented and closed. If that's true it would have been useful to mention that on this thread, I think.

Anyway, I wonder if fillBytes is ever likely to be used with a non-zero byte. Wouldn't zeroBytes have been just as good?


On Wed, Nov 12, 2014 at 03:21:17PM +0100, Alex Petrov wrote:
Hi everyone,

Currently the memory allocated with (Foreign.Marshal.Alloc) malloc may potentially be dirty. In C this problem is usually solved by using memset.

This would be extremely useful for FFI / C interop, when a data structure is allocated within Haskell code. With memset, you can do something like

customMem <- malloc
_         <- memset (castPtr customMem) 0 #{size custom_t}
This will fill a block of allocated memory with zeroes.

For example, I've been working on FFI bindings for collectd, and when I was allocating memory, previously used within the process, it was dirty, so my CString contained a value of

"custom name7e0700490, test_plugin_LTX_module_register): symbol nd"
Instead of just

"custom name"
After using memset and filling everything with zeroes, problem never appeared again.

This can be implemented in user applications, too, although it would be really nice to have it by default.
This is a common practice in C world, and this function is not only useful for cases when the memory was just allocated from Haskell process, but also when one receives a dirty memory buffer from C code. 

The patch for proposal was implemented and is available on Phabricator: https://phabricator.haskell.org/D465


Please share your thoughts.


Alex
https://twitter.com/ifesdjeen
http://clojurewerkz.org/

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries


_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries