Memory Barriers with STM and MVar

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory Barriers with STM and MVar

Andrew Martin
I've been poking around in old threads and in the GHC source, and I cannot find this documented anywhere. What are the guarantees that GHC provides around load/store reordering when it comes to using MVar or TVar+retry+atomically. For example:

import Data.Primitive
import Control.Concurrent.MVar
...
foo :: IO ()
foo = do
  ...
  writeByteArray myArr 13 (42 :: Word)
  putMVar myMVar ()

If there is another thread that calls takeMVar followed by `readByteArray myArr 13`, is it guaranteed by GHC to see the 42 that's been written to the array. Same question applies for situations using STM facilities to accomplish blocking behavior that MVar gives us. Any documentation or pointers to places in GHC source where these barriers are guaranteed would be appreciated.

--
-Andrew Thaddeus Martin

_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Memory Barriers with STM and MVar

Andrew Martin
Perhaps it may be better to open an issue for this information to be added to the GHC user manual. Neither the word "barrier" nor the word "fence" show up in the context of reordering in the user manual. Having some kind of guarantee is extremely important for writing certain types of applications. I suspect that both atomically and all the MVar operations imply a full memory barrier, but I haven't been able this documented anywhere.

On Fri, May 24, 2019 at 1:38 PM Andrew Martin <[hidden email]> wrote:
I've been poking around in old threads and in the GHC source, and I cannot find this documented anywhere. What are the guarantees that GHC provides around load/store reordering when it comes to using MVar or TVar+retry+atomically. For example:

import Data.Primitive
import Control.Concurrent.MVar
...
foo :: IO ()
foo = do
  ...
  writeByteArray myArr 13 (42 :: Word)
  putMVar myMVar ()

If there is another thread that calls takeMVar followed by `readByteArray myArr 13`, is it guaranteed by GHC to see the 42 that's been written to the array. Same question applies for situations using STM facilities to accomplish blocking behavior that MVar gives us. Any documentation or pointers to places in GHC source where these barriers are guaranteed would be appreciated.

--
-Andrew Thaddeus Martin


--
-Andrew Thaddeus Martin

_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Memory Barriers with STM and MVar

Carter Schonwald
what the precise guarantees are actually may depend on the platform.

if you look for the barriers, they're in the RTS c codebase
should get you started, theres also read_barrier and write_barrier in the rts

On Tue, May 28, 2019 at 3:16 PM Andrew Martin <[hidden email]> wrote:
Perhaps it may be better to open an issue for this information to be added to the GHC user manual. Neither the word "barrier" nor the word "fence" show up in the context of reordering in the user manual. Having some kind of guarantee is extremely important for writing certain types of applications. I suspect that both atomically and all the MVar operations imply a full memory barrier, but I haven't been able this documented anywhere.

On Fri, May 24, 2019 at 1:38 PM Andrew Martin <[hidden email]> wrote:
I've been poking around in old threads and in the GHC source, and I cannot find this documented anywhere. What are the guarantees that GHC provides around load/store reordering when it comes to using MVar or TVar+retry+atomically. For example:

import Data.Primitive
import Control.Concurrent.MVar
...
foo :: IO ()
foo = do
  ...
  writeByteArray myArr 13 (42 :: Word)
  putMVar myMVar ()

If there is another thread that calls takeMVar followed by `readByteArray myArr 13`, is it guaranteed by GHC to see the 42 that's been written to the array. Same question applies for situations using STM facilities to accomplish blocking behavior that MVar gives us. Any documentation or pointers to places in GHC source where these barriers are guaranteed would be appreciated.

--
-Andrew Thaddeus Martin


--
-Andrew Thaddeus Martin
_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries

_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Memory Barriers with STM and MVar

Andrew Martin
Thanks. The does have the information I was looking for. While write_barrier gets used in a number of places, store_load_barrier never actually gets used anywhere. This makes me suspect that MVar and STM cannot be used to coordinate access to memory in this way my example tries to. That is unfortunate.

On Fri, May 31, 2019 at 12:52 PM Carter Schonwald <[hidden email]> wrote:
what the precise guarantees are actually may depend on the platform.

if you look for the barriers, they're in the RTS c codebase
should get you started, theres also read_barrier and write_barrier in the rts

On Tue, May 28, 2019 at 3:16 PM Andrew Martin <[hidden email]> wrote:
Perhaps it may be better to open an issue for this information to be added to the GHC user manual. Neither the word "barrier" nor the word "fence" show up in the context of reordering in the user manual. Having some kind of guarantee is extremely important for writing certain types of applications. I suspect that both atomically and all the MVar operations imply a full memory barrier, but I haven't been able this documented anywhere.

On Fri, May 24, 2019 at 1:38 PM Andrew Martin <[hidden email]> wrote:
I've been poking around in old threads and in the GHC source, and I cannot find this documented anywhere. What are the guarantees that GHC provides around load/store reordering when it comes to using MVar or TVar+retry+atomically. For example:

import Data.Primitive
import Control.Concurrent.MVar
...
foo :: IO ()
foo = do
  ...
  writeByteArray myArr 13 (42 :: Word)
  putMVar myMVar ()

If there is another thread that calls takeMVar followed by `readByteArray myArr 13`, is it guaranteed by GHC to see the 42 that's been written to the array. Same question applies for situations using STM facilities to accomplish blocking behavior that MVar gives us. Any documentation or pointers to places in GHC source where these barriers are guaranteed would be appreciated.

--
-Andrew Thaddeus Martin


--
-Andrew Thaddeus Martin
_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries


--
-Andrew Thaddeus Martin

_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Memory Barriers with STM and MVar

Andrew Martin
I feel like what I really need is a primop for sticking a store_load_barrier at an arbitrary place in the code. That would give me the guarantee I'm looking for. Then I could write:

foo :: IO ()
foo = do
  ...
  writeByteArray myArr 13 (42 :: Word)
  storeLoadBarrier
  putMVar myMVar ()

On Fri, May 31, 2019 at 1:44 PM Andrew Martin <[hidden email]> wrote:
Thanks. The does have the information I was looking for. While write_barrier gets used in a number of places, store_load_barrier never actually gets used anywhere. This makes me suspect that MVar and STM cannot be used to coordinate access to memory in this way my example tries to. That is unfortunate.

On Fri, May 31, 2019 at 12:52 PM Carter Schonwald <[hidden email]> wrote:
what the precise guarantees are actually may depend on the platform.

if you look for the barriers, they're in the RTS c codebase
should get you started, theres also read_barrier and write_barrier in the rts

On Tue, May 28, 2019 at 3:16 PM Andrew Martin <[hidden email]> wrote:
Perhaps it may be better to open an issue for this information to be added to the GHC user manual. Neither the word "barrier" nor the word "fence" show up in the context of reordering in the user manual. Having some kind of guarantee is extremely important for writing certain types of applications. I suspect that both atomically and all the MVar operations imply a full memory barrier, but I haven't been able this documented anywhere.

On Fri, May 24, 2019 at 1:38 PM Andrew Martin <[hidden email]> wrote:
I've been poking around in old threads and in the GHC source, and I cannot find this documented anywhere. What are the guarantees that GHC provides around load/store reordering when it comes to using MVar or TVar+retry+atomically. For example:

import Data.Primitive
import Control.Concurrent.MVar
...
foo :: IO ()
foo = do
  ...
  writeByteArray myArr 13 (42 :: Word)
  putMVar myMVar ()

If there is another thread that calls takeMVar followed by `readByteArray myArr 13`, is it guaranteed by GHC to see the 42 that's been written to the array. Same question applies for situations using STM facilities to accomplish blocking behavior that MVar gives us. Any documentation or pointers to places in GHC source where these barriers are guaranteed would be appreciated.

--
-Andrew Thaddeus Martin


--
-Andrew Thaddeus Martin
_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries


--
-Andrew Thaddeus Martin


--
-Andrew Thaddeus Martin

_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Memory Barriers with STM and MVar

Elliot Cameron-2
In reply to this post by Andrew Martin
For my education, can you describe your use case a bit more? Using MVar as you did in your example seems like a strange use case. Normally you'd just put the pointer to the structure in the MVar itself.

_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Memory Barriers with STM and MVar

Andrew Martin
What I'm doing is using STM (but it could be done with MVar instead) to guard access to a rotating queue that is a large mmapped region of memory (in my situation, it also happens to be backed by a file on disk). There can only be one writer at a time. There can be many readers, but the readers cannot read from the head of the queue if the writer is still working on it. They must wait until the writer is finished. So, every thread has the same base address and then they are all working at different offsets into the queue. But when the writer tries to tell the readers "I'm done writing to this area", how can I be sure that the readers will actually see the writes to memory? That's the difficulty I'm having.

On Sat, Jun 1, 2019 at 9:31 AM Elliot Cameron <[hidden email]> wrote:
For my education, can you describe your use case a bit more? Using MVar as you did in your example seems like a strange use case. Normally you'd just put the pointer to the structure in the MVar itself.


--
-Andrew Thaddeus Martin

_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Memory Barriers with STM and MVar

Andrew Martin
After thinking about this more, it appears that a store-store barrier (not a store-load barrier) is actually sufficient for this. I just need to be sure that other threads see the writes to memory before the writes to the MVar (or before whatever STM writes to when atomically completes). However, I don't see any calls to write_barrier in the STM code or in the MVar code, so GHC may not currently provide the behavior I am looking for.

On Sat, Jun 1, 2019 at 10:11 AM Andrew Martin <[hidden email]> wrote:
What I'm doing is using STM (but it could be done with MVar instead) to guard access to a rotating queue that is a large mmapped region of memory (in my situation, it also happens to be backed by a file on disk). There can only be one writer at a time. There can be many readers, but the readers cannot read from the head of the queue if the writer is still working on it. They must wait until the writer is finished. So, every thread has the same base address and then they are all working at different offsets into the queue. But when the writer tries to tell the readers "I'm done writing to this area", how can I be sure that the readers will actually see the writes to memory? That's the difficulty I'm having.

On Sat, Jun 1, 2019 at 9:31 AM Elliot Cameron <[hidden email]> wrote:
For my education, can you describe your use case a bit more? Using MVar as you did in your example seems like a strange use case. Normally you'd just put the pointer to the structure in the MVar itself.


--
-Andrew Thaddeus Martin


--
-Andrew Thaddeus Martin

_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries
Reply | Threaded
Open this post in threaded view
|

Re: Memory Barriers with STM and MVar

Carter Schonwald
you could use the STM based sleep where the try to read an STMvar and then retry to sleep until the write was done

if you only care about AMD64,  https://en.wikipedia.org/wiki/Memory_ordering#In_symmetric_multiprocessing_(SMP)_microprocessor_systems , you're probably not going to have the issues I think you're worrying about

other platforms, ehhhhh

On Mon, Jun 3, 2019 at 11:34 AM Andrew Martin <[hidden email]> wrote:
After thinking about this more, it appears that a store-store barrier (not a store-load barrier) is actually sufficient for this. I just need to be sure that other threads see the writes to memory before the writes to the MVar (or before whatever STM writes to when atomically completes). However, I don't see any calls to write_barrier in the STM code or in the MVar code, so GHC may not currently provide the behavior I am looking for.

On Sat, Jun 1, 2019 at 10:11 AM Andrew Martin <[hidden email]> wrote:
What I'm doing is using STM (but it could be done with MVar instead) to guard access to a rotating queue that is a large mmapped region of memory (in my situation, it also happens to be backed by a file on disk). There can only be one writer at a time. There can be many readers, but the readers cannot read from the head of the queue if the writer is still working on it. They must wait until the writer is finished. So, every thread has the same base address and then they are all working at different offsets into the queue. But when the writer tries to tell the readers "I'm done writing to this area", how can I be sure that the readers will actually see the writes to memory? That's the difficulty I'm having.

On Sat, Jun 1, 2019 at 9:31 AM Elliot Cameron <[hidden email]> wrote:
For my education, can you describe your use case a bit more? Using MVar as you did in your example seems like a strange use case. Normally you'd just put the pointer to the structure in the MVar itself.


--
-Andrew Thaddeus Martin


--
-Andrew Thaddeus Martin

_______________________________________________
Libraries mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/libraries