Proposal: Stop enforcing single-writer-multi-reader file access

[Ian Lynagh]

Hi all,

Haskell currently requires that multiple-reader single-writer locking is
used on files. I understand the motivation for this was to try to
protect people from running into problems due to lazy IO, but in
practice I think it just causes more problems than it solves.

The decision may also have been influenced by the belief that this
behaviour is enforced for all programs on Windows, but I am told that it
is possible to get lock-free behaviour on Windows too.

I propose that we remove all the automatic locking from the libraries,
and let the user do any locking that they wish to do themselves.

Attached are patches for ghc, base, process and unix. (not yet tested on
Windows).


Suggested discussion period: Until 9 Nov 2011

Related GHC ticket: http://hackage.haskell.org/trac/ghc/ticket/4363


Thanks
Ian


_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Evan Laforge]

> I propose that we remove all the automatic locking from the libraries,
> and let the user do any locking that they wish to do themselves.

+1 as long as it works on windows, it's easier to explain to someone
why they shouldn't use lazy IO than to explain why their strict IO
code is getting lock errors.

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Felipe Almeida Lessa]

+1

This is the most confusing bit I've ever seen in Haskell.

On Wed, Oct 26, 2011 at 8:59 PM, Ian Lynagh <[hidden email]> wrote:

>
> Hi all,
>
> Haskell currently requires that multiple-reader single-writer locking is
> used on files. I understand the motivation for this was to try to
> protect people from running into problems due to lazy IO, but in
> practice I think it just causes more problems than it solves.
>
> The decision may also have been influenced by the belief that this
> behaviour is enforced for all programs on Windows, but I am told that it
> is possible to get lock-free behaviour on Windows too.
>
> I propose that we remove all the automatic locking from the libraries,
> and let the user do any locking that they wish to do themselves.
>
> Attached are patches for ghc, base, process and unix. (not yet tested on
> Windows).
>
>
> Suggested discussion period: Until 9 Nov 2011
>
> Related GHC ticket: http://hackage.haskell.org/trac/ghc/ticket/4363
>
>
> Thanks
> Ian
>
>
> _______________________________________________
> Libraries mailing list
> [hidden email]
> http://www.haskell.org/mailman/listinfo/libraries
>
>

--
Felipe.

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Conrad Parker]

On 27 October 2011 07:10, Evan Laforge <[hidden email]> wrote:
>> I propose that we remove all the automatic locking from the libraries,
>> and let the user do any locking that they wish to do themselves.
>
> +1 as long as it works on windows, it's easier to explain to someone
> why they shouldn't use lazy IO than to explain why their strict IO
> code is getting lock errors.

+1

Conrad.

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Herbert Valerio Riedel]

On Wed, 2011-10-26 at 23:59 +0100, Ian Lynagh wrote:
> I propose that we remove all the automatic locking from the libraries,
> and let the user do any locking that they wish to do themselves.

+1


_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Max Bolingbroke-2]

On 26 October 2011 23:59, Ian Lynagh <[hidden email]> wrote:
> Attached are patches for ghc, base, process and unix. (not yet tested on
> Windows).

Assuming that this works on Windows, I heartily support the proposal!
This locking policy is a really strange wart that has confused me a
few times.

Max

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Edward A Kmett]

+1!

On Wed, Oct 26, 2011 at 6:59 PM, Ian Lynagh <[hidden email]> wrote:

Hi all,

Haskell currently requires that multiple-reader single-writer locking is
used on files. I understand the motivation for this was to try to
protect people from running into problems due to lazy IO, but in
practice I think it just causes more problems than it solves.

The decision may also have been influenced by the belief that this
behaviour is enforced for all programs on Windows, but I am told that it
is possible to get lock-free behaviour on Windows too.

I propose that we remove all the automatic locking from the libraries,
and let the user do any locking that they wish to do themselves.

Attached are patches for ghc, base, process and unix. (not yet tested on
Windows).


Suggested discussion period: Until 9 Nov 2011

Related GHC ticket: http://hackage.haskell.org/trac/ghc/ticket/4363


Thanks
Ian


_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Edward Z. Yang]

+1

But I kind of wonder if there don't exists programs that will break
due to this. Probably only those that people have actually put in
some attention to cross-platform compatibility.

Edward

Excerpts from Ian Lynagh's message of Wed Oct 26 18:59:38 -0400 2011:

>
> Hi all,
>
> Haskell currently requires that multiple-reader single-writer locking is
> used on files. I understand the motivation for this was to try to
> protect people from running into problems due to lazy IO, but in
> practice I think it just causes more problems than it solves.
>
> The decision may also have been influenced by the belief that this
> behaviour is enforced for all programs on Windows, but I am told that it
> is possible to get lock-free behaviour on Windows too.
>
> I propose that we remove all the automatic locking from the libraries,
> and let the user do any locking that they wish to do themselves.
>
> Attached are patches for ghc, base, process and unix. (not yet tested on
> Windows).
>
>
> Suggested discussion period: Until 9 Nov 2011
>
> Related GHC ticket: http://hackage.haskell.org/trac/ghc/ticket/4363
>
>
> Thanks
> Ian

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Bas van Dijk-2]

On 27 October 2011 00:59, Ian Lynagh <[hidden email]> wrote:
> I propose that we remove all the automatic locking from the libraries,
> and let the user do any locking that they wish to do themselves.

+1

Of course this breaks compatibility with Haskell 2010:

http://www.haskell.org/onlinereport/haskell2010/haskellch41.html#x49-32800041.3.4

Should a Haskell' proposal be filed?

http://hackage.haskell.org/trac/haskell-prime/wiki/Process

Regards,

Bas

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Max Bolingbroke-2]

On 27 October 2011 16:31, Bas van Dijk <[hidden email]> wrote:
> Should a Haskell' proposal be filed?
>
> http://hackage.haskell.org/trac/haskell-prime/wiki/Process

Ian mentioned this issue in the GHC ticket at
http://hackage.haskell.org/trac/ghc/ticket/4363. He says:

"""
I think the next step should be a  Haskell' proposal to remove the
locking from the report (including a clarification that a no-locking
implementation is possible on Windows).

Some people might argue that we should have an implementation first, though.
"""

So I guess we are seeing this patch as part of an "implementation
first" approach.

Max

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Bas van Dijk-2]

On 27 October 2011 17:55, Max Bolingbroke <[hidden email]> wrote:

> On 27 October 2011 16:31, Bas van Dijk <[hidden email]> wrote:
>> Should a Haskell' proposal be filed?
>>
>> http://hackage.haskell.org/trac/haskell-prime/wiki/Process
>
> Ian mentioned this issue in the GHC ticket at
> http://hackage.haskell.org/trac/ghc/ticket/4363. He says:
>
> """
> I think the next step should be a  Haskell' proposal to remove the
> locking from the report (including a clarification that a no-locking
> implementation is possible on Windows).
>
> Some people might argue that we should have an implementation first, though.
> """
>
> So I guess we are seeing this patch as part of an "implementation
> first" approach.
>
> Max
>

Right and to be clear, I'm totally fine with it.

It does create the situation that once this change gets released in a
new base package, the haskell2010 package will loose Haskell 2010
compatibility because it depends on base for its IO functions. We
could then of course add custom IO functions to haskell2010 that are
compatible with Haskell 2010. However I think it's best to just
quickly change the Haskell 2010 report and have a short period where
the haskell2010 package is incompatible with Haskell 2010.

Bas

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Antoine Latter-2]

On Thu, Oct 27, 2011 at 11:09 AM, Bas van Dijk <[hidden email]> wrote:

>
> Right and to be clear, I'm totally fine with it.
>
> It does create the situation that once this change gets released in a
> new base package, the haskell2010 package will loose Haskell 2010
> compatibility because it depends on base for its IO functions. We
> could then of course add custom IO functions to haskell2010 that are
> compatible with Haskell 2010. However I think it's best to just
> quickly change the Haskell 2010 report and have a short period where
> the haskell2010 package is incompatible with Haskell 2010.
>
> Bas
>

Wouldn't the Haskell2010 report remain unchanged by this? I imagine
this would be incorporated into Haskell2012, or whatever the next
revision is going to be.

Antoine

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Ian Lynagh]

On Thu, Oct 27, 2011 at 04:55:41PM +0100, Max Bolingbroke wrote:

> On 27 October 2011 16:31, Bas van Dijk <[hidden email]> wrote:
> > Should a Haskell' proposal be filed?
> >
> > http://hackage.haskell.org/trac/haskell-prime/wiki/Process
>
> Ian mentioned this issue in the GHC ticket at
> http://hackage.haskell.org/trac/ghc/ticket/4363. He says:
>
> """
> I think the next step should be a  Haskell' proposal to remove the
> locking from the report (including a clarification that a no-locking
> implementation is possible on Windows).
>
> Some people might argue that we should have an implementation first, though.
> """
>
> So I guess we are seeing this patch as part of an "implementation
> first" approach.

Actually, I realised when revisiting the ticket that this is really only
a change in the libraries, and we've taken the approach that library
changes are handled by the library proposal process, and the standard
will follow along in the next release.


Thanks
Ian


_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Bas van Dijk-2]

On 27 October 2011 19:03, Ian Lynagh <[hidden email]> wrote:
> Actually, I realised when revisiting the ticket that this is really only
> a change in the libraries, and we've taken the approach that library
> changes are handled by the library proposal process, and the standard
> will follow along in the next release.

Great, that's settled then.

It's probably best to just put an upper version constraint on the
dependency on base of the haskell2010 package so that it won't depend
on the new base package.

Bas

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Ian Lynagh]

On Thu, Oct 27, 2011 at 11:14:05AM -0500, Antoine Latter wrote:

> On Thu, Oct 27, 2011 at 11:09 AM, Bas van Dijk <[hidden email]> wrote:
>
> > It does create the situation that once this change gets released in a
> > new base package, the haskell2010 package will loose Haskell 2010
> > compatibility because it depends on base for its IO functions. We
> > could then of course add custom IO functions to haskell2010 that are
> > compatible with Haskell 2010. However I think it's best to just
> > quickly change the Haskell 2010 report and have a short period where
> > the haskell2010 package is incompatible with Haskell 2010.
>
> Wouldn't the Haskell2010 report remain unchanged by this? I imagine
> this would be incorporated into Haskell2012, or whatever the next
> revision is going to be.

Yes, and really we ought to add wrappers that do the locking to the
haskell2010 and haskell98 packages.

(which also means we wouldn't apply my ghc patch, now I think about it).

(in actual fact, I doubt anyone actually uses these packages, and
personally I think the standard shouldn't define the libraries, but
that's another debate!)

(hmm, haskell2010 has 3 revdeps, according to
    http://revdeps.hackage.haskell.org/~roel/cgi-bin/hackage-scripts/revdeps/haskell2010-1.1.0.0
haskell98 claims to have 294, but I bet most don't compile).


Thanks
Ian


_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Ian Lynagh]

On Thu, Oct 27, 2011 at 07:10:27PM +0200, Bas van Dijk wrote:

> On 27 October 2011 19:03, Ian Lynagh <[hidden email]> wrote:
> > Actually, I realised when revisiting the ticket that this is really only
> > a change in the libraries, and we've taken the approach that library
> > changes are handled by the library proposal process, and the standard
> > will follow along in the next release.
>
> Great, that's settled then.
>
> It's probably best to just put an upper version constraint on the
> dependency on base of the haskell2010 package so that it won't depend
> on the new base package.

Well, haskell2010 comes with ghc, so it needs to use the version of base
that GHC comes with.

If we stopped shipping it and set dependencies such that it won't build
with the new version of base, then no-one with the latest GHC would be
able to use it.


Thanks
Ian


_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Bryan O'Sullivan]

On Wed, Oct 26, 2011 at 3:59 PM, Ian Lynagh <[hidden email]> wrote:

I propose that we remove all the automatic locking from the libraries,
and let the user do any locking that they wish to do themselves.

+50

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Johan Tibell-2]

On Thu, Oct 27, 2011 at 3:10 PM, Bryan O'Sullivan <[hidden email]> wrote:

On Wed, Oct 26, 2011 at 3:59 PM, Ian Lynagh <[hidden email]> wrote:

I propose that we remove all the automatic locking from the libraries,
and let the user do any locking that they wish to do themselves.

+50

+1. In general I don't like when low-level libraries try to do too many smart things behind my back. That normally ends up in me having to write my own version of the library without those smarts.

-- Johan
 

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Duncan Coutts-4]

On Wed, 2011-10-26 at 23:59 +0100, Ian Lynagh wrote:

> Hi all,
>
> Haskell currently requires that multiple-reader single-writer locking is
> used on files. I understand the motivation for this was to try to
> protect people from running into problems due to lazy IO, but in
> practice I think it just causes more problems than it solves.
>
> The decision may also have been influenced by the belief that this
> behaviour is enforced for all programs on Windows, but I am told that it
> is possible to get lock-free behaviour on Windows too.
>
> I propose that we remove all the automatic locking from the libraries,
> and let the user do any locking that they wish to do themselves.
>
> Attached are patches for ghc, base, process and unix. (not yet tested on
> Windows).

I have to say, I'm a little reticent about this.

Certainly it's useful to be able deliberately to have multiple writers,
or to be able deliberately to read a file while also writing to it. But
doing so accidentally seems like a bug waiting to happen doesn't it?

Do we really want to make this the default everywhere? What about
keeping the check by default but adding something to turn off the
locking when that's intentional. That'd mean a param of openFile (which
incidentally is ripe for converting to take a record of these various
parameters, like we have for createProcess).

For Windows, it'd be fine to switch from using OS-level file locking to
intra-process RTS-level locking as we use on Unix. That is, only
protecting ourselves from bugs in our own code where we read and write,
not protecting from other processes.

And if you switch the default and users have to do their own locking,
how can they do it? I'm not sure there is any sensible way to do it. Any
bit of IO code can open any file. If they're already tightly coupled and
can share an MVar it's fine, but that's not the sort of accidental
situation that the current implicit locking protects against.

Duncan


_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Duncan Coutts-4]

On Tue, 2011-11-08 at 11:54 +0000, Duncan Coutts wrote:
> On Wed, 2011-10-26 at 23:59 +0100, Ian Lynagh wrote:
> > Hi all,
> >
> > Haskell currently requires that multiple-reader single-writer locking is
> > used on files. I understand the motivation for this was to try to
> > protect people from running into problems due to lazy IO, but in
> > practice I think it just causes more problems than it solves.

> > I propose that we remove all the automatic locking from the libraries,
> > and let the user do any locking that they wish to do themselves.

> I have to say, I'm a little reticent about this.
>
> Certainly it's useful to be able deliberately to have multiple writers,
> or to be able deliberately to read a file while also writing to it. But
> doing so accidentally seems like a bug waiting to happen doesn't it?

BTW, I realise this proposal has received a good deal of support,
however I suspect this is from people who are thinking about the current
inability to opt-out from locking in the various cases where it would be
useful for them to do so.

Perhaps the people who have spoken up in favour of this proposal could
comment on the following:

        We all seem to agree that its important to be able to open a
        file without any locking.

        There is a distinction between being able to opt-out from
        locking and removing all locking by default.
       
              * do you think it important to remove all locking?
              * would you be satisfied (or put up with) having to
                explicitly opt-out of locking when opening a file?
              * do you see any value at all in locking by default to
                catch bugs where files are read and written concurrently
                by accident?

Duncan


_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries