ANN: network 3.0.0.0

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

ANN: network 3.0.0.0

evan@evan-borden.com
Announcing the release of network 3.0.0.0

Version 3.0.0.0 of network is less platform dependent, easier to build, easier to use correctly (yay no more String api), more extensible, and all around safer. Kazu Yamamoto and Tamar Christina deserve a special shout out, they put an exceptional amount of thought and work into this version and it really shows. Thanks to network's contributors for kicking the tires and continuing to keep quality high.

Version 3.0.0.0

* Breaking change: the Network and Network.BSD are removed.
  Network.BSD is provided a new package: network-bsd.
* Breaking change: the signatures are changed:
```
old fdSocket :: Socket -> CInt
new fdSocket :: Socket -> IO CInt

old mkSocket :: CInt -> Family -> SocketType -> ProtocolNumber -> SocketStatus -> IO Socket
new mkSocket :: CInt Socket
```
* Breaking change: the deprecated APIs are removed: send, sendTo, recv, recvFrom, recvLen, htonl, ntohl, inet_addr, int_ntoa, bindSocket, sClose, SocketStatus, isConnected, isBound, isListening, isReadable, isWritable, sIsConnected, sIsBound, sIsListening, sIsReadable, sIsWritable, aNY_PORT, iNADDR_ANY, iN6ADDR_ANY, sOMAXCONN, sOL_SOCKET, sCM_RIGHTS, packSocketType, getPeerCred.
* Breaking chage: SockAddrCan is removed from SockAddr.
* Socket addresses are extendable with Network.Socket.Address.
* "socket" is now asynchronous-exception-safe.
* "recvFrom" returns (0, addr) instead of throwing an error on EOF.
* All APIs are available on any platforms.
* Build system is simplified.
* Bug fixes.

--
Evan Borden

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: network 3.0.0.0

Bryan Richter-2
Excellent!

FYI Hackage isn't showing Haddocks for network yet, although I've noticed this for a few packages again recently.

On Sat, 19 Jan 2019, 17.35 [hidden email] <[hidden email] wrote:
Announcing the release of network 3.0.0.0

Version 3.0.0.0 of network is less platform dependent, easier to build, easier to use correctly (yay no more String api), more extensible, and all around safer. Kazu Yamamoto and Tamar Christina deserve a special shout out, they put an exceptional amount of thought and work into this version and it really shows. Thanks to network's contributors for kicking the tires and continuing to keep quality high.

Version 3.0.0.0

* Breaking change: the Network and Network.BSD are removed.
  Network.BSD is provided a new package: network-bsd.
* Breaking change: the signatures are changed:
```
old fdSocket :: Socket -> CInt
new fdSocket :: Socket -> IO CInt

old mkSocket :: CInt -> Family -> SocketType -> ProtocolNumber -> SocketStatus -> IO Socket
new mkSocket :: CInt Socket
```
* Breaking change: the deprecated APIs are removed: send, sendTo, recv, recvFrom, recvLen, htonl, ntohl, inet_addr, int_ntoa, bindSocket, sClose, SocketStatus, isConnected, isBound, isListening, isReadable, isWritable, sIsConnected, sIsBound, sIsListening, sIsReadable, sIsWritable, aNY_PORT, iNADDR_ANY, iN6ADDR_ANY, sOMAXCONN, sOL_SOCKET, sCM_RIGHTS, packSocketType, getPeerCred.
* Breaking chage: SockAddrCan is removed from SockAddr.
* Socket addresses are extendable with Network.Socket.Address.
* "socket" is now asynchronous-exception-safe.
* "recvFrom" returns (0, addr) instead of throwing an error on EOF.
* All APIs are available on any platforms.
* Build system is simplified.
* Bug fixes.

--
Evan Borden
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: network 3.0.0.0

Niklas Hambüchen
In reply to this post by evan@evan-borden.com
Hey Evan,

it's great that work is being done on network, that is very appreciated.
I do find one thing problematic though:

Why do breaking type changes like `fdSocket` and `mkSocket` not go through a deprecation cycle?

It's a best-practice to not break types of fundamental packages unless absolutely necessary.
In this case it seems like these functions could have been deprecated, kept forever, and new functions with the improved types could have been added.
This would avoid breakage for network's many users.

Given that rest of the changelog in which deprecations are mentioned, it seems `network` contributers generall understand that and agree on that. Why was the deprecation approach not chosen for these functions?

What in general is network's decision process for deprecation vs breaking change? Was any analysis performed that showed that only the fewest of network's 926 Hackage dependencies use these functions? Or should we expect major ecosystem breakages?

Also interesting is that in https://github.com/haskell/network/commit/d69c3072071859a28a442ff713e90f8499882d21 `fdSocket` was deprecated, but that deprecation was about a slightly different change and never made it into any Hackage release so users never saw an advance hint that `fdSocket` would soon be changed.

Personally, it bugs me out that we can't get deprecations right in Haskell, when other ecosystems have understood and successfully executed the "deprecate and make better functions" approach for a very long time. But I'd be happy to hear why I'm wrong and this approach couldn't be used here.

Niklas
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: network 3.0.0.0

evan@evan-borden.com
Niklas,

I appreciate and empathize with your concerns. Care was taken to utilize a deprecation cycle for as much of the API as possible, as we knew that these changes would cause churn in the ecosystem. As you noted, some breaking changes were not communicated early through these means. Instead we decided to mark an epoch change in network to signal that there would be significant breakage. This follows conventions established by Ed Kmett, which are documented here https://pvp.haskell.org/faq/. I'm not sure a specific convention exists for a situation where the type of a function will change in a future version. I'd love to hear ideas.

Thank you very much for the feedback. Constructive criticism is necessary to improve our handling of these issues and is always welcome with open arms.

On Sat, Jan 19, 2019 at 9:56 AM Niklas Hambüchen <[hidden email]> wrote:
Hey Evan,

it's great that work is being done on network, that is very appreciated.
I do find one thing problematic though:

Why do breaking type changes like `fdSocket` and `mkSocket` not go through a deprecation cycle?

It's a best-practice to not break types of fundamental packages unless absolutely necessary.
In this case it seems like these functions could have been deprecated, kept forever, and new functions with the improved types could have been added.
This would avoid breakage for network's many users.

Given that rest of the changelog in which deprecations are mentioned, it seems `network` contributers generall understand that and agree on that. Why was the deprecation approach not chosen for these functions?

What in general is network's decision process for deprecation vs breaking change? Was any analysis performed that showed that only the fewest of network's 926 Hackage dependencies use these functions? Or should we expect major ecosystem breakages?

Also interesting is that in https://github.com/haskell/network/commit/d69c3072071859a28a442ff713e90f8499882d21 `fdSocket` was deprecated, but that deprecation was about a slightly different change and never made it into any Hackage release so users never saw an advance hint that `fdSocket` would soon be changed.

Personally, it bugs me out that we can't get deprecations right in Haskell, when other ecosystems have understood and successfully executed the "deprecate and make better functions" approach for a very long time. But I'd be happy to hear why I'm wrong and this approach couldn't be used here.

Niklas


--
--
Evan Borden

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: network 3.0.0.0

Niklas Hambüchen
Thanks for the quick reply!

> I'd love to hear ideas.

I recommend the following way:

* Never change the type of an existing function unless absolutely necessary.
* Do not change the the behaviour/semantics of a function unless it's an obvious fix/improvement.
  (In other words, if somebody may in some reasonable way rely on the old behaviour, don't change it).
* Improve functions by introducing new functions with names, marking the old functions as deprecated.
* GHC will show deprecation warnings to users.
** In the deprecation message, point out the intended replacement.
** If a deprecated function is planned to be removed in the next release, say it in the message.
* Consider removing deprecated functions after some years. But only if e.g. maintainability demands it.
** In some cases, consider moving the functions to an `.Old` module or similar.

Key point:
* Use deprecations and new functions liberally to make progress. Almost never break existing functions.

I like to call this approach "the Java way of deprecation", and it helps an ecosystem to move forward swiftly while keeping people happy a whole lot. This is because it allows incremental transitions instead of hard cutoff points that often don't align with people's schedules or test plans.

How this relates to the PVP:
This thought process happens *before* PVP considerations enter the stage.
After you've made your changes (hopefully as few breaking ones as possible), you can use the PVP to determine what the new version should be.
The reverse logic should not be applied: If some change made demands a major version bump (e.g. removal of an old deprecated function that nobody uses), that does not "allow" other functions to be removed more liberally. The rationale is that our goal is not to do as much as a given version jump allows, but to minimise breaking changes for users even when we know that some breaking changes must be made.

(Also, while the PVP as mentioned doesn't tell you what to do with your package contents and applies afterwards, the diagram in https://pvp.haskell.org/pvp-decision-tree.svg mentions "Consider renaming the function instead" next to "Did the behaviour of any exported functions change", so I think what I recommend is in the spirit of those that came up with the process).

Concrete example:

This is how I imagine applying the above to network 3.0.0.0 would have worked:

* Deprecate `send`, `sendTo`, `recv`, etc. (As correctly done in network-2.7)
* Do NOT remove them. Leave at least 2 years time before touching them.
* In this case, move them to Network.Socket.String, instead of removing them. For 2 reasons:
** These functions have been this ways since forever and lots of code will use them. Putting them into a legacy module will allow projects to trivially get into a compiling state again with one `import` change, vs having to fix every use site. This eases migration.
** While certainly not a good idea, the String based functions aren't so fatally flawed that they have no justification for further existence, so keeping them in a legacy module would do no harm.
* Introduce `socketToFd :: Socket -> IO CInt`
* Deprecate `fdSocket`, saying why it's bad and that `socketToFd` is the replacement
* Keep `fdSocket` forever

(Just to be clear, I think the `String` based network API is bug; a reasonable and modern programming ecosystem shouldn't have that, and this should be addressed. So I'm not recommending this approach because I think these functions are great, but because I think this is how to transition away from design bugs in general.)

Then, looking at the above changes, if we follow PVP, we'd look at the PVP decision tree, and determine what the new version should be that way. If (after a long time period) we'd do the "move to Network.Socket.String", it would tell us that a major bump is necessary.
Of course one can also introduce a major version bump when PVP says that a minor bump would be sufficient if one wants to "mark an epoch change" -- that is at the liberty of the maintainer.

Discussion:

I found this approach to work very well and am convinced of it.
But I'm happy to hear other people's opinion about it to see if it's really as agreeable as I think.
If yes, I'd be happy to write down these deprecation guidelines in some repo so that projects can refer to them and say "we follow that general approach" (similar to the PVP FAQ link).

What do you think?

Niklas
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: network 3.0.0.0

Tom Ellis-5
On Sat, Jan 19, 2019 at 06:38:09PM +0100, Niklas Hambüchen wrote:
> I recommend the following way:
[...]
> I'd be happy to write down these deprecation guidelines in some repo so
> that projects can refer to them

I think that's a great idea.

Opaleye roughly follows your specification.  I try to announce deprecations
in major version N, implement the deprecation in N+1 and remove the
deprecated entity no earlier than N+2.

NB that network could still be changed to follow your plan if a new version
were released and 3.0.0.0 deprecated (hah!).

Tom
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: network 3.0.0.0

Alexander Kjeldaas
In reply to this post by Niklas Hambüchen

While those are good suggestions, the biggest problem is not whether a given deprecation strategy is correct or not.

The biggest problem with the PVP in this case is that it does not say anything about how to release a "pre" release so any deprecation strategy could be tested by the community.

This makes the concept of learning painful, because mistakes are way too expensive.

Can this be fixed first, then later we can figure out how to do deprecations? Or am I wrong?

Alexander

On Sat, Jan 19, 2019 at 6:38 PM Niklas Hambüchen <[hidden email]> wrote:
Thanks for the quick reply!

> I'd love to hear ideas.

I recommend the following way:

* Never change the type of an existing function unless absolutely necessary.
* Do not change the the behaviour/semantics of a function unless it's an obvious fix/improvement.
  (In other words, if somebody may in some reasonable way rely on the old behaviour, don't change it).
* Improve functions by introducing new functions with names, marking the old functions as deprecated.
* GHC will show deprecation warnings to users.
** In the deprecation message, point out the intended replacement.
** If a deprecated function is planned to be removed in the next release, say it in the message.
* Consider removing deprecated functions after some years. But only if e.g. maintainability demands it.
** In some cases, consider moving the functions to an `.Old` module or similar.

Key point:
* Use deprecations and new functions liberally to make progress. Almost never break existing functions.

I like to call this approach "the Java way of deprecation", and it helps an ecosystem to move forward swiftly while keeping people happy a whole lot. This is because it allows incremental transitions instead of hard cutoff points that often don't align with people's schedules or test plans.

How this relates to the PVP:
This thought process happens *before* PVP considerations enter the stage.
After you've made your changes (hopefully as few breaking ones as possible), you can use the PVP to determine what the new version should be.
The reverse logic should not be applied: If some change made demands a major version bump (e.g. removal of an old deprecated function that nobody uses), that does not "allow" other functions to be removed more liberally. The rationale is that our goal is not to do as much as a given version jump allows, but to minimise breaking changes for users even when we know that some breaking changes must be made.

(Also, while the PVP as mentioned doesn't tell you what to do with your package contents and applies afterwards, the diagram in https://pvp.haskell.org/pvp-decision-tree.svg mentions "Consider renaming the function instead" next to "Did the behaviour of any exported functions change", so I think what I recommend is in the spirit of those that came up with the process).

Concrete example:

This is how I imagine applying the above to network 3.0.0.0 would have worked:

* Deprecate `send`, `sendTo`, `recv`, etc. (As correctly done in network-2.7)
* Do NOT remove them. Leave at least 2 years time before touching them.
* In this case, move them to Network.Socket.String, instead of removing them. For 2 reasons:
** These functions have been this ways since forever and lots of code will use them. Putting them into a legacy module will allow projects to trivially get into a compiling state again with one `import` change, vs having to fix every use site. This eases migration.
** While certainly not a good idea, the String based functions aren't so fatally flawed that they have no justification for further existence, so keeping them in a legacy module would do no harm.
* Introduce `socketToFd :: Socket -> IO CInt`
* Deprecate `fdSocket`, saying why it's bad and that `socketToFd` is the replacement
* Keep `fdSocket` forever

(Just to be clear, I think the `String` based network API is bug; a reasonable and modern programming ecosystem shouldn't have that, and this should be addressed. So I'm not recommending this approach because I think these functions are great, but because I think this is how to transition away from design bugs in general.)

Then, looking at the above changes, if we follow PVP, we'd look at the PVP decision tree, and determine what the new version should be that way. If (after a long time period) we'd do the "move to Network.Socket.String", it would tell us that a major bump is necessary.
Of course one can also introduce a major version bump when PVP says that a minor bump would be sufficient if one wants to "mark an epoch change" -- that is at the liberty of the maintainer.

Discussion:

I found this approach to work very well and am convinced of it.
But I'm happy to hear other people's opinion about it to see if it's really as agreeable as I think.
If yes, I'd be happy to write down these deprecation guidelines in some repo so that projects can refer to them and say "we follow that general approach" (similar to the PVP FAQ link).

What do you think?

Niklas
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: ANN: network 3.0.0.0

Kazu Yamamoto (山本和彦)
In reply to this post by Niklas Hambüchen
Hello,

> * Never change the type of an existing function unless absolutely necessary.

The change of mkSocket and fdSocket is inevitable because the old
definition of Socket is:
data Socket = MkSocket CInt Family SocketType ProtocolNumber (MVar SocketStatus)
while new one is:
data Socket = Socket !(IORef CInt) CInt {- for Show -}

Why this drastic change? Well, believing or not, old Socket cannot be
GCed because of MVar! mkWeakMVar does not solve this issue.

So, we decided to get rid of MVar. But without SocketStatus (ie Socket
CInt), "close" becomes unsafe if "close" is called multiple time.  To
fix this, IORef was introduced. Now a closed Socket contains (-1).

If you type "fdScoket" or "GC" to the github repo search, you can find
a lot of discussions about this.

--Kazu
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.