Haskell Platform proposal: Add the vector package

[Johan Tibell-2]

Hi,

I am, with Roman's support, making a formal proposal to have the
vector package included in the Haskell Platform:

    http://trac.haskell.org/haskell-platform/wiki/Proposals/vector

See the wiki page for details, and a list of open issues for discussion.

The vector package itself can be found on Hackage:

    http://hackage.haskell.org/package/vector

I have set the deadline to 13 August 2012 (~2 months' time), with
comments due by 13 July 2012 (~one month).  There's plenty of time
before the next platform release anyway (scheduled for 12 November
2012).

-- Johan

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Henning Thielemann]

On Fri, 15 Jun 2012, Johan Tibell wrote:

> I am, with Roman's support, making a formal proposal to have the
> vector package included in the Haskell Platform:
>
>    http://trac.haskell.org/haskell-platform/wiki/Proposals/vector
>
> See the wiki page for details, and a list of open issues for discussion.
>
> The vector package itself can be found on Hackage:
>
>    http://hackage.haskell.org/package/vector

I thought about migrating storablevector to vector, but it seems that the
vector package needs some GHC-only extensions like type families. I do not
plan to make the storablevector package obsolete, but I think it would be
useful if both packages would use the same datatype. Is it possible to put
the Storable part of 'vector' into a separate package? Would this one be
more portable?

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Bas van Dijk-2]

On 15 June 2012 23:45, Johan Tibell <[hidden email]> wrote:

> Hi,
>
> I am, with Roman's support, making a formal proposal to have the
> vector package included in the Haskell Platform:
>
>    http://trac.haskell.org/haskell-platform/wiki/Proposals/vector
>
> See the wiki page for details, and a list of open issues for discussion.
>
> The vector package itself can be found on Hackage:
>
>    http://hackage.haskell.org/package/vector
>
> I have set the deadline to 13 August 2012 (~2 months' time), with
> comments due by 13 July 2012 (~one month).  There's plenty of time
> before the next platform release anyway (scheduled for 12 November
> 2012).

+1

I like the idea of the vector-safe package. Are you also proposing to
add this package to the HP? (I would also be +1 on that)

I see that the trustworthiness of the .Safe modules is conditional on
whether bound checking is enabled in vector:

#if defined(VECTOR_BOUNDS_CHECKS)
{-# LANGUAGE Trustworthy #-}
#endif

The VECTOR_BOUNDS_CHECKS pragma would not be directly available in
vector-safe. But I guess, by using the install-includes cabal field,
vector can export a header file that exports this symbol when bound
checking is enabled.

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Johan Tibell-2]

On Sun, Jun 17, 2012 at 4:14 AM, Henning Thielemann
<[hidden email]> wrote:
> I thought about migrating storablevector to vector, but it seems that the
> vector package needs some GHC-only extensions like type families. I do not
> plan to make the storablevector package obsolete, but I think it would be
> useful if both packages would use the same datatype. Is it possible to put
> the Storable part of 'vector' into a separate package? Would this one be
> more portable?

I'll let Roman address this. I suggest we consider this a separately
from whether we should add vector to the platform (perhaps we should
start a new thread from that.)

-- Johan

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Johan Tibell-2]

On Mon, Jun 18, 2012 at 9:54 AM, Bas van Dijk <[hidden email]> wrote:
> I like the idea of the vector-safe package. Are you also proposing to
> add this package to the HP? (I would also be +1 on that)

I think it makes sense as a separate package, but I don't think it
makes sense to add to the HP. SafeHaskell isn't used enough to warrant
that.

> I see that the trustworthiness of the .Safe modules is conditional on
> whether bound checking is enabled in vector:
>
> #if defined(VECTOR_BOUNDS_CHECKS)
> {-# LANGUAGE Trustworthy #-}
> #endif
>
> The VECTOR_BOUNDS_CHECKS pragma would not be directly available in
> vector-safe. But I guess, by using the install-includes cabal field,
> vector can export a header file that exports this symbol when bound
> checking is enabled.

That sounds like a reasonable solution.

-- Johan

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Ryan Newton]

+1

This is one of those packages that I periodically forget is not in HP.

On Mon, Jun 18, 2012 at 2:39 PM, Johan Tibell <[hidden email]> wrote:

On Mon, Jun 18, 2012 at 9:54 AM, Bas van Dijk <[hidden email]> wrote:
> I like the idea of the vector-safe package. Are you also proposing to
> add this package to the HP? (I would also be +1 on that)

I think it makes sense as a separate package, but I don't think it
makes sense to add to the HP. SafeHaskell isn't used enough to warrant
that.

> I see that the trustworthiness of the .Safe modules is conditional on
> whether bound checking is enabled in vector:
>
> #if defined(VECTOR_BOUNDS_CHECKS)
> {-# LANGUAGE Trustworthy #-}
> #endif
>
> The VECTOR_BOUNDS_CHECKS pragma would not be directly available in
> vector-safe. But I guess, by using the install-includes cabal field,
> vector can export a header file that exports this symbol when bound
> checking is enabled.

That sounds like a reasonable solution.

-- Johan

_______________________________________________
Haskell-platform mailing list
[hidden email]
http://projects.haskell.org/cgi-bin/mailman/listinfo/haskell-platform

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Roman Leshchinskiy]

On 17/06/2012, at 12:14, Henning Thielemann wrote:

> On Fri, 15 Jun 2012, Johan Tibell wrote:
>
>> I am, with Roman's support, making a formal proposal to have the
>> vector package included in the Haskell Platform:
>>
>>   http://trac.haskell.org/haskell-platform/wiki/Proposals/vector
>>
>> See the wiki page for details, and a list of open issues for discussion.
>>
>> The vector package itself can be found on Hackage:
>>
>>   http://hackage.haskell.org/package/vector
>
> I thought about migrating storablevector to vector, but it seems that the vector package needs some GHC-only extensions like type families. I do not plan to make the storablevector package obsolete, but I think it would be useful if both packages would use the same datatype. Is it possible to put the Storable part of 'vector' into a separate package? Would this one be more portable?

There are type families, rank-n types, unboxed types and other goodies deep in the guts of vector so the Storable part is very much GHC-specific. To be honest, I don't think being portable is feasible for high-performance code at the moment, the language standard simply doesn't have enough tools for this. Which is a shame, really.

FWIW, Storable vectors are fundamentally broken, anyway, since a Storable instance can perform arbitrary I/O in its methods but a pure vector based on Storable will necessarily have to unsafePerformIO these operations. Storable should *really* live in ST but it's too late for that now. Which reminds me, I should dig up and finish my ST-based Storable alternative one of these days and provide a safe vector type for interoperating with C.

Roman



_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Roman Leshchinskiy]

On 18/06/2012, at 19:39, Johan Tibell wrote:

> On Mon, Jun 18, 2012 at 9:54 AM, Bas van Dijk <[hidden email]> wrote:
>> I like the idea of the vector-safe package. Are you also proposing to
>> add this package to the HP? (I would also be +1 on that)
>
> I think it makes sense as a separate package, but I don't think it
> makes sense to add to the HP. SafeHaskell isn't used enough to warrant
> that.

I fully agree with Johan and I wouldn't even really want to maintain this separate package. It is a lot of work for something I don't use and don't entirely understand. The *.Safe modules in vector are currently bitrotted since I forget to update them when I add new operations and I'm not really sure what is and isn't "safe" anyway. Is anybody interested in this code at all?

>> I see that the trustworthiness of the .Safe modules is conditional on
>> whether bound checking is enabled in vector:
>>
>> #if defined(VECTOR_BOUNDS_CHECKS)
>> {-# LANGUAGE Trustworthy #-}
>> #endif
>>
>> The VECTOR_BOUNDS_CHECKS pragma would not be directly available in
>> vector-safe. But I guess, by using the install-includes cabal field,
>> vector can export a header file that exports this symbol when bound
>> checking is enabled.
>
> That sounds like a reasonable solution.

VECTOR_BOUNDS_CHECKS is defined in vector.cabal as follows:

  if flag(BoundsChecks)
    cpp-options: -DVECTOR_BOUNDS_CHECKS

Doesn't Cabal provide access to the flags that a package has been compiled with? It seems like it should and depending on the BoundsChecks flag would be cleaner than depending on a CPP symbol.

Roman



_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Henning Thielemann]

On Mon, 18 Jun 2012, Roman Leshchinskiy wrote:

> There are type families, rank-n types, unboxed types and other goodies
> deep in the guts of vector so the Storable part is very much
> GHC-specific. To be honest, I don't think being portable is feasible for
> high-performance code at the moment, the language standard simply
> doesn't have enough tools for this. Which is a shame, really.

I am not mainly interested in the efficient implementation. I am
completely ok with having the definition of (Vector a) in a separate
package, such that it can be used by vector (GHC only) and storablevector
(portable).

However, I have just looked into Vector.Storable and it looks like

   data Vector a = Vector Int (ForeignPtr a)

I thought it was

   data Vector a = Vector {len :: Int, allocated :: ForeignPtr a, start :: Ptr a}

ByteString looks like:

   data ByteString = PS {allocated :: ForeignPtr Word8, start, length :: Int}

Both forms allow efficient slicing.
How do you perform efficient 'take' and 'drop' ?


> FWIW, Storable vectors are fundamentally broken, anyway, since a
> Storable instance can perform arbitrary I/O in its methods but a pure
> vector based on Storable will necessarily have to unsafePerformIO these
> operations.

That's unfortunately true.

> Storable should *really* live in ST but it's too late for that now.

How would this prevent from broken pointer arithmetic?

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Henning Thielemann]

On Mon, 18 Jun 2012, Roman Leshchinskiy wrote:

> VECTOR_BOUNDS_CHECKS is defined in vector.cabal as follows:
>
>  if flag(BoundsChecks)
>    cpp-options: -DVECTOR_BOUNDS_CHECKS
>
> Doesn't Cabal provide access to the flags that a package has been
> compiled with?

Do you mean this way:
    http://hackage.haskell.org/trac/hackage/ticket/836
?

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Bas van Dijk-2]

On Jun 19, 2012 12:16 AM, "Henning Thielemann" <[hidden email]> wrote:
>
>
> On Mon, 18 Jun 2012, Roman Leshchinskiy wrote:
>
>> There are type families, rank-n types, unboxed types and other goodies deep in the guts of vector so the Storable part is very much GHC-specific. To be honest, I don't think being portable is feasible for high-performance code at the moment, the language standard simply doesn't have enough tools for this. Which is a shame, really.
>
>
> I am not mainly interested in the efficient implementation. I am completely ok with having the definition of (Vector a) in a separate package, such that it can be used by vector (GHC only) and storablevector (portable).
>
> However, I have just looked into Vector.Storable and it looks like
>
>  data Vector a = Vector Int (ForeignPtr a)
>
> I thought it was
>
>  data Vector a = Vector {len :: Int, allocated :: ForeignPtr a, start :: Ptr a}
>
> ByteString looks like:
>
>  data ByteString = PS {allocated :: ForeignPtr Word8, start, length :: Int}
>
> Both forms allow efficient slicing.
> How do you perform efficient 'take' and 'drop' ?

Slicing is done by directly updating the pointer in the ForeignPtr:

{-# INLINE basicUnsafeSlice #-}
basicUnsafeSlice i n (Vector _ fp) =
    Vector n (updPtr (`advancePtr` i) fp)

{-# INLINE updPtr #-}
updPtr :: (Ptr a -> Ptr a) -> ForeignPtr a -> ForeignPtr a
updPtr f (ForeignPtr p c) =
    case f (Ptr p) of { Ptr q -> ForeignPtr q c }

This saves an Int.

Regards,

Bas

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Johan Tibell-2]

On Mon, Jun 18, 2012 at 3:53 PM, Bas van Dijk <[hidden email]> wrote:

> Slicing is done by directly updating the pointer in the ForeignPtr:
>
> {-# INLINE basicUnsafeSlice #-}
> basicUnsafeSlice i n (Vector _ fp) =
>     Vector n (updPtr (`advancePtr` i) fp)
>
> {-# INLINE updPtr #-}
> updPtr :: (Ptr a -> Ptr a) -> ForeignPtr a -> ForeignPtr a
> updPtr f (ForeignPtr p c) =
>     case f (Ptr p) of { Ptr q -> ForeignPtr q c }
>
> This saves an Int.

(This is off-topic as far as the proposal.)

ByteString has an extra Ptr field so that accessing the data is fast,
given that ForeignPtrs can't be unpacked.

-- Johan

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Roman Leshchinskiy]

On 18/06/2012, at 23:16, Henning Thielemann wrote:

> On Mon, 18 Jun 2012, Roman Leshchinskiy wrote:
>
>> There are type families, rank-n types, unboxed types and other goodies deep in the guts of vector so the Storable part is very much GHC-specific. To be honest, I don't think being portable is feasible for high-performance code at the moment, the language standard simply doesn't have enough tools for this. Which is a shame, really.
>
> I am not mainly interested in the efficient implementation. I am completely ok with having the definition of (Vector a) in a separate package, such that it can be used by vector (GHC only) and storablevector (portable).

By Vector a you mean just the data type, not the type classes, right? What would the package contain apart from the type definition?

> However, I have just looked into Vector.Storable and it looks like
>
>  data Vector a = Vector Int (ForeignPtr a)
>
> I thought it was
>
>  data Vector a = Vector {len :: Int, allocated :: ForeignPtr a, start :: Ptr a}

The ForeignPtr already stores an Addr#:

data ForeignPtr a = ForeignPtr Addr# ForeignPtrContents

I just manipulate it directly which saves a pointer per vector. This might not seem like a big deal but sometimes this pointer will be threaded through a loop clobbering registers which *is* a big deal. Simon Marlow says that there is no requirement that the Addr# in the ForeignPtr must point to the start of the block.

I would make this more explicit by manually unpacking the ForeignPtr but alas, ForeignPtrContents (the actual type name) isn't exported from GHC.ForeignPtr so I can't.

Incidentally, this is another portability issue.

>> FWIW, Storable vectors are fundamentally broken, anyway, since a Storable instance can perform arbitrary I/O in its methods but a pure vector based on Storable will necessarily have to unsafePerformIO these operations.
>
> That's unfortunately true.
>
>> Storable should *really* live in ST but it's too late for that now.
>
> How would this prevent from broken pointer arithmetic?

It wouldn't but it would rule out this:

data T = T

class Storable T where
  peek p = do
    print "Peeking"
    spam ghcBugTracker
    return T

Which, unfortunately, is a perfectly valid implementation of peek.

Roman



_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Roman Leshchinskiy]

On 18/06/2012, at 23:20, Henning Thielemann wrote:

>
> On Mon, 18 Jun 2012, Roman Leshchinskiy wrote:
>
>> VECTOR_BOUNDS_CHECKS is defined in vector.cabal as follows:
>>
>> if flag(BoundsChecks)
>>   cpp-options: -DVECTOR_BOUNDS_CHECKS
>>
>> Doesn't Cabal provide access to the flags that a package has been compiled with?
>
> Do you mean this way:
>   http://hackage.haskell.org/trac/hackage/ticket/836
> ?

Indeed. I'm surprised Cabal doesn't provide this. But now I'm confused. How do I export a CPP symbol that depends on a flag to other packages? Do I have to generate a header file during configuration for that?

Roman



_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Roman Leshchinskiy]

On 19/06/2012, at 00:03, Johan Tibell wrote:

> On Mon, Jun 18, 2012 at 3:53 PM, Bas van Dijk <[hidden email]> wrote:
>> Slicing is done by directly updating the pointer in the ForeignPtr:
>>
>> {-# INLINE basicUnsafeSlice #-}
>> basicUnsafeSlice i n (Vector _ fp) =
>>     Vector n (updPtr (`advancePtr` i) fp)
>>
>> {-# INLINE updPtr #-}
>> updPtr :: (Ptr a -> Ptr a) -> ForeignPtr a -> ForeignPtr a
>> updPtr f (ForeignPtr p c) =
>>     case f (Ptr p) of { Ptr q -> ForeignPtr q c }
>>
>> This saves an Int.
>
> (This is off-topic as far as the proposal.)
>
> ByteString has an extra Ptr field so that accessing the data is fast,
> given that ForeignPtrs can't be unpacked.

ForeignPtrs can be unpacked, just not manually (which is a GHC bug, IMO, I should report it):

data Vector a = Vector {-# UNPACK #-} !Int
                       {-# UNPACK #-} !(ForeignPtr a)

ByteString just has some room for optimisation here ;-)

Roman



_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Johan Tibell-2]

On Mon, Jun 18, 2012 at 4:09 PM, Roman Leshchinskiy <[hidden email]> wrote:
> ForeignPtrs can be unpacked, just not manually (which is a GHC bug, IMO, I should report it):
>
> data Vector a = Vector {-# UNPACK #-} !Int
>                       {-# UNPACK #-} !(ForeignPtr a)
>
> ByteString just has some room for optimisation here ;-)

I probably just remembered the ByteString optimization incorrectly. My
apologies. I think the Addr# field was moved to the top of ForeignPtr
(not to the PS constructor itself) to support unpacking.

-- Johan

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Simon Peyton-Jones]

| ForeignPtrs can be unpacked, just not manually (which is a GHC bug, IMO,
| I should report it):

Yes, do!


_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Henning Thielemann]

On Tue, 19 Jun 2012, Roman Leshchinskiy wrote:

> On 18/06/2012, at 23:16, Henning Thielemann wrote:
>
>> On Mon, 18 Jun 2012, Roman Leshchinskiy wrote:
>>
>>> There are type families, rank-n types, unboxed types and other goodies deep in the guts of vector so the Storable part is very much GHC-specific. To be honest, I don't think being portable is feasible for high-performance code at the moment, the language standard simply doesn't have enough tools for this. Which is a shame, really.
>>
>> I am not mainly interested in the efficient implementation. I am completely ok with having the definition of (Vector a) in a separate package, such that it can be used by vector (GHC only) and storablevector (portable).
>

> By Vector a you mean just the data type, not the type classes, right?

yes

> What would the package contain apart from the type definition?

If the implementation of Vector functions requires GHC extensions then the
pure Vector data type definition would be the only definition. However, if
Vector is defined as it is and these direct manipulations of ForeignPtr
are not portable, then there is even no benefit in putting the Vector
definition in a separate package. We should then leave the 'vector' and
the 'storablevector' packages as they are and have to convert explicitly
between these types.

>>> Storable should *really* live in ST but it's too late for that now.
>>
>> How would this prevent from broken pointer arithmetic?
>
> It wouldn't but it would rule out this:
>
> data T = T
>
> class Storable T where
>  peek p = do
>    print "Peeking"
>    spam ghcBugTracker
>    return T

I see. But how would I define an ST-Storable instance for a new type, say
LLVM.Vector (the CPU vector type in 'llvm') without unsafeIOtoST? I could
still lift the 'spam' command into ST. However, ST might make people think
more thoroughly whether the lifted operations are appropriate for ST.

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Bryan O'Sullivan]

On Fri, Jun 15, 2012 at 5:45 PM, Johan Tibell <[hidden email]> wrote:

I am, with Roman's support, making a formal proposal to have the
vector package included in the Haskell Platform:

Very much in favour of this. Thanks for running the proposal! 

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries

[Simon Marlow-7]

On 18/06/2012 23:06, Roman Leshchinskiy wrote:

> On 18/06/2012, at 19:39, Johan Tibell wrote:
>
>> On Mon, Jun 18, 2012 at 9:54 AM, Bas van Dijk <[hidden email]> wrote:
>>> I like the idea of the vector-safe package. Are you also proposing to
>>> add this package to the HP? (I would also be +1 on that)
>>
>> I think it makes sense as a separate package, but I don't think it
>> makes sense to add to the HP. SafeHaskell isn't used enough to warrant
>> that.
>
> I fully agree with Johan and I wouldn't even really want to maintain
> this separate package. It is a lot of work for something I don't use
> and don't entirely understand. The *.Safe modules in vector are
> currently bitrotted since I forget to update them when I add new
> operations and I'm not really sure what is and isn't "safe" anyway.
> Is anybody interested in this code at all?

I respectfully disagree with this approach, I think it's heading in the
wrong direction.

We should be moving towards safe APIs by default, and separating out
unsafe APIs into separate modules.  That is what SafeHaskell is about:
it's not an obscure feature that is only used by things like "Try
Haskell", the boundary between safety and unsafety is something we
should all be thinking about.  In that sense, we are all users of
SafeHaskell.  We should think of it as "good style" and best practice to
separate safe APIs from unsafe ones.

I would argue against adding any unsafe APIs to the Haskell Platform
that aren't in a .Unsafe module.  (to what extent that applies to vector
I don't know, so it may be that I'm causing trouble for the proposal here).

Cheers,
        Simon

>>> I see that the trustworthiness of the .Safe modules is conditional on
>>> whether bound checking is enabled in vector:
>>>
>>> #if defined(VECTOR_BOUNDS_CHECKS)
>>> {-# LANGUAGE Trustworthy #-}
>>> #endif
>>>
>>> The VECTOR_BOUNDS_CHECKS pragma would not be directly available in
>>> vector-safe. But I guess, by using the install-includes cabal field,
>>> vector can export a header file that exports this symbol when bound
>>> checking is enabled.
>>
>> That sounds like a reasonable solution.
>
> VECTOR_BOUNDS_CHECKS is defined in vector.cabal as follows:
>
>    if flag(BoundsChecks)
>      cpp-options: -DVECTOR_BOUNDS_CHECKS
>
> Doesn't Cabal provide access to the flags that a package has been compiled with? It seems like it should and depending on the BoundsChecks flag would be cleaner than depending on a CPP symbol.
>
> Roman
>
>
>
> _______________________________________________
> Haskell-platform mailing list
> [hidden email]
> http://projects.haskell.org/cgi-bin/mailman/listinfo/haskell-platform
>

_______________________________________________
Libraries mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/libraries