Distributed and persistent events bus in Haskell

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Distributed and persistent events bus in Haskell

Arnaud Bailly-2
Hello,

I am implementing an application using event sourcing as primary storage for data, which implies I need a way to durably and reliably store streams of events on stable storage. I also need to be able to have an event distribution system on top of that persistent storage so that components can subscribe to stored events.

So far I have implemented a simple store, e.g. a flat file, which reuses the format of Apache Kafka (just in case...). Not very robust nor sophisticated but can work for moderate loads. Now I am looking for the event distribution part in the hope of being able to reuse some distributed event bus system that might exist somewhere and not having to roll my own. 

I have had a look couple of months ago at Vaultaire, Marquise and friends, but I am not sure they are really suited to my use case: They seem to be geared toward very high workload and throughput, like log or huge data streams analysis. 

Thanks for any pointer you might share,
-- 
Arnaud Bailly

twitter: abailly
skype: arnaud-bailly

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Distributed and persistent events bus in Haskell

Kyle Marek-Spartz
If you do end up going with the Kafka route, there is a native Haskell
client:

https://github.com/tylerholien/milena



Arnaud Bailly writes:

> Hello,
>
> I am implementing an application using event sourcing as primary storage
> for data, which implies I need a way to durably and reliably store streams
> of events on stable storage. I also need to be able to have an event
> distribution system on top of that persistent storage so that components
> can subscribe to stored events.
>
> So far I have implemented a simple store, e.g. a flat file, which reuses
> the format of Apache Kafka (just in case...). Not very robust nor
> sophisticated but can work for moderate loads. Now I am looking for the
> event distribution part in the hope of being able to reuse some distributed
> event bus system that might exist somewhere and not having to roll my own.
>
> I have had a look couple of months ago at Vaultaire, Marquise and friends,
> but I am not sure they are really suited to my use case: They seem to be
> geared toward very high workload and throughput, like log or huge data
> streams analysis.
>
> Thanks for any pointer you might share,

--
Kyle Marek-Spartz
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Distributed and persistent events bus in Haskell

Arnaud Bailly-2
Cool! however, I would rather avoid having to manage kafka, if something simpler exists :-) 

I know there is also a mature zeromq client so given I already have the persistent part, I could probably leverage that but I thought somebody might have already treaded that path...

Thanks a lot for the pointer, anyway.

-- 
Arnaud Bailly

twitter: abailly
skype: arnaud-bailly

On Tue, Apr 7, 2015 at 5:26 PM, Kyle Marek-Spartz <[hidden email]> wrote:
If you do end up going with the Kafka route, there is a native Haskell
client:

https://github.com/tylerholien/milena



Arnaud Bailly writes:

> Hello,
>
> I am implementing an application using event sourcing as primary storage
> for data, which implies I need a way to durably and reliably store streams
> of events on stable storage. I also need to be able to have an event
> distribution system on top of that persistent storage so that components
> can subscribe to stored events.
>
> So far I have implemented a simple store, e.g. a flat file, which reuses
> the format of Apache Kafka (just in case...). Not very robust nor
> sophisticated but can work for moderate loads. Now I am looking for the
> event distribution part in the hope of being able to reuse some distributed
> event bus system that might exist somewhere and not having to roll my own.
>
> I have had a look couple of months ago at Vaultaire, Marquise and friends,
> but I am not sure they are really suited to my use case: They seem to be
> geared toward very high workload and throughput, like log or huge data
> streams analysis.
>
> Thanks for any pointer you might share,

--
Kyle Marek-Spartz


_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: Distributed and persistent events bus in Haskell

timmy tofu
In reply to this post by Arnaud Bailly-2

We're working on something with 0MQ,  if you (or anyone else reading) go that route and want to compare notes, hit me off-list.

On Apr 8, 2015 8:01 AM, <[hidden email]> wrote:
Send Haskell-Cafe mailing list submissions to
        [hidden email]

To subscribe or unsubscribe via the World Wide Web, visit
        http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
or, via email, send a message with subject or body 'help' to
        [hidden email]

You can reach the person managing the list at
        [hidden email]

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Haskell-Cafe digest..."


Today's Topics:

   1. Distributed and persistent events bus in Haskell (Arnaud Bailly)
   2. Re: Distributed and persistent events bus in Haskell
      (Kyle Marek-Spartz)
   3. Re: Distributed and persistent events bus in Haskell
      (Arnaud Bailly)
   4. Re: MinGHC for GHC 7.10.1 available (Aaron Contorer)
   5. status of deb.haskell.org? (Jeremy)
   6. help w/ improving custom Stream for parsec (Maurizio Vitale)
   7. Anyone interested in taking over network-uri? (Johan Tibell)
   8. Re: Anyone interested in taking over network-uri? (Alexey Shmalko)


----------------------------------------------------------------------

Message: 1
Date: Tue, 7 Apr 2015 16:50:49 +0200
From: Arnaud Bailly <[hidden email]>
To: Haskell Cafe <[hidden email]>
Subject: [Haskell-cafe] Distributed and persistent events bus in
        Haskell
Message-ID:
        <[hidden email]>
Content-Type: text/plain; charset="utf-8"

Hello,

I am implementing an application using event sourcing as primary storage
for data, which implies I need a way to durably and reliably store streams
of events on stable storage. I also need to be able to have an event
distribution system on top of that persistent storage so that components
can subscribe to stored events.

So far I have implemented a simple store, e.g. a flat file, which reuses
the format of Apache Kafka (just in case...). Not very robust nor
sophisticated but can work for moderate loads. Now I am looking for the
event distribution part in the hope of being able to reuse some distributed
event bus system that might exist somewhere and not having to roll my own.

I have had a look couple of months ago at Vaultaire, Marquise and friends,
but I am not sure they are really suited to my use case: They seem to be
geared toward very high workload and throughput, like log or huge data
streams analysis.

Thanks for any pointer you might share,
--
Arnaud Bailly

twitter: abailly
skype: arnaud-bailly
linkedin: http://fr.linkedin.com/in/arnaudbailly/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20150407/ee65ca06/attachment-0001.html>

------------------------------

Message: 2
Date: Tue, 07 Apr 2015 10:26:26 -0500
From: Kyle Marek-Spartz <[hidden email]>
To: Arnaud Bailly <[hidden email]>
Cc: Haskell Cafe <[hidden email]>
Subject: Re: [Haskell-cafe] Distributed and persistent events bus in
        Haskell
Message-ID: <[hidden email]>
Content-Type: text/plain

If you do end up going with the Kafka route, there is a native Haskell
client:

https://github.com/tylerholien/milena



Arnaud Bailly writes:

> Hello,
>
> I am implementing an application using event sourcing as primary storage
> for data, which implies I need a way to durably and reliably store streams
> of events on stable storage. I also need to be able to have an event
> distribution system on top of that persistent storage so that components
> can subscribe to stored events.
>
> So far I have implemented a simple store, e.g. a flat file, which reuses
> the format of Apache Kafka (just in case...). Not very robust nor
> sophisticated but can work for moderate loads. Now I am looking for the
> event distribution part in the hope of being able to reuse some distributed
> event bus system that might exist somewhere and not having to roll my own.
>
> I have had a look couple of months ago at Vaultaire, Marquise and friends,
> but I am not sure they are really suited to my use case: They seem to be
> geared toward very high workload and throughput, like log or huge data
> streams analysis.
>
> Thanks for any pointer you might share,

--
Kyle Marek-Spartz


------------------------------

Message: 3
Date: Tue, 7 Apr 2015 17:33:01 +0200
From: Arnaud Bailly <[hidden email]>
To: Kyle Marek-Spartz <[hidden email]>
Cc: Haskell Cafe <[hidden email]>
Subject: Re: [Haskell-cafe] Distributed and persistent events bus in
        Haskell
Message-ID:
        <CAL4zPaoAoBBxAF2hZ_fTOFuKgavt2D5V=[hidden email]>
Content-Type: text/plain; charset="utf-8"

Cool! however, I would rather avoid having to manage kafka, if something
simpler exists :-)

I know there is also a mature zeromq client so given I already have the
persistent part, I could probably leverage that but I thought somebody
might have already treaded that path...

Thanks a lot for the pointer, anyway.

--
Arnaud Bailly

twitter: abailly
skype: arnaud-bailly
linkedin: http://fr.linkedin.com/in/arnaudbailly/

On Tue, Apr 7, 2015 at 5:26 PM, Kyle Marek-Spartz <
[hidden email]> wrote:

> If you do end up going with the Kafka route, there is a native Haskell
> client:
>
> https://github.com/tylerholien/milena
>
>
>
> Arnaud Bailly writes:
>
> > Hello,
> >
> > I am implementing an application using event sourcing as primary storage
> > for data, which implies I need a way to durably and reliably store
> streams
> > of events on stable storage. I also need to be able to have an event
> > distribution system on top of that persistent storage so that components
> > can subscribe to stored events.
> >
> > So far I have implemented a simple store, e.g. a flat file, which reuses
> > the format of Apache Kafka (just in case...). Not very robust nor
> > sophisticated but can work for moderate loads. Now I am looking for the
> > event distribution part in the hope of being able to reuse some
> distributed
> > event bus system that might exist somewhere and not having to roll my
> own.
> >
> > I have had a look couple of months ago at Vaultaire, Marquise and
> friends,
> > but I am not sure they are really suited to my use case: They seem to be
> > geared toward very high workload and throughput, like log or huge data
> > streams analysis.
> >
> > Thanks for any pointer you might share,
>
> --
> Kyle Marek-Spartz
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20150407/570a7fb2/attachment-0001.html>

------------------------------

Message: 4
Date: Tue, 7 Apr 2015 08:51:18 -0700 (PDT)
From: Aaron Contorer <[hidden email]>
To: [hidden email]
Cc: [hidden email], [hidden email]
Subject: Re: [Haskell-cafe] MinGHC for GHC 7.10.1 available
Message-ID: <[hidden email]>
Content-Type: text/plain; charset="utf-8"

For those who don't know, MinGHC is a minimal installer of GHC for Windows,
including GHC, cabal-install, and MSYS. Further discussion on this
announcement is available
at http://www.reddit.com/r/haskell/comments/30h52i/minghc_for_ghc_710/ and
a short blog post with a bit more info is
at https://www.fpcomplete.com/blog/2015/03/minghc-ghc-7-10 .

On Friday, March 27, 2015 at 2:01:27 AM UTC-7, Michael Snoyman wrote:
>
> I've just uploaded a new release of MinGHC, including GHC 7.10.1 and
> cabal-install 1.22.2.0. This release can be downloaded from:
>
>
> https://s3.amazonaws.com/download.fpcomplete.com/minghc/minghc-7.10.1-i386.exe
>
> In the process, I also needed to upload a cabal-install binary for
> Windows, which is available at:
>
>
> https://s3.amazonaws.com/download.fpcomplete.com/minghc/cabal-install-1.22.2.0-i386-unknown-mingw32.tar.gz
>
> I've tested this distribution, but only lightly. Feedback from others
> would be useful :)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20150407/66bc0517/attachment-0001.html>

------------------------------

Message: 5
Date: Tue, 7 Apr 2015 09:15:17 -0700 (MST)
From: Jeremy <[hidden email]>
To: [hidden email]
Subject: [Haskell-cafe] status of deb.haskell.org?
Message-ID: <[hidden email]>
Content-Type: text/plain; charset=us-ascii

The haskell status page says that deb.haskell.org has been down since
mid-February. Are there any plans to bring it back up?



--
View this message in context: http://haskell.1045720.n5.nabble.com/status-of-deb-haskell-org-tp5768430.html
Sent from the Haskell - Haskell-Cafe mailing list archive at Nabble.com.


------------------------------

Message: 6
Date: Tue, 7 Apr 2015 12:25:12 -0400
From: Maurizio Vitale <[hidden email]>
To: Haskell Cafe <[hidden email]>
Subject: [Haskell-cafe] help w/ improving custom Stream for parsec
Message-ID:
        <CAAeLbQKR+EHDtAke84sDX=44NMGXHWnWNq_zmxhU=[hidden email]>
Content-Type: text/plain; charset="utf-8"

I need a custom stream that supports insertion of include files and
expansions of macros.
I also want to be able to give nice error messages (think of clang
macro-expansion backtrace), so I cannot use the standard trick of
concatenating included files and expanded macros to the current input with
setInput/getInput (I think I can't maybe there's a way of keeping a more
complex "position" and since the use in producing an error backtrac is
rare, it migth be worth exploring; if anybody has ideas here, I'm listening)

Assuming I need a more compelx stream, this is what I have (Macro and File
both have a string argument, but it will be more compicated, a list of
expansions for Macro for instance).

Is there a better way for doing this?
What are the performance implications with backtracking? I'll be
benchmarking it, but if people see obvious problems, let me know.

Thanks a lot,
  Maurizio

{-# LANGUAGE FlexibleInstances #-}
{-# LANGUAGE FlexibleContexts #-}
{-# LANGUAGE InstanceSigs #-}
{-# LANGUAGE MultiParamTypeClasses #-}

module Parsing where

import Text.Parsec

type Parser s m = ParsecT s () m

data VStream = File String | Macro String deriving Show

newtype StreamStack = StreamStack [VStream] deriving Show

instance (Monad m) ? Stream VStream m Char where
  uncons ? VStream -> m (Maybe (Char, VStream))
  uncons (File (a:as)) = return $ Just (a, File as)
  uncons (File []) = return Nothing
  uncons (Macro (a:as)) = return $ Just (a, File as)
  uncons (Macro []) = return Nothing



instance (Monad m) => Stream StreamStack  m Char where
  uncons (StreamStack []) = return Nothing
  uncons (StreamStack (s:ss)) =
    case uncons s of
     Nothing ? uncons $ StreamStack ss
     Just Nothing ? uncons $ StreamStack ss
     Just (Just (c, File s')) ? return $ Just (c, StreamStack (File s': ss))
     Just (Just (c, Macro s')) ? return $ Just (c, StreamStack (Macro
s':ss))
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20150407/3e28a995/attachment-0001.html>

------------------------------

Message: 7
Date: Tue, 7 Apr 2015 23:18:35 +0200
From: Johan Tibell <[hidden email]>
To: haskell-cafe <[hidden email]>
Subject: [Haskell-cafe] Anyone interested in taking over network-uri?
Message-ID:
        <[hidden email]>
Content-Type: text/plain; charset="utf-8"

Hi!

I find myself with less time to hack lately and I'm trying to reduce the
number of libraries I maintain so I can invest time in new things. Would
anyone be interested taking over maintenance of network-uri?

It should not be much work at all. Bump a few dependencies and fix a bug or
two. The package is very widely used so I suggest that the new maintainer
should avoid breaking changes whenever possible*.

* In other words, if you want to completely rethink network URI handling,
this package is probably not the place for it.

-- Johan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20150407/da29f7b3/attachment-0001.html>

------------------------------

Message: 8
Date: Wed, 8 Apr 2015 12:38:57 +0300
From: Alexey Shmalko <[hidden email]>
To: Johan Tibell <[hidden email]>
Cc: haskell-cafe <[hidden email]>
Subject: Re: [Haskell-cafe] Anyone interested in taking over
        network-uri?
Message-ID:
        <CAFC2PC4=[hidden email]>
Content-Type: text/plain; charset="utf-8"

Hi, Johan,

I have a little spare time and would be interested in taking over
maintenance of network-uri.

However I'm not sure if I'm a right person for that. I've never maintained
any haskell package before, but would be glad to. I'm experienced with
maintenance issues applied to C/C++ but not Haskell. However, I'm sure they
are pretty much the same.

I looked through github issues; as far as I understand, the maintainer's
responsibilities are tracking things work fine with newer versions of
dependencies, fixing bugs that doesn't break source-level compatibility and
rejecting other proposals. I believe I would handle this.

Regards,
Alexey


On Wed, Apr 8, 2015 at 12:18 AM, Johan Tibell <[hidden email]>
wrote:

> Hi!
>
> I find myself with less time to hack lately and I'm trying to reduce the
> number of libraries I maintain so I can invest time in new things. Would
> anyone be interested taking over maintenance of network-uri?
>
> It should not be much work at all. Bump a few dependencies and fix a bug
> or two. The package is very widely used so I suggest that the new
> maintainer should avoid breaking changes whenever possible*.
>
> * In other words, if you want to completely rethink network URI handling,
> this package is probably not the place for it.
>
> -- Johan
>
>
> _______________________________________________
> Haskell-Cafe mailing list
> [hidden email]
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20150408/9b32ff13/attachment-0001.html>

------------------------------

Subject: Digest Footer

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe


------------------------------

End of Haskell-Cafe Digest, Vol 140, Issue 9
********************************************

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe