Design of an extensible system

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Design of an extensible system

Marc Busqué
Hi there,

I'm currently programming my first non toy project in Haskell. I come from Ruby
background, where everything is possible no matter how much entropy the
universe gains after you finish your work, so I'm still not sure how to do
certain things without this side-effect :)

I'll try to be concise. My idea is that the project would consist of several
different Haskell packages. The central one, call it Core, would define a
certain data type (with info about certain URLs). Other packages would define
each one an strategy to do something with that data type (processing the URL
info). Each strategy would have its own dependencies, and as a user probably
won't need all of the strategies, I would like to keep them in separated
packages so she or he has no need to download the unneeded stuff.

The problem is that Core should act as the central dashboard, whether through a
CLI or even a web frontend. For example, in the CLI case, I would like to be
able to install Core package along with package StrategyA and package StrategyB
and then do something like the following:

```
bash> core --action=fetch --strategy=A
bash> core --action=fetch --strategy=B
```

Two generic ideas come to my mind but I don't know how to translate them
to Haskell:

- Infer module name from the `--strategy` argument and call it dynamically.
   However, as far as I know, in Haskell modules are not first class citizens,
   so no way I can dynamically call them. I have read something to achieve this
   through Template Haskell or through outdaded
   [plugins](http://hackage.haskell.org/package/plugins) package, but I'm not
   sure whether the need is the same and anyway it seems quite unnatural to
   Haskell (I would not like to go against it, but to learn to do thing in its
   way).

- Define from Core some kind of global list where strategy packages can
   register a map between CLI argument value and a defined instance of the core
   data type. But, again, how to do global things in Haskell (and I would prefer
   not to need external tools like RDBMS, filesystem or whatever)? And, anyway,
   surely I don't want to do this because concurrency and everything...

Any idea?

Thanks in advance!

Marc Busqué
http://waiting-for-dev.github.io/about/
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Design of an extensible system

Brandon Allbery
plugins is outdated because the functionality is now in ghc-api and exposed in more convenient ways by the hint package. (Some parts of it were always in ghc-api; plugins provided the parts it originally didn't, notably safe unloading of modules.) That said, dynamic loading is something of a bad fit for Haskell; often, the easiest way to do this is to build a new program including the module and run it in place of the original. See for example how xmonad configuration works. The dyre package also provides a form of this.

Another way to do this is to use programs instead of modules, and the Core driver provides the UI and plumbs the programs together with sockets or other IPC mechanisms. This is slower in general. (Either way requires some form of serialization and marshaling of data, so you will have that overhead regardless.)


On Sun, Jul 22, 2018 at 11:27 AM Marc Busqué <[hidden email]> wrote:
Hi there,

I'm currently programming my first non toy project in Haskell. I come from Ruby
background, where everything is possible no matter how much entropy the
universe gains after you finish your work, so I'm still not sure how to do
certain things without this side-effect :)

I'll try to be concise. My idea is that the project would consist of several
different Haskell packages. The central one, call it Core, would define a
certain data type (with info about certain URLs). Other packages would define
each one an strategy to do something with that data type (processing the URL
info). Each strategy would have its own dependencies, and as a user probably
won't need all of the strategies, I would like to keep them in separated
packages so she or he has no need to download the unneeded stuff.

The problem is that Core should act as the central dashboard, whether through a
CLI or even a web frontend. For example, in the CLI case, I would like to be
able to install Core package along with package StrategyA and package StrategyB
and then do something like the following:

```
bash> core --action=fetch --strategy=A
bash> core --action=fetch --strategy=B
```

Two generic ideas come to my mind but I don't know how to translate them
to Haskell:

- Infer module name from the `--strategy` argument and call it dynamically.
   However, as far as I know, in Haskell modules are not first class citizens,
   so no way I can dynamically call them. I have read something to achieve this
   through Template Haskell or through outdaded
   [plugins](http://hackage.haskell.org/package/plugins) package, but I'm not
   sure whether the need is the same and anyway it seems quite unnatural to
   Haskell (I would not like to go against it, but to learn to do thing in its
   way).

- Define from Core some kind of global list where strategy packages can
   register a map between CLI argument value and a defined instance of the core
   data type. But, again, how to do global things in Haskell (and I would prefer
   not to need external tools like RDBMS, filesystem or whatever)? And, anyway,
   surely I don't want to do this because concurrency and everything...

Any idea?

Thanks in advance!

Marc Busqué
http://waiting-for-dev.github.io/about/_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.


--
brandon s allbery kf8nh                               sine nomine associates
[hidden email]                                  [hidden email]
unix, openafs, kerberos, infrastructure, xmonad        http://sinenomine.net

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Design of an extensible system

John Lato-2
In reply to this post by Marc Busqué
I think you're attempting to optimize the wrong thing. In a dynamic language like Ruby, this approach makes sense because download times are a significant fraction of the user's time, and otherwise the user experience isn't affected much because the code is basically interpreted either way.

Haskell is primarily a compiled language. The total download size probably won't be significantly larger from downloading all modules. The cost of recompiling/dynamically loading plugins will be expensive (relative to the rest of the user time), and payed with every execution. Plus it's additional work for the developer to set up.

In all likelihood you should just build a binary with all available strategies and distribute that.

On Sun, Jul 22, 2018, 08:27 Marc Busqué <[hidden email]> wrote:
Hi there,

I'm currently programming my first non toy project in Haskell. I come from Ruby
background, where everything is possible no matter how much entropy the
universe gains after you finish your work, so I'm still not sure how to do
certain things without this side-effect :)

I'll try to be concise. My idea is that the project would consist of several
different Haskell packages. The central one, call it Core, would define a
certain data type (with info about certain URLs). Other packages would define
each one an strategy to do something with that data type (processing the URL
info). Each strategy would have its own dependencies, and as a user probably
won't need all of the strategies, I would like to keep them in separated
packages so she or he has no need to download the unneeded stuff.

The problem is that Core should act as the central dashboard, whether through a
CLI or even a web frontend. For example, in the CLI case, I would like to be
able to install Core package along with package StrategyA and package StrategyB
and then do something like the following:

```
bash> core --action=fetch --strategy=A
bash> core --action=fetch --strategy=B
```

Two generic ideas come to my mind but I don't know how to translate them
to Haskell:

- Infer module name from the `--strategy` argument and call it dynamically.
   However, as far as I know, in Haskell modules are not first class citizens,
   so no way I can dynamically call them. I have read something to achieve this
   through Template Haskell or through outdaded
   [plugins](http://hackage.haskell.org/package/plugins) package, but I'm not
   sure whether the need is the same and anyway it seems quite unnatural to
   Haskell (I would not like to go against it, but to learn to do thing in its
   way).

- Define from Core some kind of global list where strategy packages can
   register a map between CLI argument value and a defined instance of the core
   data type. But, again, how to do global things in Haskell (and I would prefer
   not to need external tools like RDBMS, filesystem or whatever)? And, anyway,
   surely I don't want to do this because concurrency and everything...

Any idea?

Thanks in advance!

Marc Busqué
http://waiting-for-dev.github.io/about/_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Design of an extensible system

Marc Busqué
On Mon, 23 Jul 2018, John Lato wrote:

> I think you're attempting to optimize the wrong thing. In a dynamic language like Ruby, this approach makes sense because download times are a
> significant fraction of the user's time, and otherwise the user experience isn't affected much because the code is basically interpreted either
> way.
> Haskell is primarily a compiled language. The total download size probably won't be significantly larger from downloading all modules. The cost
> of recompiling/dynamically loading plugins will be expensive (relative to the rest of the user time), and payed with every execution. Plus it's
> additional work for the developer to set up.

What you say makes a lot of sense in terms of optimization. But, what
about extensibility? Take for example the Pandoc package, which would be
similar in design of what I'm trying to do. Say I want to add a
converter to some mark-up format not supported by Pandoc, and that
Pandoc team doesn't agree to merge my work in its repo or simply I don't
have the time to wait until it happens. In this sense, it would be nice
that Pandoc supported dynamically loading of extra plugins on demand.

Marc Busqué
http://waiting-for-dev.github.io/about/
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Design of an extensible system

Paul

> that Pandoc supported dynamically loading of extra plugins on demand.

It means .dll/.so. But also there are applications with pre-built "plugins" - you can not load them dynamically, but can turn-on/off dynamically (from a list of pre-installed plugins). It's useful too.


24.07.2018 09:07, Marc Busqué wrotes:
On Mon, 23 Jul 2018, John Lato wrote:

I think you're attempting to optimize the wrong thing. In a dynamic language like Ruby, this approach makes sense because download times are a
significant fraction of the user's time, and otherwise the user experience isn't affected much because the code is basically interpreted either
way.
Haskell is primarily a compiled language. The total download size probably won't be significantly larger from downloading all modules. The cost
of recompiling/dynamically loading plugins will be expensive (relative to the rest of the user time), and payed with every execution. Plus it's
additional work for the developer to set up.

What you say makes a lot of sense in terms of optimization. But, what
about extensibility? Take for example the Pandoc package, which would be
similar in design of what I'm trying to do. Say I want to add a
converter to some mark-up format not supported by Pandoc, and that
Pandoc team doesn't agree to merge my work in its repo or simply I don't
have the time to wait until it happens. In this sense, it would be nice
that Pandoc supported dynamically loading of extra plugins on demand.

Marc Busqué
http://waiting-for-dev.github.io/about/


_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.


_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Design of an extensible system

Jeremy O'Donoghue


On 24 Jul 2018, at 15:16, Paul <[hidden email]> wrote:

> that Pandoc supported dynamically loading of extra plugins on demand.

It means .dll/.so. But also there are applications with pre-built "plugins" - you can not load them dynamically, but can turn-on/off dynamically (from a list of pre-installed plugins). It's useful too.

Pandoc plug-ins are separate executables rather than shared libraries. Haskell data types are serialised by Pandoc then de-serialised, modified and re-serialised in each plugin.

The Pandoc approach is more like a data pipeline with Pandoc itself at both ends.

Jeremy 

24.07.2018 09:07, Marc Busqué wrotes:
On Mon, 23 Jul 2018, John Lato wrote:

I think you're attempting to optimize the wrong thing. In a dynamic language like Ruby, this approach makes sense because download times are a
significant fraction of the user's time, and otherwise the user experience isn't affected much because the code is basically interpreted either
way.
Haskell is primarily a compiled language. The total download size probably won't be significantly larger from downloading all modules. The cost
of recompiling/dynamically loading plugins will be expensive (relative to the rest of the user time), and payed with every execution. Plus it's
additional work for the developer to set up.

What you say makes a lot of sense in terms of optimization. But, what
about extensibility? Take for example the Pandoc package, which would be
similar in design of what I'm trying to do. Say I want to add a
converter to some mark-up format not supported by Pandoc, and that
Pandoc team doesn't agree to merge my work in its repo or simply I don't
have the time to wait until it happens. In this sense, it would be nice
that Pandoc supported dynamically loading of extra plugins on demand.

Marc Busqué
http://waiting-for-dev.github.io/about/


_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Design of an extensible system

Paul

Hmm, so this is easier even. Plugin must have standard interface, available through stdin/stdout/pipes/sockets. Stdin/stdout/stderr is more simple and it's easy to debug (as first version, at least), also can be "mapped" to sockets easy (in inetd-style, for example). "Driver" can looks like "cmd" module in Python (it's better than to use command line arguments, due to length limitation). Even more, it's easy convertable to other protocols/formats which allows to switch to microservice architecture without big effort ;) (I did it in D: communication can be automatically mapped to JSON/XML/Show-Read/etc). It sounds as interesting task.

I like Powershell: it extends standard way of Unix scripting languages communicating with strings (like Tcl, shell) to objects instead. So binary protocol can be supported too (MessagePack, BSON, Xdr, etc). It's totally language neutral plugin architecture ;) "Pipelining" can be done in WSGI style - with middleware which can help the "plugging" of protocols :)

Dispatching can be achieved with special directory layout/some config file (better).

24.07.2018 10:30, Jeremy O'Donoghue wrotes:


On 24 Jul 2018, at 15:16, Paul <[hidden email]> wrote:

> that Pandoc supported dynamically loading of extra plugins on demand.

It means .dll/.so. But also there are applications with pre-built "plugins" - you can not load them dynamically, but can turn-on/off dynamically (from a list of pre-installed plugins). It's useful too.

Pandoc plug-ins are separate executables rather than shared libraries. Haskell data types are serialised by Pandoc then de-serialised, modified and re-serialised in each plugin.

The Pandoc approach is more like a data pipeline with Pandoc itself at both ends.

Jeremy 

24.07.2018 09:07, Marc Busqué wrotes:
On Mon, 23 Jul 2018, John Lato wrote:

I think you're attempting to optimize the wrong thing. In a dynamic language like Ruby, this approach makes sense because download times are a
significant fraction of the user's time, and otherwise the user experience isn't affected much because the code is basically interpreted either
way.
Haskell is primarily a compiled language. The total download size probably won't be significantly larger from downloading all modules. The cost
of recompiling/dynamically loading plugins will be expensive (relative to the rest of the user time), and payed with every execution. Plus it's
additional work for the developer to set up.

What you say makes a lot of sense in terms of optimization. But, what
about extensibility? Take for example the Pandoc package, which would be
similar in design of what I'm trying to do. Say I want to add a
converter to some mark-up format not supported by Pandoc, and that
Pandoc team doesn't agree to merge my work in its repo or simply I don't
have the time to wait until it happens. In this sense, it would be nice
that Pandoc supported dynamically loading of extra plugins on demand.

Marc Busqué
http://waiting-for-dev.github.io/about/


_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.


_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Design of an extensible system

Marc Busqué
In reply to this post by Paul
On Tue, 24 Jul 2018, Paul wrote:

> you can not load them dynamically, but can turn-on/off dynamically
> (from a list of pre-installed plugins).

Sorry if it's an obvious question, but, how do you do it?

Marc Busqué
http://waiting-for-dev.github.io/about/
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Design of an extensible system

Paul
24.07.2018 13:29, Marc Busqué wrotes:
> On Tue, 24 Jul 2018, Paul wrote:
>
>> you can not load them dynamically, but can turn-on/off dynamically
>> (from a list of pre-installed plugins).
>
> Sorry if it's an obvious question, but, how do you do it?
Depends on architecture. Suppose, plugins will be compiled with the
application and will be "registered" in some container (list, for
example). So, the turn-off will remove them from the list / will mark
them with some flag as "disabled". So, your pipeline will skip them.

As I understand you have cases:
- plugins are DLLs
- plugins are statically linked with application but can be disabled
individually
- plugins are EXEs, so easy can be found/called...

> Marc Busqué
> http://waiting-for-dev.github.io/about/

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Design of an extensible system

Joachim Durchholz
In reply to this post by Marc Busqué
Am 24.07.2018 um 08:07 schrieb Marc Busqué:
>  But, what
> about extensibility? Take for example the Pandoc package, which would be
> similar in design of what I'm trying to do. Say I want to add a
> converter to some mark-up format not supported by Pandoc, and that
> Pandoc team doesn't agree to merge my work in its repo or simply I don't
> have the time to wait until it happens.

You fork Pandoc and add the code you need.
Then you put your extension into a public source repository so that
others can share your work.
And you ask the Pandoc developers to make their code friendly for extension.

I.e. you do the plugins at the source code level, not at the binary level.
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.