status of dynamic loading

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

status of dynamic loading

Evan Laforge
I have an application where I'd like to load and run code at runtime.
As far as I know, the possibilities are:

Use hs-plugins.  I'm not sure if this still works on modern ghc, or if
it works on my platform (darwin intel), but I managed to get this
compiled after "fixing" a use of STArray whose arity apparently
changed, but I probably broke it because now I get "Error in array
index" all the time.  I gather Austin Seipp has been doing some work
on this.  Is there a newer version that should work?  An easy fix for
the error?

Then there's the ghc api, which I gather is seen as the way forward
from hs-plugins, saving it from hi-parsing brittleness and platform
specific breakage.  I think Yi uses this, apparently in in the Shim
package.  I poked around some, but wasn't too clear to me what was
going on, if I was looking in the right place, and if this is a
reasonable way forward.

lambdabot also appears to have its own dynamic loading mechanism, I
haven't looked at the source enough yet to understand what it's doing.
 Maybe there's some documentation out there.

The low-tech solution, used by xmonad, would be to pass the "dynamic"
code to main and just recompile the app for each instance, passing in
an instance specific set of config arguments.  Of course, this isn't
really dynamic at all, and compiling all the various instances could
be clunky and awkward, but I know it works and could possibly be
reasonable if I also write my own little language for the stuff that
really has to be dynamic (anyone got one of these?  lisplike would be
easy and ok, an haskell subset might be nicer...  I would like to
avoid having *two* languages to write extensions in).

The last solution I can think of is to export an RPC API instead of
dynamically loading code.  The difficulty here is coming up with a
vocabulary that can be flattened and shipped to another program
without becoming an excessive maintenance issue or being too
inflexible.


All in all, the dynamic loading approach of hs-plugins seems like the
easiest and most flexible, provided I can get its staged type
checking.  Before I go trying to hack hs-plugins into working, or
poking around inside yi or lambdabot, can anyone point me at some
documentation, resources, advice, etc.?

Thanks!
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: status of dynamic loading

Niklas Broberg
>  Use hs-plugins.  I'm not sure if this still works on modern ghc, or if
>  it works on my platform (darwin intel), but I managed to get this
>  compiled after "fixing" a use of STArray whose arity apparently
>  changed, but I probably broke it because now I get "Error in array
>  index" all the time.  I gather Austin Seipp has been doing some work
>  on this.  Is there a newer version that should work?  An easy fix for
>  the error?

Which version have you tried to use? plugins-1.2 available on hackage
works out of the box with ghc-6.8.2 on both linux and windows, we're
using it for happs-hsp. Haven't tried it on Mac though, but I doubt it
would contain any arity mismatches there, which leads me to believe
that you're using an older one.

Cheers,

/Niklas
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: status of dynamic loading

Gwern Branwen
In reply to this post by Evan Laforge
On 2008.05.20 15:34:07 -0700, Evan Laforge <[hidden email]> scribbled 2.2K characters:
> I have an application where I'd like to load and run code at runtime.
> As far as I know, the possibilities are:
>
> Use hs-plugins.  I'm not sure if this still works on modern ghc, or if
> it works on my platform (darwin intel), but I managed to get this
> compiled after "fixing" a use of STArray whose arity apparently
> changed, but I probably broke it because now I get "Error in array
> index" all the time.

It's probably going to break again soon; at the very least, newer Cabals break it here. So I don't find that surprising. hs-plugins has been problematic for a long time; I don't think anyone could recommend it (unless they're dons) in good conscience - I certainly couldn't.

> I gather Austin Seipp has been doing some work
> on this.  Is there a newer version that should work?  An easy fix for
> the error?

I personally haven't heard of anything; vaguely something with the GHC API comes to mind, but nothing about hs-plugins.

> Then there's the ghc api, which I gather is seen as the way forward
> from hs-plugins, saving it from hi-parsing brittleness and platform
> specific breakage.  I think Yi uses this, apparently in in the Shim
> package.  I poked around some, but wasn't too clear to me what was
> going on, if I was looking in the right place, and if this is a
> reasonable way forward.

GHC API will be the future, at some point, but it's definitely in flux. I know I had a rough time adapting some of the good docs on the 6.6 GHC API to work with 6.8.2; which I suppose is a good thing in the long run, but causes trouble for the here and now.

Yi itself I think uses the API now, yes. The previous reload stuff from 0.3 wasn't too hot, and so JYP went and added in all this Shim stuff I don't understand.

> lambdabot also appears to have its own dynamic loading mechanism, I
> haven't looked at the source enough yet to understand what it's doing.
>  Maybe there's some documentation out there.

So far as I know, lambdabot uses the 'old' architecture Yi was using: <http://www.cse.unsw.edu.au/~dons/papers/SC05.html>. That is, a small kernel which loads in fresh code and data. (On the other hand, the lambdabot.cabal claims to use plugins, so what do I know?)

> The low-tech solution, used by xmonad, would be to pass the "dynamic"
> code to main and just recompile the app for each instance, passing in
> an instance specific set of config arguments.  Of course, this isn't
> really dynamic at all, and compiling all the various instances could
> be clunky and awkward, but I know it works and could possibly be
> reasonable if I also write my own little language for the stuff that
> really has to be dynamic (anyone got one of these?  lisplike would be
> easy and ok, an haskell subset might be nicer...  I would like to
> avoid having *two* languages to write extensions in).

This is actually a very nice solution in many ways; I think it gets you a lot of power and extensibility at a very minimal cost in complexity and bugginess. At least, XMonad has been much better at providing dynamic changes and updates via xmonad.hs, for very few bugs or problems, than anything I see with Yi (which has many, many fewer users).

If you can use this solution, you probably should. I added in an XMonad-style of configuration to the autoproc package <http://hackage.haskell.org/cgi-bin/hackage-scripts/package/autoproc>, and it worked very nicely and was relatively easy to code up.

> The last solution I can think of is to export an RPC API instead of
> dynamically loading code.  The difficulty here is coming up with a
> vocabulary that can be flattened and shipped to another program
> without becoming an excessive maintenance issue or being too
> inflexible.
>
> All in all, the dynamic loading approach of hs-plugins seems like the
> easiest and most flexible, provided I can get its staged type
> checking.  Before I go trying to hack hs-plugins into working, or
> poking around inside yi or lambdabot, can anyone point me at some
> documentation, resources, advice, etc.?
>
> Thanks!
--
gwern
CACI NSO Al Chelsea UKUSA HTCIA insurgency BUDS Ft. Blacklisted

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe

attachment0 (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: status of dynamic loading

Austin Seipp
In reply to this post by Evan Laforge
Excerpts from Evan Laforge's message of Tue May 20 17:34:07 -0500 2008:
> Use hs-plugins.  I'm not sure if this still works on modern ghc, or if
> it works on my platform (darwin intel), but I managed to get this
> compiled after "fixing" a use of STArray whose arity apparently
> changed, but I probably broke it because now I get "Error in array
> index" all the time.

What version of GHC, and were are you pulling the hs-plugins code from?
If you're using at least 6.8 and you do:

darcs get http://code.haskell.org/~dons/code/hs-plugins

It should work just fine. Cale replaced a lot of the brittle parts
(interface file parsing) with supplements that use the GHC API, so it
should be at least a little easier to update, and last time I built it,
it installed and worked nicely (I have heard no word on whether it does
so for darwin/intel.)

If memory serves me, there was a version put on hackage a few weeks ago
from that repository that I would think work just as good.

> I gather Austin Seipp has been doing some work on this.

Just playing around. ;)

> Then there's the ghc api, which I gather is seen as the way forward
> from hs-plugins, saving it from hi-parsing brittleness and platform
> specific breakage.  I think Yi uses this, apparently in in the Shim
> package.  I poked around some, but wasn't too clear to me what was
> going on, if I was looking in the right place, and if this is a
> reasonable way forward.

>From my experiences, getting the GHC API to work properly like I wanted
was a bit of a pain point. In many instances, if I just threw away the
session after changing the code and making a new session and reloading,
the old version is still apparently resides in memory and is still
executed. I spent a bit of time playing with this, etc. etc. but never
got it to fully work.

I did hear later that by simply setting the session you already used to
be empty and then compiling modules into it works like what you could get
from hs-plugins, though (a la load -> unload -> load or just call reload.)

If you still want to give the GHC API a try, I have a package that just
does some of the basic, quick stuff and it's on hackage:

http://hackage.haskell.org/cgi-bin/hackage-scripts/package/metaplug

It's not been updated for a long time though (since way before GHC 6.8.)
The code is really small and the changes should be trivial, though.

There is also hint which embeds GHC into your code and is a wrapper like
my metaplug package:

http://hackage.haskell.org/cgi-bin/hackage-scripts/package/hint

I haven't tried it, but it might be worth investigating if hs-plugins
doesn't turn to work out, and it looks far more well done than mine
(which was forged in a time of haskell-naivety, for the most part.)

--
"It was in the days of the rains that their prayers went up,
not from the fingering of knotted prayer cords or the spinning
of prayer wheels, but from the great pray-machine in the
monastery of Ratri, goddess of the Night."
 Roger Zelazny
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: status of dynamic loading

Evan Laforge
Great stuff everyone, thanks so much.  I did indeed manage to download
an old hs-plugins, and the new one seems to work fine.  I'll
definitely check out hint and metaplug, and for most of the stuff that
doesn't need to be easily changeable at runtime I can do the xmonad
thing.

On the down side, my binary zoomed up to 25 unstripped MB, but since
that's how big ghc is, I sort of expected that.  I wonder if hugs
could be embedded in a ghc program for the same sort of thing?  Idly
wondering, of course.
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|

Re: status of dynamic loading

Spencer Janssen-2
In reply to this post by Gwern Branwen
On Tue, May 20, 2008 at 07:39:12PM -0400, Gwern Branwen wrote:
> It's probably going to break again soon; at the very least, newer Cabals break it here. So I don't find that surprising. hs-plugins has been problematic for a long time; I don't think anyone could recommend it (unless they're dons) in good conscience - I certainly couldn't.

Newer Cabal doesn't actually break hs-plugins, the problem is mixing an older
version of Cabal linked against GHC's API and a newer version of Cabal.  This
is the same version mismatch issue that has become rather common with
ByteString.  To fix the build on my system, I simply replaced "Cabal >= 1.2.3"
with "Cabal == 1.2.3.0".


Cheers,
Spencer Janssen
_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe