Quantcast

ANN: Portackage - a hackage portal

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

ANN: Portackage - a hackage portal

Andrew Seniuk
Hello,

This is to announce a provisional webpage to summarise hackage.haskell.org.

http://www.fremissant.net/portackage

It is a lot of data, so will be a bit slow to load. Columns can be
sorted by clicking the headers, but this will also take some seconds
on a typical client. The initial sort is by module name.  Clicking a
table row links to the project page on hackage.

Although I believe this is a complete list of all packages/modules in
hackage, it is probably still buggy, so if you see anything amiss
please let me know. Also, a subsequent version in the near future will
update automatically based on the
http://hackage.haskell.org/packages/archive/recent.rss RSS feed, but
for now it's based on a recent snapshot. Haskell code which prepares
the page will be released after a few more days testing and
refinement.

Some previous work [1] provided a list of all modules, but is not
being maintained, and I was unable to rearouse the author's interest.

Feature requests and suggestions are welcome!

Kind Reg'ds,
Andrew Seniuk

[1] http://folk.ntnu.no/hammar/explore-hackage/

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Haskell] ANN: Portackage - a hackage portal

Andrew Seniuk
Jonathan,

On Thu, Apr 26, 2012 at 4:13 PM, Jonathan Daugherty <[hidden email]> wrote:
>> Although I believe this is a complete list of all packages/modules in
>> hackage, it is probably still buggy, so if you see anything amiss
>> please let me know.
>
> It might not be showing the most recent releases of everything.  As an
> example, it shows a much older version of my vty-ui library (1.0) and
> its metadata, instead of the latest version, 1.5.  As a result the
> version, module list, description, homepage, etc., are not current.

Thank-you, I'd realised this issue but forgot to address it. It is
due to a false assumption on my part that the .cabal entries in the
00-index.tar file were in increasing version order.

I'll address this problem today.

Cheers,
Andrew

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Haskell] ANN: Portackage - a hackage portal

Andrew Seniuk
Should be fixed in that respect now, thanks again for pointing it out.

On Thu, Apr 26, 2012 at 4:23 PM, Andrew Seniuk <[hidden email]> wrote:

> Jonathan,
>
> On Thu, Apr 26, 2012 at 4:13 PM, Jonathan Daugherty <[hidden email]> wrote:
>>> Although I believe this is a complete list of all packages/modules in
>>> hackage, it is probably still buggy, so if you see anything amiss
>>> please let me know.
>>
>> It might not be showing the most recent releases of everything.  As an
>> example, it shows a much older version of my vty-ui library (1.0) and
>> its metadata, instead of the latest version, 1.5.  As a result the
>> version, module list, description, homepage, etc., are not current.
>
> Thank-you, I'd realised this issue but forgot to address it. It is
> due to a false assumption on my part that the .cabal entries in the
> 00-index.tar file were in increasing version order.
>
> I'll address this problem today.
>
> Cheers,
> Andrew

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ANN: Portackage - a hackage portal

Andrew Seniuk
In reply to this post by Andrew Seniuk
Somewhat lighter, using summarised module lists. A more sophistocated
UI with per-column filtering is in the works. I invite your comments.

Andrew

On Thu, Apr 26, 2012 at 4:05 PM, Andrew Seniuk <[hidden email]> wrote:

> Hello,
>
> This is to announce a provisional webpage to summarise hackage.haskell.org.
>
> http://www.fremissant.net/portackage
>
> It is a lot of data, so will be a bit slow to load. Columns can be
> sorted by clicking the headers, but this will also take some seconds
> on a typical client. The initial sort is by module name.  Clicking a
> table row links to the project page on hackage.
>
> Although I believe this is a complete list of all packages/modules in
> hackage, it is probably still buggy, so if you see anything amiss
> please let me know. Also, a subsequent version in the near future will
> update automatically based on the
> http://hackage.haskell.org/packages/archive/recent.rss RSS feed, but
> for now it's based on a recent snapshot. Haskell code which prepares
> the page will be released after a few more days testing and
> refinement.
>
> Some previous work [1] provided a list of all modules, but is not
> being maintained, and I was unable to rearouse the author's interest.
>
> Feature requests and suggestions are welcome!
>
> Kind Reg'ds,
> Andrew Seniuk
>
> [1] http://folk.ntnu.no/hammar/explore-hackage/

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Haskell] ANN: Portackage - a hackage portal

Andrew Seniuk
Excuse the delay, I have no home internet.

> On Fri, Apr 27, 2012 at 1:16 AM,  <[hidden email]> wrote:
> Great idea.  Maybe the format can start as compactly as possible,
> i.e. 1 entry per line to make it easy to browse as many entries
> as possible in one page.
>
> It looks as though disabling just the dependencies would go a
> long way to accomplish that.

Yep, and in fact the version I uploaded while reading your
message has the deps column elided (and one or two others).
Enforcing single-line rows is a but a few tweaks away.

The thing is, the module namespace was the main motivator for
this project, so I'm also seeing whether I can get away with a
single view, package-oriented as presently, and still give good
focus to the modules names.

Soon I hope to include "accordion" views of the module namespace,
with cookie-based (?) persistent state so user's collapse/expand
tree state is preserved.

> I'd like to vote to make the synopsis to be the second column.
>
> will each column type have an enable/disable ?

I like the idea (although that one wasn't in the upload).
Hopefully when I explore the JavaScript UI tools a bit more,
the columns (key structure) will be totally configurable
by drag-and-drop, with columns sizable, and keys selected from
a listbox. (?) stuff like that

> it's probably best if a click on a specific target, say the actual
> name of the package, sends you to the hackage entry instead of a
> click anywhere on the row.  I find myself clicking on the page
> by accident and it sending me off to some random hackage entry.

:) That's a feature!

But seriously, you're right, and as a bonus the links can be
specific to the columns (package -> hackage, modules -> API pages,
homepages -> hps, maintainer -> sendto [1]).

(A 2-finger or Mod- click anywhere on a row might still be nice, but
a bare tap is admittedly way too accident prone with touch input.)

> I really like the idea - makes your hackage surfing
> much more enjoyable !
>
> Brian

And thank-you for sharing your thoughts! I'm committed to improving
it, until the ideas run out.

> On Fri, Apr 27, 2012 at 4:12 AM, Christoph Breitkopf <[hidden email]> wrote:
> this already looks great. So here's my wish-list, in case you have
> too much time on yours hands ;-)

Thanks! Yes I suppose I do have too much time on my hands, but the
pain of that is conterbalanced by all the excitement.

> Quality/popularity indicators (I'm aware that some of these are
> real work and better provided by hackage itself):
> - used by how many other packages (number of edges to that package
>    in the dependency graph of all packages)
> - haddock coverage, test coverage, etc.
> - like, +1, whatnot button
> - link to issue tracker, if there is one

If there's interest, this could become integrated with hackage
at some point in the future.

Next thing is to get a "last upload" field which can replace
the version field. The codependency graph :) would be a nice metric
to add, I'll look into it. I remember seeing a package that does
this someplace on hackage. The +1 button is a good one!

In short I'll try to implement all these ideas, thanks to you both
for the encouraging comments.

-Andrew

[1] I omitted the maintainer column, thinking its inclusion might
     be harmful, making the emails so easy for a bot to collect.

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Haskell] ANN: Portackage - a hackage portal

Andrew Seniuk
Hi Derek,

Thanks for your comments. Yes, I am probably going to focus on making
it work as a web service and stay away from platform-specific mobile
apps. Not been thinking mobile much at all, since the quantities of
data are taxing even to my laptop, and also I'm inclined to believe
people are mostly at a laptop or desktop when coding Haskell.

However, I'm planning to try server-side sorting and filtering, and
compare performance. No DB is used on the server yet, but it would be
simple enough I suppose -- all this is quite new territory for me.
Also my shared hosting resource limits may become a problem if this
gets a lot of use.

On the other hand, was planning to use persistence (cookies i guess?)
to save the data on the client, so that patches based on Hackage RSS
could be applied incrementally. Also, I think the sorting could be
faster by sending per-column lists of indices. But maybe it is just
the browser rendering which is taking all the time?

I'll take this opportunity to announce a few improvements:
 - shift-click a row to see the Hackage page (bare click does nothing)
 - per-column filtering widgets

Presently working on getting the "last updated" column, this
information is not in a .cabal file but I'm hoping it is in the TAR
metadata...

The link again:  http://www.fremissant.net/portackage

A.

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Haskell] ANN: Portackage - a hackage portal

Andrew Seniuk
> On Apr 27, 2012 6:18 PM, "Andrew Seniuk" <[hidden email]> wrote:
> Oh, the idea wasn't that you would make the app, but that by providing a web
> service interface you would make it easier for me or anyone else to make an
> app or other alternative interface.  It would also reduce bandwidth usage
> over screen scraping.

Yes, well as I mentioned in the part of my response you didn't quote,
I agree it would be desireable, and am working on it, but my
motivation was to provide people with something immediately useful.
Once I get something reasonably efficient worked out, I'll look at
uploading the Haskell code to Hackage, providing JSON data, etc..
Meanwhile if someone wanted to work on their own app or interface,
it's not too hard to get CSV from the table.

I made a mistake and the link was down for about 12 hours.  It's back
now, with a number of improvements and up-to-date hackage info.  It is
an expensive page to run, you may get "Script not responding" pop-ups
depending on your hardware, but I think it's worth it.

http://fremissant.net/portackage

Kind Reg'ds,
Andrew

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Haskell] ANN: Portackage - a hackage portal

Simon Michael
Great work! Thanks.

Does it include only packages without executables ? Eg I see hledger-lib but not the hledger or gist packages. It would
be nice to have all of hackage there.

Best - Simon


_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Haskell] ANN: Portackage - a hackage portal

Andrew Seniuk
First let me mention, there is now a complete modules tree with API
links (in a very preliminary state), at
http://fremissant.net/portackage/modules.php and the package view is
still http://fremissant.net/portackage

On Mon, Apr 30, 2012 at 11:35 AM, Simon Michael <[hidden email]> wrote:
> Great work! Thanks.
>
> Does it include only packages without executables ? Eg I see
> hledger-lib but not the hledger or gist packages. It would be nice
> to have all of hackage there.

Simon, thank you. And, yes, as a temporary measure, I elided the
packages that expose no modules, to try to tame the slowness a bit. I
could provide them on a similar, separate page, until figure out some
optimisations.

Derek, if you're reading, sorry if I seemed defensive there before. :)

Okay, this bagel is screaming "Eat Me!!"...

Kind Reg'ds,
Andrew

On Mon, Apr 30, 2012 at 11:35 AM, Simon Michael <[hidden email]> wrote:

> Great work! Thanks.
>
> Does it include only packages without executables ? Eg I see hledger-lib but
> not the hledger or gist packages. It would be nice to have all of hackage
> there.
>
> Best - Simon
>
>
> _______________________________________________
> Haskell-Cafe mailing list
> [hidden email]
> http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Haskell] ANN: Portackage - a hackage portal

Andrew Seniuk
...and the "binary" packages (those exposing no modules) are now up at
http://fremissant.net/portackage/binary.php

Not nice to have them in a separate table, but for now that's the case. :(

So the three links in summary:

module tree: http://fremissant.net/portackage/modules.php
library packages: http://fremissant.net/portackage/binary.php
other packages: http://fremissant.net/portackage/portackage.php

On Mon, Apr 30, 2012 at 1:42 PM, Andrew Seniuk <[hidden email]> wrote:

> First let me mention, there is now a complete modules tree with API
> links (in a very preliminary state), at
> http://fremissant.net/portackage/modules.php and the package view is
> still http://fremissant.net/portackage
>
> On Mon, Apr 30, 2012 at 11:35 AM, Simon Michael <[hidden email]> wrote:
>> Great work! Thanks.
>>
>> Does it include only packages without executables ? Eg I see
>> hledger-lib but not the hledger or gist packages. It would be nice
>> to have all of hackage there.
>
> Simon, thank you. And, yes, as a temporary measure, I elided the
> packages that expose no modules, to try to tame the slowness a bit. I
> could provide them on a similar, separate page, until figure out some
> optimisations.
>
> Derek, if you're reading, sorry if I seemed defensive there before. :)
>
> Okay, this bagel is screaming "Eat Me!!"...
>
> Kind Reg'ds,
> Andrew
>
> On Mon, Apr 30, 2012 at 11:35 AM, Simon Michael <[hidden email]> wrote:
>> Great work! Thanks.
>>
>> Does it include only packages without executables ? Eg I see hledger-lib but
>> not the hledger or gist packages. It would be nice to have all of hackage
>> there.
>>
>> Best - Simon
>>
>>
>> _______________________________________________
>> Haskell-Cafe mailing list
>> [hidden email]
>> http://www.haskell.org/mailman/listinfo/haskell-cafe

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Haskell] ANN: Portackage - a hackage portal

Andrew Seniuk
Oops, the three links in summary are:

module tree: http://fremissant.net/portackage/modules.php
library packages: http://fremissant.net/portackage/portackage.php
other packages: http://fremissant.net/portackage/binary.php

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ANN: Portackage - a hackage portal

Andrew Seniuk
Hi ___,

Thank you for your suggestion to provide JSON, which seconds Derek's
earlier suggestion. I have anonymised your name since you wrote to me
but didn't CC the list. :)

As much as I wish the web interface was up to snuff, it is really too
slow. While I continue to try to resolve that, there is something like
a JSON feed available now. I'm new to these things and may be doing it
wrong. Any advice would be appreciated, either in accelerating the web
interface or in providing the JSON. I have no idea whether it parses
or not.

JSON Schema:
http://www.fremissant.net/portackage/packages.schema.json.txt

Hopefully compressed JSON data:
http://www.fremissant.net/portackage/packages.php

Raw JSON data in case the compressed doesn't work:
http://www.fremissant.net/portackage/packages.json

The hackage data updated as of today (June 14).

For now this is only the packages which expose modules (i.e. not
including binary packages). No particular reason, I just ran out of
time this afternoon and thought I'd post what I've got.

The fields present in the JSON are those available on the HTML table
version. There are several dozen other fields which come up more or
less infrequently in the Hackage packages, and they could all be
included. The complete list of modules is included (rather than the
summary versions used in the HTML table version).

A next step would be for the server to incrementally update the JSON
data based on the Hackage RSS feed.

Kind Regards,
Andrew

On Wed, Jun 13, 2012 at 7:58 AM, ___ wrote:

> Hi Andrew
>
> Portackage is a great site for browsing packages, but it's really
> really slow for filtering per column. What about offering an
> additional non-javascript version of the pages -- searching with plain
> text in a browser works quite well.
>
> On 4/30/12, Andrew Seniuk <[hidden email]> wrote:
>> Oops, the three links in summary are:
>>
>> module tree: http://fremissant.net/portackage/modules.php
>> library packages: http://fremissant.net/portackage/portackage.php
>> other packages: http://fremissant.net/portackage/binary.php
>>
>> _______________________________________________
>> Haskell-Cafe mailing list
>> [hidden email]
>> http://www.haskell.org/mailman/listinfo/haskell-cafe
>>
>
>
> --
> ___

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ANN: Portackage - a hackage portal

Andrew Seniuk
I discovered http://jsonlint.com/ and got it to validate. :)

On Thu, Jun 14, 2012 at 6:18 PM, Andrew Seniuk <[hidden email]> wrote:

> Hi ___,
>
> Thank you for your suggestion to provide JSON, which seconds Derek's
> earlier suggestion. I have anonymised your name since you wrote to me
> but didn't CC the list. :)
>
> As much as I wish the web interface was up to snuff, it is really too
> slow. While I continue to try to resolve that, there is something like
> a JSON feed available now. I'm new to these things and may be doing it
> wrong. Any advice would be appreciated, either in accelerating the web
> interface or in providing the JSON. I have no idea whether it parses
> or not.
>
> JSON Schema:
> http://www.fremissant.net/portackage/packages.schema.json.txt
>
> Hopefully compressed JSON data:
> http://www.fremissant.net/portackage/packages.php
>
> Raw JSON data in case the compressed doesn't work:
> http://www.fremissant.net/portackage/packages.json
>
> The hackage data updated as of today (June 14).
>
> For now this is only the packages which expose modules (i.e. not
> including binary packages). No particular reason, I just ran out of
> time this afternoon and thought I'd post what I've got.
>
> The fields present in the JSON are those available on the HTML table
> version. There are several dozen other fields which come up more or
> less infrequently in the Hackage packages, and they could all be
> included. The complete list of modules is included (rather than the
> summary versions used in the HTML table version).
>
> A next step would be for the server to incrementally update the JSON
> data based on the Hackage RSS feed.
>
> Kind Regards,
> Andrew
>
> On Wed, Jun 13, 2012 at 7:58 AM, ___ wrote:
>> Hi Andrew
>>
>> Portackage is a great site for browsing packages, but it's really
>> really slow for filtering per column. What about offering an
>> additional non-javascript version of the pages -- searching with plain
>> text in a browser works quite well.
>>
>> On 4/30/12, Andrew Seniuk <[hidden email]> wrote:
>>> Oops, the three links in summary are:
>>>
>>> module tree: http://fremissant.net/portackage/modules.php
>>> library packages: http://fremissant.net/portackage/portackage.php
>>> other packages: http://fremissant.net/portackage/binary.php
>>>
>>> _______________________________________________
>>> Haskell-Cafe mailing list
>>> [hidden email]
>>> http://www.haskell.org/mailman/listinfo/haskell-cafe
>>>
>>
>>
>> --
>> ___

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ANN: Portackage - a hackage portal

Andrew Seniuk
Now all packages, including those without exposed modules, are
included. Also, I've attempted to parse the author lists into JSON
arrays. Due to the free format of many fields in the .cabal package
descriptions, including author list, there are many exceptional cases
where they are ill-parsed, but at least the JSON file is well-formed
and validates at jsonlint.com.

Once more:

JSON Schema:
http://www.fremissant.net/portackage/packages.schema.json.txt

Hopefully compressed JSON data:
http://www.fremissant.net/portackage/packages.php

Raw JSON data in case the compressed doesn't work:
http://www.fremissant.net/portackage/packages.json

The hackage data updated as of yesterday (June 14). Will work on daily
incremental updates from Hackage RSS feed soon.

-Andrew

On Thu, Jun 14, 2012 at 7:05 PM, Andrew Seniuk <[hidden email]> wrote:

> I discovered http://jsonlint.com/ and got it to validate. :)
>
> On Thu, Jun 14, 2012 at 6:18 PM, Andrew Seniuk <[hidden email]> wrote:
>> Hi ___,
>>
>> Thank you for your suggestion to provide JSON, which seconds Derek's
>> earlier suggestion. I have anonymised your name since you wrote to me
>> but didn't CC the list. :)
>>
>> As much as I wish the web interface was up to snuff, it is really too
>> slow. While I continue to try to resolve that, there is something like
>> a JSON feed available now. I'm new to these things and may be doing it
>> wrong. Any advice would be appreciated, either in accelerating the web
>> interface or in providing the JSON. I have no idea whether it parses
>> or not.
>>
>> JSON Schema:
>> http://www.fremissant.net/portackage/packages.schema.json.txt
>>
>> Hopefully compressed JSON data:
>> http://www.fremissant.net/portackage/packages.php
>>
>> Raw JSON data in case the compressed doesn't work:
>> http://www.fremissant.net/portackage/packages.json
>>
>> The hackage data updated as of today (June 14).
>>
>> For now this is only the packages which expose modules (i.e. not
>> including binary packages). No particular reason, I just ran out of
>> time this afternoon and thought I'd post what I've got.
>>
>> The fields present in the JSON are those available on the HTML table
>> version. There are several dozen other fields which come up more or
>> less infrequently in the Hackage packages, and they could all be
>> included. The complete list of modules is included (rather than the
>> summary versions used in the HTML table version).
>>
>> A next step would be for the server to incrementally update the JSON
>> data based on the Hackage RSS feed.
>>
>> Kind Regards,
>> Andrew
>>
>> On Wed, Jun 13, 2012 at 7:58 AM, ___ wrote:
>>> Hi Andrew
>>>
>>> Portackage is a great site for browsing packages, but it's really
>>> really slow for filtering per column. What about offering an
>>> additional non-javascript version of the pages -- searching with plain
>>> text in a browser works quite well.
>>>
>>> On 4/30/12, Andrew Seniuk <[hidden email]> wrote:
>>>> Oops, the three links in summary are:
>>>>
>>>> module tree: http://fremissant.net/portackage/modules.php
>>>> library packages: http://fremissant.net/portackage/portackage.php
>>>> other packages: http://fremissant.net/portackage/binary.php
>>>>
>>>> _______________________________________________
>>>> Haskell-Cafe mailing list
>>>> [hidden email]
>>>> http://www.haskell.org/mailman/listinfo/haskell-cafe
>>>>
>>>
>>>
>>> --
>>> ___

_______________________________________________
Haskell-Cafe mailing list
[hidden email]
http://www.haskell.org/mailman/listinfo/haskell-cafe
Loading...