Crossreferenced codebase for Stackage LTS 9.2 available

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Crossreferenced codebase for Stackage LTS 9.2 available

Robin Palotai
To find out who else uses your favorite functions and how: http://stuff.codereview.me/lts/9.2 .

For example, click 'async' at http://stuff.codereview.me/lts/9.2/#async-2.1.1.1/Control/Concurrent/Async.hs?corpus&signature&line=175 to get all the callsites of that method.

There are additional files available at http://stuff.codereview.me/ if you want to serve the xrefs locally, or merge your own packages into the index (assuming you use the same stackage lts version).

A request: the crossreferencing is pretty usable, but far from complete. If you find any issues (missing crossreferences or bad ones), please have a look at the buglist at https://github.com/google/haskell-indexer/issues, and thumbs-up any that you find hurting, or open new ones!

Thank you & happy code browsing,
Robin

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Crossreferenced codebase for Stackage LTS 9.2 available

Michael Snoyman
I don't have anything useful to contribute, but just wanted to say: this is really cool, thank you!

On Thu, Aug 31, 2017 at 8:41 AM, Robin Palotai <[hidden email]> wrote:
To find out who else uses your favorite functions and how: http://stuff.codereview.me/lts/9.2 .

For example, click 'async' at http://stuff.codereview.me/lts/9.2/#async-2.1.1.1/Control/Concurrent/Async.hs?corpus&signature&line=175 to get all the callsites of that method.

There are additional files available at http://stuff.codereview.me/ if you want to serve the xrefs locally, or merge your own packages into the index (assuming you use the same stackage lts version).

A request: the crossreferencing is pretty usable, but far from complete. If you find any issues (missing crossreferences or bad ones), please have a look at the buglist at https://github.com/google/haskell-indexer/issues, and thumbs-up any that you find hurting, or open new ones!

Thank you & happy code browsing,
Robin

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.


_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Crossreferenced codebase for Stackage LTS 9.2 available

Ara Adkins
In reply to this post by Robin Palotai
No useful contributions from me, but I just wanted to say that this is a really cool idea! 

_ara

On 31 Aug 2017, at 06:41, Robin Palotai <[hidden email]> wrote:

To find out who else uses your favorite functions and how: http://stuff.codereview.me/lts/9.2 .

For example, click 'async' at http://stuff.codereview.me/lts/9.2/#async-2.1.1.1/Control/Concurrent/Async.hs?corpus&signature&line=175 to get all the callsites of that method.

There are additional files available at http://stuff.codereview.me/ if you want to serve the xrefs locally, or merge your own packages into the index (assuming you use the same stackage lts version).

A request: the crossreferencing is pretty usable, but far from complete. If you find any issues (missing crossreferences or bad ones), please have a look at the buglist at https://github.com/google/haskell-indexer/issues, and thumbs-up any that you find hurting, or open new ones!

Thank you & happy code browsing,
Robin
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Crossreferenced codebase for Stackage LTS 9.2 available

Niklas Hambüchen
In reply to this post by Robin Palotai
Hey Robin,

This is really awesome and useful.

I think it will help a lot to decide whether or not we can
change/deprecate a certain function, as now we can really easily judge
how many callers it has.

How long does that take to generate?
And how did you run it to do it for all of Stackage?

Thanks!

On 31/08/17 06:41, Robin Palotai wrote:
> To find out who else uses your favorite functions and
> how: http://stuff.codereview.me/lts/9.2 .
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Crossreferenced codebase for Stackage LTS 9.2 available

Robin Palotai
Hi Niklas - sounds like a nice usecase.

Though we need to make sure no usages are missing to make that decision (there are some edge cases yet, for example around TH splice application, or type families). Also it's only valid on the set of indexed packages.

I'll merge later, but https://github.com/robinp/haskell-indexer/commit/0592c3d934515c4661c69c5b998e26a543b93b52 has the changes necessary. Note:
- It didn't index ~50 packages
- For some it complained that C libs are missing (like SDL, though I was using the full stack-build Docker image)
- I had to kill a few huge compilations that ate my RAM (Agda, language-python, and some yi packages)

I would estimate 8-12 hours, but didn't follow closely (also it was idling quite some due to the huge compilations).

By the way the per-package entries are available for download in a tarball, so if you just index your local packages and dump the entries along with those of your dependencies into the graphstore, you can get a ~full local index without reexecuting the whole indexing. In the future it would be nice to have tooling that magically fetches the pre-generated entries you need, and merges them locally.

2017-08-31 13:23 GMT+02:00 Niklas Hambüchen <[hidden email]>:
Hey Robin,

This is really awesome and useful.

I think it will help a lot to decide whether or not we can
change/deprecate a certain function, as now we can really easily judge
how many callers it has.

How long does that take to generate?
And how did you run it to do it for all of Stackage?

Thanks!

On 31/08/17 06:41, Robin Palotai wrote:
> To find out who else uses your favorite functions and
> how: http://stuff.codereview.me/lts/9.2 .


_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Crossreferenced codebase for Stackage LTS 9.2 available

Michael Snoyman

Which Docker image did you use for building? The (inappropriately named but hysterical raisins and all that) snoyberg/stackage Docker images are what we use for doing the actual Stackage builds.


On Thu, Aug 31, 2017, 2:37 PM Robin Palotai <[hidden email]> wrote:
Hi Niklas - sounds like a nice usecase.

Though we need to make sure no usages are missing to make that decision (there are some edge cases yet, for example around TH splice application, or type families). Also it's only valid on the set of indexed packages.

I'll merge later, but https://github.com/robinp/haskell-indexer/commit/0592c3d934515c4661c69c5b998e26a543b93b52 has the changes necessary. Note:
- It didn't index ~50 packages
- For some it complained that C libs are missing (like SDL, though I was using the full stack-build Docker image)
- I had to kill a few huge compilations that ate my RAM (Agda, language-python, and some yi packages)

I would estimate 8-12 hours, but didn't follow closely (also it was idling quite some due to the huge compilations).

By the way the per-package entries are available for download in a tarball, so if you just index your local packages and dump the entries along with those of your dependencies into the graphstore, you can get a ~full local index without reexecuting the whole indexing. In the future it would be nice to have tooling that magically fetches the pre-generated entries you need, and merges them locally.

2017-08-31 13:23 GMT+02:00 Niklas Hambüchen <[hidden email]>:
Hey Robin,

This is really awesome and useful.

I think it will help a lot to decide whether or not we can
change/deprecate a certain function, as now we can really easily judge
how many callers it has.

How long does that take to generate?
And how did you run it to do it for all of Stackage?

Thanks!

On 31/08/17 06:41, Robin Palotai wrote:
> To find out who else uses your favorite functions and
> how: http://stuff.codereview.me/lts/9.2 .

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Crossreferenced codebase for Stackage LTS 9.2 available

Robin Palotai
I used fpco/stack-build:lts-9.2. Might explain, thank you!
Is the GHC directory layout the same in
snoyberg/stackage? Just asking since the derived container hijacks the GHC executable for the indexer.

2017-08-31 14:18 GMT+02:00 Michael Snoyman <[hidden email]>:

Which Docker image did you use for building? The (inappropriately named but hysterical raisins and all that) snoyberg/stackage Docker images are what we use for doing the actual Stackage builds.


On Thu, Aug 31, 2017, 2:37 PM Robin Palotai <[hidden email]> wrote:
Hi Niklas - sounds like a nice usecase.

Though we need to make sure no usages are missing to make that decision (there are some edge cases yet, for example around TH splice application, or type families). Also it's only valid on the set of indexed packages.

I'll merge later, but https://github.com/robinp/haskell-indexer/commit/0592c3d934515c4661c69c5b998e26a543b93b52 has the changes necessary. Note:
- It didn't index ~50 packages
- For some it complained that C libs are missing (like SDL, though I was using the full stack-build Docker image)
- I had to kill a few huge compilations that ate my RAM (Agda, language-python, and some yi packages)

I would estimate 8-12 hours, but didn't follow closely (also it was idling quite some due to the huge compilations).

By the way the per-package entries are available for download in a tarball, so if you just index your local packages and dump the entries along with those of your dependencies into the graphstore, you can get a ~full local index without reexecuting the whole indexing. In the future it would be nice to have tooling that magically fetches the pre-generated entries you need, and merges them locally.

2017-08-31 13:23 GMT+02:00 Niklas Hambüchen <[hidden email]>:
Hey Robin,

This is really awesome and useful.

I think it will help a lot to decide whether or not we can
change/deprecate a certain function, as now we can really easily judge
how many callers it has.

How long does that take to generate?
And how did you run it to do it for all of Stackage?

Thanks!

On 31/08/17 06:41, Robin Palotai wrote:
> To find out who else uses your favorite functions and
> how: http://stuff.codereview.me/lts/9.2 .

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.


_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Crossreferenced codebase for Stackage LTS 9.2 available

Michael Snoyman
It looks like it should be using the same method of installing GHC (Herbert's PPA). I'm not sure what you mean by hijacking the GHC executable though.

On Thu, Aug 31, 2017 at 3:27 PM, Robin Palotai <[hidden email]> wrote:
I used fpco/stack-build:lts-9.2. Might explain, thank you!
Is the GHC directory layout the same in
snoyberg/stackage? Just asking since the derived container hijacks the GHC executable for the indexer.

2017-08-31 14:18 GMT+02:00 Michael Snoyman <[hidden email]>:

Which Docker image did you use for building? The (inappropriately named but hysterical raisins and all that) snoyberg/stackage Docker images are what we use for doing the actual Stackage builds.


On Thu, Aug 31, 2017, 2:37 PM Robin Palotai <[hidden email]> wrote:
Hi Niklas - sounds like a nice usecase.

Though we need to make sure no usages are missing to make that decision (there are some edge cases yet, for example around TH splice application, or type families). Also it's only valid on the set of indexed packages.

I'll merge later, but https://github.com/robinp/haskell-indexer/commit/0592c3d934515c4661c69c5b998e26a543b93b52 has the changes necessary. Note:
- It didn't index ~50 packages
- For some it complained that C libs are missing (like SDL, though I was using the full stack-build Docker image)
- I had to kill a few huge compilations that ate my RAM (Agda, language-python, and some yi packages)

I would estimate 8-12 hours, but didn't follow closely (also it was idling quite some due to the huge compilations).

By the way the per-package entries are available for download in a tarball, so if you just index your local packages and dump the entries along with those of your dependencies into the graphstore, you can get a ~full local index without reexecuting the whole indexing. In the future it would be nice to have tooling that magically fetches the pre-generated entries you need, and merges them locally.

2017-08-31 13:23 GMT+02:00 Niklas Hambüchen <[hidden email]>:
Hey Robin,

This is really awesome and useful.

I think it will help a lot to decide whether or not we can
change/deprecate a certain function, as now we can really easily judge
how many callers it has.

How long does that take to generate?
And how did you run it to do it for all of Stackage?

Thanks!

On 31/08/17 06:41, Robin Palotai wrote:
> To find out who else uses your favorite functions and
> how: http://stuff.codereview.me/lts/9.2 .

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.



_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Crossreferenced codebase for Stackage LTS 9.2 available

Robin Palotai
I mean the build/indexing scripts takes the GHC in the container, and replaces it with the indexer wrapper (that will call both the original GHC, and the indexer). This is to capture the compiler arguments.

If the paths are the same, it's easy to adopt to the other container.

2017-08-31 16:06 GMT+02:00 Michael Snoyman <[hidden email]>:
It looks like it should be using the same method of installing GHC (Herbert's PPA). I'm not sure what you mean by hijacking the GHC executable though.

On Thu, Aug 31, 2017 at 3:27 PM, Robin Palotai <[hidden email]> wrote:
I used fpco/stack-build:lts-9.2. Might explain, thank you!
Is the GHC directory layout the same in
snoyberg/stackage? Just asking since the derived container hijacks the GHC executable for the indexer.

2017-08-31 14:18 GMT+02:00 Michael Snoyman <[hidden email]>:

Which Docker image did you use for building? The (inappropriately named but hysterical raisins and all that) snoyberg/stackage Docker images are what we use for doing the actual Stackage builds.


On Thu, Aug 31, 2017, 2:37 PM Robin Palotai <[hidden email]> wrote:
Hi Niklas - sounds like a nice usecase.

Though we need to make sure no usages are missing to make that decision (there are some edge cases yet, for example around TH splice application, or type families). Also it's only valid on the set of indexed packages.

I'll merge later, but https://github.com/robinp/haskell-indexer/commit/0592c3d934515c4661c69c5b998e26a543b93b52 has the changes necessary. Note:
- It didn't index ~50 packages
- For some it complained that C libs are missing (like SDL, though I was using the full stack-build Docker image)
- I had to kill a few huge compilations that ate my RAM (Agda, language-python, and some yi packages)

I would estimate 8-12 hours, but didn't follow closely (also it was idling quite some due to the huge compilations).

By the way the per-package entries are available for download in a tarball, so if you just index your local packages and dump the entries along with those of your dependencies into the graphstore, you can get a ~full local index without reexecuting the whole indexing. In the future it would be nice to have tooling that magically fetches the pre-generated entries you need, and merges them locally.

2017-08-31 13:23 GMT+02:00 Niklas Hambüchen <[hidden email]>:
Hey Robin,

This is really awesome and useful.

I think it will help a lot to decide whether or not we can
change/deprecate a certain function, as now we can really easily judge
how many callers it has.

How long does that take to generate?
And how did you run it to do it for all of Stackage?

Thanks!

On 31/08/17 06:41, Robin Palotai wrote:
> To find out who else uses your favorite functions and
> how: http://stuff.codereview.me/lts/9.2 .

_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.




_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Crossreferenced codebase for Stackage LTS 9.2 available

Mihai Maruseac
In reply to this post by Robin Palotai
Hi,

This sounds great.

Can you prepare an entry for HCAR
(https://wiki.haskell.org/Haskell_Communities_and_Activities_Report)
for this project? Seems like a good addition to the ecosystem.

On Wed, Aug 30, 2017 at 10:41 PM, Robin Palotai <[hidden email]> wrote:

> To find out who else uses your favorite functions and how:
> http://stuff.codereview.me/lts/9.2 .
>
> For example, click 'async' at
> http://stuff.codereview.me/lts/9.2/#async-2.1.1.1/Control/Concurrent/Async.hs?corpus&signature&line=175
> to get all the callsites of that method.
>
> There are additional files available at http://stuff.codereview.me/ if you
> want to serve the xrefs locally, or merge your own packages into the index
> (assuming you use the same stackage lts version).
>
> A request: the crossreferencing is pretty usable, but far from complete. If
> you find any issues (missing crossreferences or bad ones), please have a
> look at the buglist at https://github.com/google/haskell-indexer/issues, and
> thumbs-up any that you find hurting, or open new ones!
>
> Thank you & happy code browsing,
> Robin
>
> _______________________________________________
> Haskell-Cafe mailing list
> To (un)subscribe, modify options or view archives go to:
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
> Only members subscribed via the mailman list are allowed to post.



--
Mihai Maruseac (MM)
"If you can't solve a problem, then there's an easier problem you can
solve: find it." -- George Polya
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Crossreferenced codebase for Stackage LTS 9.2 available

Joachim Durchholz
In reply to this post by Niklas Hambüchen
Am 31.08.2017 um 13:23 schrieb Niklas Hambüchen:
> I think it will help a lot to decide whether or not we can
> change/deprecate a certain function, as now we can really easily judge
> how many callers it has.

It does not (and cannot) tell you about caller code that is not on Hackage.
If you have a low-level library then that information is still better
than nothing, but if your code is a public API this information is
almost worse than nothing.

Just saying, having seen some people deprecate stuff just to be caught
in the backlash :-)

HTH
Jo
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.
Reply | Threaded
Open this post in threaded view
|

Re: Crossreferenced codebase for Stackage LTS 9.2 available

Niklas Hambüchen
I consider usages on Hackage an OK estimator of usages outside of
Stackage, certainly enough to get a first expression.

Also, with haskell-indexer becoming really easy to use, the maintainers
of off-Hackage code bases can themselves more easily determine whether
they use functions in question and chime into the discussion with real data.

On 31/08/17 19:31, Joachim Durchholz wrote:
> It does not (and cannot) tell you about caller code that is not on Hackage.
> If you have a low-level library then that information is still better
> than nothing, but if your code is a public API this information is
> almost worse than nothing.
_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
Only members subscribed via the mailman list are allowed to post.