Registering GHC for Coverity SCAN

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Registering GHC for Coverity SCAN

Austin Seipp-4
Hello all,

On IRC today, Nicolas Trangez brought up the idea of registering GHC
for Coverity's SCAN project. SCAN is essentially a free service run by
Coverity, which runs their Static Analyzer on open source projects
('open source' being defined by OSI) and gives the results back to
developers. Coverity is widely regarded as having the absolute best
tool in the business for C codebases as it stands. The idea is to try
and rattle bugs out of the runtime system if possible.

I think this is a good idea, but it needs some public discussion.
Namely, Coverity requires an official Point of Contact within the
project (maintainer/author/governing body) to register the project in
their name. We are left up to determine who this is and verify it. Bug
reports which are found are not also made public, as they could be
potentially serious security vulnerabilities.*

So, if done, this is a rather piece-by-piece project which requires a
bit of commitment, because it will require the registree to not only
assess the bugs, but sensibly move them into the issue tracker or fix
it themselves.

It's also a question of whether it's worth it. Personally, I think it
has the chance to uncover some real bugs. But the RTS is highly
unusual C code (by some standards) and it will probably require tuning
and patience to get tangible results. Bugs will also need to be
assessed by someone. False positives happen etc. This is the double
edged sword of static analysis.** Anyone who does it should be very
aware of this.

So, what do you all think? I'd particularly like input from Simon M
here since he's the primary author of the RTS, of course. :) And
anyone who does this will undoubtedly have to be in contact with him
to some degree, for all the aforementioned reasons.

If nobody else would step up for this, I would be interested in
contacting them if it seems worth it and giving it a shot.

* I think the chance of this in our case is very very low of course.
The cases we really care about for a safety perspective are
type-safety violations at the Haskell level, obviously. Nonetheless
this requires an individual who is committed to sifting through
reports and assessing them in a detailed way.

** Coverity at one point found out the hard way that more analysis is
not necessarily better, and people don't respond well to the results
sometimes. Or they're too complicated to explain. The following is a
great read: http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later/fulltext

--
Regards,
Austin - PGP: 4096R/0x91384671


Reply | Threaded
Open this post in threaded view
|

Registering GHC for Coverity SCAN

Edward Z. Yang
Austin brings up an interesting point, which is that the last time
we had a security vulnerability situation was when it was discovered
that CGI scripts could pass +RTS options to Haskell programs and
get programs to do things that they shouldn't.  And, AFAICT, we haven't
ever had any CVEs issued against GHC.  So we don't really have any
infrastructure in place for dealing with security problems (not even
a "private security bug tracker").

While in principle having this infrastructure is probably a good idea,
I don't think it should block a motivated person who wants to do
some Coverity analysis.

Edward

Excerpts from Austin Seipp's message of Wed May 08 17:36:49 -0700 2013:

> Hello all,
>
> On IRC today, Nicolas Trangez brought up the idea of registering GHC
> for Coverity's SCAN project. SCAN is essentially a free service run by
> Coverity, which runs their Static Analyzer on open source projects
> ('open source' being defined by OSI) and gives the results back to
> developers. Coverity is widely regarded as having the absolute best
> tool in the business for C codebases as it stands. The idea is to try
> and rattle bugs out of the runtime system if possible.
>
> I think this is a good idea, but it needs some public discussion.
> Namely, Coverity requires an official Point of Contact within the
> project (maintainer/author/governing body) to register the project in
> their name. We are left up to determine who this is and verify it. Bug
> reports which are found are not also made public, as they could be
> potentially serious security vulnerabilities.*
>
> So, if done, this is a rather piece-by-piece project which requires a
> bit of commitment, because it will require the registree to not only
> assess the bugs, but sensibly move them into the issue tracker or fix
> it themselves.
>
> It's also a question of whether it's worth it. Personally, I think it
> has the chance to uncover some real bugs. But the RTS is highly
> unusual C code (by some standards) and it will probably require tuning
> and patience to get tangible results. Bugs will also need to be
> assessed by someone. False positives happen etc. This is the double
> edged sword of static analysis.** Anyone who does it should be very
> aware of this.
>
> So, what do you all think? I'd particularly like input from Simon M
> here since he's the primary author of the RTS, of course. :) And
> anyone who does this will undoubtedly have to be in contact with him
> to some degree, for all the aforementioned reasons.
>
> If nobody else would step up for this, I would be interested in
> contacting them if it seems worth it and giving it a shot.
>
> * I think the chance of this in our case is very very low of course.
> The cases we really care about for a safety perspective are
> type-safety violations at the Haskell level, obviously. Nonetheless
> this requires an individual who is committed to sifting through
> reports and assessing them in a detailed way.
>
> ** Coverity at one point found out the hard way that more analysis is
> not necessarily better, and people don't respond well to the results
> sometimes. Or they're too complicated to explain. The following is a
> great read: http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later/fulltext
>


Reply | Threaded
Open this post in threaded view
|

Registering GHC for Coverity SCAN

Simon Marlow-7
In reply to this post by Austin Seipp-4
On 09/05/13 01:36, Austin Seipp wrote:

> Hello all,
>
> On IRC today, Nicolas Trangez brought up the idea of registering GHC
> for Coverity's SCAN project. SCAN is essentially a free service run by
> Coverity, which runs their Static Analyzer on open source projects
> ('open source' being defined by OSI) and gives the results back to
> developers. Coverity is widely regarded as having the absolute best
> tool in the business for C codebases as it stands. The idea is to try
> and rattle bugs out of the runtime system if possible.
>
> I think this is a good idea, but it needs some public discussion.
> Namely, Coverity requires an official Point of Contact within the
> project (maintainer/author/governing body) to register the project in
> their name. We are left up to determine who this is and verify it. Bug
> reports which are found are not also made public, as they could be
> potentially serious security vulnerabilities.*
>
> So, if done, this is a rather piece-by-piece project which requires a
> bit of commitment, because it will require the registree to not only
> assess the bugs, but sensibly move them into the issue tracker or fix
> it themselves.
>
> It's also a question of whether it's worth it. Personally, I think it
> has the chance to uncover some real bugs. But the RTS is highly
> unusual C code (by some standards) and it will probably require tuning
> and patience to get tangible results. Bugs will also need to be
> assessed by someone. False positives happen etc. This is the double
> edged sword of static analysis.** Anyone who does it should be very
> aware of this.
>
> So, what do you all think? I'd particularly like input from Simon M
> here since he's the primary author of the RTS, of course. :) And
> anyone who does this will undoubtedly have to be in contact with him
> to some degree, for all the aforementioned reasons.

By all means go ahead.  But I'm dubious that they'd be able to find
anything useful, at least without adding some GHC-specific knowledge
about conventions in the RTS to their static analyser.  We hardly use
malloc/free at all, for example.

Also I can't guarantee that I'll have much time to look into any
potential issues they uncover, but I'll try my best.

Cheers,
        Simon


> If nobody else would step up for this, I would be interested in
> contacting them if it seems worth it and giving it a shot.
>
> * I think the chance of this in our case is very very low of course.
> The cases we really care about for a safety perspective are
> type-safety violations at the Haskell level, obviously. Nonetheless
> this requires an individual who is committed to sifting through
> reports and assessing them in a detailed way.
>
> ** Coverity at one point found out the hard way that more analysis is
> not necessarily better, and people don't respond well to the results
> sometimes. Or they're too complicated to explain. The following is a
> great read: http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later/fulltext
>



Reply | Threaded
Open this post in threaded view
|

Registering GHC for Coverity SCAN

Nicolas Trangez
In reply to this post by Austin Seipp-4
On Wed, 2013-05-08 at 19:36 -0500, Austin Seipp wrote:

> Hello all,
>
> On IRC today, Nicolas Trangez brought up the idea of registering GHC
> for Coverity's SCAN project. SCAN is essentially a free service run by
> Coverity, which runs their Static Analyzer on open source projects
> ('open source' being defined by OSI) and gives the results back to
> developers. Coverity is widely regarded as having the absolute best
> tool in the business for C codebases as it stands. The idea is to try
> and rattle bugs out of the runtime system if possible.
>
> I think this is a good idea, but it needs some public discussion.
> Namely, Coverity requires an official Point of Contact within the
> project (maintainer/author/governing body) to register the project in
> their name. We are left up to determine who this is and verify it. Bug
> reports which are found are not also made public, as they could be
> potentially serious security vulnerabilities.*
>
> So, if done, this is a rather piece-by-piece project which requires a
> bit of commitment, because it will require the registree to not only
> assess the bugs, but sensibly move them into the issue tracker or fix
> it themselves.
>
> It's also a question of whether it's worth it. Personally, I think it
> has the chance to uncover some real bugs. But the RTS is highly
> unusual C code (by some standards) and it will probably require tuning
> and patience to get tangible results. Bugs will also need to be
> assessed by someone. False positives happen etc. This is the double
> edged sword of static analysis.** Anyone who does it should be very
> aware of this.
>
> So, what do you all think? I'd particularly like input from Simon M
> here since he's the primary author of the RTS, of course. :) And
> anyone who does this will undoubtedly have to be in contact with him
> to some degree, for all the aforementioned reasons.
>
> If nobody else would step up for this, I would be interested in
> contacting them if it seems worth it and giving it a shot.

Although I believe this is a good idea (who'd have thought ;-)), I won't
step up as a potential 'lead' for this due to a lack of knowledge, and I
can't be sure I'll be able to spend enough time on the project.

If someone does give it a shot, I'd love to help out triaging etc (a
project 'owner' can provide access to the result to others as well if
I'm not mistaken).

As Simon M. mentioned, the analysis might be confused since the RTS is
in several places not 'your average C code'. This makes sense (although
adding some annotations in comments can help), but maybe some false
positives caused by this could also point at 'too magic' code, which
might need some simplification love?

Lua, Mono & Racket are some projects already part of the program which
might have similar code patterns etc.

Nicolas