Re: Report on Coverity

On 10/21/2012 12:03 PM, Kinkie wrote:

> so far I have checked 134 defects uncovered by Coverity out of 334,
> I think I have seen enough to report some numbers.
> There are 49 false positives, and 24 intentional risky behaviors.
> 61 are bugs; but in most cases they are not real issues, just poor
> practice: things like undocumented assumptions on callers' handling of
> buffer sizes.
>
> I hope this can be enough help you understand whether Coverity is a
> good deal - triaging without fixing is a bit of a drag.
> The UI is nice but maybe due to me not sitting on the server it's not
> really as responsive as it could be.

My understanding is that at most two "real bugs" were exposed by these
tests so far. And by a "real bug", for the lack of a better word, I mean
a defect that may affect Squid runtime behavior (e.g., cause a crash or
a wrong response). The two "real bugs" candidates are:

* ACLHierCodeData::empty() has out-of-bounds access to values[] array.
* Our chroot() calls are missing chdir("/") follow up.

There are many defects that point to low quality code. Since Squid runs
OK after many years of no Coverity testing, we can assume that when that
low quality code leads to "real bugs" most are caught by our usual
review/testing/deployment practices (or not caught by Coverity).

Needless to say that it would be nice to discover those real bugs
sooner, just by running [Coverity] tests. However, the probability of
that happening is unclear to me based on your feedback so far.

I think that probability should be the primary factor in deciding how
valuable Coverity testing is for the Squid Project: We can estimate the
pain associated with late discovery of an average "real bug", but it is
difficult to estimate the probability of Coverity eliminating an average
"real bug" before it causes pain because only a couple of such bugs were
exposed by these tests.

If we simplify a bit, we are essentially trying to distinguish between
these two cases:

  A) Squid code review practices eliminate nearly all real bugs
     that static analysis can find. Thus, SA is not very helpful.

  B) We have already found (the "hard way") and fixed nearly all
     real bugs so static analysis cannot find them until new bugs
     are added. When they are added, SA will be helpful so that we
     do not need to find those bugs the "hard way".

I wonder if it makes sense to test a much earlier version of Squid
(e.g., 3.1.0 or perhaps even 3.0.1). That way, we can see whether
Coverity can detect the real bugs that we have found the "hard way" (and
since fixed)?

Thank you,

Alex.
Received on Wed Oct 24 2012 - 16:56:28 MDT