SimonHF's Blog

Just another site

libsxe code metrics February 6, 2011

Filed under: Uncategorized — simonhf @ 7:08 pm

There aren’t too many quality assurance statistics available online about things like ratio of test lines of code to production lines of code, or ratio of tests to production lines of code. So I thought I’d publish a quick snapshot of such information for libsxe:

[libsxe]# find | grep -v /build | grep /lib-sxe | egrep “\.c” | xargs cat | wc -l
17712 <– Number of lines in .c files; production source code and test source code

[libsxe]# find | grep -v /build | grep /lib-sxe | egrep “\.c” | egrep “test-” | xargs cat | wc -l
7839 <– Number of lines in .c files; test source code only

[libsxe]# find | grep -v /build | grep /lib-sxe | egrep “\.c” | egrep -v “test-” | xargs egrep “SXE[ERLD]” | wc -l
680 <– Number of lines of permanent instrumentation; production source code only

[libsxe]# find | grep -v /build | grep /lib-sxe | egrep “\.c” | egrep “test-” | xargs grep “plan_tests” | perl -lane ‘($c) = $_ =~ m~plan_tests\s*\(\s*(\d+)~; $total+=$c; sub END { printf qq[%d <– Minimum tests run per platform before P4 submit\n], $total; }’
1291 <– Minimum automated tests run per platform to achieve 100% code coverage before source code check-in

Ratio of production lines to test lines: 9873 : 7839 or 1.3 : 1
Ratio of production lines to tests run: 9873 : 1291 or 7.6 : 1
Ratio of production lines to permanent instrumentation: 9873 : 680 or 14.5 to 1


4 Responses to “libsxe code metrics”

  1. John Says:

    What exactly do these numbers prove?

  2. Tobias Werner Says:

    John makes a great point. What does this ratio prove?

    What if 33% of the test lines were useless? Then you have a rwti of 9873:6615:2268.
    Where 9873 is the number of production lines.
    Where 6615 is the number of useful test lines.
    Where 2268 is the number of useless test lines.

    How does measuring the ratio of test lines to proruction lines of code love anything at all?

    I hop you’re not getting paid based on some ridiculous ratio like this.

    • simonhf Says:

      Thanks for the questions, John & Tobias.

      Superficially, the numbers prove that libsxe is well tested and has a clean API abstraction which is even used to test itself:

      On 100% coverage *and* lines ratio: Sorry, I guess what I left out of the metrics blog is the fact that the libsxe build mechanism enforces 100% code coverage. This is a useful context to view the figures in, otherwise you’re right, they don’t mean that much. Anybody who has tried to get code coverage in the 90% to 100% range will tell you that it’s difficult to achieve high coverage like that. Most agile teams shoot for 80% to 90% code coverage and end up struggling to do that. When creating the tests for libsxe then we initially concentrate on meeting the 100% code coverage requirement. In addition we do team level brain-storming to come up with test scenarios above and beyond achieving 100% coverage. One of our goals is to keep the source code for both production and test code as small as possible by eliminating source code whenever possible via merciless refactoring.

      On API abstraction *and* lines ratio: What’s most interesting about the libsxe ratios is that the lines of test code are *less* than the lines of productions code. Recently I looked at a project very similar to libsxe in some ways; a high performance, http proxy written in C. I won’t name the project but the ratio of production lines to test lines is 2000 to 5000. Remember that the libsxe ratio of production lines to test lines is 9873 to 7839. Why the big discrepancy? This is because the authors of the http proxy in question did not go to the trouble of cleanly separating generic network code from http proxy business logic. This means that the test code for the http proxy could not reuse generic network code from its production source code. Hence, each test program must re-invent the wheel and have its own version of the network code which greatly inflates the total test lines.

      If you know of any other projects which have either 100% or close to 100% code coverage then please let me know because I’d be interested in comparing figures 🙂

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s