Embedded Academy

Forwarded from Embedded Academy

✅ Where Complexity Fails Us

✍ Jack Ganssle

Engineering is about numbers. Do you specify a ±5% resistor or ±1%? Do the math! Will all of the signals arrive at the latch at the proper time? A proper analysis will reveal the truth. How hot will that part get? Crunch the numbers and pick the proper heat sink.

Alas, software engineering has been somewhat resistant to such analysis. Software development fads seem more prominent than any sort of careful analysis. Certainly, in the popular press "coding"1 is depicted as an arcane art practiced by gurus using ideas unfathomable to "normal" people. Measure stuff? Do engineering analysis? No, that will crowd our style, eliminate creativity, and demystify our work.

I do think, though, that in too many cases we've abdicated our responsibility as engineers to use numbers where we can. There are things we can and must measure.

One example is complexity, most commonly expressed via the McCabe Cyclomatic Complexity metric. A fancy term, it merely means the number of paths through a function. One that consists of nothing more than 20 assignment statements can be traversed exactly one way, so has a complexity of one. Add a simple if and there are now two directions the code can flow, so the complexity is two.

There are many reasons to measure complexity, not the least is to get a numerical view of the function's risk (spoiler: under 10 is considered low risk. Over 50: untestable.)

To me, a more important fallout is that complexity tells us, in a provably-correct manner, the minimum number of tests we must perform on a function to guarantee that it works. Run five tests against a function with a complexity of ten, and, for sure, the code is not completely tested. You haven't done your job.

What a stunning result! Instead of testing to exhaustion or boredom we can quantify our tests.

Alas, though, it only gives us the minimum number of required tests. The max could be a much bigger number.

Consider:

if ((a && b) (c && d) (e && f))...
Given that there's only two paths (the if is taken or not taken) this statement has a complexity of 2. But it is composed of a lot of elements, each of which will affect the outcome. A proper test suite needs a lot more than two tests. Here, complexity has let us down; the metric tells us nothing about how many tests to run.

Thus, we need additional strategies. One of the most effective is modified condition/decision coverage (MC/DC). Another fancy term, it means making sure every possible element in a statement is tested to ensure it has an affect on the outcome.

Today some tools offer code coverage: they monitor the execution of your program and tag every statement that has been executed, so you can evaluate your testing. The best offer MC/DC coverage testing. It's required by the most stringent of the avionics standards (DO-178C Level A), which is partly why airplanes, which are basically flying computers, aren't raining out of the sky.

Use complexity metrics to quantify your code's quality and testing, but recognize its limitations. Augment it with coverage tools.


Footnotes:

1 I despise the word "coding." Historically coding was the most dreary of all activities: the replacement of plain text by encrypted cipher. Low-level functionaries, or even machines, did the work. Maybe "coding" is an appropriate term for script kiddies or HTML taggers. If we are coders you can be certain that in very short order some AI will replace us. No, we in the firmware world practice2 software engineering: implementing complex ideas in software using the precepts of careful engineering. These include analysis, design, negotiating with customers, implementation and measurements of our implementations.

2 Bob Dylan got it right: "he not busy being born is busy dying". We should be forever practicing software engineering. Practice: "perform (an activity) or exercise (a skill) repeatedly or regularly in order to improve or maintain one's proficiency." Unless we're constantly striving to improve we'll be dinosaurs awaiting the comet of our destruction.

@embedded

www.tgoop.com/embedded/2080

4.5K viewsNov 19, 2023 at 16:35

✅ Where Complexity Fails Us

✍ Jack Ganssle

Engineering is about numbers. Do you specify a ±5% resistor or ±1%? Do the math! Will all of the signals arrive at the latch at the proper time? A proper analysis will reveal the truth. How hot will that part get? Crunch the numbers and pick the proper heat sink.

Alas, software engineering has been somewhat resistant to such analysis. Software development fads seem more prominent than any sort of careful analysis. Certainly, in the popular press "coding"1 is depicted as an arcane art practiced by gurus using ideas unfathomable to "normal" people. Measure stuff? Do engineering analysis? No, that will crowd our style, eliminate creativity, and demystify our work.

I do think, though, that in too many cases we've abdicated our responsibility as engineers to use numbers where we can. There are things we can and must measure.

One example is complexity, most commonly expressed via the McCabe Cyclomatic Complexity metric. A fancy term, it merely means the number of paths through a function. One that consists of nothing more than 20 assignment statements can be traversed exactly one way, so has a complexity of one. Add a simple if and there are now two directions the code can flow, so the complexity is two.

There are many reasons to measure complexity, not the least is to get a numerical view of the function's risk (spoiler: under 10 is considered low risk. Over 50: untestable.)

To me, a more important fallout is that complexity tells us, in a provably-correct manner, the minimum number of tests we must perform on a function to guarantee that it works. Run five tests against a function with a complexity of ten, and, for sure, the code is not completely tested. You haven't done your job.

What a stunning result! Instead of testing to exhaustion or boredom we can quantify our tests.

Alas, though, it only gives us the minimum number of required tests. The max could be a much bigger number.

Consider:

if ((a && b) (c && d) (e && f))...
Given that there's only two paths (the if is taken or not taken) this statement has a complexity of 2. But it is composed of a lot of elements, each of which will affect the outcome. A proper test suite needs a lot more than two tests. Here, complexity has let us down; the metric tells us nothing about how many tests to run.

Thus, we need additional strategies. One of the most effective is modified condition/decision coverage (MC/DC). Another fancy term, it means making sure every possible element in a statement is tested to ensure it has an affect on the outcome.

Today some tools offer code coverage: they monitor the execution of your program and tag every statement that has been executed, so you can evaluate your testing. The best offer MC/DC coverage testing. It's required by the most stringent of the avionics standards (DO-178C Level A), which is partly why airplanes, which are basically flying computers, aren't raining out of the sky.

Use complexity metrics to quantify your code's quality and testing, but recognize its limitations. Augment it with coverage tools.


Footnotes:

1 I despise the word "coding." Historically coding was the most dreary of all activities: the replacement of plain text by encrypted cipher. Low-level functionaries, or even machines, did the work. Maybe "coding" is an appropriate term for script kiddies or HTML taggers. If we are coders you can be certain that in very short order some AI will replace us. No, we in the firmware world practice2 software engineering: implementing complex ideas in software using the precepts of careful engineering. These include analysis, design, negotiating with customers, implementation and measurements of our implementations.

2 Bob Dylan got it right: "he not busy being born is busy dying". We should be forever practicing software engineering. Practice: "perform (an activity) or exercise (a skill) repeatedly or regularly in order to improve or maintain one's proficiency." Unless we're constantly striving to improve we'll be dinosaurs awaiting the comet of our destruction.

@embedded

BY Embedded Academy