Ben Northrop

  Decisions and software development
Essays   |   Popular   |   Cruft4J   |   RSS    

Is it Irrational to Comment your Code?

(8 comments)
Lessons for Software Engineering from the Prisoners Dilemma Game
  November 11th 2007

Imagine this...

You and five other developers have just been hired as contractors for a 4 month project at Acme Corporation. Congratulations! Going in, you've been told that there's a chance some of you could be hired at the end of these 4 months to continue working on the system, and since you all prefer the stability of full time work, you're encouraged by this. You also learn that for Acme, releasing the software in 120-days is absolutely critical, and accordingly, your manager (who, incidentally determines whether you stay or go at the end of the contract) is most concerned with whether the system works, not how you make it work. Finally, you've all been given different functional areas to work on, so you'll be working relatively independently of your fellow contractors.

Sound good? Now here's my question for you: will you comment your code?

Deep down, you know writing comments is a good thing to do, but if the situation is as I described it, I'm guessing you might not. Here's why:

Analyzing the Decision

The decision you're faced with, to comment or not comment your code, can be modeled in a simple two-by-two matrix, where rows represent your decision, columns represent the other contractors' decision, and the cells represent the unique possible outcomes (i.e. the intersection of your decision and the other contractors' decision). For each outcome, there are two numbers: one which represents the benefit (or “utility) that you would receive for that outcome, and the other representing the benefit everyone else individually receives.

Everyone Comments Everyone Doesn't Comment
You Comment You: 8, Everyone: 8 You: -2, Everyone: 2
You Don't Comment You: 10, Everyone: 6 You: 0, Everyone: 0

The utility payouts for the four possible outcomes are derived as follows:

1. Everyone Comments (upper-left): If you comment your code and so do the other contractors, everyone does equally well. Though you all had to expend a little effort to write good comments, the upside is that you're left with a well documented system at the end of four months, and so if you do indeed get extended, your life will be much easier in the long run for maintenance and enhancements. Assuming a +2 utility for each person (other than yourself) whose comments you get to read and -2 utility for the time expenditure of having to write comments, each person receives an 8.

2. Everyone but you Comments (lower-left): If you don't comment and everyone else does, you get all the benefit of reading their comments (+2 x 5 = 10) with none of the cost of writing your own. Everyone else, however, still had to take the time to write their own comments (-2), but at least gets the benefit of reading all but your comments (+2 x 4 = 8), so they each get a utility of 6.

3. Only you Comment (upper-right): If you do write comments but everyone else doesn't, you lose big, since you expended the effort of commenting (-2) and don't get any of the benefits of the rest of the system being commented (0). Everyone else, on the other hand, gets to use your comments (+2 x 1) but never had to spend any energy commenting themselves, so all get a 2.

4. No One Comments (lower-right): Lastly, if neither you nor anyone else writes comments, then you all get a score of zero - no benefits but no costs either.

Given these payouts, how do you decide what to do?

Well, assuming you are rational (that is to say, you're interested in maximizing your own utility) then you subtract away everyone else's payouts, and look at the decision matrix like this:

Everyone Comments Everyone Doesn't Comment
You Comment You: 8 You: -2
You Don't Comment You: 10 You: 0

What you realize, then, is that no matter what everyone else does, you're always better off not commenting your code! For example, if you think everyone else will comment their code, you're still better off not commenting, because you can get all the benefit of reading their comments with none of the cost of writing them yourself .

Everyone Comments
You Comment You: 8
You Don't Comment You: 10

If, however, you think none of your colleagues will comment their code, again, you're better off not commenting - if you're not going to get any benefit of reading other people's comments, you mine as well not exert the effort yourself!

Everyone Doesn't Comment
You Comment You: -2
You Don't Comment You: 0

You begin to understand, sadly, that it just isn't in your best interests to comment your code, and so you end up not doing it. The unfortunate crux of this situation, however, is that your fellow contractors use similar logic and decide not to comment as well. In the end, you're all stuck with a utility of zero – a completely uncommented system.

Everyone Comments Everyone Doesn't Comment
You Comment You: 8, Everyone: 8 You: -2, Everyone: 2
You Don't Comment You: 10, Everyone: 6 You: 0, Everyone: 0

“But wait a minute”, you think, “it didn't have to be this way”! If you all had just ignored your self-interest and written comments, you all would have been much better off with a utility of 8, instead of the 0 that you actually received.

Everyone Comments Everyone Doesn't Comment
You Comment You: 8, Everyone: 8 You: -2, Everyone: 2
You Don't Comment You: 10, Everyone: 6 You: 0, Everyone: 0

In other words, the group as a whole was made worse off because each person followed their own individual best interests!

Prisoner's Dilemma

In the language of game theory, this situation is called a "Prisoner's Dilemma", and refers to any two player, two decision game with the following payout structure:

Cooperate Defect
Cooperate win, win lose much, win much
Defect win much, lose much lose

The essential element of the game is that the Nash equilibrium (i.e. the outcome that rational agents are driven to – in our case, don't comment/don't comment) is Pareto inefficient, which just means that there is another outcome in which all players would be better off (e.g. comment/comment). In Prisoner's Dilemma terminology, doing the rational thing to the detriment of the group (in our case, not commenting your code) is called “defecting”. Conversely, ignoring your own rational self interest to the benefit of the group (e.g. commenting your code) is called “cooperating”.

So what does this all mean for code comments in software systems? Would we really be irrational to comment our code? From a rational choice perspective , the answer is “it depends”. If the situation was similar to Acme's 4-month long project, then yes, it would be irrational for developers to comment their code – it's just not in their best interests. Not all companies, however, operate like Acme. In many cases, specific environmental, organizational, or cultural factors can work to augment the incentives in the decision such that a developer's self interest can align with the team's best interest. The question is, then, what are these factors that contribute to win-win outcomes? How can a software development organization encourage cooperative software development behavior, whether it's commenting code, writing unit tests, refactoring, or writing design documentation?

1. Evaluate Internal Quality

Acme's first mistake was focusing solely on external behavior, to the neglect of internal quality. Had Acme manangers evaluated developers not just on functional, but on extra-functional concerns (like maintainability, extensibility, etc.) as well, then each developer's incentive structure would have been rebalanced to the extent that commenting would be in their obvious best interest - not commenting would be risking their future at Acme. To this end, managers who are interested in the long-term viability of a system must find some way to measure internal quality (whether by attending code reviews or analyzing reports from static source code analysis tools like PMD, CheckStyle, or FindBugz) and then tie incentives to it.

2. Harness the Power of Reputation

Even if Acme's managers didn't evaluate internal quality, had their developers been working collaboratively and reviewing each others code, then reputation alone could have been a strong enough force to compel them to write good code. For most developers, their reputation is their most valuable asset – more important than any credential or experience on a resume when finding future work. Software organizations need to use the power of reputation to incent good coding practices, and they can do this by promoting collaboration, enforcing design and code reviews, or publishing code quality metrics per component or developer to a Wiki. When a developer knows others will see his code, he will naturally do his best. If, on the other had, a developer is disconnected from his colleagues, the temptation to take short-cuts may be too great.

3. Encourage Reciprocation through Long Term Relationships

Players in a Prisoner's Dilemma intuitively understand that when playing many games instead of just one, cooperating in the short-term will yield benefits in the long– in other words, if I scratch your back now, you may scratch mine later. Acme, however, hired short-term contractors who had no guarantee of a long-standing relationship with Acme, and so reciprocation wasn't much of a factor in their individual comment-or-not-comment decision calculus. Had they known they'd be working with their colleagues in the future, then even if they were working on isolated features, they'd know that down the road one of their colleagues might need to debug their code (or vice-versa) and comments would help. By writing clean, commented code for others, others might write clean, commented code for them. Companies, therefore, when possible, should try to foster long-term relationships with their developers and by so doing will increase the likelihood of cooperation.

4. Start out on the Right Foot

When determining whether to defect or cooperate, a player of an iterated Prisoner's Dilemma game looks to his opponents' past actions. If he finds himself in a situation where everyone consistently defects, then he'll defect too, so not to be the “sucker”. The same holds for developers when presented with a system of poor internal quality – “if no one else takes the time to write good code”, they think, “why should I”? For this reason, setting an early precedent of good development practices is key. A project that gets off on the wrong footing will have great difficulty regaining its balance.

Conclusion

Each of these four principles is important for encouraging good development practices – though no single principle alone is imperative. For example, a company that hires short-term contractors can still create high quality software if they set an early precedent of good coding, increase visibility of developer's work through code reviews or static code analysis tools, and tie incentives to internal quality. An organization, however, that neglects each of these principles - hires short-term developers, evaluates only whether the system works and not how, allows developers to work in isolation, and starts out with a poorly coded system – will be doomed to buggy software at best, and failed projects at worst.

Resources

  • In 1979, Robert Axelrod held his now famous computing tournament, which pitted different Prisoner's Dilemma playing agents against each other to see which strategy would yield the greatest utility. Does it really pay to defect? To everyone's surprise, the agent that won was coded with the most straightforward strategy: tit-for-tat. In Evolution of Cooperation, Axelord tells the story of this tournament and explains what it means for cooperation at all levels (animals, people, countries).
  • Apparently, Bill Clinton made each person in his cabinet read Nonzero - for whatever that's worth;). A feel-good pop-science book that asserts that evolution naturally tends toward greater cooperation.
  • The Origins of Virtue explains that morality (i.e. cooperation) is neither some rational construction of ours nor is it merely a divine mandate, but rather it's hard-coded into our fundamental design because it gives us the best chance for survival.
  • Hume disagrees that we are naturally cooperative, and instead argues that it is government's role to encourage cooperation amongst its citizenry.



I believe that software development is fundamentally about making decisions, and so this is what I write about (mostly). I'm a Distinguished Technical Consultant for Summa and have two degrees from Carnegie Mellon University, most recently one in philosophy (thesis here). I live in Pittsburgh, PA with my wife, 3 energetic boys, and dog. Subscribe here or write me at ben at summa-tech dot com.



Got a Comment?

Name:
Website:
Are you human:
Comment:

Comments (8)



Jean-Baptiste Potonnier December 14, 2007
Hi Ben!
As a ex AI worker, and now a short term contractor I find your post interresting and funny.

It is very difficult to keep good quality in a project, simply because quality of your job is often not evaluated like that. Most of the time,
the criteria is the time you take to complete a task.
If you're not a team player, you could commit you code as quick as possible, and complete your task, you will be seen as a good developper. Letting the others correcting your bugs and messy code.

And if someone finds out your fault, no problem, you're already far ;)

In my experience, extreme programming in a good mean to build team spirit and code quality, but it requires discipline

bye & please excuse my bad english...

mccoyn December 14, 2007
Good post. I was having trouble trying to figure out why my code quality (comments and organization) is lower in my own opinion after switching jobs. Your argument associating the existing code base with the prisoners dilemma explains it pretty well.

I think there is a way out of this dilemma. If I comment my code in a way that will make it easier for myself to change it down the road, thats a benefit to me irrespective of whether everyone else comments their code. Unfortunately, my comments to me are bound to be less valuable than the comments others have wrote, probably a +1 in your examples, but a +2 is required to break the dilemma.

Ben Nadel December 14, 2007
Very interesting view point on commenting. I happen to be a huge fan of commenting as when I re-examine code, it really helps for me to understand what I was thinking at the time. Of course, on a small, short-term project, this is not as important.

Incidentally, in philosophy class, we had a similar view of believing in God. The four quadrants were defined by You Believe, You Do NOT Believe, He Exists, He Does Not Exist. Using what seemed like arbitrary numbers and infinity, you can basically say it is in your best interest to believe in God. This has nothing to do with programming, but I drew the comparison in my head.

Ben Northrop December 14, 2007
Thanks for your comments!

Ben Nadel: The decision about whether to believe in God or not is Pascal's famous wager - definitely applicable to what I was thinking in that it boils down a decision to the simple question "what's in my best interests?".

Jean-Baptiste: I definitely agree that XP drives/encourages programmers to produce better quality code...and I think some of the reason is because it incorporates a few of the factors I mentioned, namely reciprocation (through iterations) and reputation (through pair programming). Not sure what Kent Beck would agree with me...;)

Rob Whelan December 15, 2007
Interesting article, though I definitely have some bones to pick with the approach. :)

A few underlying assumptions I disagree with:
* commenting is a binary decision (you either write them or don't)
* writing comments for the individual developer is a net negative (it takes more time to write them than will be saved during dev & bug-fixing)
* each developer is spending an large amount of time working on code the other developers wrote.

Fast-coding a project successfully in my experience *requires* a certain level of commenting far more than when you have the time to "do it right." You don't have time to find the elegant & clean approach... you're have to leave in the ugly first-shot hack that gets the job done. If you don't comment that as you go, you're going to be in hell during bug-fixing... and the features you're responsible for may not get to release-level quality at all by the deadline. You can't often just commit and forget, unless they're skipping the QA process entirely. It's also often the fastest way to turn some complicated logic into code -- walk through the process in plain English, make sure you've accounted for all inputs/outputs, *then* expand into code.

There are other classes of comments that are much more useful to maintenance programmers that you probably *won't* put in -- for example, higher-level comments explaining the place of the class in the entire project.

Final point -- especially if the deadline is tight, they're going to split up the work as much as possible. Every time a developer has to work on another's code, there's a large cost (especially when the code is rushed and ugly... as it will be). Who fixes the bug? The developer who's responsible for that feature.... So every developer during these 4 months will be spending at least 90% of their time working on their *own* code, not benefitting from any comments in the other developers' code. After the release, sure... but that's after the hire decision and irrelevant to this exercise.

There are also other factors, like if your code is easy to work with but you don't get hired, the ones who *do* get the job may actually recommend you next time a position opens up... reputation *always* matters, even if not now. :)

Anyway, I'm not tackling the adjusted grid (I can't imagine the formatting will work...) so that's an exercise left for the reader....

Thoughts?

About Pascal's wager: I suppose this isn't the blog to get into it deeply, but it falls into one of my pet obsessions.... First, we've figured out quite a lot of mysteries formerly ascribed to "God, or we-have-no-clue" over the past 300+ years since Pascal's wager was published. Second, religious belief and support of religion *does* come with a severe cost (perhaps more than evident in many of the world's conflicts today...). Finally, Pascal didn't live in the global religious context that we do -- there are tons of religions who propose his wager equally... and they also pretty much universally assert that if you choose the *wrong* one (even the wrong branch of Christianity...), you're going to hell.

I guess there's also the concept that no one actually founds a belief on a cost-benefits analysis like the wager; if you judge something as irrational, the best you can do is *act* as if you believed it.

Wow, this is way more than enough for a blog comment, though.....

Ben Northrop December 17, 2007
Rob: thanks for the comments! I agree with you, some *huge* simplifying assumptions had to be made in order to shoe-horn the decision about whether to comment or not into the Prisoner's Dilemma structure. I felt a little like the professor that starts a proof by saying "first let's assume this circle is a square...";) In general, I think this is an inherent deficiency of game theory itself - for anything but the basely simple problem, the mathematics are intracably hard - and so you need to make all sorts of assumptions, warrented or not, to reduce it to a form where anything useful can be gleaned. You pointed out some of the blatant assumptions, though, that I probably should have referenced: foremost that commenting is *not* a binary decision and that developers of course *do* get some benefit from their own comments.

Overall, I think that Prisoner's Dilemma situations are omnipresent in development organizations - though they're very subtle. We are often confronted as developers with decisions that pit the team's best interest (at some cost to ourselves) against our own. Development organizations, I think, could benefit by thinking in terms of cooperation and defection. I have seen first-hand projects that were very similar to the one I described, where at the end of 4 months the contractors were let go, the company pondered why code quality slipped, and the developers left over were left to maintain thousands of lines of uncommented, poorly-written code.

Lastly, re: Pascal, because not believing in God incurs an infinite cost (i.e. eternal damnation!!), even if you ascribe a very low probability to His existence and even if the worldly costs are high, it's still in your best interests to believe. You make a good point that in this day and age, there are multiple Gods for which we can profess our faith...so it's weighing one infinite cost against another...however, a loop-hole in this would be to simplify (as many do) and say, "yes...they may be different religions, but it's really the same God"...so you're back to the binary "God Exists"/"God Doesn't Exist" outcome.

Rob Whelan December 17, 2007
The problem definitely exists -- even if a complete lack of commenting may harm the developer directly to a degree, that doesn't mean they'll recognize that. Plus, the commenting approaches I mentioned above don't even come into play in the sadly common "keep changing stuff until it works" approach to coding... which is the source of the nastiest code of all, because you end up with confused, confusing code littered with unused variables, unreachable and unnecessary code, and possibly wrong comments (washed along in the cut-and-paste, usually), which are even worse than no comments at all.


So the Prisoner's Dilemma as you laid it out probably does apply if you assume all of the coders are similar and comments are a net loss to them. Part of my response was based simply on my personal reaction, i.e. that doesn't apply to me, at least, since I develop faster with some level of commenting.


Re Pascal: I think you're breaking the rule of "make everything as simple as possible, but not simpler." First of all, we don't have actual "belief" to offer by choice. Once I've rationally concluded that there is no god, the best I can do (with no further evidence) is "fake it". Is that good enough? The loophole of "it's all the same God" also only works if you only listen to selected religious leaders (who are trying to dampen conflict...) instead of looking for the most "authoritative" source of religious knowledge, like the official texts of the various religions. They're not so forgiving.

You won't get into Mormon heaven unless you accept Joseph Smith; you won't get to Muslim heaven if you're Christian or Jewish (depending on what part of the Koran you read, you should be kindly encouraged to accept Muhammad and convert to save yourself, or you should just be shunned or killed with the rest of the infidels), and the Christian gospels have similar stuff about accepting Christ... lake of fire if you don't, etc.. Most of the evangelical Christian churches in the US are very outspoken about the incompatibilities -- you can't even just be Christian; Catholics go to hell because they have an idolatrous relationship with Mary and the saints, for example. It's kind of freaky once you dig into it a bit.


Hey, maybe they're all wrong -- but we have no *other* basis for belief in God, so we have to go with what we have. The documentation of the religions are the closest thing to proof of miracles, divine action, etc. that exist, so you have to either accept it all or build it up from scratch... which puts you in Flying Spaghetti Monster territory.


So the potential cost is still infinite... but if you weigh all of the logical inconsistencies in the claims (and in the scriptures themselves), the possibility of this particular God -- one who will punish me for eternity regardless of how morally virtuous I am, simply based on the fact that I failed to *fake* a belief in him.... well, I'm comfortable that possibility goes to nil.


...now let's see if I have paragraphs... it seems like simple HTML tags aren't stripped out. :) How about more complicated ones?

Lars D February 10, 2008
Good article! It is nice to see things explained in a way that makes the blog post a good link to send to others.