Jason Cohen of SmartBear recently posted an article about “survivor bias“. His basic message is that you have to look at failures as well as successes in order to get a fair picture of what works and what doesn’t. By some estimates, 95% or more of the projects on open source portals like SourceForge and Google Code are inactive; looking at why they didn’t pan out is at least as informative as looking at the few that did. The problem is, it’s much harder to do: magazines aren’t as interested in failure as they are in success (unless the failure is really dramatic), and it’s much harder to track down people who’ve walked away from a project than it is to make contact with ones who are still actively involved.

We try to address this by having a post mortem at the end of every undergraduate project. Everyone who has been involved has to say what went right and what went wrong, i.e., what we should repeat next time and what we should fix or avoid. What comes up most often in the second list is, “Better time management.” Because students have fixed deadlines for assignments in other courses, but are setting their own goals in this one, it’s very easy for them to keep pushing their project aside (“I know I need to work on it, but Stats is due Thursday and the AI midterm is Friday”). This is one of the reasons we’re moving to two-week Scrum-style iterations: students get to decide what they’re going to do during each one, but they will then be graded against the goals they themselves set.