Hong Jin's blog about Software Engineering

On Goodhart's law and Hacktoberfest

It's the time of the year again, when project maintainers on GitHub have to close dozens (or hundreds, for very popular repositories) of spam pull requests.

As they say, the road to hell is paved with good intentions.

A few years ago, Hacktoberfest started with good intentions, incentivizing people who were not familiar with open source contributions to start contributing to open source. If one contributed to open source, one gets a free t-shirt. It's a good deal.

How do we count contributions though?

Oh, the number of Pull Requests made in a month is a measure of the quantity of contributions, I suppose.

Hacktoberfest probably helped with DigitalOcean's branding, giving them good publicity. It also (probably) actually really introduced many people to new open source projects, or at least, to learn the process to contribute to the open source project.

Things were nice at the beginning.

Fast forward a few years ahead, at the present time, there is a perversion of this process. Many new contributors now are far more likely to be one-off contributors in search of trivial PRs. They'll probably fix a typo, or even introduce a new typo. This is fine to them, since their sole objective is to complete enough Pull Requests to get a free shirt.

I can't help but think of Goodhart's Law. "When a measure becomes a target, it ceases to be a good measure". Just like how Lines of Code (LoC) were once used to evaluate software developers' productivity, the number of Pull Requests are now used for measuring open source contributions.

Just like how developers gamed their LoC metrics to look good to their managers (and to get a pay increase), these days, people game their #PRs to look good to DigitalOcean (and to get a free t-shirt).

- 1 toasts