An Uneasy Case for Project Management Tools
Last week I described the history of project management software, leading us to a world of convergence where all the various options (Jira, Asana, etc.) have largely the same feature-set, which seems to get more and more bloated each year. But the primary question remains: what value is all this actually adding?
Marketing claims
It’s not hard to find grandiose claims from purveyors of project management software or their trade association Project Management Institute (PMI).
PMI claims that “organizations that undervalue project management as a strategic competency for driving change report an average of 67 percent more of their projects failing outright”. I’m not sure who would actually admit that they undervalue their project management?
PMI also described “champion” organizations, which “have higher project success rates (92% versus 32% of underperformers), enjoy more successful business outcomes, and waste significantly less money due to poor project performance”. If we look at how they define champions, it’s those organizations that have 80% or more projects completed on time, on budget, and meeting original goals/business intent. So this is just a circular claim that tells us nothing.
monday.com claims “3 hours a week” saved per employee, a “288% ROI” and “$525,095 team productivity improvement value” in their TEI report. Of course, as you might suspect, these TEI studies are commissioned by the software vendor, and interview a handpicked set of very successful and excited customers. It’s about as far from a controlled study as you can get.
Asana, in their Anatomy of Work Index, claims that workers spend 60% of their time on “work about work” (implied to be unnecessary). To me this feels unrealistically high. I’m also not sure that the takeaway from this is that we should start using more tools that require even more work about work?
Atlassian only claims that 80% of the Fortune 500 use their products, which at least doesn’t make any claims about efficacy or value, so it’s hard to find fault with it. Of course it also doesn’t really suggest that you should be rushing to buy their product, unless the only way you make business decisions is by copying whatever is popular.
Research
There has been relatively little independent research on the efficacy of project management on the success of projects. This is probably because (a) it is essentially impossible to do controlled trials and (b) even uncontrolled data is hard to get since it requires opt-in from businesses (typically via surveys).
Some of the only major studies on project success come from the Standish Group, and their CHAOS reports. In their 2012 report they looked at “factors of success” for projects, in which tools were ranked in last place because they “can help a project succeed, but like any tool they can also hurt”. They gave them an estimate of being responsible for only 1% of project success. Even “project management expertise” in general was rated as being responsible for only 6% of project success. Compared to “executive management support” at 19% and “user involvement” at 18%, these factors barely matter.
Their 2015 report continued on this theme, stating that “[p]roject management expertise, process methods, and tools are affected by the physics law of diminishing returns”. And they strengthened this even further in their 2020 report, where they “found that these types of tools actually decreased success and doubled the cost”, a result they found “very surprising”. As one summary put it: “the data shows the project managers fail more often than no project manager, and you put your success at risk if you use projects, project managers or project management tools when developing software”.
Of course we should be careful not to put too much stock into one single source. In The Rise and Fall of the Chaos Report Figures, the authors point out several flaws in the methodology. Most significantly, a successful project is defined as one that is completed within its original estimates of time and cost. It doesn’t consider any actual outcomes such as usage, profit, and so on. Essentially it’s just measuring how accurately you are doing forecasting–it would be easy to get a high score on this metric simply by adding more padding to the time and cost estimates.
The most favorable interpretation we can give is that we should ignore the reports because of this poor methodology, which just leaves us with essentially no evidence that project management tooling (or project management practices in general) have any positive effect on project success.
Placebo?
Despite the lack of any evidence supporting project management tools (and indeed some weak evidence for the opposite), why do they still feel essential for organizations? Why do we keep using them? I can identify a few reasons.
Visibility for leadership
Whether or not project management software helps teams to ship, it certainly does help leadership have visibility into what they are doing. With the huge rise of remote, distributed work it is harder to just “pop by” for a chat to get a status update, so it creates a desire among leadership to use software to address this visibility gap. Of course, it still doesn’t mean that this software actually accelerates shipping the end product. Earlier I mentioned that “executive management support” was found to be a good predictor for project success, so if executives are more likely to be involved in a project that uses these tools, that could indirectly be a benefit. Yet in that case a good question might be whether leadership accept less visibility if it meant the product would ship faster?
Mistaking the map for the territory
It’s easy to confuse all the fancy status reports, burndown charts, and so on for the deliverables themselves. You don’t need to actually understand the state of your project if you see a bunch of green status dots all lined up in a row. But, as Alfred Korzybski said, “the map is not the territory”, and if you don’t actually know where this data is coming from it’s easy to completely misunderstand the true state of the world.
Selection bias
Teams that function well tend to follow “best practices”. Thus, it is likely that teams that ship high-quality work on time will use the “best practice” project management software. Conversely, teams that are poorly run are less likely to use this software and less likely to ship on time. This can create an effect where it seems that the successful teams are the ones using the software, but in this case it is correlation, not causation.
What are we to do?
As discussed above, the evidence for project management software improving teams’ ability to ship faster and with higher quality is at best mixed. There’s much more evidence that certain practices matter, but these don’t require the highly structured and complex tools that now permeate this space. My view is that we would get much better outcomes if we let teams pick their own tools based on how they want to work, rather than mandating any top-down solution, and that teams should also err on the side of selecting the lightest tool that meets their needs.
It’s time to move back toward the original Agile philosophy of individuals and interactions over processes and tools.