CMMI Level 5: Worth the Effort?
By Ed Carroll, sales executive, Agilis Solutions
Whenever I get into a discussion about implementing software engineering processes, I invariably hear the question, “What is the ROI for implementing CMMI to Level 5?” The question might have variations such as, “I can see taking a team to Level 3, but what is the point of going for Level 5?” Sometimes the question is more straightforward, “How do you justify the cost?” To answer these questions, I will draw from lessons learned in a real-life achievement, the six-year-long effort of the long-time software development partner of Agilis Solutions.
In accordance with the program, our partner documented operations at CMM Level 5 in early 2003, was assessed at CMM Level 5 in March 2004, and assessed again at CMMI Level 5 in March 2006.
Background – what is the Capability Maturity Model?
In 1984, the Department of Defense, frustrated by frequent failures in software development, requested that Carnegie Mellon University devise a system to assess a vendor’s ability to successfully complete a project. In response, the Software Engineering Institute (SEI) was created, and it proceeded to model software engineering best practices. The SEI published its findings as the Capability Maturity Model (CMM) in 1991. Today the CMM has evolved into the CMM Integration (CMMI) Product Suite (see http://www.sei.cmu.edu/ for details), which provides an extensive roadmap for improving organization and software engineering processes.
The basic Capability Maturity Model
The CMM frameworks help organizations increase the maturity of their human resources, process and technology assets to improve long-term business performance. The SEI has developed CMMs for software, people and software acquisition, and helped develop CMMs for systems engineering and integrated product development. The latest development in CMMs is the CMM Integration (CMMI) Product Suite (http://www.sei.cmu.edu/cmmi/).
The basic model has five maturity levels where success builds on previous efforts:
-
Initial
-
Repeatable
-
Defined
-
Managed
-
Optimizing
For more about the CMMI, please click on this link: http://www.sei.cmu.edu/cmmi/adoption/cmmi-start.html.
How and why it started
The Agilis Solutions partner started operating much like any other company: opportunistically. However, company executives quickly recognized that to be competitive and to really grow, they needed to be more than opportunistic; they had to become better, and not just a little better, but significantly better than their competition.
Several very large software services companies were touting their assessment at CMM Level 5 as a competitive advantage, so our partner sought them out. As it happened, several founders of the partner are mathematicians, and the systematic approach of the CMM appealed to their sense of the right way to proceed. After learning that these companies (all of them in India) were growing at exceptional rates, our partner naturally decided to emulate their example.
Level 2 – value gained
During these early days, our partner struggled, like any new company, between two harsh realities. On the one hand, company executives knew that they had to please their customers, yet on the other hand their processes were so immature that they continuously made costly mistakes. In fact, they were characteristic of organizations operating at Level 1 in that they did not believe they had the problems they did have. Going through the effort to advance along the CMM provided them in this early stage with valuable awareness of their real lack of capability.
The CMM helped our partner understand that a concise and consistent project management methodology is essential to doing good software engineering, again and again. After a couple of years’ effort there was real recognition across the management team that there was a right and wrong way to manage projects, the result being the implementation of repeatable successful methods.
Real growth – in both size and maturity
As our partner achieved CMM Level 3, it also gained competitiveness. It established a successful track record of on-time and on-budget delivery. These successful local (in-country) projects caught the attention of larger and more widespread (international) customers looking for predictability in their project cost and schedule. (Note that low cost was not the leading objective of these customers.)
Level 3 – value gained
It took our partner more than three years to achieve CMM Level 3 (from initial start). By this time, the effort to reach this achievement was simply driven by the need to grow the business. Doing things this way seemed the most natural or the most systematic way to proceed. At the time, the growth did not seem unnatural, but a natural outcome of the company’s hard efforts. It had doubled in size and revenue every year since implementing CMM. Many companies that expend the effort to achieve this level of success (particularly in the United States) stop the improvement process, assuming that their success and the improvements will continue.
Optimizing for sustained growth
The reasons that our partner continued its CMM efforts were not overtly or rationally contrived. The models went further, so it seemed natural to continue the effort. One might say that the leadership team had developed a passion for success. They rationalized that if CMM Level 3 could produce the success they had achieved, then Levels 4 and 5 should produce even greater success. Particularly important is the fact that 100 percent of the leadership team, from the CEO downward, believed passionately in the CMM.
Level 4 and 5 – value gained
Interestingly, the statistics that justify the return on investment for implementing CMM process improvements did not truly materialize until our partner fully implemented the metrics program necessary to understand how to improve. It takes a high level of maturity to understand what to measure in order to affect improvements to processes.
Our partner discovered along its way to Level 5 that it was not simple to collect the right metrics consistently. To solve this problem, being a software company, it automated the metrics-gathering process and to-date has one of the most sophisticated metrics environments in the marketplace (see my previous article A Software Metrics Case Study).
Real value has been gained in the metrics data gathered. Our partner quickly recognized that its data was a gold mine in that it provided the information needed to help the company stay competitive in a tough market. With this information, company executives knew precisely what to improve and by how much. With this information they can validate the assertions of the SEI, that Level 5 teams will typically develop software in 40 percent less time and cost, with significantly fewer (if any) defects than a Level 1 team. Moreover, the total lifecycle savings of these projects are huge, due to the low defect rates alone.
By-products of our partner’s extensive metrics program have proven equally advantageous. For example, project cost and schedule data has validated a parametric software estimating model to an accuracy of less than 9 percent error, even though the estimates are made very early in the requirements phase of the project lifecycle. In contrast, the industry average for project estimates made early in the requirements phase have an error rate of 200 percent (and often much more).
Clearly, the success of our partner validates the establishment of repeatable processes, understanding what processes work well and which do not, as well as the efforts to gather copious amounts of data as a driver for continuous process improvement.
Conclusion
As is often the case with life in general, we begin something based on only partial information. Our partner’s executives did not have the results of a definitive analysis that justified the effort required to achieve the highest level of the Capability Maturity Model before deciding to start down that path; instead, the CMM presented a systematic approach that appealed to their mathematically oriented minds. However, now that they have the data, they can prove an astounding return on investment for the real work required to achieve CMMI Level 5.
The awareness our partner’s executives gained of the successes and failures within their processes has evolved to real clarity that enables them to respond to client needs by addressing process improvements quickly and at a very micro level. Their metrics program enables conversations about what went wrong and what could be done better that are much more open and constructive. For the cost of leadership focus, buy-in from the entire staff, the development of some useful (internal) tools and a small percentage of resources continuously dedicated to process improvement, our partner has reaped the benefits of being able to compete in the market as a truly world-class software engineering organization.
About the author
Ed Carroll has been building software products for more than 20 years, with particular expertise in automating economic analyses, decision support and supply-chain management processes. He has provided strategic technology leadership in roles such as the vice president of engineering for Egghead.com, director of technology at Nike and director of software engineering at Boeing. He can be reached at EdCarroll@AgilisSolutions.com.
|
