The Right QA, at the Right Time, for the Right Job

By Ed Carroll, sales executive, Agilis Solutions

Eileen Boerger, general manager, Agilis Solutions, led the Quality Assurance discussion at the September 7, 2006 Engineering Executive Forum. Boerger was eminently qualified to lead such a discussion thanks to her background in delivering outsourced engineering services and her extensive experience in the development and management of software systems and applications. This includes the outsourcing of software development and maintenance projects to Vietnam, India and New Zealand.

Before joining Agilis Solutions, Boerger was vice president of operations at QualityLogic, where she was responsible for delivering QA services. She has held multiple vice president positions in engineering at Mentor Graphics Corporation and was vice president of software technology at Unisys Corporation.

As a peer-to-peer group of engineering executives, members of the Engineering Executive Forum are not shy about sharing their views or asking pointed questions, but Boerger was more than able to handle the challenge. Topics covered all levels of development, including small and large teams, Agile and traditional methods, in-house and outsourced, onshore and offshore. To download her presentation, please click here.

Boerger launched the discussion by defining two typical industry terms. First, she defined quality assurance as “confidence.” Then she defined software vs. process QA as “confidence in the software vs. the quality of the development lifecycle.”

Establishing confidence
Emphasis was placed on establishing the processes that are “right” for the particular team and the particular project. Processes form a framework around which work is performed, with the intent of reminding the engineers of those easily forgotten or skipped steps. Processes can vary from team to team and project to project. For example, it is likely that developing a healthcare system requires more stringent processes, oversight and in-depth review than developing a marketing application.

Boerger reminded participants that there are two sides to establishing the right processes, and that once a process is defined, it is equally important that processes are followed. It does not matter what the process is if it is not followed. I know of only one successful engineering team with very few established processes. However, this team is comprised entirely of highly experienced individuals who work independently or in two- or three-person collaborations. In essence, each individual or subteam has its own processes. Perhaps, in fact, these individuals or small groups have many, many processes.

Use metrics to define the right testing
Another discussion focused on when and how to test in relation to using Agile or traditional methods. Some of the executives explained that they were struggling to implement Agile methods and wanted to understand how others were implementing this tenet of the Agile method. It seems that testing is one area where Agile methods can be pushed either too far or not far enough.

Boerger advocated the use of metrics to help find the right balance. This is a concept not often implemented, but the results can provide a significant improvement in software defect rates. For example, Boerger described how her teams combine predictions of a defect rate, calculated from continuously updated historical defect rate metrics (from previous project actual results). These predictions define the number and characteristics of defects the team should find within its own work as that work progresses from phase to phase, module to module, and prior to integration at the end of a sprint.

At this point one executive asked, “Do you mean to say that you test between phases?” To which she replied, “Yes, don’t you?” – and the room filled with nervous laughter. This led the discussion to the concept of Test First, as defined in the Agile methods. Most of the executives present expressed difficulty ensuring that engineers comply with Test First precesses, and I wonder if such difficulty is a holdover from poor habits. Many engineers find it easy to consider testing as something that comes after they have completed their work; after all, they don’t make mistakes, so why should they test, right?

Conclusion – Continuous Process Improvement
Boerger closed her discussion on the topic of Continuous Process Improvement (CPI), which she described as the “Holy Grail” of software engineering. As she pointed out, too often a process-implementation effort is considered only when something is obviously broken. Then, a band-aid is applied, only to have things quickly revert back to the normal level of chaos.

The Capability Maturity Model (CMM) identifies CPI as the highest level of process maturity. I wonder if some people feel that they must wait until everything is fixed before implementing CPI. Boerger suggested that a simple feedback loop can be established to enable everyone to contribute and thereby encourage the continuous flow of improvement ideas – backed up by good data (metrics, of course).

About the author
Ed Carroll has been building software products for more than 20 years, with particular expertise in automating economic analyses, decision support and supply-chain management processes. He has provided strategic technology leadership in roles such as the vice president of engineering for Egghead.com, director of technology at Nike and director of software engineering at Boeing. He can be reached at EdCarroll@AgilisSolutions.com.