Adapting Engineering to Growth

By Ed Carroll, Sales Executive, Agilis Systems

The economy seems to be improving – investors are investing, employers are hiring, revenue reports are up and expansion is in the wind. What does expansion (or contraction, for that matter) do to a well-tuned engineering program? The Engineering Executive Forum will explore this issue in the February 28 forum. For those of you who want a little preview, those of you who cannot wait until the 28th, or those of you who just like to read my prose, I’ll give you a flavor of the discussions to come. We’ll break the discussions into four areas, 1) Process, 2) Architecture, 3) Product Road Map, and 4) People.

Process

When an organization experiences rapid growth, a common answer to the inherent chaos caused by the increase of new people into the organization who don’t have the long-held cultural and technical understandings of the original team is to add processes in order to “control” engineering.

Thirteen years ago, shortly after I was newly minted as Director of Software Engineering, the company senior executives decided that they were tired of the chaos (we were experiencing significant growth) and that we needed to improve our processes by introducing me and the rest of engineering middle management (there were about a dozen of us leading some 300 engineers) to the Capability Maturity Model (CMM) from the Software Engineering Institute of Carnegie Mellon University. (For an intro to CMM see my previous article: Will CMM Help?)

My story on implementing process control is a little unusual. While other groups in the company experienced significant growth before the implementation of CMM (the reason for our chaos), my group was only newly formed. Our efforts in continuous process improvement resulted in explosive growth, rather than the other way around.

At the start of this endeavor we put a huge amount of time and effort into training (convincing) our management and staff on the virtues of Continuous Process Improvement (CPI). Each manager was given a topic to champion, not only with their own team, but throughout the entire engineering staff. The affect was two-fold, 1) it meant that we, as managers, were not around to “lead” our own team while we were off training the others (a positive attribute), and 2) it meant that the engineering staff was exposed to all of the managers (a negative attribute). Toward the end of this massive classroom effort we evaluated ourselves according to the key performance areas of CMM. Interestingly, all of the engineering staff pegged us as having very immature processes, and all of the managers (except me, of course) felt we were already well along on the maturity curve. In fact, it took another two years for most of the managers to agree with the staff’s assessment. Personally, I consider this epiphany we brought to the management team to be one of our greatest achievements.

In the beginning all groups seemed to be following the CMM coaching faithfully, starting with the creation of a CPI program and the naming of a Software Engineering Process Group (SEPG). That was the fun part, because we exposed some definite weaknesses and recognized some very talented people. Then we tackled the standardization of our project management practices. It was a major eye opener to many, in particular senior management, when they discovered that no two project managers followed the same process. It is interesting to note in hindsight that this fact alone should have clued management in on where we stood on the CMM scale, but it did not.

As I mentioned, in the beginning everyone was on board with our efforts, even enthusiastically supportive – both managers and engineering staff. We managed to smooth out a lot of bumpy processes and establish a common language around what was “supposed” to happen on a project or through a release cycle (we both developed commercial software products and custom solutions).

Enthusiasm started to wane when we tried to establish engineering standards, and fell off completely in some groups when we held them accountable to the standards. Perhaps this is because the staff felt that defining project management procedures was like telling management what to do. However, when we started to establish engineering standards, even as simple as code objects or annotation or common libraries, the staff felt like management was telling them what to do again (the old model was back). This should point out the value of having the people who must do a process participate in the decision of what that process is all about.

We lost a lot of momentum in many groups. Nearly half of the engineering groups stopped contributing to the improvement process. I’ve always felt this was unfortunate. My group and a few of the others continued the program, achieving a very impressive record of reduced cost, accurate predictions, and (best of all) a reputation far exceeding the other groups for the ability to develop solutions to virtually any problem.

The real lesson learned was the direct relationship between process improvement and growth in revenue. Word of our capability spread quickly – fueled by our happy customers – resulting in a 10-fold growth in revenue in just two years.

The product road map and technical architecture

I once worked for a CEO who was (or thought he was) the consummate salesman. It seemed as if he would close a deal every time he met with a prospective customer. This CEO loved to read the trade magazines, looking for new ideas. Then he would run out and talk about what he had read with prospective clients. The only problem was that he was selling features that our software product could not do. Sometimes he would even sneak down into engineering and talk an engineer into creating a little prototype shell application (just look and feel, no substance, integration, or QA). The first I would hear of the app was in his announcement that he had sold this new feature to some customer. I could have pulled my hair out trying to control my boss, the CEO (but then I would look like my three bald brothers). It would have been nice to have had a well-ordered process where someone in Marketing created the ideas – writing them down in neat, detailed, descriptive requirement documents. Yet, if I had that process, I probably would not have learned how to talk to customers. I learned a lot about being flexible to new ideas and how to incorporate change into a product release cycle.

Prior to my education from the eager CEO, back when I was implementing CPI, engineering managers controlled what features went into the next release. We would apply our superior engineering-based, systematic “logic” litmus test to feature requests from salespeople. Fortunately for us, we were building software to be sold to other engineers. Then my eager CEO unintentionally taught me that not all features requested by a customer are re-sellable to other customers. I developed the attitude that it was Marketing’s responsibility to define the features of our products; all I cared about was whether the product was well-built, on-time and on-budget. It took me a long time to learn that I also had a sales role to play. In hindsight, it seems obvious that when things worked best was when there was a collegial equality to the give and take between Sales/Marketing/Consulting (what, what order) and Engineering (how, how much, when), with ownership of the product marketing or product management process belonging to the group closest to the customer.

People

Once, while announcing some new procedure that I expected the staff to follow, I was accused of trying to turn the place into Hewlett-Packard. That comment floored me. My snappy comeback was something like, “And that would be so bad, why?” After all, HP is a billion-dollar company and considered successful by many measures. Imagine our situation: A year previous to the comment, we had merged two companies together, resulting in an engineering department of about 80 members, split into two teams. One team considered themselves very entrepreneurial, while the other felt that they were at the peak of process efficiency. Guess which team had a problem with being like HP? Ironically, they both felt that they were better than HP, but we were a long way from HP’s revenue or profit margins.

Another similar comment made, again while introducing new processes, was, “All of these procedures are stupid. They stifle my creativity.” Heaven forbid that we stifle anyone’s creativity; we are only trying to build a sellable product. What procedure did this creative engineer disagree with? He was opposed to writing out a design specification. A real team player he was, but then I guess he was never asked to maintain code developed by someone no longer with the group, or work on a team of developers larger than one (himself).

In another situation, we had a senior manager retire. His replacement came in and did what all new senior managers do, he reorganized. What amazed me as a young manager at the time was that even though we assured the team that the reorganization had absolutely no affect on my group, every last person in my 15 person team expressed concern that they might lose their job.

The common thread in these three examples is that software engineers truly are a lot like cats. Perhaps it is because we spend our days myopically focused on a computer screen. When we rarely lift our head up and look around, we are surprised to see that things have changed and it scares us. The lesson I’ve learned in managing people is to always consider the individual. When someone complains about too much process, perhaps they are saying that they do not understand the objectives. When someone complains about their creativity being stifled, perhaps they are saying that they have not been heard. When someone worries about things that do not affect them, perhaps they are saying that they do not feel included. Our society tends to look for blame too quickly in my view. I prefer the adage, “Blame the process, not the person.” As organizations grow, these types of issues multiply many-fold. If we can remember to consider the individual, perhaps growing pains will not be so painful.

Conclusion

Revenue, organization and responsibility growth have significant impact to continued or future success. The negative impact to engineering happens when there is a mismatch in the appropriateness of layered processes, product definition and personnel oversight. The real trick is to apply control slightly behind the growth curve. That little bit of chaotic edge helps to keep people interested, excited, and looking around, but not overwhelmed.

About the author
Ed Carroll has been building software products for more than 20 years, with particular expertise in automating economic analyses, decision support and supply-chain management processes. He is currently a sales executive with Agilis Solutions and has provided strategic technology leadership in roles such as the vice president of engineering for Egghead.com, director of technology at Nike and director of software engineering at Boeing. He can be reached at EdCarroll@AgilisSolutions.com.