The Big Projects Always Fail: Taking an Enterprise Agile

Roger Valade
Vice President of Technology, Entertainment Publications

[This whitepaper was published in the proceedings of the Agile 2008 (Toronto) conference and formed the basis of the presentation I gave there. You can buy the official IEEE copy of this paper here.]

Abstract

My overall approach to software development has been to decompose large efforts into smaller, tractable components, and failing fast when things appear to be heading in the wrong direction. Though not a particularly unusual model, I have been able to road-test it over about eight years of agile experience, including a stint at DaimerChrysler on an Extreme Programming (XP) team that included some of the members of the original C3 payroll project. This pattern has been a keystone of our direction at the Technology division of Entertainment Publications, where we began our agile transformation about two years ago. Though it was originally a small pilot with an unspecified roll-out plan, we now use an agile approach for all our software development and support. This paper highlights the lessons we’ve learned in the process — some items for you to either avoid or embrace.

1. Introduction

Entertainment Publications is a 46-year-old publisher of coupon books that was born here in metro Detroit, where it continues to be headquartered, and now serves approximately 170 North American markets. It has a very unique business model that centers on the distribution of discounts and promotions (coupons) through both the consumer and fundraising channels. The products that are vehicles for these discounts include both traditional print and web delivery models. This paper will describe how we overturned the previous software development model, which varied among teams and was distributed along a spectrum from waterfall to unplanned chaos. When I stepped into this environment in 2005, my focus was on the online product side of the development organization. While I completely enjoyed the entrepreneurial nature of the development approach, the disorganization and unpredictability were becoming more and more of a concern for our business customers. We needed to make a change.

Our IT organization of roughly ninety team members is composed of an application development team of forty spread across both our web-oriented Java platform as well as a PeopleSoft Enterprise Resource Planning (ERP) solution for our back-office functions. Since the beginning of our agile path, we’ve created collaborative team rooms without significant expense and implemented a system for tracking and managing our story cards with a lightweight commercial application that we’ve modified for our needs. We’ve embraced weekly iterations for planning and all of our applications have the opportunity to be released to production every week. We are constantly tailoring our processes using a fail-fast model to ensure that we’re improving on an ongoing basis.

2. Why Agile?

My own experience with agile began with Extreme Programming back in 1999 at DTE Energy, where I introduced an agile approach after learning about XP at UML World 1999, I think from a presentation by none other than Martin Fowler. This small, successful project bolstered my admiration of the model and, in 2001, I was fortunate enough to work on an XP-by-the-book project at DaimlerChrysler with a few of the members of the original C3 payroll project. I also participated in the local Michigan Extreme Programming Enthusiasts (MXPE) group, where I met a number of the fine agilists at Menlo Innovations in Ann Arbor, a software development shop that does a wonderful job evangelizing the approach.

When a large site rewrite project at Entertainment was initiated in 2006, I asked a number of the team’s members to attend one of Menlo’s agile workshops. A week later there were makeshift boards on the walls, the team was starting to arrange its workspaces in a collaborative configuration, and daily stand-up meetings had been scheduled. A grassroots revolution was born. As Menlo’s Richard Sheridan had said about his own implementation of XP, “We had reached a point where the risk of not changing was greater than the risk of change.” Unfortunately, we were in the same place; fortunately, the team realized it.

3. Agile Web Development

Our first agile project was a rewrite of one of our core websites. It was a complex project because the user interface was undergoing a relatively dramatic change and our immature processes didn’t provide the confidence we needed in ourselves or that our customers needed in us. Regardless, with the ambition, passion, energy, and hope that a new approach provides, we jumped in head first. We learned a number of lessons during our first agile implementation:

1. Be prepared for your business customers to be skeptical. Even though they may (and need to) support the change, they will have a lot of questions and wonder if this is really the right thing to do if they don’t have some sort of agile background. In our case, we believed agile’s transparency and steerability were obviously good things, but these qualities also demand a certain planning mindset on the part of the customers that they may have never had before. I was not prepared when, early on in the adoption process, one of our customer executives told me (paraphrasing): “I don’t like agile because now nothing is your fault — we have to plan everything ahead of time and you simply have to execute.” I thought I was helping to enable a sea change, but, as with most things, it is critical to understand what your customers think. Hopefully you have their trust for the first agile project, but you will need to constantly remind them of the speed and complexities of the changes ahead and that the road, on a day-to-day basis, will be rocky.

2. Transparency is your frienemy. Our business customers love the aforementioned steerability in general. But they hate the planning process. They love to prioritize. But they hate it when cross-functional resource conflicts leave their hands tied. If you don’t have an established mechanism for project and program management, prioritization, and planning, your new agile process might expose that in a way that actually makes it “feel” like an IT process problem. This happened to us rather dramatically and the ROI of doing the proper expectation management upfront cannot be overstated. Obviously, how much work you need to do depends on the mentality of your business customers, and gauging how your customers might feel about this sort of transparency is a critical success factor.

3. Embracing the empirical management model isn’t easy. Like most IT customers, ours are very date-centric. The variability of weekly planning and some of the deadline uncertainty that can come out of a noisy project is probably their biggest dislike of our development process. We’re slowly migrating to a Scrum sprint model to smooth out the variability that the empirical management model, which Ken Schwaber [1] describes so well, fosters. This approach has been met with appreciation by a number of our customers already and is probably where we should have started two years ago.

4. Don’t forget a Program Management Office (PMO). I’ve heard of companies that decided to close down their PMO when they moved to agile because such a group either has a waterfall flavor or a heavyweight connotation. I have to say that I can’t imagine implementing agile without a clear set of project managers and ideally business analysts in the mix. Agile is a great software development and implementation methodology, but you still need people to oversee the project initiation and governance model, as well as the deployment process. Yes, tech leads can do this, but ultimately most of them really don’t want to. Having a dedicated team to oversee and manage the process is crucial. Our development managers operate much like coaches or Scrum Masters and keep the team and our customers focused on the appropriate approach to development when new challenges arise.

5. Some of your developers will hate it. Even though agile is a lightweight and fairly simple process to implement and follow, there will always be frictional feedback from some members of the development team. The great thing about agile is that these developers’ disposition will become extremely obvious in a short period of time, and you’ll have the opportunity to either course correct or decide whether these folks really should be on your team. There are a few different facets of this mentality that we experienced:

1. Developers who dislike meetings. We have a number of developers who consider meetings as anathematic as filling out time sheets or imbibing caffeine-free beverages. Stand-ups, even though only 15 minutes in length (if not shorter), will feel to them like an eternity. I consider stand-ups to be more of a social event — getting information out to people who may not know they even need it, planning for meetings and obstacle-removal conversations scheduled for the day (“Hey, we need to meet to discuss the new mapping service”), and generally demonstrating human interest in your colleagues and what they are doing. The stereotype of developers being isolated and anti-social is occasionally true (though more often it is not). If any developers on your team have these tendencies, they will need coaching on how to fit into the new culture you are creating.

2. Developers who dislike talking in public. Even if members of your development team do like meetings, a larger number of them do not like to speak publicly when the topic isn’t their latest design or a technology in which they are passionately interested. The routine of the stand-up is a proactive opportunity to overcome this dislike specifically in order to make public speaking something that comes more naturally to them and to improve the conversationality of the team spaces.

3. Developers who dislike “management.” It was a difficult day when I realized, much later than I should have, that the developers in our organization considered me more a manager and less a developer. One of the exciting things that I had always looked forward to was finally being a leader who understood development and was interested in building an organization and culture that was designed around sensible and effective processes and tools. Unfortunately, unless you’ve grown into your leadership role from within the company that you’ve been a developer in, you may be perceived to be a Dilbertesque manager with evil intent if agile is considered to be your idea. Regardless of how you get your agile implementation off the ground and how many evangelists you have in place, there will always be the curmudgeonly cynical who consider any process change as a attempt by “the boss” to control his or her day, and you need to be extremely sensitive to that viewpoint. My own development experience with agile was a good talking point, but it still was something that happened in the past and wasn’t real to my current team. Fortunately, there were a few thought leaders in the PMO and development ranks who provided enough momentum to overcome most of these concerns.

This initial project suffered a number of business direction twists and turns, and in many ways was eventually abandoned (some of the new work made it into the existing website; much of it never made it out of Subversion). However, so many members of the development organization were excited by the original effort with agile that we were soon using this approach for all of our web development and support projects (which was a project portfolio of roughly five sizable projects and about twenty-five developers). It is no small point to note that the methodology itself survived the project’s demise. During this larger roll out, we learned a few more lessons:

1. Embrace team rooms or spaces. We had been talking with our facilities manager about our interest in team rooms for a while when our new CEO decided that the executive team should move out of the executive suite to be closer with their teams, who were dispersed around our building. The facilities manager called and asked if we would like to move our development teams into these six now-vacant, beautifully decorated offices. It didn’t take us terribly long to say yes. We moved out the furniture, ran out to IKEA and bought thirty new worktables on casters and a few new chairs, ran some power and network cables, and we were in business.

2. Don’t forget about personal space when planning your team rooms. In our eagerness to move into the team rooms and get collaborative, we overlooked some of the basic needs of our most valued assets: the developers themselves. The reactions to a collaborative team space vary widely, but it is a really good idea to make sure that you provide some personal zones for people to move away, decompress, and focus when they need to. Many developers need relative silence and few distractions to be effective. Others love the collegial din of a hypercommunicative and collaborative space. But everyone needs to call their doctor, spouse, or car dealership every once in a while, and having an area in which to do so is an important facilities feature. We are finally rebuilding our team spaces this year and won’t make this mistake again.

3. You need a system to store, sort, prioritize, and archive your story cards. My original agile teachers were firm that stories must be on real index cards, handwritten, and handled. The physical modality of the cards was an intellectual and nearly sensual part of the methodology. Our friends at Menlo Innovations, who have a great workspace that makes you want to sit down and start coding whenever you walk into their office, had found that the building material Homasote was a great, inexpensive, cork-board-like surface for your story boards. We purchased a ton of it for $300 and had more surface than we really needed. After two years they are still in great shape. However, the cards become a management problem when you are trying to run a number of major projects across a sizable team of developers. We purchased a lightweight project management tool called Copper, tuned it to our needs, and haven’t looked back. Having 24/7 access to the cards, their status, and the overall project health is invaluable and something that the ritual of physical cards couldn’t duplicate. It also enables us to give the developers a view into what is expected of them for the current iteration through the window they typically love the most: a web browser. However, we do still print the cards in a 5×7 index card format and post them to our swimlanes so that we don’t lose the simple visibility story boards provide.

4. Expect to iterate the structure of your daily stand-ups. We started out with our small teams doing their own stand-ups independently and simultaneously. At 9:30 every morning, each aisle in our workspace became a makeshift meeting room. It was noisy and chaotic and I loved it, but it was logistically difficult for anyone to attend more than one stand-up. We then switched to an all-hands stand-up in a bigger space — the uninhabited and unfinished office suite next door — and went around the room with nearly fifty people. Unfortunately, the level of detail that we were getting out of everyone diminished as the psychological pressure of getting through the entire group in fifteen minutes naturally condensed each developer’s talk time. We failed fast and have moved back to small team stand-ups staggered in fifteen minute increments from 8:45 AM through 10:00 AM so that our managers can attend all the meetings they need to. I also have a leadership stand-up at 9:00 AM that is less focused on development and more on status and obstacles. For all our stand-ups, we take Wednesday mornings off in acknowledgement of our corporate “no meeting” day. This works well because four daily stand-ups only consume one hour every week, which is more palatable for those developers who don’t like meetings.

5. Avoid static team composition. Our business customers love the concept of dedicated resources on their projects. This dedication breeds domain knowledge, provides a consistent face for these customers to interact with, and can provide additional productivity over time as the developers expand their understanding with the system and the business it supports. However, there is a downside to this dedication as well: developers grow weary of the same domain and application and begin to suffer from burnout. And, in the case of attrition, there can be an imbalanced concentration of expertise that leaves your organization and puts you in a precarious position. Originally we were working in a very rotation-centric model that varied the specific developers associated with any one project on a weekly iteration basis, but we kept the tech lead constant to provide the needed oversight and direction. This widened domain knowledge and reduced project weariness, but was too dynamic for our customers. We now take a more case-by-case approach to resource dedication, drawing allocation lines at project milestones or at some many-iteration increment, which provides us with a good balance of variety and dedication.

6. What about business analysts? At first, we tried to enable our project owners to be the business analysts and provide us with atomic requirements in the form of story cards. We held informal training sessions to show them how to write story cards, how to break down requirements into estimatable tasks, and generally how to provide us with the input we needed to estimate and plan our work. This tactic did not succeed, so we failed fast and started to ask our quality assurance team to step up and play the business analyst role, thinking that this would give them a head start when they needed to test the new features they had helped specify. This was better, but our QA team of three had a hard time keeping up and there was some sentiment from a number of our executives that this capability should really reside outside of IT. Now we are in a somewhat ambiguous state where the business or requirements analysis role is shared. I’m still not that happy with this arrangement but given the size of our team it is the only practical option we have.

Obviously, no software process is magical, and we continue to have our fair share of prioritization, planning, requirements, and deployment challenges. However, we don’t have projects that run significantly over budget or schedule without an early warning, and this is an invaluable benefit. We’ve also seen other, non-IT teams adopt some of our methods in order to organize and prioritize their own work.

4. Agile Enterprise Resource Planning

After the success of the online team’s implementation of agile, we began to extend the methodology into our PeopleSoft team. We have a fairly robust and mature ERP implementation that includes customer relationship management (CRM), supply chain management (SCM), finance, and human resources modules. The work that falls into this groups’ queue is mostly just that: a production support queue. We used a straightforward ticketing system to manage this effort and were excited by the possibilities of renovating both our tool-set and our approach to this area of our business. After about a year, we’ve learned:

1. Running a production support queue like an agile project is actually really easy. The majority of the work that falls into the ERP team’s plate is standard fare: this user isn’t seeing the data in the report she expects; this account isn’t balancing because this transaction didn’t post; this employee’s security role is misconfigured. We took the same agile approach that we were using for new development requests and simply pointed it at this stream of incoming work instead. Each non-urgent ticket is treated like a story card that can be estimated and planned as a part of our normal, weekly prioritization process by the same business customers who are responsible for our normal planning. This approach has caused some logistical challenges for our development managers who have to manage two systems — our project management system, which includes the non-ticket enhancements, and our ticketing system — but it has not been too overwhelming and the benefit and sanity that this approach has provided to us is non-trivial.

2. Production issues can derail you if you don’t explicitly plan for them. When the site goes down, the critical process is failing, or the deployment needs to rollback, the best laid plans start to crumble. The process makes adjusting to change extremely simple, but sometimes the process isn’t the problem, it is the established expectations that now need to be managed. When one of our business customers is told that yesterday’s production issue means that the task they were hoping would be completed this week might not make it, they generally understand. However, when someone less directly involved with the outage is negatively affected by a cascade of dependencies, things tend to be a little more challenging. As mentioned above, transparency is a great thing and a sore point simultaneously. Overall, though, our agile approach makes handling this change no more difficult than any other change. We also realized that reserving a certain percentage of development time every week for production issues and tickets greatly simplified the planning process and contained the amount of chaos that any one issue introduced into the development and deployment cadence.

3. Breaking items down into plannable and estimatable tasks is particularly hard within an ERP. Our development teams are focused on specific ERP modules. More so than with our other application development teams, requests span these team boundaries. As an example, an order needs to be created in CRM, passed to SCM for fulfillment, and then the transaction posted into Finance. An adjustment to one of these core objects needs to be synchronized across all of these teams. Though we don’t use the term internally, the scrum of scrums approach [2] is helpful in resolving these issues. We also empower our development managers to override our customers’ prioritization in these cases and this mechanism helps plan through such complexities.

4. Unit testing and source code control are particular challenges with an ERP. The web side of our development practice is based on traditional open-source technologies that have become the de facto standards for many teams: Java, Apache, JBoss, Hibernate, Subversion, Eclipse, CruiseControl, and JUnit. We have not found affordable source code control systems for our PeopleCode assets, nor have we identified an efficient way to build unit tests and implement continuous integration in a PeopleSoft environment. Fortunately, we’ve been able to use our over-arching processes to manage the work as we do in the Java space regardless of the implementation details at a day-to-day level, which provides us with the same discipline and predictability we so desperately sought.

5. Ultimately, it is all about communication and relationships. The expectation management issues mentioned above are all tractable problems with a business customer who has faith in the team and understands, at least at a skin-deep level, the difficulties of software development and support. The relationship that agile enables with your development teams and their respective customers is unlike that I have ever seen with any other methodology. Our success or failure is mutual, and the better the durability and trust that the relationship holds, the more likely a successful outcome over the long term. I’ve been consistently surprised with the speed with which we’ve either built or extended solid relationships using a software development approach that many, in the beginning, could never imagine would work.

In some ways, the agile implementation for our ERP area was actually easier than in our online team’s. It was mostly the relaxation of some waterfall practices and the implementation of a transparent prioritization process that helped manage customer expectations in a cleaner way than had been implemented before.

5. Embrace Change and Fail Fast

Wired ran an interview of Freeman Dyson [3] in which he had the following exchange with Stuart Brand:

Brand: So you’re saying just go ahead and try stuff and you’ll sort out the right way.
Dyson: That’s what nature did. And it’s almost always true in technology. That’s why computers never really took off until they built them small.
Brand: Why is small good?
Dyson: Because it’s cheaper and faster, and you can make many more. Speed is the most important thing — to be able to try something out on a small scale quickly.
Brand: Fail fast.
Dyson: Yes. These big projects are guaranteed to fail because you never have time to fix everything.

“Fail fast” has been a mantra of mine ever since, and one of the attractive components of an agile approach is its empirical nature. When things go wrong, steer them right. Find out about problems early, when you have the time to fix, or at least respond, to them. This same concept needs to guide your agile implementation as well. Throw away the things that don’t work quickly and embrace the things that do. Break your projects into increments — make them small — because the big projects always fail. It is a bold approach because of its simplicity and straightforwardness, and, most importantly — and sometimes amazingly — it actually works.

6. Acknowledgements

I’d like to thank the wonderful members of the Entertainment technology leadership and PMO team, without whom none of our success could have happened: Tristan Boutros, Jennifer Cardella, Andy Clark, Jason Gamet, Lisa Kloock, Dave Limer, Gordon McCowey, Jim McDonald, Noelle Missant, Stacey O’Connor, Greg Pogarch, Lisa Pollard, Tim Purdie, and Carrie Weidenbach.

7. References

[1] Schwaber, K., Agile Project Management with Scrum, Microsoft Press, 2004.
[2] Cohn, Mike, “Advice on Conducting the Scrum of Scrums Meeting,” May 7, 2007, http://www.scrumalliance.org/articles/46-advice-on-conducting-the-scrum-of-scrums-meeting.
[3] Brand, Stuart, “Freeman Dyson’s Brain,” Wired, February, 1998, http://www.wired.com/wired/archive/6.02/dyson.html.