How Agile Can Remedy a Bad System

How Agile Can Remedy a Bad System

Do you want to know who you can blame for the latest fighter jets running hundreds of billions of dollars over budget, getting delayed for years and never living up to their potential?

You can blame me!

Well, you can blame me and thousands of others who have worked on fighters like the F-22 and F-35. These “next generation” planes are examples of what happens when the people making them don’t communicate with one another very well.

Let me tell you a story of a usual day working on the avionics for the F-22, back in 1990.

I was a software engineer working on some code that helped the F-22 communicate digitally with other planes. This was something fighters hadn’t done before and it was an important “feature” of the jet.

If the F-22 could receive digital radar information from a large radar plane 100 miles back behind the fight, it wouldn’t have to use its own radar and expose itself.

A bad system

So this new feature required unique software and hardware. It also used encrypted data, which classified the work as secret. Much of the rest of the jet had parts that were secrets as well. Maybe the name of the jet and the wheels were the only things that weren’t secret.

So, in developing this software for prototype hardware in development, I often had questions for the people who made the prototype. Unfortunately these people were in a different state.

In fact, people working in more than 35 states created the F-22. The reason for this was that in order for congress to approve the enormous cost of the plane, most congress people had to have constituents working on it. So Lockheed Martin broke up the work to ensure congress would approve.

Now it’s hard enough for people separated by a few time zones to communicate, but when they have to communicate about something with a secret classification, it gets downright near impossible.

So the conversation process was set up like this:

-I had to request permission from security to have a conversation on the phone.
-Security had to contact the hardware people and schedule a conversation. The earliest was a week away.
-I had to find something else to do for a week.
-When the appointment arrived, I was escorted to a secure room that had a scrambled phone.
-A secure room had “pink noise” (which sounds like a washing machine filling up with water) to mask any sound from leaking out.
-The scrambled phone degrades the sound quality of whoever is speaking.
-A security officer has to be present on both sides of the call.

Finally when I was able to speak to the hardware engineer about the problem I was having with his hardware, we were not allowed to have any casual conversation to start (in case I learn something about his classified details).

With the pink noise and scrambled phone connection, he was very hard to hear. So I started yelling a question into the receiver about my problem, but before I could get my first sentence out, the security officer in the room interrupts me and says I can’t go into specifics on secret technology.

I try to ask him what the point of having the call in the secure room and the scrambled phone was, but he wasn’t budging.

The conversation I waited a week for was reduced to: “I’m having a problem with something you made.”

Obviously the hardware engineer could offer little help beyond suggesting I reboot the hardware.

As a result, we had to put off getting much of the avionics to work for a year until we were able to integrate it all in a shared lab we all visited.

As you may imagine, integration was a disaster and delayed the plane for at least a year. However, we did solve the digital communication problem in a different way, described below.

Fixing a bad system

The managers I worked for on the F-22 program knew the policies of secure communication and had some control over them, but they didn’t fix them, and there was no urgency to do so.

We developers had a sense of urgency, but no power to change the policy, so we stopped communicating with our distant colleagues, which was bad.

The obvious solution is to couple the sense of urgency with the power to create change. This is a core principle in agile. It raises the urgency for fixing these problems to those in power through transparency, and it raises the power and accountability of the developers who are closest to the problems through self-organization.

Creating transparency

Transparency is created through frequent inspection of “what” we are making and “how” we are making it. In Scrum, every one to three weeks, we are reconciling what we achieved in a sprint review with what we forecasted we thought we might achieve in sprint planning.

The cost and impact of practices like secure phone conversations would become more apparent because they slow the velocity of adding features such as digital communication.

It’s harder to hide behind a complex schedule or hope that the problems will magically go away in some distant integration phase. We integrate as much as possible every sprint.

This transparency and velocity are tools for experimenting with inspecting the policy of secure communications and adapting to new policies that should increase velocity.

Creating self-organization

Scrum also defines the retrospective that allows the developers to discuss what is slowing them down every iteration, and to experiment with new ways of working to improve their velocity.

If developers don’t have this power to try new ways of working, this virtuous cycle does not occur. Fortunately we had a manager that trusted us and let us experiment.

Developers on secret programs have little power to override security protocols, but we can innovate within them to improve “how” we work.

In this case, when requested, our manager gave us permission to fly out and meet with our distant colleagues instead of calling them. For some reason, security policies allowed us to be left alone in a secure room with the hardware to discuss whatever we felt like.

Although it took more time to fly than to dial, we were able to solve many integration problems quickly in a day of meeting together. As the young, single engineer, I was the one that traveled the most to discuss the hardware and, after a few trips, became a limited expert on the hardware.

From then on, whenever a software engineer at our location had a problem with the hardware, they’d first come to me. If I couldn’t fix their problem, I would add it to my list of things to talk about during my next trip.

The same goes for games, actually

As a result of this new way of working, the digital communication part of the avionics did not have the integration problems that other areas did.

When I switched my career to video game development, I found similar problems occurring with distributed teams.

Although the security policies weren’t to blame, there were other problems with culture and organization that separated disciplines and put a systemic burden on their work, much like I experienced earlier.

So when someone asks me why it’s valuable to have cross-discipline teams who are co-located as much as possible, I remember the F-22.

Distributed or discipline-centric teams might not have the communication barriers I did in 1990, but when there are thousands of cross-discipline conversations that need to occur over the course of development, any barrier in the way of those, such as management tools or chains of communication or a long hallway to walk down, add delay – or worse: eliminate crucial conversations.

The moral of the story is that if your teams aren’t performing, take a look at how they’re organized. As W. Edwards Deming said, “A bad system will beat a good person every time.”


Clinton Keith is an independent agile coach and Certified Scrum Trainer with 20 years of video game development experience.  Clinton introduced the game industry to Scrum in 2003 and Lean/Kanban in 2006. He has coached teams at many studios.   He is the author of "Agile Game Development with Scrum”.

Learn More
comments powered by Disqus