This article is the second part of the series Your Company Will Never Be Agile. The previous part is Your Company Will Never Be Agile - Intro.

In order to be good at software development, you need an organization that's optimized for software development Samir Talwar

Samir told me that when I first told him that I wanted to do an essay / talk about “Your Company Will Never Be Agile”. His Theory: Many traditional organizations have organizational structures that are optimized for something else - Banks are optimized for banking, telecommunication companies are optimized for burrying cables in the ground and routing calls, and so on.

And I think it’s even worse than that: Many companies have organizational structures that are optimized for how those businesses worked several years / decades ago! They evolved over a long, long time, and now cannot change easily anymore.

If your company does not actively work on optimizing it’s organizational structure for software development, you probably don’t have a structure that’s optimized for software development.

Impediments to Agility

How can your organizational structure affect your agility in a negative way?

Remember: We are talking about true business agility here - The ability of a business to react quickly to changed circumstances. The benchmark here is basically the lead time for a small change in direction: How long does it take from the moment you recognize a change is necessary until you actually changed direction.

Such a “small change in direction” might or might not involve software development. But if IT is involved, we need to be able to deliver the new or changed functionality as quickly as possible, or we will slow down the whole organization.

Here’s how your organizational structure can impede your agility:

Wasteful activities: People do some activity - filling out forms, writing emails to different departments, … - that does not directly provide value for your end users. This is often caused by your organizational structure - When your departments and teams are organized in a way that is not optimal for delivering value.

Waiting time: You need someting from somebody in another team, but she is overworked: Queueing. You need answers from different people in different teams / departments, so you write emails with 10 people on CC and have to wait for the discussion to finish: Coordination. You need someone to sign a document before you can start implementing, testing or deploying: Clearance.

Hand-offs: Say you have two teams or departments who have to work together to provide some functionality: Team one does the first half, team two the second half of the work. Now you have a hand-off. This does not only cause waiting, it’s even worse: You lose knowledge in the handoff. You probably need written documentation to coordinate the hand-off, but then you always risk that someone will not understand it correctly. And they won’t ask, because they’ll simply do what the document says.

Communication bandwidth: People in the same team will communicate more and better - Even if they are not in the same room.

Departments / Silos

Many companies I have worked with so far had at least some of the functions listed below in separate departments:

Development: They implement functionality and fix defects. Maybe they do some unit testing, or even test driven development.

Release Management: Some central department that takes care what is currently released on which production and staging system and when a team / software system can do a rollout. You probably need clearance from them to release something to production or even to staging.

Operations: They deploy your application to the production systems and solve minor issues themselves (like re-starting cluster nodes). You probably need clearance from them too to do a rollout, and you also need at least one person from them during the rollout.

Quality Assurance: They test all new functionality you implement and also automate those tests.

Here’s a hypothetical, but not entirely unrealistic scenario. It’s a bit exaggerated, but I’ve seen things like that happen at past clients.

“Release Management” keeps a schedule of all releases to production. They coordinate multiple software systems, and multiple teams work on each of those systems. They also have to provide clearance (usually takes 1-2 days) for each of those releases. This means every team can only release so often - otherwise they would be completely overworked. Our team is allowed to do one release every two months.

In order to get clearance from release management, you need a changelog and a test protocol. The testing department tests most new features within 1-2 days after they are declared “finished”, then they need another 1-2 days to automate the test. When they find a defect in a new feature, it goes back to development and it takes another 2-3 days until it’s developed and tested again. Before each release, they also insist on 2 days of smoke testing before providing a test protocol.

Operations wants the deployment unit at least 1 day before the release, but they only accept it once there is clearance from Release Management.

So, in order to reliably do a release, we have to have a code freeze 10 days before the release: 1 for operations, 2 for clearance, 2 for smoke testing, and 5 for “test-fix defect-test again” of the last few features.

Features from earlier iterations/sprints will be delayed even longer, since they have to wait until the next release, which is up to two months away.

IT itself: Say you have four departments - A, B, C and a centralized IT department. Does it make sense to have IT as a separate department? You probably have the IT department because someone saw “synergies” in centralizing it. But often the IT department is then organized in three groups: A, B and C, where group A serves department A, B serves B and C serves C. They don’t even talk much with each other. So in this case, there are no synergies, but you have complicated the communication structure between each department and the corresponding group in IT. Sometimes it might make sense to have a separate IT department, but sometimes it would be better to move IT closer to the people who need them.

Communication Paths

In many companies, the established communication channels make it hard to get the information you need quickly when you need it. And decisions take longer, because you need people from multiple teams/departments to make a decision, but you can only communicate with them through the established channels.

“Reduce the distance between problems and problem-solvers” is one of the points of “Agile Doctrine”, as coined by Jason Yip. And this distance is still huge in many companies. Not necessarily the physical distance - All persons might be in the same building. But the “organizational distance” can still be big: How easy is it to communicate with all the persons you need information from?

Team to end user: I have worked with companies in the past where developers are not allowed to talk to end users. The PO does the talking. But actually, the PO talks to business analysts. But they don’t talk to end users either - they talk to the boss of the end users. That’s 5 rounds of Chinese Whispers between end users and developers.

Availability of key persons: Maybe you are allowed to talk to everyone you need information from, but only at certain points in time. In the past, I worked with teams where we only could talk to key stakeholders during project initiation phases - They just wouldn’t have any time for us later. So the business analysts had to document everything they said in writing, and later we hoped that we understood the documentation correctly (Remember: Documentation Lies). And the key stakeholders could not change their minds without a change request.

Team to other teams: You want to do something, but in order to achieve your goal, you have to coordinate multiple people from different teams (operations, ESB, other application teams, …). When this happens too often, it is an indication that your team structure is not aligned with your business goals.

Strategy / Goals / Budgets: Often, companies try to hide the details of the company strategy, the tactical goals and the current financial situation from their “line workers”. They do so with the best of intentions: The line workers don’t necessarily need to know this, so why should they spend time on learning / processing this information? The problem is that self organization cannot work without information. Decision making and problem solving takes longer, since information about the problem first has to get to a level in the hierarchy where all necessary information and power is concentrated, but it will arrive there diluted. The solution will be watered down too when it finally reaches the people who need it. And the feedback loop is very long.

Inappropriate communication channels will slow you down considerably. And if you are slow to take and implement decisions, you cannot be truly agile.

Example: The Enterprise Service Bus

If you have an Enterprise Service Bus (or ESB), this will slow down your development. You might ask: How can an architectural detail like an ESB be a problem of the organizational structure? Here’s how:

So, you have different software systems and subsystems that communicate with each other over a network. To make sure that you have well-defined communication protocols in place, and also for some flexibility, you bought a piece of software that manages and channels all messages between your software systems - an ESB.

If you have an ESB, you probably also have an ESB team. The ESB does not only route messages, it also filters them, provides auditing, can translate messages, implement fallback, and so on. You need someone to implement and maintain all this functionality. Your ESB team does this - After all, it would be a mess if all teams added functionality to the ESB themselves. Also, not every team can have an ESB expert on the team, right?

This is actually an example for both “Departments / Silos” and “Inappropriate Communication Channels”. Because now you have a bottleneck: The ESB team. You need them for all changes that involve any communication between any two of your subsystems. That’s potentially a lot of changes - Especially if you really want to have an Evolving Architecture.

Even worse: The functionality has to be deployed to production on the ESB before it can be deployed on the subsystems. If you have scheduled deployments, like in the example above, and the ESB team deployes every two weeks, then you have to wait for your next deployment after their next deployment after all teams were done implementing the change. This adds weeks to your lead time!

End-to-End in the Value Chain

You have to “Reduce the distance between problems and problem-solvers”.

You will move fastest if you have small, cross-functional teams that can work on the whole value chain - From a problem some end user has to deployed software on the production systems.

You have to turn your company structure by 90 degrees. Bernd Greifeneder

What would have been different departments or teams in the past - Development, QA, Operations, … - should now be different roles in cross-functional teams. Teams that can do anything that’s required to deliver value to the company, along the whole value chain.

This article is part of an on-going series. Want to get notified when I write the next article in this series? Subscribe to my newsletter (you should also see a popup below) and you’ll get a mail with my next article.