Why it is so hard to change established processes in organizations

Or why so many DSL projects fail for non-technical reasons

Business Process Improvement with DSLs

Most of my DSL-related projects are ultimately about business process improvement: we’re trying to improve the processes how subject matter finds its way into software. We achieve that improvement by developing languages that are used by subject matter experts to directly and formally express and test that subject matter, and then automating the translation to source code.

As we do that, the existing processes must go. This should be obvious. And companies only hire me and my colleagues if they understand that there’s something wrong with their existing process. So there is some degree of awareness that there’s a problem. And problems there are! You wouldn’t believe how broken some (not all!) of these processes are, even for companies that are leaders in their respective markets. So replacing these broken, slow, and low-quality processes should be easy, right?

Challenges

Of course I wouldn’t write this if that were the case. Streamlining processes requires buy-in from all the people and roles involved in the current process. And processes that are especially broken often involve especially many roles — that’s often a major reason why they are broken. You have to convince each involved party that the new process makes their life easier while making sure they remain “important” to the overall story — nobody wants to lose status. And most changes, when you propose them, don’t seem to people that they make their life easier because people tend to a) confuse the transient state of learning something new with a potentially more productive steady state after the learning is done, b) psychologically value loss (of the current way of working) higher than benefit (of a new improved way) and c) oh, DSLs really look like programming, and I hate programming.

Leadership (not pressure!) from management can help overcome these problems by painting a picture of where the organization should evolve to and how that improves the business and everybody’s everyday work. But good leaders in management aren’t always available. And often management is too far away from the particular processes for them to convincingly talk to the people who must change their ways.

So what? Well, such projects then tend to fail. Or they require so many compromises in the new process that many of the benefits of the new approach disappear (which then leads to the new process not delivering, and probably dying later). The organization muddles on … until their competitor does a better job of improving their process.

The Accidental Process

So, you might ask, how did the processes become so broken in the first place? Why did all the people buy into the broken processes when that one was introduced? Say hello to path dependence. Here’s the definition from Wikipedia (with my adaptation in brackets):

Path dependence is when the decisions presented to people are dependent on previous decisions or experiences made in the past. Path dependence exists when a feature of [the process] is not based on current conditions, but rather has been formed by a sequence of past actions each leading to a distinct outcome.

If you were to try to introduce the existing process in its current (broken) state into the current (often large) organization you’d fail because the brokenness of the process is obvious to everybody. So what happened is that the process has become broken through small, incremental changes based on the sequence of previous small, incremental changes:

• time pressure didn’t allow for a reasonable change, so you “improvise”
• people had certain skills and didn’t know about a better solution
• “we already use Excel for X, so let’s also use it for A, B, C and D” (even though it’s not a good fit)
• “let’s go with this industry standard” (even though what we try to do is quite different from what the standard is built upon)

Through chains of incremental changes — each of them somehow kinda justifiable — organizations paint themselves into a corner that makes them slow and ineffective. If you ask people why things are as they are, they often can’t explain. And in fact, there isn’t a good reason. It just evolved this way, the current state is basically an accident. A corollary of this is that the statement, “well, we do it this way, so it must make sense” just isn’t true. When I was a teenager, frustrated about some of the ways my flying club conducted its business, my buddies and I used the term “Tradition statt Vernunft” to describe this situation; tradition instead of reason.

How can it be changed?

When you try to change the process to something more reasonable, it’s often not feasible to rely on incremental changes. It would take too long, and the outcome is hard to predict. To achieve meaningful business process change, some people have to change how they work significantly. Or people might have to be educated or retrained. Some people might become more “important” while others will lose some of their status. Or maybe you have to change the structure of your workforce (letting some people go and hiring others with different skills). Obviously there is resistance that requires management attention. Nothing can be achieved with just a better tool!

These problems make this kind of change really hard. Which is also why it happens relatively rarely (and often frustrates Yours Truly). I’ll write some more about some of this in the future.

Zooming out

By the way: path dependence and the resulting “unexplainable” ways of how some things are done is of course not just a thing in business. It’s a feature of our society and political system as well. And the same reasons explain why obviously nonsensical things persist for so long in a society, even though the environment might have changed a lot. This frustrates the citizen Markus just as much as it frustrates the consultant Markus.