Agile & Architecture

Confronted with the problems of Waterfall approaches, organisations are all jumping on the Agile bandwagon. This is often a very good thing. Big Up-Front Design (BUFD) has many drawbacks in volatile environments and much of what happens in IT happens in rather volatile and uncertain environments. The volatility and uncertainty come from everywhere, including the organisation itself, as I’ve argued in my book. Hence, the book suggests a much more agile and practical approach to enterprise architecture. But the book is still written from a perspective where projects are ‘classic’, with a design upfront. What about Agile?

The fact that the book is written from a more classical project management perspective (the one I was in, and in which I applied the Chess approach to enterprise architecture) doesn’t mean I’m supporting BUFD, and neither do decent project management technologies such as PRINCE2. PRINCE2, for instance, is centered around being able to adapt and using the concept of a ‘business case’ to decide. In the real world, however, project management generally tries to prevent adaption, and has to do with an imperfect way of steering change: based purely on time & budget estimates, making promises about these (and then failing because the world is too unpredictable and complex).

So, while we can even be somewhat agile in situations that are ‘up-front design’, the real agile methodologies (Scrum, etc.) work from a more radical adaptability perspective. And — as said — this can even be good. But as many have noted: there are tensions between agile short-termism and architecture quality. Agile can be an excuse to create poor architecture and rack up the technical debt.

Philippe Kruchten writes on his blog:

There is a naïve thinking that just by being agile, an architecture will gradually emerge, out of bi-weekly refactorings. This belief was amplified by a rather poorly worded principle #11 in the agile manifesto, which states that:

“The best architectures, requirements, and designs emerge from self-organizing teams.”

and cemented by profuse amount of repeated mantras like: YAGNI (You Ain’t Gonna Need It) or No BUFD (No Big Up-Front Design), or “Defer decision to the last responsible moment”. (This principle is neither prescriptive, not can it be tested, as Séguin et al. showed in […], so it is probably not a principle, but merely an observation or a wish.)

In 2010 Philippe gave a talk with the title What Colours Is Your Backlog? (link to updated version from 2013), in which he presented a effective way to look at the work to be done  (development targets) from a perspective like this:

This is not exactly Philippe’s version, but one I have somewhat adapted. I changed Philippe’s Visible/Invisible to Actual/Potential. That is because visible/invisible is often a matter of perspective. Kruchten uses for instance ‘infrastructure’ as an example: this is invisible for the user. But the performance that the infrastructure provides, isn’t invisible. The way infrastructure creates performance is as invisible to the user as the way the software creates it or functionality. By the way, as visibility is a matter of perspective, people often attribute many causes from invisible things to visible things. E.g. while the architecture is the real reason, infrastructure may be blamed for poor performance. While a poor experience design is the real reason, users may be blamed for being unwilling to learn. Related to this (in terms of: what is visible to the user?): I also think that Non-functional Requirement (NFR) is a seriously broken concept.

Technical Debt

The idea of Technical Debt is widely used, but it has some issues. Management, for instance, actually sees debt as an enabler, so the term debt may send the wrong message to management as technical debt is a disabler). This is why Chris Matts and Steve Freeman came up with the Selling a Naked Call Option analogy.

In finance, a Call Option is the right to buy something at a certain price. E.g. if a share costs $100 now, and you expect it to rise a lot, you may buy a Call Option from someone with the right-to-buy it in three years time for $140. If, after three years, the price has risen to $200, you make $60. After all, at that time you can buy it for $140 and you can sell it immediately for $200. But if the price has languished and is $130, your right-to-buy at $140 is worthless.

Such an option costs money, e.g. it might cost you $10 to buy the right. This money goes to the one selling you the right to buy (the call option), which in finance terms is called ‘writing a call option’. Suppose this person has such a share and it is now worth $100. He makes a certain $10 to sell you that right. His future looks like this: if the share remains under $140, he has made $10 + whatever made or lost on the share itself. But when after three years the share has gone to $200, he has to sell you his share at $140, so he misses out on $60 profit.

So far so good, but the seller can do something else: he can sell you the right to buy the share at $140 without personally actually owning that share. This is called a Naked Call Option. Very easy money, no investment needed: he doesn’t need to own the share, he just signs a contract. He has nothing and he makes a certain $10 (or a million, with large contracts, in effect an option contract is generally per lot of 100 underlying assets). But with a catch, because when after three years the share is at $200, he has to sell you a share for $140. He doesn’t have that share, so he has to buy one at $200 to sell it to you immediately for $140. Not ‘missing out on a $60 profit’, but a ‘$60 real loss‘. And the higher the share goes, the larger his loss. Writing a Naked Call Options is therefore something with an immediate limited upside and a potential unlimited future downside.

That comes close to how poor architecture can affect you. If you don’t fix the architectural defect, you save money in the short term, but there is a chance of an ‘unlimited’ cost in the long term. The difference with writing a Call Option is that you don’t make actual money, you just avert paying cost.

Of course, poor architecture may also cost you immediately, e.g. if because of poor architecture, operation is expensive, performance is poor, etc.. In that case, Technical Debt actually works as a real debt. You pay interest as long as you do not eliminate the debt. But in that case, the debt is also very visible, after all you have those high maintenance cost and you can make a business case for fixing the architectural problem. Hence, Technical Debt that behaves like real debt can be seen as an ordinary visible defect.

In summary: poor architecture can appear in two different ways:

• As something that visibly creates a loss (e.g. operating cost, lost revenue)
• As something that potentially creates cost in the future (e.g. making change expensive)

I like to classify the former as an actual Defect and the latter as a potential, and that is the category that I label Technical Debt.

An overview of cost and benefit for the four categories could look like this:

 

The upper and lower row are each other’s mirror. Fixing a defect is like creating a feature. Subtracting -1 is like adding +1 (I always love a little math for fun — link to an article about calculating zero divided by zero). Fixing technical debt is like creating a better architecture. The left column makes for easy and transparent business cases (though sometimes the benefits/losses are just ‘perceived’).

It is easy to see why it is so easy to accrue Technical Debt. If you don’t fix a Defect, the loss is real. So, you weigh certain cost against certain benefits. But if you do not fix Technical Debt, the loss is not ‘real’. So, you weigh real cost against unreal benefits.  That is true for architecture in general, but the positive side of architecture often fits well with strategic goals and plans, and for those that invest the money, strategy is very real — it is what they do. That explains why management is often happy to invest in ‘architecture’ (a.k.a. strategic transformation), but loath to invest in architecture to solve the mess of the past.

The right column is also the one that is generally not ‘simple to plan and do’. Architecture, after all, is also “things that people perceive as hard to change” (Martin Fowler, Who Needs an Architect?).

Checks & Balances

In the Enterprise Chess approach to enterprise architecture (see the book or the presentation), the natural tension between Solutions (or ‘Products’ in Agile) and Enterprise Architecture is managed by creating ‘consent’-based collaboration and checks & balances between the solutions perspective and the enterprise perspective. I’m a fan of working with consent-based approaches. They tend to be less sluggish (until downright deadlocked) than consensus-based. I’m a fan of collaborative approaches, because they tend to have a higher quality in complex endeavours than ‘split’ or ‘separated’ ones (e.g. collaboration versus supply/demand, DevOps versus dev/ops).

It is maybe good to explain the idea behind consent. With consent, you don’t necessarily strive to agree in full (consensus), but the parties will work to look for a minimally acceptable outcome from their own perspective. (‘Minimally acceptable outcome’, by the way, sounds like ‘minimally viable product’ from agile, a nice example of cultural memes popping up concurrently everywhere.) In other words, central enterprise architecture reviews project architectures, but the review is split into ‘show stopper’ items (no consent) and advice (a request to improve, but consent: acceptable if the project fails to do so). This is a fundamental part of the Enterprise Chess approach to enterprise architecture: Conflicts are minimised to the ones that really matter (another definition of architecture: “the important stuff”). Only when ‘no consent’ items are not resolved, the matter escalates. The same approach is followed in the architecture board discussions (though the ‘no consent’ here is technically advice for the lead architect, though participants may/will of course then seek their own escalation elsewhere). Note: the architecture board is not a democracy. More in the book.

Note that consent doesn’t mean ‘agree to disagree’. There is a single outcome and it is accepted by all. Sometimes consensus on content is important. E.g. if two companies build a rocket together and both use different units (e.g. centimetre versus inches), the rocket will fail. So, in a consent-based approach, the important design issues to reach a shared conclusion on are identified and a design choice is made. If that design choice is “use centimetres” that the people arguing for “use inches” accept that outcome (or escalation happens until the important issue is resolved in full).

More on consent: Consent for Decision-Making in Agile Organizations and Using consent over consensus for decision making.

Governing Architecture in Agile Projects

I’ve been inspired to write this post by Eltjo Poort‘s talk on the Dutch national enterprise architecture congress (LAC 2016), last November. He described how they were using Kruchten’s approach in an architecture road mapping setting. See this article. From his presentation comes the following slide:

 

I liked this immediately, but I had one issue with this when I saw it. Who decides what goes in the respective backlogs? In agile methods, that is the Product Owner and that means that checks & balances from the perspective of the organisation-as-a-whole (overall quality aspects, coherence, and strategic directions and choices) are weakly governed. Basically, you’re weak in the aspect of architecture that exceeds the scope of the project if you don’t do checks & balances based governance..

Apart from the checks & balances issue, there is the internal coherence issue. Kruchten also comes with very good advice: manage all your colours together. The question then becomes: how? Let’s start with a simple picture for an agile approach, using Kruchten’s visual language for types of development targets:

 

What this shows is that the Product Owner is in charge of the Solution Backlog and the Spint Backlogs. The arrows denote accountability, not so much doing the work. I’ve been friendly and shown that this Product Owner has an eye for architecture, so architecture work ends up in the sprints. In badly run agile projects, or those where the teams do not only consist of superstars (Agile depends enormously on the quality of the skills of the teams unless you have effective architecture & quality governance), sprints are all green and red until in the end a product emerges with a lot of black.

But enterprise architecture requires checks & balances between projects and the whole. For that reason, it is a good idea to introduce an Architecture Backlog, so it is easy to focus. Kruchten however warns:

Having multiple repositories of things to do, managed by multiple or different people, based on different criteria is a bad idea. It leads to delays, frustrations, accumulated technical debt, reduced velocity, distrust….

And I agree. But I’m not talking about moving from a collaborative approach to a separated approach. Having a separate backlog, doesn’t mean working with it isn’t part of a collaborative approach. Adding it, we get:

 

The architect of the project manages the Architecture Backlog. It is of course still the Product Owner who is responsible for the content of the Solution Backlog and the Sprint Backlogs. We can introduce the overall enterprise architecture checks & balances as follows:

 

Two things change: first, filling the Solution Backlog — while still the responsibility of the Product Owner — becomes a consent-based collaboration between Product Owner and Enterprise Architecture (collaborating create&review roles, see below). And also: Enterprise Architecture can add items to the Architecture Backlog.

Now, as in the Enterprise Chess approach for more classical projects, Enterprise Architecture’s role is checks & balances from the organisation-as-a-whole perspective. Hence, the Product Owner (in classic Prince2 terms, the Project Executive) is in charge of his project. But if there is no consent (e.g. Enterprise Architecture finds the level of neglect for architecture or technical debt unacceptably high, or architectural choices are made that will create problems elsewhere in the organisation’s landscape), then an escalation occurs to the level above the Product Owner, if need be up to the board of the organisation (after all: enterprise architecture’s checks & balances are from the perspective of the organisation-as-a-whole, for which the board is responsible).

Let’s compare this to a more classical ‘design-up-front’ approach (from Enterprise Chess), which looks like this:

 

Remember: the arrows denote accountability. I’ve been less friendly here: all the plans are obviously green and yellow, but the result contains black and red. Note: there are ways to manage that too in up-front situations, namely the acceptance check at the end. But that is often difficult to make stick. Anyway: even if the Solution Architect creates the Project Architecture, it is the Project Executive who has to accept it.

Governance, roles, architecture board, design authorities

Finally, we get to organising the consent-based checks & balances. Let’s start with an overview of the Enterprise Chess approach to governing project architecture:

 

In this approach, EA, supported by architects from various ‘domains’ in the organisation in an architecture board, reviews the up-front architecture from a project from the company-as-a-whole perspective. There is much to be said about that (all in the book) and it sounds much more bureaucratic than it is (it isn’t, it is a collaborative approach, based on clear roles and responsibilities).

Now,these days, we also see another pattern. Instead of a central architecture review, each (agile) project has its own Design Authority. In Enterprise Chess terms, that would look like this:

 

All kinds of different design authorities for each project has a disadvantage of course (see Kruchten’s warning above), in that it becomes a lot more difficult to get an overall consistent approach to the development of the landscape. One can counter that by making sure there are a lot of the same persons in each design authority. An advantage of the design authority approach is that each design authority can be staffed with the best people for the project. A disadvantage is that input from a fresh perspective by the different domain architects has turned out to be very effective in the first variant. Such input also leads to a ‘learning organisation’ in the architecture field.

But for the agile projects, the second approach may be more optimal. And it is identical to  the previous one, but with a few different roles:

 

The enterprise architecture role works together with the solution architect to create the architecture backlog. This way, architectural requirements from the organisation-as-a-whole perspective may end up in the Architecture Backlog. The Product Owner decides what is in the Solution Backlog, but this gets reviewed by enterprise architecture. If (from the perspective of the organisation-as-a-whole) important architectural requirements are not met and Lead Architect and Product Owner cannot agree (consent-based), the issue gets escalated. The Lead Architect also must keep an eye on the Solution Backlog to make sure the ‘organisation-as-a-whole’ items do not stay in the Architecture Backlog indefinitely.

The Design Authority approach has different repositories managed by different sets of people, so Kruchten’s warning applies. The up-front setup from Enterprise Chess doesn’t have that because it handles multiple repositories (the future state, the plans for each project) with a single set of people. The negative effect of what Kruchten warns for (dev/ops instead of DevOps) needs to be countered in the Design Authority approach or you will lose architectural coherence in your organisation (which means: Technical Debt). It can be countered by having key people active in managing multiple repository, namely the people from enterprise architecture and the domain architects.

Managing Details

One important point. Design Authorities are often also engaged in various details. Now, the Enterprise Chess approach to details is that there is a difference between relevant details and the rest, and architecture includes the relevant details. As in real chess: sometimes a detail is important: a pawn position may win or lose you the game. Hence, the Solution Architect works with the team to create the architecture (which in part ’emerges’ in the agile approach), but the architecture needs to be documented and discussed and the Design Authority is for checks & balances. So, my approach to a Design Authority might be different than what happens in practices where the Solution Architect is in charge.

A Word of Warning

All this architecture governance remains brittle and imperfect and it always will remain brittle and imperfect. We should all memorise this Tweet from Tim Ottinger:

 

Conclusion

The Enterprise Chess approach is easily adaptable to agile instead of up-front design approaches to projects. The introduction of Architecture Backlogs and the checks & balances governance of Architecture & Solution backlogs support organisation-wide coherence and good-quality architecture work in agile projects, just as the approach in the book does the same for up-front designed projects.

The Design Authority approach to architecture governance relies on key people playing the same role for multiple Design Authorities. In the Enterprise Chess approach, the Design Authority is part of checks & balances for the solution/project; Solution Architect & team are the ones that create the architecture and there the Solution Architect is responsible.

Afterword

Chess and the Art of Enterprise Architecture describes a way to govern enterprise architecture that is consent-based. But I wasn’t aware of the literature on this when I wrote it (let alone when I first tried it out). If I were to create a new release of the book, this would be strengthened and also this blog post would be added.

This article has also been published on the Enterprise Chess site: Agile & Architecture. You can find more articles by me on Enterprise Architecture there.