How to make well architected reviews work for organisations (1)
Mike looks at part 1 of the SCORP Review Cycle with his insights into carrying out well architected reviews.
Security, Cost, OpEx, Reliability & Performance (SCORP)
All three leading Cloud Providers (AWS, Azure, GCP) have produced well-architected frameworks and guidance for teams and organisations. They are designed to guide engineering and architecture approaches within their platforms.
What is the well architected framework?
AWS Well-Architected helps cloud architects build secure, high-performing, resilient, and efficient infrastructure for their applications and workloads. The framework is based on five pillars — operational excellence, security, reliability, performance efficiency, and cost optimisation . AWS Well-Architected provides a consistent approach for customers and partners to evaluate architectures. And to implement designs that can scale over time.
The AWS Well-Architected Framework started as a single white paper. But it has expanded to include domain-specific lenses, hands-on labs, and the AWS Well-Architected Tool. The AWS WA Tool is available at no cost in the AWS Management Console. It provides a mechanism for regularly evaluating your workloads, identifying high-risk issues, and recording your improvements.
How do organisations embrace well architected?
How organisations embrace this guidance has been of particular interest to me in my day job as a Technical Architect. Encouraging teams to adopt the Well-Architected Framework in their architecture and engineering practices truly is a significant step forward. When an organisation applies it at scale it can achieve economies of scale. And create healthier development environments for their teams.
My organisation has started promoting use of Well-Architected and we have begun to experience the benefits it brings. We are in the early stages of adoption, currently driving it at an individual team and workload level. More recently, I have been thinking about getting more productive with it through supporting dedicated facilitation. And using it as a basis for achieving broader economies of scale to driving continuous improvement for our teams. Around six months ago, I started with the thought/hypothesis that:
‘If the organisation invested in facilitating Well-Architected-Framework into their core ways of working then we would quantifiably see organisational improvements in the quality of our team execution and the quality of our outcomes.’
What is behind my hypothesis on organisational Well Architected Facilitation?
I am a Technical-Architect for a large engineering organisation with multiple development teams, not necessarily on the same engagements. As a result, there are many aspects of our operation that I am responsible for and need to consider, such as:
Reducing the distance between our most effective teams and the least effective (emerging) — More commonly measured in the DORA Metrics.
Defining and supporting the architectural enabling constraints to guide all teams? Consistency and benchmarking of engineering standards.
Creating Efficacy (Efficiency and Effectiveness) through prevention of duplication in terms of problem solving and creation of situational awareness between teams. Reuse.
Highlighting and Celebrating good engineering rigor and exposing shortcutting and technical debt.
General engineering proficiencies and development of those proficiencies.
Inspired from recently reading the book Atomic Habits, I realised good psychology in slow, deliberate and sustainable incremental improvement. In meeting the responsibilities, slow continuous and gradual improvement is the approach I take. I just needed a framework to base my plan for supporting this incremental improvement around. A direction to steer individuals and teams towards AWS’s Well-Architected Framework gave me the solid fundamentals to work from.
Why does the well architected framework tick the boxes of organisational scale?
The Well Architected Framework is a commoditised set of opinionated standards. This means that I, or other architects, don’t have to invest a lot of time crafting and agreeing standards. The subject matter experts from cloud providers have already done that and will continue to evolve them with their platforms. You are correct in thinking that not all these standards apply all of the time.
As an architect, I focus on the principles and guidance that apply. And I get right into the ways we can implement or meet the said standard. The Well Architected Lenses are an essential subset of the Well Architected Framework tailored for specialised domains. (Serverless, FinTech, Machine Learning, IoT etc.).
The power of portability
The framework gives us the power of portability. When a developer moves from one team or organisation to another, their experiences and expectations are consistent. Their skills and experience, have become very transferable. My challenge was to create a process that brought Well Architected Reviews, collaboration, enterprise direction and integration with team backlog. The process would need to allow me to meet the teams where they were at (experience and capability wise). And bring them together in the spirit of learning and collaboration.
This allowed us to make and celebrate small wins through deliberate, incremental improvement. A process that would ultimately be sustainable and not overbearing on the development process.
Working with ten squads, I attempted to make my hypothesis a reality. I integrate Well-Architected into our ways of working through a Process that supported regular facilitation and collaborative coaching techniques. Read my story so far.
Building on
Before I introduced the SCORP process cycle, I had facilitated around 15 Well Architected Reviews with different teams. The process itself (when performed correctly), was quite enjoyable for me and for the Architect and the engineers participating (even if I do say so). The review provided an entry point to meet the team. And allowed me to understand the nuts and bolts of the architectural implementation of their workload. As well as being able to advise and share thoughts and direction, the team would escalate and discuss challenges.
Successes and experiences are invaluable in terms of facilitating the reviews with other teams. I was effectively a conduit for well-architected practice across teams. The well architected reviews would typically end up with me updating the AWS Well Architected Tool. And creating a milestone. I followed up to see what the team had added to their backlogs, after a few days.
My observations were the same, three months later. Teams were progressing, but was it enough? I wasn't disappointed with lack of progress, but I knew that we could be better. We should be covering more ground. And we were missing tons of low-hanging fruit type improvements. A typical review can last anywhere between 2–10 hours depending on the lens being used. This can be a tiring exercise if the team hasn’t made much progress. And it can create a sense of frustration if continued.
Build Trap
What I was observing was that the teams were falling into a sort of build trap. By being able to move fast, teams typically were more focused on shipping to production. And maybe not as invested in engineering excellence around the Five Pillars as they could be. The focus was not on continuous improvement (proactivity); it was more on reactive corrections and rapid delivery. In some cases where teams start with small workloads, and the dev-ops overhead is negligible. With time, the team scaled out the workload and the dev-ops overhead increased significantly. In these cases, the teams always ended up with epics to resolve issues with tech debt or reliability.
These sorts of trends are not, in my experience, trends that can be corrected by directing technical leads at books. Or by meeting once a quarter for a technical review. The teams are typically succumbing to the everyday pressures of delivery. These trends in my experience require leadership to engage in the delivery process itself. It requires clarity of direction and a way for the teams to introspect themselves continuously. As well as performance and capacity to act. It also needs to be consistent for all teams regardless of how they were engaged.
I set about this, with the introduction of SCORP Process Cycle of the well architected review. We will explore this in part 2 of this blog series.
Originally published on The Serverless Edge