We want {CompanyName} to be a fun, challenging and rewarding place to work. Help us realize our vision of a world-class Agile work environment by completing the DevOps Health Assessment. This assessment is your opportunity to help shape the future of our team and improve your own experience in the process. Your feedback will be compiled and used to guide further positive changes across the team and organization.
Building the right product is measured by having happy customers who feel the product meets their needs.
To what extent do you agree with the statements below? Please rate according to the PRE-CRAWL (1-2), CRAWL(3-4), WALK(5-6), RUN(7-8), FLY(9-10) scale keeping in mind each level assumes you've achieved the previous level.
Customer Engagement
When building applications, the team should continually validate what is being built through customer feedback to ensure they are on the right track and are incorporating that feedback along the way in order to build the right product.
Pre-Crawl (1-2): We do not solicit customer feedback and/or understand how to handle customer feedback when it is received.
Crawl (3-4): We understand the value of having customer feedback loops but have no practices in which to capture feedback.
Walk (5-6): We have practices in which to solicit and capture customer feedback but do not know how to apply the feedback.
Run (7-8): We gain insight into customer needs both through their feedback and by reviewing their actual experience/journey in our applications. We are experimenting with practices to allow us to fully utilize this learning as we improve our products.
Fly (9-10): Our applications are driven by satisfying our customer needs and this has become our competitive advantage.
Pre-Crawl: 1-2 | Crawl: 3-4 | Walk: 5-6 | Run: 7-8 | Fly: 9-10
Lead Times
Being able to communicate predictable lead times to our customers to set expectations and working to reduce lead times both elevate customer satisfaction.
Pre-Crawl (1-2): We are not measuring lead time in our practice and/or do not yet understand the benefits.
Crawl (3-4): We are learning about the benefits of tracking and measuring lead time.
Walk (5-6): We are tracking lead time and exploring how to leverage the benefit of this information.
Run (7-8): We use lead time to forecast delivery times for our products, services, and delivery. We share our learnings with other teams and our CoP.
Fly (9-10): We actively analyze lead time to determine where opportunities for improvement lie so that we are constantly improving our delivery process. We are sharing our proven practice across our company.
Recovery Times
A high performing team is able to quickly recognize and recover from system failures so that there is little to no impact to the customers.
Pre-Crawl (1-2): We are slow to recognize that a failure exists in the system. Sometimes we learn of these issues from our end customers. Recovery time is lengthy.
Crawl (3-4): We are learning how to use sources of information such automated tests, monitoring/alerting, and application logging to inform us when there is a failure.
Walk (5-6): We are beginning to use sources of information such as automated tests, monitoring/alerting and application logging to inform us when there is a failure. Our recovery time is improving.
Run (7-8): We are able to easily identify production issues because we have a full suite of automated tests that run as well as monitoring & alerting that tells us of a problem. When an issue is discovered, we are able to resolve quickly. Our recovery time is improving.
Fly (9-10): When production issues arise, we have means to quickly identify changes that have happened in the system recently so that we can initiate automated recovery to the last known running state. There is virtually no customer impact.
A/B Testing
A high performing team deploys different implementations of the same functionality to determine which one our customers prefer (A/B Testing).
Pre-Crawl (1-2): We don't know how to do A/B Testing, or what value it would bring to our organization.
Crawl (3-4): We are learning about how to implement A/B Testing. We are working with our Product Owners to identify opportunities to use A/B Testing to improve our applications.
Walk (5-6): We have selected an A/B Testing tool/method and have implemented a few A/B Testing experiments. We believe that A/B Testing will result in higher customer/user satisfaction.
Run (7-8): We routinely use A/B Testing in all of our products/applications. A/B Testing allows us to quickly choose features that result in greater customer/user satisfaction and more value to our organization.
Fly (9-10): Our organization embraces A/B Testing as a key practice in our application development process. It helps us deliver better applications in less time, with improved customer satisfaction and value to our organization.
Add any additional thoughts about "Building the Right Product - Customer Satisfaction"
Building the right product is measured by ensuring alignment between the delivery and the enterprise direction and goals.
Systems Thinking
A high performing team uses Systems Thinking to understand the big picture and optimize the whole system, rather than just individual components.
Pre-Crawl (1-2): We operate in silos, focused only on our part and do not understand the system as a whole.
Crawl (3-4): We understand the system as a whole. We have obstacles preventing us from leveraging system thinking.
Walk (5-6): We are beginning to work together to build solutions with an end-to-end mindset; with the goal of optimizing the system as a whole.
Run (7-8): From a systems perspective, we approach solutioning with a holistic point-of-view. We seek to optimize the whole system rather than simply improving individual components. We are beginning to form emergent outcomes.
Fly (9-10): We have advanced our systems mindset for complex problem solving and transitioned to a circular economy.
Unified Backlog
A cross-functional team works from one backlog which not only includes development items, but also operational and non-functional requirements (NFRs) such as performance, usability, load, security testing, etc.
Pre-Crawl (1-2): Our backlog is feature-driven and non-functional requirements are not identified as part of product delivery.
Crawl (3-4): Non-functional requirements are part of a separate operations backlog.
Walk (5-6): Our team identifies NFRs and operational requirements as part of the product backlog, but are implemented at the end of the delivery, which often causes rework.
Run (7-8): Our team includes NFRs and operational requirements in the unified backlog and prioritizes them appropriately. Some NFRs are setup to automatically provide feedback during builds and deploys, while others are incorporated into the monitoring and alerting mechanisms.
Fly (9-10): Our team has fully automated testing of our NFRs so that we are ensuring that the system meets these requirements continually. We incorporate our NFR's into our process as we do functional requirements.
A cross-functional team should be working from one backlog which not only includes development items, but also operational and non-functional items such as performance, usability, load, security testing, etc.
Pre-Crawl (1-2): Our teams don't have an understanding of the value in one backlog over continuing with two separate backlogs (product backlog and operations backlog)
Crawl (3-4): The operations and delivery teams have their own backlogs, each with their own priorities. We have coordination activities that help us align on delivery, but often struggle to make this happen seamlessly. We understand why a unified backlog is important, but do not know how to take the next steps.
Walk (5-6): Our teams are analyzing how we can move from siloed operation backlogs to unified backlogs. We are tracking and targeting different experiments but have not begun to unify.
Run (7-8): Our teams create backlogs containing development, operations, and NFRs, driven by a shared vision goals. The unified backlogs are prioritized to efficiently implement the solution and eliminate waste. Team members use agile practices to ensure correct implementation, to build skills, and to share knowledge
Fly (9-10): Our backlog is comprised of operations and product backlog items, aligned to a shared vision and common goals. We plan the execution of the backlog items as one team focused on value delivery not task completion. Hand-offs are non-existent. We share our learnings and proven practices across our company.
Business Outcomes
High performing teams are aligned to the business initiatives and value streams of the organization so there is clarity on the benefit being delivered to our customers. They are solving real business problems.
Pre-Crawl (1-2): We are not clear on how the work we are doing provides business value.
Crawl (3-4): We believe there is value in our work but we do not know the actual business impact.
Walk (5-6): Our work is aligned to key initiatives that are important to our business but we are unsure if we are making a difference.
Run (7-8): Our team's work is aligned to key business initiatives and we clearly see how our work is making a difference.
Fly (9-10): We can measure the effectiveness and impact of our work to drive real business outcomes.
Prioritize
High performing teams work on projects that are aligned to key business goals and prioritized according to the highest business value.
Pre-Crawl (1-2): There is no visibility at the team level as to how priority is determined. Often the team is faced with conflicting priorities.
Crawl (3-4): Our organization is working on creating a framework to prioritize initiatives. Work is assigned to teams based on resource availability.
Walk (5-6): Our organization is refining the framework for prioritization to be based around team capacity rather than resource availability.
Run (7-8): Our organization is utilizing our framework for prioritizing initiatives to support enterprise goals in achieving business outcomes. They funnel the work to teams, in small, outcome-based slices.
Fly (9-10): Our organization's culture prioritizes initiatives that support enterprise goals in achieving business outcomes. Our alignment to value streams simplifies our prioritization by narrowing the scope of the initiatives.
Add any additional thoughts about "Building the Right Product - Enterprise Alignment"
Focusing on building the right product results in the delivery of value.
Lead Time to Deploy/Change
On average, how long does it take from final code commit to code successfully running in production?
Pre-Crawl (1-2): 1+ month.
Crawl (3-4): Within 1 or 3 weeks.
Walk (5-6): Within 2+ days.
Run (7-8): Within 1 day.
Fly (9-10): Less than 1 hour or within minutes.
Mean Time To Detection
For the primary application or service you work on, how long does it generally take to detect an incident has occurred?
Pre-Crawl (1-2): More than a day
Crawl (3-4): 1 day or less
Walk (5-6): Less than 8 hours
Run (7-8): Less than 30 minutes
Fly (9-10): less than 15 minutes
How well does this team deliver on the planned business outcomes or goals?
Pre-Crawl (1-2): We don’t define measurable business outcomes/metrics or understand the value/impact of our work to our customers.
Crawl (3-4): We deliver 50% of our planned outcomes.
Walk (5-6): We deliver 60%+ of our planned outcomes.
Run (7-8): We deliver 75%+ of our planned outcomes.
Fly (9-10): We deliver 95+ of our planned outcomes.
Add any additional thoughts about "Building the Right Product - Value"
High performing delivery teams understand that they can achieve continuous flow by implementing smaller batches of work that are much easier to develop, test, and deploy through an automated Continuous Delivery Pipeline.
Batch Sizes
Small batch sizes enable a more continuous flow of value to the customer; pivoting according to the business needs is faster and it is easier to measure flow and team performance.
Pre-Crawl (1-2): We bundle multiple features for a single, large deploy often monthly or longer. This can cause large pools of defects, integration is difficult and unpredictable, and time to market is negatively impacted.
Crawl (3-4): We are working to break the large features into smaller batches. However these batches often contain component parts of the whole feature that in and of themselves do not deliver end customer value.
Walk (5-6): Our team is learning to identify features and break them into valuable vertical slices that can be quickly implemented.
Run (7-8): Features are delivered in small. correlated batches (small end-to-end vertical slices), i.e. a little UI, a little business logic, a little data, a little testing, etc.
Fly (9-10): By organizing our work in small batches of valuable features, we are able to deliver value to our customer quickly and validate that we are building the right features/product.
Delivery Pipeline
High performing teams utilize a delivery pipeline that enables them to go from code commit to deployed to production while meeting security, compliance, and automated testing, all through automation.
Pre-Crawl (1-2): Our builds and deploys are manual. We don't know how to implement an automated delivery pipeline.
Crawl (3-4): We are learning about automated delivery pipelines and planning to implement one in our environment.
Walk (5-6): We are beginning to implement an automated delivery pipeline in our environment. It includes version control, automated tests that run when code is checked in, security/compliance audits, and automated deployment.
Run (7-8): Our automated delivery pipeline enables us to deploy to production after passing functional and performance tests and meeting security and compliance requirements.
Fly (9-10): Our delivery pipeline is the orchestration of CI, CD, Test Automation, Security, and Audit allowing us to go from code check in to production automatically.
Continuous Integration
Continuous Integration ensures quality and provides immediate feedback to the developers through short lived branches and passing automated tests.
Pre-Crawl (1-2): Everyone does branching different which causes a lot of problems. When pulling the latest code, it is often broken.
Crawl (3-4): We have long-lived code branches; the merging process takes a long time and is error prone. Developers are asked to make sure their code compiles before committing.
Walk (5-6): We are merging our branches regularly and experimenting with integrating code more frequently utilizing a centralized build that helps ensure that code always compiles. Our unit tests still fail and branching is error prone.
Run (7-8): We minimize code branch lifespan with merging done frequently. Errors caused by merging code branches happen less often and we are actively applying our learning to reduce errors. Our CI build verifies the unit tests are passing and the team fixes any broken tests immediately.
Fly (9-10): We have one mainline branch into which code merging is done often to reduce disruptions in the development process. Our CI process has been expanded to provide additional quality feedback to the developers including code coverage, code metrics, and static code analysis.
Feature Flags
Techniques like Feature Flags allow teams to control the features that are available to customers in production without doing a deployment and enables separating releases from deployments.
Pre-Crawl (1-2): We don't know how to use feature flags, but we would like to be able to control the functionality that is available to our customers.
Crawl (3-4): We are learning about feature flags and planning how to implement them in our applications.
Walk (5-6): We are beginning to implement a feature flags framework that allows us to control the functionality available to our customers and separate the code deployment from the release.
Run (7-8): We routinely use and maintain feature flags in our applications to separate the code deployment from the release to customers. Feature flags allows us to rapidly develop, test, and have more control of our applications.
Fly (9-10): Our organization uses feature flags to allow multiple teams to rapidly develop and test features in the same application, without collisions, to separate feature availability from releases.
Staying Shippable
Maintaining the continuous delivery pipeline (CI, CD, SDP, Test, Automation) so that software applications can be deployed at any point in time with automation and quality built in allows teams to deliver value to customers faster.
Pre-Crawl (1-2): We are not able to deploy our applications to production on demand. Our build, test, and deploy process is manual and non-functional testing requirements are often skipped to get the release out.
Crawl (3-4): We are learning how to use an automated delivery pipeline to consistently deploy to production.
Walk (5-6): We are beginning to implement an automated delivery pipeline to deploy to production, but don't always keep them up to date.
Run (7-8): We maintain our automated delivery pipeline (i.e. Builds are never left broken) for most of our applications so that we can deliver our applications at any point in time with automation which allows us to get features to our customers faster.
Fly (9-10): Our organization expects our teams to maintain each automated delivery pipeline so that we can deliver value to our customers at the touch of a button. We share our proven practice across our company.
Release Management
By decoupling deployments from releases, teams are able to roll out software to targeted sets of customers to minimize the potential impact and learn from the feedback loops.
Pre-Crawl (1-2): Deployments go out to all users at once. We do not have the ability to deliver to a subset of users at this time.
Crawl (3-4): We wish we could deploy updates to only a targeted group of customers or user roles but do not have the ability to do so.
Walk (5-6): Our team deploys updates to key groups of Internal users first to ensure they are performing properly before deploying to all customers.
Run (7-8): Our team deploys new features and updates to targeted customer user roles for feedback and performance before deploying to all customers.
Fly (9-10): We have the ability to control the deployment of new features to targeted users in our production environment in order to gather feedback and verify performance before deploying to all customers.
Add any additional thoughts about "Faster Value Delivery - Continuous Flow"
High performing delivery teams have achieved the ability to treat infrastructure as a flexible resource that can be provisioned, scaled-up, and version controlled alongside each application.
App-driven Infrastructure
Applications deployed to purpose-built, dedicated infrastructure allows teams to deliver value to customers whenever needed.
Pre-Crawl (1-2): We share infrastructure resources with other applications, we run into delays because we have to coordinate our deployments.
Crawl (3-4): We are experimenting with segregating some application environments but still heavily share infrastructure resources with other applications and are faced with delays due to coordinating our deployments.
Walk (5-6): We work closely with our infrastructure team to ensure the environments meet the needs of the applications.
Run (7-8): Our team creates and configures infrastructure that is dedicated to our applications which allows us to deploy whenever we need to.
Fly (9-10): Our team creates infrastructure that is purpose built for our application, using a self-service process, which allows us to make changes as needed with no other dependencies.
Infrastructure as Code
Teams are able to automatically provision infrastructure and environments consistently and repeatably when infrastructure configuration is scripted and maintained in version control.
Pre-Crawl (1-2): Infrastructure is provisioned by the Operations Teams using manual processes. We often run into challenges with inconsistencies between environments.
Crawl (3-4): Operations Teams are learning how to use tools/scripts to provision infrastructure.
Walk (5-6): Development and Operations Teams are working together to use automated tools/scripts to provision infrastructure. We are actively experimenting to leverage scripts that are maintained in source control. Most of our environment configurations are consistent.
Run (7-8): Infrastructure resources are provisioned using scripts that are maintained in source control. The scripts are parameterized so they can be used in all environment levels, i.e. dev, test, qa, beta, prod.
Fly (9-10): Our team is proficient at writing Infrastructure as Code and all of our applications include such provisioning.
Emergent Architecture
The best architectures will emerge over time and are based upon product needs.
Pre-Crawl (1-2): We believe we have to define all of the detailed architecture upfront and that we cannot make changes over time
Crawl (3-4): We are learning about emergent architecture patterns.
Walk (5-6): We are experimenting with implementing emergent architecture patterns to evolve our technical decisions.
Run (7-8): Our team uses emergent architectural patterns which delivers a continuous flow of value.
Fly (9-10): Our organization embraces the principle of that the best architecture will emerge over time and should be based upon needs of the application. We have a proven practice on how we approach emergent architecture that we share across our company.
Architecture is a collaborative activity guided by both an enterprise view and application needs so that teams can build the most effective, emergent architectures for their applications.
Pre-Crawl (1-2): Architecture decisions are made outside of our team and do not evolve over time.
Crawl (3-4): We have some influence but typically only meet with the architecture team when we need sign off on a design. The architecture remains relatively static.
Walk (5-6): Our team has some collaboration with Enterprise Architecture and Solution Architecture, it is slowly evolving and the approach is improving but is focused mostly on projects.
Run (7-8): We meet regularly as an architecture community of practice to collaboratively solve our needs. We are practicing just-in-time architecture and find value in our emergent architecture practices.
Fly (9-10): We meet regularly as an architecture community of practice to deliver just-in-time and just enough solutions. There is collaboration and alignment on proven emergent architectural practices that we share across our company.
A system architected with small, independent, testable services allows organizations to optimize the flow of value to the customer.
Pre-Crawl (1-2): We design and develop monolithic applications that are tightly coupled and difficult to work with. Our architecture is set at the start of our project. Changes to the system are costly.
Crawl (3-4): We see the value of emergent architecture and designing smaller, service-oriented applications. We have begun to strategize on how we could accomplish this.
Walk (5-6): Our team is taking a methodical approach to transform our applications into smaller, service-oriented applications and adopt this strategy for new application design. We are developing design patterns that enable emergent architecture.
Run (7-8): Our team has broken the monolithic application out into multiple services, however some dependency still exists. Most applications are designed as smaller, service-oriented applications and we have begun to refactor existing applications to follow this pattern. Our architecture evolves with the system.
Fly (9-10): Our applications are comprised of multiple services which each encapsulate a single business capability. Developers are able to build, test, and deploy independent of one another and can be used in a self-service fashion. We have completely decoupled all services which eliminates deployment dependencies on other services and enables faster deliver to our customers.
Technical Practices
Teams build applications with an operational mindset and instrument their code so that they have visibility into the health of the system through automatic monitoring. Exceptions and failures can be resolved quickly.
Pre-Crawl (1-2): We don't know how our code deals with failures in production.
Crawl (3-4): We are learning how to identify potential exceptions and failures so we can instrument our code to provide troubleshooting information when they occur in production.
Walk (5-6): We are beginning to instrument our code to provide troubleshooting information for exceptions and failures when they occur in production.
Run (7-8): We always instrument our code to provide troubleshooting information for exceptions and failures when they occur in production. This practice has dramatically reduced downtime and increased customer satisfaction.
Fly (9-10): Our culture embraces the practice of thoroughly instrumenting our code to provide troubleshooting information for exceptions and failures. We are implementing automatic remediation of exceptions and failures where possible.
Flexibility & Scalability
High performing applications can automatically scale up performance and infrastructure based upon the system and customer demands.
Pre-Crawl (1-2): Our system is constrained to the performance/capacity of the shared infrastructure. We do not have insight into potential performance bottlenecks or capacity limitations.
Crawl (3-4): We are working with the operations teams to get information about our system performance so that we can begin to make adjustments to our infrastructure and applications.
Walk (5-6): We work with the operations teams to identify performance bottlenecks and capacity limitations. We are working together to experiment with different infrastructure configurations.
Run (7-8): Our team works together to design and implement infrastructure configurations that can be scaled up to meet the performance and capacity needs of our customers.
Fly (9-10): Our system has the ability to automatically scale up and down, based on capacity needs.
High performing teams are empowered to leverage both new application and infrastructure resources to create solutions that meet the needs of our business and customers.
Pre-Crawl (1-2): We have a pre-defined set of tools and are not allowed to leverage tools or technologies.
Crawl (3-4): We are aware of the newer infrastructure resources available to us but only have access to the infrastructure resources provided to us.
Walk (5-6): Our team is experimenting with leveraging both new application and infrastructure resources as needed to speed up development and the deliver capabilities.
Run (7-8): Our team has access to the infrastructure resources necessary to delivery within our team. We have applied our learnings and are leveraging both new application and infrastructure resources as needed to speed up development and deliver capabilities our customers expect.
Fly (9-10): Acting as one team, we have realized value in leveraging both new application and infrastructure resources to quickly develop and deliver capabilities to our customers.
Add any additional thoughts about "Faster Value Delivery - Scalable Architectures"
Focusing on the faster delivery of value results in flow.
Throughput Increase
How much improvement have you had over your baseline (example: first month of Agile adoption) in throughput? You can measure your number of work items OR number of story points delivered.
Pre-Crawl (1-2): We have improved by 0-20%.
Crawl (3-4): We have improved by 20-40%.
Walk (5-6): We have improved by 40-60%.
Run (7-8): We have improved by 60-80%.
Fly (9-10): We have improved by >80%.
Predictability
How predictable is your team? What % of your target work items (or SLA if you are a support team) do you meet within your time-box (ex: iteration/week)?
Pre-Crawl (1-2): We don’t set targets OR we meet less than 50% of our planned items / SLA.
Crawl (3-4): We meet 50%+ of our planned items (we miss 5 out of 10).
Walk (5-6): We meet 70%+ of our planned items (we miss 3 out of 10).
Run (7-8): We meet 80%+ of our planned items (we miss 2 out of 10).
Fly (9-10): We meet 90%+ of our planned items (we miss 1 out of 10).
Story Cycle Time
On average, how long does it take us to deliver a story to production once it’s pulled by the team? If you don’t use Stories, then measure a work item.
Pre-Crawl (1-2): within 1 month+.
Crawl (3-4): within 3 weeks.
Walk (5-6): within 2 weeks.
Run (7-8): within 1 week.
Fly (9-10): Less than 3 days.
Feature Cycle Time
On average, how long does it take us to deliver a feature to production once it’s pulled by the team?
Pre-Crawl (1-2): within 3+ months.
Crawl (3-4): within 8 weeks.
Walk (5-6): within 6 weeks.
Run (7-8): within 4 weeks.
Fly (9-10): within 2 weeks.
Deployment Frequency
How often DOES your team deploy to production?
Pre-Crawl (1-2): We deploy every 3+ months.
Crawl (3-4): We deploy every month.
Walk (5-6): We deploy every 1-2 weeks.
Run (7-8): We deploy and release on demand.
Fly (9-10): We deploy continuously and release on demand.
How often COULD your team deploy to production?
Pre-Crawl (1-2): We could deploy every 5+ weeks.
Crawl (3-4): We could deploy every 3-4 weeks.
Walk (5-6): We could deploy every 1-2 weeks.
Run (7-8): We could deploy once per day.
Fly (9-10): We could deploy multiple times per day or on demand.
Add any additional thoughts about "Faster Value Delivery - Flow"
High performing delivery teams require visibility throughout the entire process and make their decisions based upon metrics and data so that improvement can measured without personal bias.
Application Monitoring
Improving production performance demands continuous monitoring of applications and collection of production metrics that illuminate continuous improvement opportunities.
Pre-Crawl (1-2): We have no visibility into our applications in production.
Crawl (3-4): There are operations teams that monitor production metrics. They see production anomalies, but don't always have the knowledge to understand the real impact to the system.
Walk (5-6): Our team collaborates with the operations team(s) that monitor production metrics, and together we are getting better at recognizing and reporting exceptions or production failures. We work with them to understand opportunities to grow our development practices to improve production performance.
Run (7-8): Our team thoroughly monitors production and collects metrics on our applications. We have fewer application outages and shorter resolution time for exceptions and failures. We incorporate the feedback in our continuous improvement practice.
Fly (9-10): Our team has automated monitors in place that enable us to diagnose application exceptions and failures in production. We analyze opportunities to correct failures based on impact and severity, immediately. Our practices are part of our everyday activities.
In order to quickly diagnose application exceptions and failures in production, teams must have the ability to effectively monitor their applications.
Pre-Crawl (1-2): We don't have the ability to recognize and report exceptions or failures in production.
Crawl (3-4): We are learning how to recognize and report exceptions or failures in production.
Walk (5-6): We are beginning to implement application monitoring and alerting to recognize and report exceptions or failures in production.
Run (7-8): We have implemented thorough application monitoring and alerting, resulting in fewer application outages and shorter resolution times for exceptions and failures.
Properly instrumented applications have automatic notifications that identify when certain thresholds are exceeded in production that could indicate problems so that teams can proactively address them and avoid impact to customers.
Pre-Crawl (1-2): We typically only know about issues if they are brought up by customers.
Crawl (3-4): Our team has the ability to identify certain thresholds when they are exceeded in production and could indicate problems.
Walk (5-6): Our team is experimenting with identifying certain thresholds when they are exceeded in production and could indicate problems. We are acting on our learning and working to refine our practice.
Run (7-8): By applying our learnings, our team has implemented automatic notifications to identify when certain thresholds are exceeded in production that could indicate problems, enabling us to act and resolve them quickly.
Fly (9-10): Our team has proven practices to automatically identify when certain thresholds are exceeded in production that could indicate problems, notify the team immediately and enable us to act and resolve them quickly.
Work Traceability
Traceability and auditing allow agile teams to connect user stories, test cases and code to more easily resolve issues when they occur.
Pre-Crawl (1-2): We do not use an electronic Agile lifecycle management tool and have no means to link requirements to codes and deployments.
Crawl (3-4): We use an electronic Agile lifecycle management tool, but do not know how to leverage it for traceability and auditing.
Walk (5-6): Our team captures user stories and is beginning to capture test cases, technical debt and bugs in an electronic Agile lifecycle management tool and linking requirements to the resulting code changes for traceability and auditing.
Run (7-8): Our team captures user stories, test cases, technical debt and bugs in an electronic Agile lifecycle management tool and links requirements to the resulting code changes for traceability and auditing.
Fly (9-10): Our team has proven practices to capture user stories, test cases, technical debt and bugs in an electronic Agile lifecycle management tool for traceability and auditing and linking requirements to the resulting code changes.
Resolving Issues
When issues occur in production systems, high performing teams not only fix the issue, but they do root cause analysis and implement a solution and testing to ensure the issue doesn't occur in the future.
Pre-Crawl (1-2): We don't have sufficient instrumentation to help us quickly diagnose and resolve issues.
Crawl (3-4): We are learning how to implement system logging, monitoring, and alerting that helps us diagnose and resolve issues.
Walk (5-6): We are beginning to instrument our most vulnerable and mission critical systems, so that we have information on issues that helps us to diagnose and resolve the problem. Time permitting, we work as a whole team to understand the root cause of the issue, and implement a permanent fix. We improve instrumentation where needed.
Run (7-8): We have instrumented most of our systems, so that the appropriate team members are alerted and provided with a description of the problem. We conduct root cause analysis and implement the fix to ensure that we won't experience that same issue in the future.
Fly (9-10): We have a proven practice to instrument our systems so that we can respond quickly when we encounter an issue. We build tests around the issue to ensure the system works as desired into the future.
Add any additional thoughts about "Higher Quality - Production Health"
High performing delivery teams have successfully implemented the necessary automated testing capabilities across several testing categories to ensure that when these tests pass they are confident in releasing the application to production.
To what extent do you agree with the statements below? Please rate according to the PRE-CRAWL(1-2), CRAWL(3-4), WALK(5-6), RUN(7-8), FLY(9-10) scale keeping in mind each level assumes you've achieved the previous level.
Security Testing
High performing teams incorporate automated security testing early in the development cycle so they ensure that they are building high quality applications and remain in compliance.
Pre-Crawl (1-2): We don't know how to perform security testing on our applications.
Crawl (3-4): We are learning about security testing tools and processes.
Walk (5-6): We are manually running some security tests against our application but only after they have been deployed.
Run (7-8): During development, we incorporate automated security testing to ensure we are always in compliance and make our non-functional security requirements visible.
Fly (9-10): We incorporate automated security testing from the beginning of development, to ensure we are always in compliance and make our security NFRs visible to the organization. Our security team is able to focus on more strategic security concerns.
Automated Testing
Teams can have confidence in their changes and ability to release to production through automated tests that validate the system behaves as it should to meet business and customer needs.
Pre-Crawl (1-2): We do not have any real testing processes for our applications
Crawl (3-4): We are manually testing our application.
Walk (5-6): We have some automated tests for our application, including unit tests, integration tests, performance tests, functional tests, and others. However, we mostly rely on manual testing.
Run (7-8): Our team consistently implements and maintains automated tests for the applications we are building. Most of the testing of our application is automated.
Fly (9-10): Our applications are fully automated, with few exceptions. When our automated tests pass, we are confident we can deploy our application to production.
Testable Requirements
Clearly defined user stories that are testable ensures the team understands what needs to be developed for the application and can write automated tests to validate that the detailed business requirements have been met. These tests can be run repeatably.
Pre-Crawl (1-2): We do not define user stories or requirements in a consistent way.
Crawl (3-4): Most of our user stories are too high level and we don't get clarity before development begins.
Walk (5-6): We break down acceptance criteria for each user story so it is clear what done looks like.
Run (7-8): We turn our acceptance criteria into test cases that can be validated to ensure the application is delivering the business needs.
Fly (9-10): Our test cases are automated and represent our testable requirements.
Application Resiliency
Resilient applications begin by implementing automated tests that validate the intended system behavior and prevent anticipated failures.
Pre-Crawl (1-2): Our systems are fragile and experience many failures, some for seemingly random errors. We don't know how to harden our systems or test for unexpected errors.
Crawl (3-4): We are learning how to make our systems more resilient and we are thinking about potential failures.
Walk (5-6): We are starting to build more resilient applications by implementing automated tests. Our tests validate the intended behavior and prevent anticipated failures. We are learning how to think about everything that can possibly break to introduce faults in our application testing.
Run (7-8): Our team regularly injects faults into our application to ensure we have built resilient applications that handle failures properly. We have proven practices to enable us to react quickly.
Fly (9-10): Our applications are self-healing, able to detect and respond to failures properly and handle recovery without impact to experience.
Performance Testing
High performing teams run both performance and load tests early in the development process as part of the continuous delivery pipeline to ensure the system always meets the defined non-functional performance requirements.
Pre-Crawl (1-2): We do not know how to do performance or capacity testing. We wait and see how the application performs when it gets to production.
Crawl (3-4): We are learning how to do performance and capacity testing, but usually only do it if there is a problem or customer complaint.
Walk (5-6): We are learning how to do automated performance and capacity testing, but usually only do it at the end of the development cycle.
Run (7-8): Our team runs automated performance and capacity tests for our applications in the deployment process and we are continuously improving our practice to ensure it is always meeting the defined performance requirements.
Fly (9-10): Our team leverages our proven practice and runs both performance and capacity tests for our application as part of our automated deploy process to ensure the system is always performing within expected boundaries.
Add any additional thoughts about "Higher Quality - Software Quality"
Focusing on Higher Quality results in Quality
Quality Confidence Rating (answered by everyone including PO and stakeholders)
How confident are you with the quality that will be produced by this team? (performance, escaped defects...)
10 = Extremely confident, 1 = Extremely not confident
Defect Removal
What percentage of the team's development time is spent removing defects?
Pre-Crawl (1-2): We spend > 50%
Crawl (3-4): We spend > 30%
Walk (5-6): We spend > 20%
Run (7-8): We spend < 5% of our time
Fly (9-10): We spend < 1% of our time
Defect Density
What is the average ratio of bugs to stories / work items in your backlog? Use either count of items or points.
Pre-Crawl (1-2): Our defect density is at 50%+ (example: We have 50 pts of defects for every 100 pts of stories)
Crawl (3-4): Our defect density is at 30%+
Walk (5-6): Our defect density is at 20%+
Run (7-8): Our defect density is <5%
Fly (9-10): Our defect density is 1% or less
MTTR (Mean Time To Recover)
On average, how long does it take to recover from a high severity production incident?
Pre-Crawl (1-2): >= 8 hrs.
Crawl (3-4): < 8 hrs.
Walk (5-6): < 6 hrs.
Run (7-8): < 2 hrs.
Fly (9-10): Minutes or seconds.
Customer Tickets
How much improvement have you had over your baseline (first month of Agile adoption ) in customer support tickets reporting issues/defects?
Pre-Crawl (1-2): We have improved by 0 - 20%.
Crawl (3-4): We have improved by 20 - 40%.
Walk (5-6): We have improved by 40 - 60%.
Run (7-8): We have improved by 60 - 80%
Fly (9-10): We have improved > 80%
Test Automation
How well have we matured towards our test automation goals?
Pre-Crawl (1-2): We don’t have any test automation goals OR We've automated 25% or less of our target goals
Crawl (3-4): We've automated 60% of our target goals
Walk (5-6): We've automated 80% of our target goals
Run (7-8): We've automated 90% of our target goals
Fly (9-10): We've fully automated 99%+ of our target goals
Defect Ratio
What is the average ratio of escaped defects to stories / work items in your backlog? Use either count of items or points.
Pre-Crawl (1-2): Our defect ratio is at 50%+ (example: We have 5 escaped defects for every 10 stories).
Crawl (3-4): Our defect ratio is at 30%+.
Walk (5-6): Our defect ratio is at 20%+.
Run (7-8): Our defect ratio is at 10%+.
Fly (9-10): Our defect ratio is 5% or less.
Change Failure Rate
What percent of your production releases over the last quarter have resulted in high severity incidents? For example, ones that lead to unplanned outages, severe performance degradations, or require a hotfix, roll-back, etc.
Pre-Crawl (1-2): ~ 40%+ (4 in every 10 releases).
Crawl (3-4): ~ 30% (3 in every 10 releases).
Walk (5-6): ~ 20% (2 in every 10 releases).
Run (7-8): ~ 10% (1 in every 10 releases).
Fly (9-10): ~ 5% or less (1 in every 20 releases).
Add any additional thoughts about "Higher Quality - Quality"
High performing delivery teams believe that everyone is responsible for customer success and therefore have developed cross functional teams that work across multiple roles to continually refine process efficiency and effectiveness to deliver the most business and customer value.
Collective Ownership
High performing teams adopt a collective ownership mindset where the quality of the applications is everyone's responsibility including the developers, testers, BA's, and operations team members.
Pre-Crawl (1-2): We have siloed areas of responsibility either by individuals or sub-teams. We rely on the responsible individual(s) to make changes to the system.
Crawl (3-4): Our team is learning how to break down silos within our team. Others may make changes to a particular system, but all changes must go through the responsible team/individual for review.
Walk (5-6): Our team actively transfers knowledge between team members so that we are building collective ownership.
Run (7-8): Our team actively shares knowledge across technical domains. Any change request can go to any member of the team with confidence.
Fly (9-10): Our team is able to deliver value to our customers faster because we have few dependencies and any team can make changes to code because it is easily understood and follows common coding standards.
Process Effectiveness
True agility means a team is able to effectively shift their delivery plan to accommodate changing business needs.
Pre-Crawl (1-2): We don't feel empowered to deviate from our work plan to respond to change.
Crawl (3-4): We cannot easily respond to change due to circumstances beyond our control.
Walk (5-6): Our team is experimenting with ways to adjust the way we work in order to effectively respond to change.
Run (7-8): Our team responds to changing business needs as they come.
Fly (9-10): Our team responds to change quickly and effectively with little waste and overhead.
High performing teams demonstrate their effectiveness for delivering business value by electronically capturing appropriate metrics.
Pre-Crawl (1-2): Our team does not know how to measure our process effectiveness.
Crawl (3-4): We are learning about metrics to measure the effectiveness our team's application development and delivery processes.
Walk (5-6): We are beginning to use metrics to measure the effectiveness our team's application development and delivery processes. As we learn, we are identifying how to collect and report our metrics using automated processes.
Run (7-8): Our team uses metrics that are collected automatically to measure the effectiveness our application development and delivery processes. We incorporate the learnings from the metrics into our continuous improvement practices.
Fly (9-10): Our culture is data-driven and embraces metrics that demonstrate the effectiveness of our application and delivery processes.
Regularly incorporating process improvement work into the regular team backlog ensures that continuous improvement activities are being completed and making an impact on the systems & applications.
Pre-Crawl (1-2): We don't have a continuous improvement process.
Crawl (3-4): We talk about ways we can work better but rarely do anything about it.
Walk (5-6): We are starting to make some small process improvements but it is very inconsistent.
Run (7-8): We regularly make process improvements and are seeing the impact of these changes.
Fly (9-10): We dedicate time every iteration to make process improvements as it is helping us become even more efficient and effective.
Cross-functional Teams
High performing teams have everyone on them needed to deliver end-to-end and those members are generalizing specialists who can work on a variety of tasks from development to operations.
Pre-Crawl (1-2): Our team does not have the necessary team members and skill sets.
Crawl (3-4): Our business, development and operations teams do not work together closely but do collaborate at times.
Walk (5-6): Business, development, and operations teams work together regularly. We consider them team members and critical in delivering an end-to-end solution. We are experimenting with creating one team that delivers new features and supports production for those features.
Run (7-8): We are a cross functional team that incorporates business, development, testing and operational team members. We frequently employ practices to share knowledge across disciplines so that our members are becoming generalizing specialists.
Fly (9-10): Our culture has embraced the cross functional team design with all the skills and knowledge necessary to deliver end-to-end; including business, development, testing and operational team members.
Add any additional thoughts about "Culture of Improvement - Efficiency & Collaboration"
High performing delivery teams continually work closely with leadership teams to ensure impediments are quickly removed and that technical debt is prioritized along with functional requirements.
Leadership
Agile leadership provides teams with the vision and proper guidance while empowering them to discover and implement the best solutions for the business.
Pre-Crawl (1-2): Our leadership takes a more command and control style when it comes to decision making. Input is not necessarily solicited from the team.
Crawl (3-4): Our leadership wants to empower teams, however when under pressure, they revert to a more command and control style.
Walk (5-6): Our leadership is taking the steps to provide our team with a vision and solicits input from the team members before making decisions.
Run (7-8): Our leadership provides us with the proper guidance and empowers our teams to discover and implement the best solutions for the business. They enable the teams to experiment, learn and grow and seek better ways to support the team as servant leaders.
Fly (9-10): Our leadership models a growth mindset. They provides us with the vision and enable our teams to discover and implement the best solutions for the business. They support the teams by removing obstacles to enable experimentation, learning and growth.
Impediment Management
High performing teams identify impediments and make them visible quickly; they take an aggressive approach to removing them. When necessary, leadership steps up to remove organizational bottlenecks when they are impeding the team from delivering.
Pre-Crawl (1-2): Most impediments that are impacting our team are talked about, but nothing gets resolved.
Crawl (3-4): We are learning about impediment management and discussing with our leadership team the benefits of working together to eliminate things that are preventing us from getting better results.
Walk (5-6): We are experimenting with our impediment management practice and ensuring visibility to the things slowing us down. We are working with leadership to remove them.
Run (7-8): Impediments are visible to everyone and our whole team works aggressively to remove them. When necessary, we escalate organizational impediments to our leadership team so that the team may move forward quickly.
Fly (9-10): Our organization takes a relentless approach to identifying and removing impediments. Our leadership teams have prompt visibility into the organizational obstacles slowing us down and works to quickly remove them.
Technical Debt Management
Teams must resolve technical debt to enable agility. Technical debt should be prioritized along with the other backlog items to ensure the team is building quality solutions.
Pre-Crawl (1-2): Our team does not track technical debt, we just kind of know the big rocks are there.
Crawl (3-4): Our team is taking steps to track our technical debt so they can ensure it is captured to be paid down.
Walk (5-6): Our team tracks our technical debt and have a practice to pay it down every iteration. We continue to continuously improve our techical debt resolution practice and overall practice to reduce new Tech Debt.
Run (7-8): Our team identifies, tracks, and measures technical debt in our unified backlog and a continuous practice of removing our technical debt to ensure we are building a quality software solution.
Fly (9-10): Our team takes a relentless approach to refactoring and removing technical debt from the system so that we are constantly able to add new features, extend old ones, and meet the business needs without undue overhead caused from looming technical debt.
Add any additional thoughts about "Culture of Improvement - Enabling Business Agility"
High performing delivery teams operate best within company cultures that incorporate learning and growth into daily work and understand feedback loops and experimentation are key drivers of innovation.
Risk Taking
Failures in the development process help drive learning experiences and enable new and innovative solutions. Leadership support of such learning is imperative.
Pre-Crawl (1-2): We do not feel empowered by our leadership team to experiment with new techniques or technologies during our development projects.
Crawl (3-4): We are learning about the value of conducting controlled experiments with new techniques and technologies. We are planning to work with our leadership team to get support to try some well defined experiments.
Walk (5-6): We have support from our leadership team to conduct some controlled experiments with new techniques and technologies. The ground rules are that the experiments are time-boxed and have well defined goals. Learning from the experiments are shared with the leadership team.
Run (7-8): We are empowered by our leadership team to experiment with new techniques and technologies. We embrace the opportunity and make it part of our team culture, ensuring we share learning from successes and failures with other teams and our CoP.
Fly (9-10): Our organization's culture celebrates experimentation in the development process and empowers teams to leverage that learning to enable new and innovative solutions. We have experimentation proven practices and celebrations that we share across our company.
Feedback Loops
Feedback from various sources, such as our customers, the system, and the application itself, help identify the best and most impactful features and capabilities of an application.
Pre-Crawl (1-2): We are not aware of any feedback processes in our organization.
Crawl (3-4): We are learning about possible feedback processes that could be established, including customer reviews, A/B Testing, performance testing, application monitoring, code reviews, functional testing, retrospectives, and others.
Walk (5-6): We are experimenting with gaining feedback from various sources and using that information to improve our applications and our delivery processes.
Run (7-8): Our team acquires and incorporates feedback from various sources of information throughout product development and delivery. We are continuously refining our practice and we share our learnings with other teams and our CoP.
Fly (9-10): Our organization is feedback-driven by insight from various sources of information early and often to identify opportunities to improve our applications and delivery processes.
Learning & Experimentation
Organizations who focus on continual learning and welcome new and innovative ideas from all of their delivery teams are better able to harness the power of agility.
Pre-Crawl (1-2): We don't have time for learning, growth and innovation; our focus is on the work at hand without time for other activities.
Crawl (3-4): Learning, growth and innovation are supported by leadership but there are too many impediments that prevent our team from getting started.
Walk (5-6): Our leadership actively supports continual learning, growth, and innovative ideas from all of our teams. Our team embraces the benefits of learning, growth and innovation.
Run (7-8): Our culture celebrates continual learning and growth, we welcome new and innovative ideas from all of our teams and celebrate results. We have a solid growth practice and are realizing the benefits regularly.
Fly (9-10): Our organization considers the investment in learning, growth and innovation as a strategic advantage. We have a proven continual learning, growth and innovation practice. We make our strategy visible and celebrate often. We have improved results in many areas due to our commitment to learn and grow.
Add any additional thoughts about "Culture of Improvement - Enabling Innovation"
Focusing on a culture of improvement results in agility.
Dependencies
Typically, how many teams does it take to complete a feature?
Pre-Crawl(1-2): We don’t track the number of teams needed to complete one feature.
Crawl (3-4): 5+ teams.
Walk (5-6): 3-4 teams.
Run (7-8): 2 teams.
Fly (9-10): 1 team.
Impediments
What % of your work is blocked by factors outside of your control? Example: Waiting for approvals, decisions, other teams, etc.
Pre-Crawl (1-2): We are usually blocked on 7 out of 10 stories.
Crawl (3-4): We are usually blocked on 5 out of 10 stories.
Walk (5-6): We are sometimes blocked on 3 out of 10 stories.
Run (7-8): We are sometimes blocked on 2 out of 10 stories.
Fly (9-10): We are rarely blocked on my stories and can often complete them with minimal wait.
Add any additional thoughts about "Culture of Improvement - Agility"
What are the top strengths of this team? What value has this team delivered? Share this team's strengths and success stories you want to celebrate!
What are this team's growth opportunities? Think of areas that could significantly improve the overall team's performance if they were addressed.
What are the significant impediments this team has been experiencing? Some may be team related and some may be organizational.
