We want {CompanyName} to be a fun, challenging and rewarding place to work. Help us realize our vision of a world-class Agile work environment by completing the AgilityHealth Assessment. This assessment is your opportunity to help shape the future of our team and improve your own experience in the process. Your feedback will be compiled and used to guide further positive changes across the team and organization.
The ART is comprised of dedicated, cross-functional teams with each of the key roles on the teams being filled. Program level roles are in place to support teams and provide leadership. Teams operate on an agreed upon development cadence and schedule for synchronized Program and Team Events.
Dedicated Teams
Crawl > Team members often split their time across multiple teams and may split their time across ARTs. Some or all of the teams are lacking key skills on the team and/or teams are highly dependent on other teams to complete work.
Walk > The teams on the Agile Release Train are formed and stable. Most, but not all, Agile team members are dedicated to a single Agile Release Train. The Teams have the skills needed to design, build, test, and deliver value.
Run > Team Members have most of their time dedicated to their team and the Agile Release Train. Due to constraints (knowledge gaps, specializations, etc.), some team members may need to take on additional tasks that are outside of their team's backlog.
Fly > Agile teams are dedicated to one Agile Release Train (ART) and teams have dedicated members. It is rare that team members have tasks outside of their team and when they do, it is work that is part of their ART.
CRAWL 1-2 | WALK 3-5 | RUN: 6-8 | FLY: 9-10
Team Roles
Crawl > Many of the teams are missing roles and are impeded by not having the leadership and skills needed to work successfully as cross-functional teams. Teams are not given full authority over their work and many decisions are made for the team.
Walk > Most if not all of the teams have the leadership and cross-functional roles filled, but there is lack of clarity on responsibilities and decision making authority. Teams are making more of their own decisions on how to complete the work.
Run > The teams on the train have all the leadership and cross-functional roles filled. A few of the critical roles are filled by people that are less than 100% dedicated. There is more clarity on the roles, teams are given full authority over their work and decisions are made in a timely manner.
Fly > Each team on the train has an identified and dedicated Scrum Master, Product Owner and cross-functional team members. The people in these roles are focused and fully filling their roles. Decisions are made quickly and the teams are empowered and accountable.
Schedule
Crawl > We are trying to create a regular cadence. The schedule changes frequently, program and team events change, iterations are shortened or lengthened and/or the PI schedule changes due to the ART struggling to stay on track.
Walk > The development cadence and schedule for Program and Team Events has been agreed upon, but we experience changes to the schedule due to people's availability, lack of alignment across groups, logistics, or other issues. The time-boxes for the events are often too short and we feel rushed or don't get everything done.
Run > The development cadence and schedule for Program and Team Events for the PI is established and followed. The time-boxes for the Program and Team events allow us to get full value for the time we spend. We only have occasional instances where the cadence and Program and Team events change and/or the time-boxes are too short to meet our needs.
Fly > The development cadence and schedule for Program and Team Events for the PI is established and followed. The time-boxes for the Program and Team events allow us to get full value for the time we spend. We only have occasional instances where the cadence and Program and Team events change and/or the time-boxes are too short to meet our needs.
Program Structure
Crawl > The program level roles are either not filled at all, only partially filled, or that some of the people in these roles have not been a good fit. The people in these roles have limited or little decision-making authority.
Walk > Most of the program level roles are filled. Some or most of the people filling these roles are new to their role on the ART. They often defer decisions to others as they do not have all the decision-making authority they need.
Run > All of the program level roles are filled by people dedicated to their role. They are a good fit for their role and they are working to improve their skills. They have authority to make most decisions but could still be more empowered to be fully effective.
Fly > Each of the program level roles for the ART are filled; System Architect/Engineer, Product Management, Release Train Engineer (RTE), and Business Owners. The people in these roles are dedicated, empowered to make decisions and have the right skills to be effective.
Add any additional thoughts about "Planning Readiness - ART Organization"
PI planning preparation includes the preparation activities around event logistics, Business and Architectural vision and priorities, and Program and Team Backlogs to ensure a highly successful planning session.
Event Space
Crawl > Problems with the event space, tech support and logistics prevent the ART from effectively making use of the planning time.
Walk > The event space and logistics are sufficient but not ideal; there are some barriers to communication within and across teams and there are a number of logistical challenges.
Run > The event space works well and supports collaboration most of the time. The event logistics cover the essentials and we have what we need to complete our deliverables. We could provide a better experience over the two days with a more thoughtful approach to our event space.
Fly > The event space is ideal and the logistics are well thought out and executed. The event space and seating arrangements are effective and support face-to-face and remote collaboration. We use video and audio connections for remote participants and quickly address issues with back-up plans and technical support. We have everything we need (supplies, post-its, markers, tools, food, etc.) to complete our deliverables and have a productive and enjoyable experience over the two days.
Business Owners Alignment
Crawl > The Business Owners are not spending enough time preparing to effectively present the business context at the PI Planning event. Business Owners, RTE and Product Management are not well aligned on the business objectives and there are outstanding questions coming into PI Planning. System Architects and other key stakeholders are not aware or do not fully understand the business objectives prior to the PI Planning event.
Walk > Business Owners are spending some time to prepare their presentations on the business context. They are not always spending enough time to speak to all key milestones and dependencies. They review the business objectives with the RTE and Product Management, but don’t regularly involve System Architects and other key stakeholders. We frequently have outstanding questions around alignment to business objectives going into the planning event.
Run > Business Owners prepare their presentation on the business context, milestones and dependencies prior to PI planning. They spend time discussing the business objectives with the RTE, Product Management, System Architects to ensure alignment. We do experience some issues with conflicting objectives, especially with stakeholders outside of the ART.
Fly > Business Owners work with the RTE, Product Management, System Architects and other key stakeholders to ensure that business objectives are understood and agreed upon prior to PI Planning. They prepare to speak to the business context, milestones and significant external dependencies during the PI Planning event.
Enabler Features & Non-Functional Requirements (NFRs)
Crawl > Program Enablers and Non-Functional Requirements are missing from planning preparation.
Walk > We identify some Program Enablers and Non-Functional Requirements but there are often questions about alignment with the business vision and priority. It is common that we discover more enablers and non-functional requirement during or after planning. The Architecture Runway is not keeping pace and we experience delays and redesign during development of the business features.
Run > We always discuss and identify the Enabler Features and Non-functional Requirements as part of the planning preparation process. Our pre-planning discussions could be better as we sometimes miss key Enabler Features or Non-functional requirements which result in occasional delays or redesign during development of the business features.
Fly > Enabler Features and Non-functional Requirements are prioritized and in alignment with the Architectural Vision, the Business Vision and the Top Business Features for the upcoming PI. The Enabler Features are selected to support building the Architectural Runway needed for developing near-term features without significant redesign and delay.
Engineering Best Practices
Crawl > There is a lack of definition and communication around engineering best practices and standards. The practices are rarely used and standards are not applied.
Walk > Some engineering practices and standards have been defined and communicated but adoption and understanding are inconsistent across teams.
Run > Engineering best practices and standards are established and published to ensure built-in quality. Regular communication of the standards could be improved as we still experience inconsistencies across teams for periods of time.
Fly > Engineering best practices and standards (Set Based Design, frequent integration, test first, pair work, collective ownership, etc.) are defined and communicated to build quality in and create consistency across teams.
Program Backlog
Crawl > The Program backlog is not ready prior to the PI planning event, causing a lack of clarity on objectives and priorities going into the event. We spend a lot of time during the PI planning event understanding, refining and breaking down Program Backlog items.
Walk > The Program backlog is ready before the event but there are last minute activities required to get everything ready. Product Owners and key stakeholders do not have much time to review, integrate and provide feedback. Teams are not regularly involved in refining the Program Backlog. Many or all of the backlog items lack acceptance criteria, benefit hypothesis and effort estimations.
Run > Product Management prepares the Program Backlog so that it is ready a couple of weeks prior to the planning event. They work with the teams to estimate effort. Most of the high-priority items have acceptance criteria and benefit hypothesis. Backlog Refinement occurs near the end of the PI so we don't have much time to discuss feasibility or break backlog items into smaller chunks.
Fly > Product Management continually reviews and updates Program Backlog items, defines acceptance criteria and develops benefit hypothesis. They work with the teams to understand technical feasibility, effort involved, and ways to split backlog items into smaller chunks of incremental value. The Backlog Refinement cadence ensures the Program Backlog is ready and socialized with Product Owners and key stakeholders 3-6 weeks prior to event.
Team Backlog
Crawl > The Team's Backlog is not prepared and the team is impeded in their ability to create a plan within the PI Planning time box.
Walk > Product Owners have backlog items to bring to planning but many of the items are not refined, estimated or prioritized. A significant amount of time is spent in the planning event getting backlogs clear enough for the team to build the plan.
Run > High priority user stories are written and reviewed with the team prior to planning. Many of the user stories are estimated and have acceptance criteria. The team has a good understanding of the work and there is just enough prepared backlog to effectively plan during the PI Planning event. Priorities are not always well understood and sometimes the team is unaware of key milestones.
Fly > The high priority items in the Team's Backlog are written and refined by the team. Refined backlog items have acceptance criteria, effort estimates and are broken down to be completed within an iteration. Team Members have a clear understanding of priorities and milestones going into the planning event.
Add any additional thoughts about "Planning Readiness - Preparation"
The PI Planning Event must be well-organized, include all ART team members, stakeholders, managers and shared resources, and result in team plans that are complete, highly visible and that identify and manage cross-team dependencies.
Preparation for Event
Crawl > The plan and agenda for event is significantly lacking; there are major gaps in planning logistics and communication.
Walk > The plan and agenda for events is sufficient but not optimal, and some communication points are missed.
Run > The plan and agenda for required events and the basic logistics are planned and communicated in advance. The plan does not incorporate specialty events, continuing education and/or special access provisions.
Fly > The RTE has prepared and announced events, agenda, attendance, venue, access provisions, specialty events, and continuing education, as appropriate. The plan is well-thought-out and all participants have good information about what is expected.
Facilitation of PI Planning Event
Crawl > Time boxes are not well managed and key planning deliverables are pushed out or dropped.
Walk > Most of the planning work is completed within the overall timeline but time management could be improved. Some start and end times are missed, with events going longer than expected.
Run > The event begins and ends on time and most of the time boxes are adhered to. The facilitation is effective at keeping things on track.
Fly > The event begins and ends on time and all time boxes are adhered to. The timeboxes for each item on the agenda have been evaluated and optimized over time. Times are effective for all involved including distributed teams, and different time zones if applicable.
Participation of Program Stakeholders & Business Owners
Crawl > Few, if any, program stakeholders and business owners attend or participate.
Walk > There is partial attendance and participation from program stakeholders and business owners.
Run > Program stakeholders and business owners attend key portions of the planning event. For the portions they attend, they actively participate. They often have conflicts that prevent them from participating throughout the planning event.
Fly > Program stakeholders and business owners make the PI Planning event a priority and are in attendance and actively participate. They present vision statement, objectives and engage with teams to answer questions and provide clarity.
Participation of Shared Resources
Crawl > Few, if any, shared resources attend or participate the PI Planning event. For those that do attend, they don't understand their role
Walk > There is partial attendance and participation from shared resources. They are confused about their role and/or how best to participate in the PI Planning event.
Run > Shared resources are beginning to understand their role in PI Planning. They routinely attend the planning sessions. We are still working to help them actively participate and contribute to the outcomes of the meeting.
Fly > Shared resources, understand their role, attend PI planning and actively participate. Their participation in planning improves the outcomes of the planning session.
Participation of 3rd Party Suppliers
Crawl > Few, if any 3rd Party suppliers attend or participate the PI Planning event. For those that do attend, they don't understand their role
Walk > There is partial attendance and participation from 3rd Party suppliers. They are confused about their role and/or how best to participate in the PI Planning event.
Run > 3rd Party suppliers are beginning to understand their role in PI Planning. They routinely attend the planning sessions. We are still working to help them actively participate and contribute to the outcomes of the meeting.
Fly > 3rd party suppliers, understand their role, attend PI planning and actively participate. Their participation in planning improves the outcomes of the planning session.
Participation of Lean-Agile Leaders
Crawl > Development managers do not engage in and actively support the planning process. They remain focused on task and work management and continue to provide solutions and direction on how to do the work. They are not well versed in Lean/Agile behaviors and/or are not supporting the change to those behaviors.
Walk > Development managers display a mix of behaviors, sometimes supporting teams and removing impediments and other times taking a command and control approach or failing to resolve issues that need their attention.
Run > Managers are shifting to becoming Lean-Agile leaders, supporting the planning process and eliminating impediments. They are becoming champions for Lean-Agile behaviors. They are still transitioning their focus on people development and serving as an Agile coach to the team.
Fly > Lean-Agile leaders embody the mindset and walk the talk. They play a supportive, empowering role in the planning process and eliminate impediments outside of program control. They emphasize continuous learning, create a safe environment, focus on developing people's skills and career paths, and coaching the team to problem solve and discover their own solutions.
Management Review & Problem Solving Meeting
Crawl > The Management Review and Problem Solving meeting is not held or it is held at a high-level or is rushed through. Issues that need management resolution are not identified or are not dealt with timely. Some key stakeholders are missing.
Walk > The Management Review and Problem Solving meeting is held and issues are being identified and discussed. Management addresses some of the most significant issues around scope, resource constraints and dependencies, but other issues that need to be dealt with are left without clear resolution.
Run > The Management Review and Problem Solving meeting is identifying and discussing most issues so that planning adjustments can be made. We are not always coming up with good resolutions for the resource constraints and dependencies. Follow-up meetings are sometimes still needed.
Fly > The Management Review and Problem Solving meeting results in the decisions needed to reach achievable objectives. The RTE facilitates bringing the primary stakeholders needed in order to collaborate with management to negotiates scope, resolve dependencies, challenges and makes planning adjustments.
Team Members Participation
Crawl > Team representation at planning is sparse; most members do not attend or miss parts or most of the meeting.
Walk > Most team members participate in planning, with some members doing other activities. Remote teams do not attend or only attend for the briefing and review activities.
Run > Team Members attend and participate in the planning meeting. Occasionally team members are pulled away for periods of time for other tasks. Remote Team members participate during planning, but struggle to participate effectively.
Fly > All team members participate in creating the PI plan for their team, in person or remotely. The team members prioritize, focus and dedicate their time to the PI Planning event. Remote team members are able to effectively participate in creating the PI Plan.
Team Planning
Crawl > Team plans are very incomplete and often unrealistic. The plans do not have estimated velocity for all iterations and many of the stories in the plans are not estimated. Very few, if any, risks and dependencies are identified.
Walk > Team plans include estimated velocity for most, if not all, iterations. There are still some stories without estimates. There are some risks and dependencies identified, but they are not capturing the majority of them yet. PI Objectives are created but do not clearly represent the team plan.
Run > Team plans are normalizing and are visible and ready for review. The majority of plans have estimated velocity for all the iterations and most stories have estimates. Many of the risks and dependencies are being identified. The PI Objectives often don't identify Stretch Objectives and/or Stretch Objectives are not well understood.
Fly > Team plans and any emerging designs align with architecture guardrails and are visible and ready for review. All iterations have estimated velocity and estimated user stories. The team has identified program risks and dependencies and created clear PI objectives that generally include Stretch Objectives.
Cross-Team Planning & Dependencies
Crawl > There is very little cross-team collaboration and dependencies are not identified.
Walk > There is informal cross-team collaboration during planning to identify dependencies, but documentation is inconsistent and some dependencies are missed. Key team members are sometimes missing during the discussions.
Run > Most of the teams are having good discussions and cross-team collaboration to address dependencies. These discussions are often being held till late in the planning event and they are rushed and not as effective or timely as they could be.
Fly > Team plans are well coordinated through cross-team collaboration, accounting for dependencies between teams. Dependencies are discussed quickly and timely throughout the planning event to ensure that team plans are effective.
Add any additional thoughts about "PI Planning Event - Planning Process"
Executives, product managers and technical leaders provide briefings at the beginning of planning to communicate vision and priorities. The deliverable from planning is well-written PI objectives that support priorities and that the teams are highly confident in delivering. The Program Board provides visibility to delivery dates, dependencies and milestones and risks are clearly identified and categorized.
Business Context & Customer Need
Crawl > The business context and customer perspective is not presented at all.
Walk > Some business context is presented at the planning event but does not provide a complete picture of the current state of the business or how solutions the teams are developing are meeting customer needs.
Run> The business context is presented at the planning event and provides a good overview of the current state of the business. It does not often address the customer's perspective and how the solutions are addressing customer problems.
Fly > The planning event includes an executive briefing to define the business context, including the current state and a perspective on how well solutions are addressing customer needs. The briefing helps us tie what we're doing to the vision and understand things from the customer's perspective and we understand the outcomes our solutions are having for our customers.
Product Vision & Context
Crawl > There is poor communication and significant confusion about the product vision and the top features the teams will plan for.
Walk > Product Management shares the top features they are bringing to planning but the product vision is not presented or well understood.
Run > Product Management provides a product briefing at each PI and they communicate the top Features in from the Product Backlog. The briefing helps us understand the context and what is intended for the top features. The briefing does not provide clarity on how the features fit into the larger product vision.
Fly > The planning event includes a product vision briefing by product management to communicate the vision for the Top features in the product backlog. The briefing helps everyone understand what those features are and what we are trying to achieve by implementing them in relation to the product vision.
Technical Vision & Context
Crawl > There is poor communication and significant confusion about the Architectural vision and enablers going into planning.
Walk > The planning event sometimes includes a briefing led by technical leadership. When there is a briefing, the information provided about the new feature Enablers and NFRs is too high-level or is incomplete.
Run > Technical Leadership provides a briefing at each PI and the briefing helps us understand the new feature enablers and NFRs. The briefing does not often, or at all, provide the context to how these new items fit into the larger technical vision.
Fly > The planning event includes a briefing by the technical leadership to communicate new feature Enablers, and Non-functional Requirements (NFRs). The briefing helps everyone understand how the new feature enablers and NFRs fit into the overall technical vision.
Feature List Prioritization
Crawl > The feature list for planning is not prioritized or communicated. Teams are not working from a common, prioritized feature set.
Walk > The feature list is prioritized for planning but it is not as well-prepared as it could be, causing communication gaps and delays during the planning event.
Run > The top features are prioritized ahead of planning but communication of the features is inconsistent. Most times it is well prepared and communicated clearly and at other times it is incomplete and/or not well communicated.
Fly > Top features are consistently prioritized ahead of time and are clearly communicated at the beginning of the planning event.
PI Objectives
Crawl > Teams have incomplete plans that have not been reviewed and are lacking objectives.
Walk > Each of the teams have visible PI Objectives, but they vary in quality, clarity and completeness. Business Owners and stakeholders struggle with assigning business value or do not assign business value at all.
Run > All teams have PI Objectives, most are clearly written and Business Owners and Stakeholders have assigned business value. PI Objectives are shared and visible.
Fly > Alignment is achieved: All teams leave with agreed to, well written PI objectives that are SMART (Specific, Measurable, Achievable, Realistic and Timely), have assigned business value points and been approved by business owners and published to all stakeholders. If there are program PI objectives they're clearly defined and well understood by all teams.
Confidence Vote
Crawl > The Confidence Vote is not taken. The level of agreement and support of the plan is unknown.
Walk > The Confidence Vote is taken but the process is not facilitated as effectively as it could be or it is not always taken (inconsistent). Some important feedback on the plan is missed.
Run > The Confidence Vote is consistently taken and discussion occurs when there are low votes. Teams re-vote after reworking their plan until consensus is reached.
Fly > The Confidence Vote is taken proactively prior to end of day 1 so the teams can work on making adjustments if the consensus is not 3 or higher. The confidence Vote is taken again at final plan review, usually results in more consensus and minimizes plan reworks needed. Confidence vote is two way and also takes into account leadership confidence in the plan.
Program Board
Crawl > The program planning board is not used at all or is so incomplete that it provides no value.
Walk > The program planning board is used but the information is somewhat incomplete or confusing.
Run > All teams are contributing to the planning board. We have a good view of major dependencies and delivery dates. We are often missing other milestones and dependencies on other ARTs.
Fly > The program planning board highlights the new feature delivery dates, feature dependencies among teams and with other ARTs, and relevant milestones.
Program Level Risks
Crawl > Risks are not identified and discussed by the teams.
Walk > Risks are identified and discussed within the team but are not well captured at the program level.
Run > Teams are identifying and making program-level risks and impediments visible but the risks are not being discussed and categorized in front of the whole train.
Fly > Teams are identifying and making visible program-level risks and impediments that could impact their ability to meet their objectives. The risks are categorized in front of the whole train and all risks have an owner.
Add any additional thoughts about "PI Planning Event - Inputs & Outputs"
Key program roles, including the RTE, Product Management and System Architect/Engineer provide leadership on process, priorities and technical direction toward accomplishing the vision and roadmap.
RTE
Crawl > The RTE role is not filled, or the RTE is taking a more traditional leadership style that is focused on work management, status reporting and centralized communication.
Walk > The RTE plays a facilitative role in the major events and processes. There are some gaps or challenges around communication and coordination and improvement efforts are inconsistent.
Run > The RTE is an effective facilitator for the major events and processes and takes a servant leadership role to ensure that impediments are escalated and addressed. The RTE is a good communicator and is moderately effective at coaching and assisting the teams in improvement.
Fly > The Release Train Engineer (RTE) acts as a servant leader, coaching the ART and effectively facilitating the major events and processes, and assisting the teams in delivering value. The RTE communicates well with Product Management, System Architect/Engineer and stakeholders. The RTE escalates impediments, helps manage risk, and drives relentless improvement.
Product Management
Crawl > Product Management does not have a good understanding of the customer and requirements are not complete or documentation is used in place of collaboration with the dev team.
Walk > Product Management provides a prioritized backlog and requirements but the connection between the customer and the development team is not optimized. At times, requirements are not ready when the team needs them.
Run > Product Management has a regular cadence for defining, prioritizing and validating requirements so that they are consistently ready when the team needs them. The voice of the customer is starting to become clear and MVP's are being defined.
Fly > Product Management takes responsibility for continuously defining, prioritizing, and validating requirements. They work in alignment with the teams' iterations, bringing the voice to the customer to the developers and the voice of development to the customer.
System Architect/Engineer
Crawl > System Architect/Engineers work independently from Product Management. There is little, if any, analysis or support work for features that are in the Roadmap.
Walk > System Architect/Engineers provide technical direction but are not always in alignment with Product Management or in collaboration with development teams. Definition of non-functional requirements, and technical and system analysis is an occasional practice.
Run > System Architect/Engineers are working well with Product Management to define non-functional requirements, and perform technical and system analysis. The work that is being done is well aligned with the technical vision in collaboration with the teams.
Fly > System Architect/Engineers collaborate with Product Management to align teams in a common technical direction toward accomplishing the Vision and Roadmap. They define higher-level functional and nonfunctional requirements, analyzing technical trade-offs and enablers, determining the major components and subsystems, and defining the interfaces and collaboration between them.
Add any additional thoughts about "PI Execution - Program Roles"
Program events support communication, collaboration and continuous learning and improvement through the PI.
Scrum of Scrums
Crawl > Scrum of Scrums does not happen at all or is of little or no value. Teams do not manage dependencies or work together to raise impediments and resolve issues.
Walk > Scrum of Scrums is held regularly but does not consistently provide value. Some issues raised in Scrum of Scrums are resolved quickly and collaboratively, but other issues are not raised or are solved without collaboration.
Run > Scrum of Scrums meeting has normalized, is time-boxed and is providing value. This meeting is the primary place where cross-team dependencies and impediments are resolved.
Fly > Scrum of Scrums meets at least weekly, is efficient and provides consistent value. Teams proactively interact outside of Scrum of Scrums to address dependencies, impediments and resolve issues.
PO Sync
Crawl > The PO Sync does not happen at all, or happens so infrequently it adds very little value, and/or the meeting happens but provides little or no value.
Walk > PO Sync is happening regularly and is somewhat effective at providing visibility toward planning and prioritization adjustments. The meeting does not consistently deliver value.
Run > PO Sync occurs regularly and is well facilitated. The meeting provides good visibility toward PI objectives and identifies issues that require resolution. The meeting would provide more value if it better supported problem resolution and led to adjustments to scope, priority or pivots.
Fly > PO Sync occurs at least weekly, is time-boxed appropriately (30-60 minutes), and provides visibility toward PI Objectives, supports resolution of problems or opportunities with Feature development, scope adjustments and other planning and prioritization needs.
System Demo of Features
Crawl > The System Demo does not happen at all or does not provide any visibility to new Features that have been delivered or is not in one integrated environment. There is a lack of feedback on the overall solution and/or lack of trust between stakeholders and teams.
Walk > The System Demo is sometimes held at the end of the iteration but does not provide a full view of all features due to technical or process limitations. When we do integrate fully, we discover some problems and risks later than we could have.
Run > The System Demo is held at the end of each iteration and often represents an integrated view of the features. There are times when 1 or more features are not part of the demo or are demonstrated in a different environment due to technical or process limitations.
Fly > The System Demo occurs at the end of every Iteration and provides an integrated, aggregated view of the new Features that have been delivered by all of the teams on the train in the most recent iteration.
System Demo Attendance
Crawl > The PI System Demo does not happen at all, is only attended by team members without feedback from sponsors, stakeholders or customers or is completely ineffective.
Walk > Only some of the sponsors and stakeholders who could provide feedback attend the PI System Demo. Conversation around business value is less than fully effective. The customer is not consistently or ever included in the session.
Run > Most of the sponsors, stakeholders and some customers attend the PI System Demo. The demo facilitates useful discussions and provides some good feedback. The business value is set but could be more collaborative. Our demos occasionally go over time when discussions get bogged down into the details.
Fly > The PI System Demo is attended by sponsors, stakeholders and customers who provide feedback and collaboratively rate the actual business value achieved for each team’s PI Objectives. The demo is time boxed to an hour or less, with the level of abstraction high enough to keep the important stakeholders engaged and providing feedback.
Inspect & Adapt Metrics Review
Crawl > Inspect & Adapt workshop does not happen or if it happens metrics are not reviewed, or used to inform continuous improvement goals.
Walk > During the Inspect & Adapt Workshop, a couple of quantitative metrics are reviewed. There is a lack of consistency in measurement across the teams which creates a lack of clarity to the data and trends. The metrics are not often used as inputs into our continuous improvement goals.
Run > During the Inspect & Adapt Workshop, several quantitative metrics are reviewed. There are some key areas that the metrics don't cover. The metrics are used to inform our continuous improvement goals, but not consistently.
Fly > During the Inspect & Adapt Workshop, teams review the quantitative metrics they have agreed to collect, then discuss the data and trends. The metrics provide a comprehensive view into how the ART is doing. Metrics are consistently used as inputs to help inform continuous improvement goals.
Inspect & Adapt Retrospective
Crawl > The Inspect & Adapt Workshop does not include a retrospective or the retrospective is poorly facilitated and does not support the teams in identifying, analyzing and resolving important issues.
Walk > Some program-level issues are addressed through retrospectives that go beyond individual teams. Stakeholders do not participate or are not consistently participating in the session. These sessions include a lot of discussion but are not facilitated in a way that consistently leads to identifying root cause.
Run > Systematic, program-level issues are identified and addressed. Stakeholders are participating in most sessions and basic root cause analysis is conducted. The workshop results in moderately effective improvement items.
Fly > During the Inspect & Adapt Workshop, a facilitated problem-solving workshop format is used to address larger, systematic program-level problems. This workshop uses root cause analysis and key ART stakeholders attend to support the teams in unblocking impediments outside the team's control. The workshop results in highly effective improvement items.
Inspect & Adapt Improvement Features & Stories Incorporated into PI Planning
Crawl > There is no process around bringing improvement stories and features from the retro and problem-solving workshop to PI planning. Process improvement stories are not tracked or prioritized.
Walk > Improvement stories and features based on the results of the retro and problem-solving workshop are sometimes brought to the next PI planning. They are not often given a high enough priority to be added into the Team Plan.
Run > Improvement stories and features based on the results of the retro and problem-solving workshop are often brought to the next PI planning. In most cases, some improvement stories are getting loaded in the iteration plans. Sometimes, not enough people and resources are dedicated to effectively improve the current state.
Fly > The team creates improvement stories and features based on the results of the retro and problem-solving workshop and prioritizes the top improvements to bring to the next PI planning. During the planning event, improvement stories are loaded into the Iteration plans, ensuring that people and resources are dedicated as necessary to improve the current state.
Innovation
Crawl > The IP Iteration does not include any time or space for creativity and innovation. The teams mostly work on finishing up the user stories that were in their PI Plan. Technical debt is growing and/or not being addressed.
Walk > The IP Iteration does include time for the teams to work on innovation. The innovation objectives the teams can work are prescribed for them or limited to a few options. It is often the case that the teams don't fully complete their innovation objectives and/or drop innovation work to finish items from the Team PI Plan.
Run > The IP Iteration regularly includes time for the teams to work on innovation and the teams often create their own innovation objectives. Occasionally teams do not fully complete their innovation objectives and/or do not consistently provide demos of the innovation results.
Fly > The IP Iteration enables innovation by providing time and space for team members to be creative. Teams are given free rein to identify Innovation objectives and they consistently provide demos of innovation results.
Continuous Learning
Crawl > The IP Iteration doesn't include any time for learning new techniques or participating in Communities of Practice but instead is used to catch up on existing or new work.
Walk > Opportunities to learn and grow are provided on occasion when circumstances allow. Communities of Practice are not used or are not effectively supported.
Run > The IP Iteration regularly provides opportunities to learn and grow. Not much time is dedicated for these activities, so the topics are often limited. There is growing support for Communities of Practice and the COPs in place are fairly new or are working to be more effective.
Fly > The IP Iteration enables continuous learning as team members have time and opportunities to learn and master new techniques. Learning is supported through mature and effective Communities of Practice. New Communities of Practice are launched as needed.
Add any additional thoughts about "PI Execution - Program Events"
The continuous delivery process supports innovation, learning and quickly meeting customer needs. Customer input and feedback is incorporated at each step. Environments match production and allow for release on demand.
Product Vision is developed collaboratively
Crawl > The product vision is not developed through collaboration with stakeholders, customers, or teams.
Walk > The product vision is is primarily developed by internal Product Management, with some input from other stakeholders and customers. Collaboration may be valued but there is little process to support it.
Run > The product vision is developed collaboratively with input from a small group of stakeholders representing some of the key roles (Agile teams, business owners, portfolio leadership and system architects/engineers). Customer input is not consistently used and is often based on customer intelligence data.
Fly > The product vision is developed through a continuous, collaborative process that solicits input from a diverse group of stakeholders including customers, Agile teams, business owners, portfolio concerns and system architects/engineers.
Product Vision is informed by customer research
Crawl > Product management does not conduct research to help establish the vision.
Walk > When customer and market data is available, it is used by Product management. The methods and standards around conducting research are inconsistent and/or are limited to customer data and market research.
Run > Product Management consistently uses research to understand the customer and help establish the vision. The majority of customer understanding comes from market research and customer data. Product Management and has started to add techniques to obtain direct customer feedback.
Fly > Product management conducts research to help establish the vision through activities such as customer visits, in-person/on-site observation of work being performed, elicitation techniques (interviews, surveys, use-case modeling, etc.), trade studies and market research.
Minimum Viable Product (MVP) & Minimum Marketable Features (MMFs)
Crawl > Features are not broken down into MVPs or MMFs. Ideas are not tested and the benefits and outcomes of features are not defined and largely unknown.
Walk > Some features in the program backlog are broken down and organized in to MVPs and MMFs. Definition of hypotheses is not practiced or is not used effectively.
Run > Most features in the program backlog are organized around feature sets (MVPs and MMFs). Hypothesis statements are used to set context for what is to be achieved and validated through the MVP or MMF. The benefits, outcomes and measurements for the hypotheses are not consistently or clearly defined.
Fly > The Program Backlog is comprised of the initial Minimum Viable Product (MVP) features set and Minimum Marketable Features (MMFs), testing hypotheses with clearly defined benefits, outcomes and measures.
Integration at the Story Level
Crawl > Integration of code across developers is not frequent enough to catch issues early. Automated testing is not something we do often or at all. We rely on manual testing to validate our work.
Walk > Team members integrate their work frequently. Code Integration is not fully automated and/or requires some manual steps. Automated testing is applied but coverage is incomplete. Typically, tests are written after the code and manual testing is the primary means to ensure existing functionality is not impacted.
Run > Team members integrate their work frequently using fully automated continuous integration. Test first development is practiced, but not always consistently. Automated testing is used and code coverage is increasing. Automated testing and targeted manual testing is used to ensure existing functionality is not impacted.
Fly > At the story level, team members integrate their individual work frequently, and apply automated continuous integration environments. Automated testing and test-first development ensures that new stories are compatible with all the existing functionality.
Integration Across Teams
Crawl > We don't have a System Team in place or they don't have all the skills, tools and resources necessary to integrate work across the teams on the ART. We struggle getting our code deployed to integration environments and need to build out the infrastructure to enable frequent integration. Our System Demos rarely demonstrate fully integrated solutions.
Walk > We have a System Team in place and they help us integrate assets across the teams. They are sometimes challenged with keeping up with the teams on the ART. We sometimes have issues with our System Demos as we find integration issues too late. We are working on building out the tools and environments we need to integrate more frequently and effectively.
Run > The System Team supports the integration of assets across the teams. They use automation and lean processes to keep pace with the teams. There are still some issues with integrating frequently enough and having System Demos ready quickly at the end of each iteration.
Fly > With the support of the System Team, the work of all teams on the ART is integrated frequently to assure that the system is evolving as anticipated. The accumulation of this work is evaluated objectively at the System Demo at the end of each iteration.
Deployment Readiness
Crawl > We don't use a staging environment or only use the staging environment just prior to major releases to Production. We don't have automated tests for validation in this environment. We perform manual tests for validation.
Walk > We have a staging environment in place and only deploy to the environment prior to planned releases. We use this environment to perform final high-level validation before moving to Production. We have some automated testing but rely on manual testing for most of the validation. During the PI, this environment does not reflect the current state of the solution as we complete each iteration.
Run > We have a staging environment in place; deployment happens frequently but not always at the end of every iteration. Testing and deployment is mostly automated.
Fly > Deployment readiness is maintained through deploying to the staging environment every iteration and the use of automated testing and deployment.
Releases are decoupled from deployment
Crawl > Releases and deployments are tightly coupled and releases are not that frequent. Our customers would like us to release more frequently. We have a ways to go to be ready for Continuous Deployment.
Walk > Deployments beyond our test environments are coupled to releases. Our releases are frequent and although we are not positioned for continuous deployment, our customers are generally satisfied with the frequency of our releases.
Run > We are moving toward fully decoupling releases from deployment. We are able to release as frequently as needed, though not completely on demand.
Fly > Releases are decoupled from deployment, so that deployment can be continuous, while releases happen on demand.
Environments
Crawl > We have quite a few differences across our development, test, staging and Production environments. It is not uncommon to find software behaving differently when it deploys across environments. We do not use version control for our configuration changes and rely on manual coordinated steps for deployment.
Walk > Our development, test and staging environments do not all match Production. Changes to the various environments are coordinated, but this is mostly a manual process. Some types of configuration changes are captured in version control, but others are not. We rely on manual steps for our deployments.
Run > Our development, test and staging environments match or very closely match Production. Most configuration changes to the various environments are coordinated using version control and some automated scripts are in use. Deployments are partially automated and require manual steps to complete.
Fly > The development, test and staging environments are managed to match production. Changes in the production environment are replicated back to all development environments using version control and automated deployment scripts wherever possible.
Release on Demand
Crawl > We do not release features incrementally. We release complete solutions to all of our customers at once when all features are complete and validated. We are not able to selectively deploy elements of the solution or to segments of our customers.
Walk > We are beginning to define and release Minimum Viable Products (MVPs) and Minimum Marketable Products (MMPs) and when these are defined well in advance of the release we are able to release them independent of additional features that are under development.
Run > We routinely release Minimum Viable Products (MVPs) or Minimum Marketable Products (MMPs) to the market. Releasing independent features incrementally takes extra effort and coordination and is not something we typically do. Product Owners can decide on when to release of an MVP or MMP, but not which elements and which users should receive the release.
Fly > Features can be released incrementally or immediately to customers based on market demand. Product owners have the ability to decide when to release, which elements of the system should be released and which end users should receive the release.
Release Management Process
Crawl > Our release management process is not well understood or consistently practiced. We bring together people as needed to work on a particular release and we often find gaps in our release process.
Walk > We have release management in place that is focused on a particular release or product. It is often organized around particular projects or programs and has a narrow scope. Our practices tend to be more ad hoc than systematic.
Run > We have release management in place and it is ongoing with a regular cadence. It is focused on the implementation of capabilities and features. We have visibility into the scope and status of each release and have the right people involved. Communication to stakeholders and leaders could be improved and the process for addressing issues and managing scope is not as effective as it could be.
Fly > Release Management coordinates the implementation of all the capabilities and features over the multiple Iterations within a release, continually managing, validating, and communicating the scope of each release as new issues, roadblocks, dependencies, over-scopes, and gaps in Vision and Backlogs are uncovered.
Add any additional thoughts about "Delivery - Continuous Delivery"
Consistent measures across teams provide visibility to determine performance and make necessary adjustments around predictability, quality and time to market.
Predictability of Features
Crawl > There is no tracking or visibility to Feature Progress or PI Burn-Down.
Walk > Feature Progress and PI Burn-Down are tracked but it is not very visible. It is not consistently used to facilitate decision-making.
Run > Feature Progress and PI Burn-Down are tracked consistently and is communicated and made visible. It is an input into decision-making but could be more effectively used to improve delivery.
Fly > The Feature Progress Report and PI Burn-down Chart provide visibility to the status of features and enablers (planned versus actual delivery) during PI execution and facilitate decisions about what changes might be necessary to successfully deliver the PI.
Program Predictability
Crawl > There is no consistency at the team level around recording planned versus actual delivery and Program Predictability is not measured.
Walk > Most teams provide Team Performance Reports, but it requires working with each team to pull together the data. Program Predictability is calculated, but there are quality and consistency problems which make aggregating the Program Predictability measure difficult.
Run > All teams provide Team Performance reports and the quality of the data is good. The data is aggregated into a measure of Program Predictability on a regular cadence. It could be more effectively communicated and used to inform decisions.
Fly > Each team provides Team Performance Reports with planned versus actual delivery of business value and these reports are aggregated to calculate the Program Predictability Measure (percent of PI objectives achieved). The Program Predictability measure is understood, well communicated and is used to inform decisions.
Build Quality-In
Crawl > Engineering best practices are not used to build quality in. Quality reviews happen after development is done.
Walk > Engineering best practices are a work in progress; some teams are using them while others are not. Some practices have not yet been adopted.
Run > Many teams have adopted all of the engineering practices to build quality in and the rest of the teams have adopted some of the practices. We are seeing improvements in quality and collective ownership.
Fly > Every team on the ART builds quality in through consistently applying the following engineering practices: Test-First, Continuous Integration, Refactoring, Pair Work, and Collective Ownership.
Customer Satisfaction with Quality
Crawl > Customer satisfaction is very low due to quality issues.
Walk > Customers expect to find some issues and there is a fair amount of rework due to quality issues.
Run > The teams are more reliably meeting customer expectations and the quality of the products are improving. There are still occasions where we encounter quality issues and/or fail to meet customer expectations.
Fly > The delivery teams consistently produce high quality, reliable products and customer satisfaction is very high.
Recovery Time
Crawl > Recovery time is unknown.
Walk > Recovery time is measured anecdotally; there is not a full picture around how often the program has to roll back features from production.
Run > Recovery over time is measured to show how often the program has had to roll back features from Production (physically or by turning off toggle features). We are not effectively tracking trends and/or not consistently using this information to drive improvements.
Fly > We measure recovery time consistently and track trends to drive improvements in quality practices and deployment processes.
Time to Market
Crawl > The product owner/business customer is very dissatisfied with the wait time for new features to be released.
Walk > Releases are regular, but not as frequent as desired. Deployment and release frequency is known, but the data is not used to drive improvement.
Run > Release frequency is increasing and there is more satisfaction with the teams product owners and customers. Deployment and release frequency is measured to show if the program is making progress toward deploying and releasing more frequently.
Fly > The delivery teams can release software on demand and we have complete and accurate measures around deployment and release frequency.
Add any additional thoughts about "Results - Performance"
There is high visibility to the status, flow and results of features in each stage of development. Hypotheses are clearly defined, tested and measured.
Kanban Board
Crawl > The Program Kanban Board is not used at all.
Walk > The Program Kanban Board is used with limited effectiveness. Features move through the board without meeting all of the standards and the board is not always kept up to date or complete to provide full visibility.
Run > The Program Kanban board has been moderately effective in ensuring features are defined and analyzed prior to PI Planning and in providing visibility to the status of features being worked on during the PI.
Fly > The Program Kanban Board is used to ensure that features are reasoned and analyzed prior to reaching a PI boundary and prioritized appropriately, and that acceptance criteria have been established. There is high visibility to which features are being worked on and which features have been completed.
Cumulative Flow Diagram
Crawl > There are no measures or visibility around cumulative flow or the efficiency of the continuous delivery pipeline.
Walk > There is some visibility and tracking around the number of features in each stage of development. The data is not tracked in a Cumulative Flow Diagram.
Run > The number of features in each stage of development is tracked and we are using a Cumulative Flow Diagram. The information is not updated frequently enough and/or it is not used regularly to make adjustments and improvements.
Fly > The number of features in each stage of development is tracked in the Cumulative Flow Diagram. (For example: Funnel, Analyzing, Backlog, Implementing, Validating on Staging, Deployment to Production, Releasing and Done). Continuous delivery pipeline efficiency is measured by tracking average touch time and average wait time in days for each step of the pipeline. The data is used to regularly make adjustments and improve processes.
Idea Validation
Crawl > Ideas are rarely - if ever - validated using hypotheses.
Walk > Hypotheses are used to test ideas, but there is little measurement around the volume and failure rate.
Run > Hypotheses are used to test ideas and we have put some measurements in place. The measurements around the volume and failure rate could be tracked and applied more effectively to sharpen our focus.
Fly > The number of hypotheses tested is measured and reported. The volume and failure rate over time is used to determine how quickly ideas are being validated so the program can focus on the good ideas.
Business Outcomes for Minimum Viable Products (MVPs) & Minimum Marketable Features (MMFs)
Crawl > MMFs and MVPs are not often used to test features, and when they are, Accounting Leading Indicators are not defined and measured.
Walk > Business outcomes for MMFs and MVPs are sometimes defined and measured but there are significant gaps in the feedback loop between delivering the feature and determining whether the expected outcome was accomplished.
Run > Business outcomes for MMFs and MVPs are regularly defined and measured and most of the feedback loops are completed and tracked.
Fly > Through the use of Accounting Leading Indicators, business outcomes for MMFs and MVPs are consistently defined, measured and reported.
Add any additional thoughts about "Results - Visibility"
What are the top strengths of this ART leadership team? What can we count as success?
What are the top ART leadership team growth opportunities? What could be improved on?
What impediments has this team been experiencing?
