Concept
Objectively measuring the agile maturity of a potential vendor during the Request for Proposal (RFP) process is a sophisticated endeavor. It requires moving beyond superficial presentations and self-assessments to a more rigorous, evidence-based evaluation. A vendor’s true agile maturity is a reflection of their organizational culture, engineering discipline, and ability to deliver value predictably and consistently.
This evaluation is a critical component of risk management and a key predictor of a successful partnership. A vendor with a high degree of agile maturity will be more responsive to change, more transparent in their operations, and more likely to deliver a high-quality product that meets your evolving needs.
The core of the challenge lies in quantifying a set of practices and a mindset that are often qualitative. The objective is to translate the principles of the Agile Manifesto into a set of measurable indicators. This is accomplished by establishing a clear framework for assessment, one that is rooted in observable behaviors and verifiable data.
This framework should be applied consistently to all potential vendors to ensure a fair and accurate comparison. The insights gained from this assessment will empower you to make a more informed decision, one that is based on a deep understanding of a vendor’s capabilities and their potential to become a true strategic partner.
Strategy
A strategic approach to assessing a vendor’s agile maturity involves a multi-faceted evaluation that examines their practices from various angles. This approach should be integrated directly into the RFP process, with specific questions and requirements designed to elicit the necessary information. The goal is to gather both qualitative and quantitative data that, when synthesized, provides a holistic view of the vendor’s agile capabilities. This process can be broken down into several key areas of focus, each with its own set of metrics and evaluation criteria.
A Pillar-Based Assessment Framework
A robust strategy for evaluating agile maturity is to use a pillar-based framework. This approach, similar to models used by industry leaders, breaks down the complex concept of “agility” into a set of more manageable and measurable components. Each pillar represents a critical aspect of agile practice, and by assessing a vendor’s maturity in each area, you can build a comprehensive profile of their capabilities. This framework provides a structured and consistent way to evaluate and compare different vendors.
The following table outlines a ten-pillar framework for assessing agile maturity, along with a description of each pillar and the key areas to evaluate:
| Agile Pillar | Description | Key Evaluation Areas |
|---|---|---|
| Culture and Mindset | The extent to which the vendor’s organizational culture embraces agile values and principles, from leadership to individual team members. | Leadership commitment to agile, evidence of a learning culture, and the prevalence of an agile mindset throughout the organization. |
| Agile Teams | The structure and composition of the vendor’s development teams, and their ability to collaborate effectively. | The use of cross-functional teams, the level of team autonomy, and the effectiveness of collaboration and communication practices. |
| Backlog Management | The vendor’s approach to managing and prioritizing the product backlog, which is the single source of requirements for the development team. | The quality and health of the product backlog, the process for backlog refinement, and the ability to effectively manage changing priorities. |
| Planning & Commitment | The vendor’s ability to effectively plan and commit to work in a way that is consistent with agile principles. | The use of agile ceremonies like sprint planning, the team’s ability to make and meet commitments, and the accuracy of their estimations. |
| Agile Practices | The specific agile practices and techniques that the vendor employs in their day-to-day work. | The use of practices like user stories, story points, and continuous integration, as well as the vendor’s overall agile governance process. |
| Engineering Maturity | The technical excellence and discipline of the vendor’s engineering teams. | The adoption of practices like test-driven development (TDD), pair programming, and a focus on building quality in from the start. |
| Focus on Automation | The extent to which the vendor leverages automation to improve efficiency, quality, and speed. | The use of automated testing, continuous integration and continuous delivery (CI/CD) pipelines, and other automation tools and practices. |
| Delivery Maturity | The vendor’s ability to consistently deliver high-quality, working software to their customers. | The frequency and reliability of their delivery cadence, their ability to deliver value in small increments, and their process for gathering and incorporating feedback. |
| Continuous Improvement | The vendor’s commitment to continuously learning and improving their processes and practices. | The use of retrospectives and other feedback mechanisms, the ability to identify and act on opportunities for improvement, and a focus on customer satisfaction. |
| Inspect and Adapt | The vendor’s ability to regularly inspect their progress and adapt their plans based on what they have learned. | The use of feedback loops at all levels of the organization, the ability to respond to change quickly and effectively, and a culture of transparency and openness. |
A pillar-based framework provides a structured and consistent way to evaluate and compare different vendors’ agile maturity.
Incorporating Agile Assessment into the RFP
To effectively assess a vendor’s agile maturity, you need to embed specific questions and requirements into your RFP. These should be designed to elicit detailed and evidence-based responses that go beyond simple affirmations of “being agile.” The goal is to gather concrete examples and data that you can use to evaluate the vendor against your chosen framework.
Here are some examples of how you can incorporate agile assessment into your RFP:
- Ask for specific examples ▴ Instead of asking “Are you agile?”, ask “Describe your process for managing changing requirements.” or “Provide an example of a time when your team had to adapt to a significant change in project scope.”
- Request access to tools and artifacts ▴ Ask to see the vendor’s product backlog, their sprint boards, and their CI/CD pipeline. This will give you a firsthand look at their actual practices.
- Conduct interviews with team members ▴ In addition to speaking with management, request interviews with developers, testers, and product owners. This will give you a more complete picture of the vendor’s agile culture.
- Include a practical exercise ▴ Consider including a small, practical exercise in the RFP process, such as asking the vendor to create a product backlog for a sample project. This will give you a sense of their capabilities in a real-world scenario.
Execution
The execution of an agile maturity assessment requires a systematic and data-driven approach. It involves gathering information through the RFP process, analyzing that information against a defined set of criteria, and then using the results to inform your vendor selection decision. This process should be as objective as possible, relying on verifiable data and observable behaviors rather than subjective impressions.
A Quantitative Scoring Model
To bring a higher degree of objectivity to your assessment, you can use a quantitative scoring model. This involves assigning a numerical score to each of the pillars in your assessment framework, based on the evidence you have gathered. This will allow you to create a “maturity score” for each vendor, which can be used to compare them in a more systematic way.
The following table provides an example of a scoring model that you can adapt for your own use. It includes a description of each maturity level and the corresponding score.
| Maturity Level | Score | Description |
|---|---|---|
| Expert | 5 | The vendor consistently excels in this area, and it is a core part of their culture. They are a leader in this practice and can demonstrate a long history of success. |
| Fully Implemented | 4 | The vendor has fully implemented this practice and it is a mandatory part of their process. It is consistently applied and audited, and they use metrics to track their performance. |
| Largely Implemented | 3 | The vendor has largely implemented this practice, and it is applied most of the time. It is process-driven, but may require some ongoing maintenance and support. |
| Minimally Implemented | 2 | The vendor has minimally implemented this practice. It is applied sometimes, but it is not a consistent or audited part of their process. It is often people-driven rather than process-driven. |
| Not Implemented | 1 | The vendor has not implemented this practice. It is not a part of their process and needs to be introduced. |
A quantitative scoring model can help to bring a higher degree of objectivity to your agile maturity assessment.
Key Metrics for Agile Maturity
In addition to the qualitative assessment of a vendor’s practices, it is also important to gather quantitative data on their performance. There are a number of key metrics that can be used to measure a vendor’s agile maturity. These metrics should be requested as part of the RFP process, and they should be used to validate the vendor’s claims about their agile capabilities.
Here is a list of key metrics for agile maturity, along with a description of each:
- Cycle Time ▴ The time it takes to complete a task from the moment work begins until it is delivered. A shorter cycle time is generally an indicator of a more efficient and mature agile process.
- Lead Time ▴ The time it takes from the moment a request is made until it is delivered. Lead time provides insight into the vendor’s overall responsiveness and their ability to deliver value to their customers.
- Velocity ▴ The amount of work a team can complete in a single sprint. Velocity is a measure of a team’s capacity and can be used to forecast future delivery.
- Defect Density ▴ The number of defects found per unit of work (e.g. per user story or per feature). A lower defect density is an indicator of higher quality and a more mature engineering process.
- Customer Satisfaction (CSAT) ▴ A measure of how satisfied customers are with the product or service they are receiving. CSAT can be measured through surveys and other feedback mechanisms.
- Team Morale/Health ▴ The overall health and morale of the development team. A happy and motivated team is more likely to be productive and to deliver high-quality work. Team morale can be assessed through anonymous surveys and other feedback channels.
References
- Ambler, S. W. (2013). Agile For Dummies. John Wiley & Sons.
- Cigniti Technologies. (2023). Agile Maturity Model ▴ How To Measure Agile Maturity. Cigniti.
- Collins, J. (2001). Good to Great ▴ Why Some Companies Make the Leap. and Others Don’t. HarperBusiness.
- Lapham, M. A. et al. (2010). Agile Methods ▴ A Good Fit for the Acquisition of Software-Reliant Systems? Carnegie Mellon University.
- Lapham, M. A. et al. (2011). RFP Patterns and Techniques for Successful Agile Contracting. Carnegie Mellon University.
- Ozcan-Top, O. & Demirors, O. (2013). A Review of Agile Maturity Models. In Software Process Improvement and Capability Determination (pp. 13-24). Springer.
- The 15th State of Agile Report. (2021). Digital.ai.
Reflection
Ultimately, the objective measurement of agile maturity is not just about selecting a vendor. It is about establishing a foundation for a successful and collaborative partnership. The process of assessing a vendor’s agile maturity should be a catalyst for a deeper conversation about how you will work together to achieve your shared goals. The insights you gain from this process will not only help you to make a more informed decision, but they will also provide you with a roadmap for how to best engage with your chosen partner to ensure a successful outcome.
Glossary
Agile Maturity
Rfp Process
Product Backlog
Quantitative Scoring Model
Scoring Model
Cycle Time
Lead Time
Velocity
Defect Density
