Given the rapid change in solution delivery over the last few years, the role of quality assurance (QA) has also evolved:
- The widespread adoption of collaborative methodologies like Agile necessitates changes in how QA is integrated into the delivery process.
- The maturity of automated delivery practices such as CI/CD pipelines has significantly changed how QA is conducted and enforced.
- Industry hype is leading organizations to invest in AI to automate QA. Adopting AI can involve significant organizational changes, but current systems, processes, and roles may not be ready or able to adopt them.
- Test requirements and scenarios are broader and more complex. Manual testing is unable to achieve the desired test coverage.
Our Advice
Critical Insight
- The perception of QA as a cost center can lead to the diversion of QA investments to other value-added capabilities. This decision may come from leadership, development teams, or other key players who prioritize value generation over cost savings.
- Significant focus is on the testing phase rather than the inclusion of QA practices throughout the solution delivery cycle.
- QA teams are unable to accommodate new and evolving security risks and technologies, aggressive performance standards, constantly changing priorities, and misunderstood quality policies.
- The marketplace for test automation and automated testing tools is crowded and difficult to navigate.
Impact and Result
- Standardize your definition of quality. Come to an organizational agreement of what attributes define a high-quality solution. Accommodate both business and IT perspectives in your definition.
- Clarify the role of QA throughout your solution delivery lifecycle. Indicate where and how QA is involved throughout solution delivery. Instill quality-first thinking in each stage of your pipeline to catch defects and issues early and motivate cross-functional collaboration.
- Adopt good QA practices to better support your quality definition and business and IT environments and priorities. Ensure your QA activities satisfy your criteria for a high-quality and successful solution with the right templates, technologies, and tactics in your toolbox.
Member Testimonials
After each Info-Tech experience, we ask our members to quantify the real-time savings, monetary impact, and project improvements our research helped them achieve. See our top member experiences for this blueprint and what our clients have to say.
9.6/10
Overall Impact
$49,332
Average $ Saved
19
Average Days Saved
Client
Experience
Impact
$ Saved
Days Saved
California Environmental Protection Agency
Guided Implementation
9/10
N/A
38
Alex is very knowledgeable and provide practical suggestions for how we can implement our QA program give our constraints. A minor issue, we had s... Read More
Louisiana Department of Health
Guided Implementation
10/10
N/A
N/A
I received some great advice that will be part of building up our team. The best part was that I actually felt like someone was listening to me. Ev... Read More
Effectus Ltd
Guided Implementation
9/10
N/A
5
Great walk through with real life examples. Was very interactive discussion which will help us immensely with our client. Kieran listened, pivot... Read More
Choctaw Nation Of Oklahoma
Workshop
8/10
$75,350
N/A
Keiren is a great instructor and did a wonderful job facilitating the topic. We don't really know how much time and money will be saved yet becaus... Read More
Kuvare US Holdings
Guided Implementation
10/10
$137K
38
There was NO worst. The best will always be insight and guidance while holding our hands and helping us avoid pitfalls and wastes of time.
Agriculture Financial Services Corporation
Guided Implementation
10/10
$10,000
5
This was our first call with Alex and really appreciate how prepared he was for the call. The feedback he provided on the deliverable was realisti... Read More
Heartland Co-op
Guided Implementation
10/10
$6,499
5
The best parts of my experience with Hans Eckman were: Clear customer focused conversation and topics that covered my needs and gave me more to t... Read More
Aipso
Guided Implementation
10/10
$32,499
50
They gave us good real-life examples. Good ideas - like rotating business users. Generally validated our thinking. Alex and Dawn are always gr... Read More
Cross Country Mortgage, Inc.
Guided Implementation
10/10
$34,649
16
This guided implementation was a great experience. Alex was able to help us adapt general software QA practices to our niche data team. We learne... Read More
New York City Housing Authority*
Guided Implementation
10/10
N/A
5
Hans was very informative and the follow up notes and materials very helpful. thank you!!
University of Texas - Arlington
Guided Implementation
10/10
N/A
5
The best part of the experience was the experience and knowledge the analyst brought to engagement. Also, we needed to change the direction of the... Read More
Shentel Management Company
Workshop
10/10
$44,099
20
Thinking and talking outside of the box... Conversations and discussions were all best parts of this workshop. The worst was that we weren't in... Read More
Oregon Public Utility Commission
Guided Implementation
9/10
$1,889
5
Colorado Housing And Finance Authority
Workshop
8/10
N/A
N/A
Scott and Allison were terrific and facilitated an quality workshop. The best parts of the class were : 1.) the breadth of information provided and... Read More
Westmoreland Mining LLC
Guided Implementation
9/10
$6,199
5
Omaha Public Power District
Guided Implementation
9/10
N/A
120
County of Clark Nevada
Workshop
7/10
N/A
N/A
Best parts of the workshop experience: Sharon and Scott engaged County Team members well. Collaborative and interactive question and answer sessi... Read More
Omaha Public Power District
Guided Implementation
8/10
$123K
120
Westmoreland Mining LLC
Guided Implementation
9/10
N/A
N/A
State of Ohio - Ohio Department of Developmental Disabilities
Guided Implementation
10/10
N/A
N/A
The best parts of our experience were the in depth discussions about best practices and the blue print for becoming a center of excellence. The ... Read More
J.R. Simplot Company
Guided Implementation
5/10
N/A
N/A
Forsyth Technical Community College
Guided Implementation
10/10
$123K
20
The best part of this experience was hearing best practices and getting tips on how to resolve some of the pain points we experience here. I can't... Read More
Saskatchewan Blue Cross
Guided Implementation
10/10
$5,000
3
BEST: Hearing Info-Tech's view on the QA process and applying a QA practice and strategy was very helpful WORST: n/a
Texas Children's Hospital
Guided Implementation
3/10
N/A
N/A
Toyota Canada Inc
Guided Implementation
9/10
N/A
2
University of Northern British Columbia
Guided Implementation
10/10
N/A
N/A
Infotech is very supportive and provided expert level advice.
Kenan Advantage Group
Guided Implementation
8/10
N/A
N/A
Workshop: Build a Software Quality Assurance Program
Workshops offer an easy way to accelerate your project. If you are unable to do the project yourself, and a Guided Implementation isn't enough, we offer low-cost delivery of our project workshops. We take you through every phase of your project and ensure that you have a roadmap in place to complete your project successfully.
Module 1: Assess Your QA Process
The Purpose
Reflect what QA means to your organization.
Key Benefits Achieved
Standardized definition of quality and list of metrics to track
Activities
Outputs
Define solution quality in your context.
- Solution quality definition
State your QA objectives and metrics.
- QA objectives
- Metrics to gauge QA success
Assess the current state of your QA practice.
- Understanding of the current state of your QA practice
Module 2: Align on Improved QA Practices
The Purpose
Define what would be your ideal QA.
Key Benefits Achieved
- Understand how can QA thinking can help your organization and what QA activities will be helped
- Defined roles and responsibilities
Activities
Outputs
Define your QA guiding principles.
- QA guiding principles
Define your QA target state.
- Target QA process and artifacts
- RACI chart of QA capabilities
- QA resource allocation approach and structure
Module 3: Build Your QA Toolbox
The Purpose
Build a solid set of testing practices.
Key Benefits Achieved
- Defined tolerance levels
- Defined tests, test data, and environment requirements
- List of desired tools
Activities
Outputs
Define your defect tolerance.
- Test defect risk tolerances
Define your tests.
- Test definitions
State your test data and environment requirements.
- Test data and environment management requirements
List your QA tools.
- List of QA solutions current available in your organization
- List of desired QA tools to be used
Module 4: Establish a QA Roadmap
The Purpose
A time-based plan that defines where your organization is, where you want to go, and how to get there.
Key Benefits Achieved
- Prioritization of QA initiatives that will help mitigate gaps
- How various roles will communication with each other and for each activity
Activities
Outputs
Report and communicate your QA activities.
- List of QA initiatives and roadmap
- Communication map
Build a Software Quality Assurance Program
Build a robust strategy and ensure quality is at the core of everything you do.
Analyst Perspective
Quality assurance (QA) has undergone a significant transformation over the years, evolving from a step in the development lifecycle to a pivotal, integrated process throughout the solution delivery cycle. This shift reflects a broader understanding that QA is not merely a gatekeeper of quality but a strategic partner in ensuring software excellence and reliability.
Today, the emphasis is on a proactive QA approach that aligns with Agile methodologies and CI/CD practices, fostering a culture where quality is everyone's responsibility. By leveraging advances in automation, AI, and machine learning (ML), modern QA practices emphasize collaboration, early defect detection, and preventive measures to ensure product robustness.
This perspective positions QA as a strategic investment, a revenue enabler, and crucial to the company's long-term value, rather than as a cost center. A modern QA strategy, therefore, must encapsulate a holistic view of quality, integrating it with business goals and leveraging automation to ensure your products consistently meet the highest standards of quality demanded in today's competitive market.
Bhavya Vora
Research Analyst,
Special Projects
Info-Tech Research Group
Andrew Kum-Seun
Research Director, Application Delivery & Management
Info-Tech Research Group
Executive Summary
Your ChallengeGiven the rapid change in solution delivery over the last few years, the role of quality assurance (QA) has also evolved. Latest disruptors helped evolve the role of QA:
|
Common Obstacles
|
Info-Tech’s Approach
|
Info-Tech Insight
QA is not a role but is a way of working; hiring QA roles can be a start to building a practice but it is not scalable. Automation can help but is limited by the roles who are developing and using it. QA must be delegated and ingrained in every aspect of work.
Insight Summary
Overarching Info-Tech Insight
QA is not a role but is a way of working; hiring QA roles can be a start to building a practice, but it is not scalable. Automation can help, but it is limited by the roles who are developing and using it. QA must be delegated and ingrained in every aspect of work.
QA performed early and throughout the solution delivery lifecycle (SDLC) improves the accuracy and effectiveness of downstream tests and reduces costs of fixing defects late in delivery. QA activities should be embedded in one’s job description.
QA is not a role. It is a mindset (way of working) that revolves around quality-first and proactive thinking. Good QA practices future-proofs solution investments by ensuring they are maintainable, scalable, and transferable.
QA is a shared responsibility. Your test plans and test cases are not static documents nor built in a single event. They are continuously updated and improved through feedback during the solution delivery process in collaboration with developers and other key stakeholders.
Start small to evaluate the fit and acceptance of new and modified roles, processes, and technologies. Aggressive initiatives and timelines can jeopardize success in big-bang deployments. Gradual and thoughtful adoption of QA ways of working helps your teams focus on practice fit rather than fighting the status quo. This approach must involve change tolerant teams, solutions, and cooperative stakeholders.
The value of QA stems from the assurance of sustainable and valuable solution delivery
What is quality and QA?
- Solution quality is the degree a system, feature, component, or process meets specified customer needs, customer expectations, and nonfunctional requirements. QA is a program of tasks and activities to ensure software quality priorities, standards and policies are met throughout the SDLC.
- Do not expect a universal definition of quality. Everyone will have a different understanding of what quality is and will structure people, processes, and technologies according to that interpretation.
What is the core of QA?
Verification
is evaluating work items to determine if they meet specified business and technical requirements.
Is the solution built right?
Validation
is evaluating the solution during and at the end of delivery to determine if they satisfy specified business and technical requirements.
Is the right solution built?
How is QA perceived in the organization?
- Efficient and effective QA practices are vital because solutions need to readily adjust to constantly evolving and changing business priorities and technologies without risking system stability and breaking business standards and expectations.
- However, investments in QA are often afterthoughts. QA is often viewed as a lower priority compared to other SDLC capabilities (e.g. design and coding) and is typically the first item cut when delivery timelines are under pressure.
Maximize the value you expect to gain from QA
Your QA program is not just about keeping pace with changes. QA is about setting a standard of software quality excellence aligned to stakeholder expectations and priorities while anticipating future challenges and opportunities.
Improved customer satisfaction
Solution issues are identified and addressed before they can negatively impact the customer. Preventative measures can then be implemented to maintain a consistent experience.
Enhanced security
QA enforces the right protocols and tactics are employed during the solution delivery. This standard is aligned to the organization’s security risk tolerance, latest trends, and industry regulations.
Business continuity
Good QA increases stakeholder confidence that the solution can reliably operate in sunny- day and rainy-day scenarios and meet defined service level agreements.
Increased resource utilization
QA practices streamline the SDLC process by reducing the time spent on fixing issues late in the lifecycle, ramping up resources unfamiliar with the solution, and paying down technical debt.
QA remains a challenge for many organizations
These challenges highlight critical gaps in our current approach, showcasing the necessity for a shift toward more integrated and automation-driven QA processes. This focus ensures that QA ultimately drives your competitive advantage.
Source: Katalon, 2023.
Increasing the QA budget does not guarantee success
Organizations are investing more into their QA practice.
Forty percent of large-scale companies are spending more than 25% of their budget on testing, with nearly 10% of enterprises spending more than 50% of their budget on testing (LambdaTest, 2023). This showcases how important quality is for all organizations.
But the reception and value of software products do not justify the money invested
78%: Average customer satisfaction score for the software industry.
Source: Fullview, 2023.
On-time, on-budget solutions do not indicate successful delivery practices.
90% of CIOs see at least some business frustration with IT’s failure to deliver value.
Source: CEO-CIO Alignment Diagnostics, Info-Tech Research Group, November 2022 to October 2023; N=115.
Design your QA approach
Stakeholders expect the speed and responsiveness of product delivery will not come at the expense of quality. A well-structured strategy streamlines delivery processes, upholds quality standards, and bolsters the solution delivery team’s reputation as a trusted partner.
Quality assurance is more than just software testing. Embrace a quality-first mindset that instills product quality accountability, fosters collaboration, and sets delivery expectations across all solution delivery lifecycle (SDLC) roles.
QA Strategy
Achieve a Quality-First Vision
- Establish clear targets to guide and measure effective QA.
- Create a unified definition of quality.
- Align to common quality standards.
Delegate QA Responsibilities
- Integrate QA early in the delivery cycle to identify critical issues sooner.
- Perform quality checks throughout the SDLC.
Instill Cross-Functional Accountability
- Include solution request accuracy and go-live sign-offs as stakeholder accountabilities.
- Empower SDLC teams to make local quality, low-risk decisions.
Build Your Toolbox
- Compile a comprehensive set of QA tactics, tools, methods, and standardized templates.
- Architect, rationalize, and monitor your QA tools and technologies.
Embrace industry good practices and leading-edge technology
- Adopt AI and automation capabilities in QA design, execution, and analysis.
- Incorporate iterative and collaboration practices (Agile, DevOps) into your QA practices.
Coach, Mentor & Support
- Guide SDLC teams and stakeholders with the priorities and tactics of the QA way of working.
- Facilitate knowledge sharing and collaboration across the different SDLC functions.
Purpose of QA Strategy
- QA Objectives and Metrics
- QA Standards and Definitions
- QA Roles and Responsibilities
- QA Communication and Reporting
- QA Process
- QA Tools and Technologies
Assess and strengthen your QA capabilities
Core Capabilities to Grow and Mature Your QA Practice
Vision and Buy-In
Clear direction and goals of the QA practice and ensuring the practice has the appropriate funding and stakeholder buy-in.
Execution Management
The tactics to manually and automatically execute the various QA activities and test types.
Planning
The creation of artifacts needed to define the scope of QA activities and for teams to confidently plan and commit to that scope.
Reporting and Communication
Preparing and delivering the outcomes of QA activities for the consumption of decision makers, SDLC teams, and dependent roles
Cross-Functional Collaboration
The collaboration among SDLC roles in QA activities and the involvement of QA in other SDLC activities.
Practice Management
Defined QA roles and responsibilities; processes, tools, and technology management; and tactics to support and improve the practice.
View the QA Current-State Assessment Tool
Your solution delivery lifecycle should embrace quality at its core
Connect all phases with a solution-centric approach that goes from the first idea all the way through to maintenance.
See our Evolve Your Software Development Lifecycle Into a Solution Delivery Lifecycle blueprint for more information
Explore the trends in the QA marketplace
-
AI and ML-Embedded QA Tools
Embedding AI and ML in QA tools increased the efficiency, scope, and accuracy of test design and execution, enabling them to:- Create test cases from functional requirements and test scripts without code.
- Analyze past testing activities to predict potential issues and defects of current delivery efforts and suggest root causes and solutions.
- Provision accurate and realistic synthetic test data using production data to train ML models.
-
Autonomous Testing
Autonomous testing involves tests and other QA activities to be created, executed, and managed through intelligent algorithms without the need for human intervention. This capability enables the- Configuration of QA activities to new requirements, testing scenarios and observations of solution delivery activities.
- Self-healing of test scenarios and scripts when issues occur.
- Immediate feedback during any phase of the SDLC of potential risks or conflicts with quality standards and industry frameworks and regulations.
-
Scriptless Automated Testing
Low- and no-code capabilities reduce or remove the technical skills traditionally needed to create, script, execute, and manage automated tests. This capability motivates the shifting of solution quality accountability earlier in the evolve-your-software-development-lifecycle-into-a-solution-delivery-lifecycle and enables the discovery of risks and defects before they cause negative impact and become expensive to fix. See Satisfy Digital End Users With Low- and No-Code for more information on low- and no-code. -
End-to-End Testing
The configuration and orchestration of automated tests to evaluate the functionality of the solution from start to finish under real user scenarios. End-to-end testing looks at:- Testing to ensure specific software layers or components work consistently and reliably across other parts of the software and system.
- Testing to ensure specific functions work smoothly across the technical stack (from the user interface down to the infrastructure).
-
QAOps
QAOps involves the embedding of QA procedures, automation, reporting, and technologies into the SDLC pipeline (BrowserStack, 2022). The goals are to guarantee high software quality consistently across teams and operationalizing QA-optimized, CI/CD processes for broader organizational adoption. QAOps shares many of the principles, behaviors, and best practices of DevOps and Agile methodologies. See Implement DevOps Practices That Work for more information. -
QA Tool Ecosystem
The team’s personal preferences for specific QA tools and technologies have been and are continually shifting away from the siloed, monolithic tooling and vendor stack that the industry standardized in the past. This demand pushed many vendors to position their solutions to build and strengthen relationships with third parties and deliver out-of-the-box plugins and customizable APIs. See Applications Priorities 2024 ass for more information on multisource ecosystems.
Extend the QA mindset beyond testing
Shift QA left and right
An emerging trend in QA is the adoption of shift-left and shift-right testing. Shift-left testing is a software testing approach that places strong emphasis on conducting testing activities earlier in the development process by shifting all testing activities to earlier development stages rather than leaving them until the very final stages (Katalon, 2023).
On the other hand, shift-right testing implies extending testing activities beyond the traditional development and release phases.
This involves performing testing activities in the production environment or closer to the end users after the software has been deployed.
QA involves testing across the SDLC
- Testing new requirements
- Testing new code
- Testing every build
- Testing every deployment
- Testing on production
Bridge your silos with DevOps
DevOps purposefully blurs the lines between these responsibilities, forcing collaboration. The developers start building the mindset of continually checking for errors in their code. The testers increase their responsibilities from validating the application to ensuring it is deployable at all times. They may even fix code as needed. All these pieces work together to ensure rapid delivery of features. The focus on the customer drives the work of the entire team.
Integrating QA into DevOps:
- Realign team structure
- Automate as much as possible
- Use metrics to track progress
- Run tests in parallel
- Have common set of processes & tools
- Continuous feedback
- Increasing viability
- Sufficient training
See Info-Tech’s Implement DevOps Practices That Work blueprint for more information.
Info-Tech’s methodology for building a software QA program
Phase Steps |
1. Assess Your QA Process1.1 List your QA objectives and metrics 1.2 Analyze your current QA state |
2. Align on Improved QA Practices2.1 Define your QA guiding principles 2.2 Define your foundational QA process |
3. Build Your QA Toolbox3.1 Define your defect tolerance 3.2 Align on your QA activities and tools |
4. Establish a QA Roadmap4.1 Build your QA roadmap |
Phase Outcomes |
|
|
|
|
Blueprint deliverables
Each step of this blueprint is accompanied by supporting deliverables to help you accomplish your goals:
QA Current-State Assessment Tool
Assess the current state of QA in your organization at team and organizational level.
Test Plan Template
Complete description of the plans and scope of tests.
Test Case Template
Specify and communicate the specific conditions and scenarios that need to be validated and verified.
Key deliverable:
QA Strategy Template
A template to help you document a comprehensive description of the QA practices for your organization. It presents several activities required to validate and verify software solutions.
Blueprint benefits
Demonstrate the value QA brings to your organization
- Level set your QA expectations with stakeholders to ensure they are achievable and aligned to their strategic goals.
- Pinpoint and optimize the QA capabilities inhibiting your team from delivering solutions your customers need.
- Measure the effectiveness of your QA practice in the language your stakeholders understand and empathize with.
- Clearly illustrate your QA vision and optimization plan through a QA strategy and good communication.
Notable Impacts
- Clear understanding of the funding needed for the QA practice.
- Executive buy-in for broader QA improvement initiatives spanning across business and IT functions.
- SDLC functions motivated to collaborate, learn from each other, and adopt a quality-first mindset.
Consistent delivery of quality across your solution portfolio and teams
- Consolidate different and siloed perspectives and definitions of quality into a single interpretation and statement to rally behind.
- Leverage a common toolbox of QA tools, tactics, and templates that were optimized to meet your quality policies and standards.
- Embed and delegate QA tasks and activities throughout your SDLC roles and processes and empower team-level decision making where possible.
- Encourage the applying QA practices to the artifacts supporting the SDLC (verification) alongside the solution being delivered (validation).
Notable Impacts
- Organizational accountability of solution quality before, during, and after the solution delivery process.
- Common interpretation of quality attributes to ensure consistent compliance with your organization’s, industry’s, and regulator's policies and standards.
- Greater customer and stakeholder trust in the solution delivery team in delivering high-quality solution.
Measure the value of this blueprint
Outcome |
Project Metrics |
Impact |
Improved software quality by reducing the number of defects |
Select and Use SDLC Metrics Effectively |
|
Increased solution delivery throughput |
Select and Use SDLC Metrics Effectively |
|
Reduction of rework due to defects found during the solution delivery process |
Select and Use SDLC Metrics Effectively |
|
Increased application and end-user satisfaction |
End User Satisfaction Diagnostic |
|
Increased IT satisfaction |
CIO Business Vision Diagnostic |
|
Increased resource effectiveness |
Fully loaded employee costs incurred per year to deliver a given solution |
|
30% improvement by second year |
Assess the Value Drivers Within Your Solutions |
|
Executive Brief Case Study
INDUSTRY: Government
SOURCE: Info-Tech Research Group Workshop
Government Agency
A government agency worked with Info-Tech to develop a strategy to mature and scale their QA practice. The QA team identified several key QA objectives that they want to achieve through their practice:
- Ensure software products meet business, functional, and nonfunctional (including security, performance, integration, and regression) requirements.
- Build a disciplined and formal QA practice.
- Increase customer and stakeholder confidence, trust, and respect.
However, they recognized key challenges standing in their way, such as:
- Low QA resource capacity.
- Low availability of business subject-matter experts.
- Lack of automated testing and test automation tools.
- Very tight project timelines resulting in the cutting of QA activities.
See our sample workshop deliverable to know how an Info-Tech Quality Assurance Workshop helps improve your QA practice.
Results
By conducting the workshop, the organization was able to:
- Build a consensus of what QA means and list the necessary changes to be successful.
- Gauge the maturity and capability of the current QA practice to define a list of optimization initiatives and build a roadmap.
- Create the initial design of target QA roles and processes of the QA practice.
- Finalize the future state of QA roles, processes, tools, and tactics, including their implementation to other products and systems, with the existing test strategy.
Info-Tech offers various levels of support to best suit your needs
DIY Toolkit
“Our team has already made this critical project a priority, and we have the time and capability, but some guidance along the way would be helpful.”
Guided Implementation
“Our team knows that we need to fix a process, but we need assistance to determine where to focus. Some check-ins along the way would help keep us on track.”
Workshop
“We need to hit the ground running and get this project kicked off immediately. Our team has the ability to take this over once we get a framework and strategy in place.
Executive & Technical Counseling
“Our team and processes are maturing; however, to expedite the journey we’ll need a seasoned practitioner to coach and validate approaches, deliverables, and opportunities.”
Consulting
“Our team does not have the time or the knowledge to take this project on. We need assistance through the entirety of this project.”
Diagnostics and consistent frameworks are used throughout all five options.
Guided Implementation
What does a typical GI on this topic look like?
Phase 1
|
Phase 2
|
Phase 3
|
Phase 4
|
A Guided Implementation (GI) is a series of calls with an Info-Tech analyst to help implement our best practices in your organization.
A typical GI is 8 to 12 calls over the course of 4 to 6 months.