Build a Software Quality Assurance Program

Build a robust strategy and ensure quality is at the core of everything you do.

Analyst Perspective

Quality assurance (QA) has undergone a significant transformation over the years, evolving from a step in the development lifecycle to a pivotal, integrated process throughout the solution delivery cycle. This shift reflects a broader understanding that QA is not merely a gatekeeper of quality but a strategic partner in ensuring software excellence and reliability.

Today, the emphasis is on a proactive QA approach that aligns with Agile methodologies and CI/CD practices, fostering a culture where quality is everyone's responsibility. By leveraging advances in automation, AI, and machine learning (ML), modern QA practices emphasize collaboration, early defect detection, and preventive measures to ensure product robustness.

This perspective positions QA as a strategic investment, a revenue enabler, and crucial to the company's long-term value, rather than as a cost center. A modern QA strategy, therefore, must encapsulate a holistic view of quality, integrating it with business goals and leveraging automation to ensure your products consistently meet the highest standards of quality demanded in today's competitive market.

Bhavya Vora

Research Analyst,
Special Projects
Info-Tech Research Group

Andrew Kum-Seun

Research Director, Application Delivery & Management
Info-Tech Research Group

Executive Summary

Info-Tech Insight

QA is not a role but is a way of working; hiring QA roles can be a start to building a practice but it is not scalable. Automation can help but is limited by the roles who are developing and using it. QA must be delegated and ingrained in every aspect of work.

Insight Summary

Overarching Info-Tech Insight

QA is not a role but is a way of working; hiring QA roles can be a start to building a practice, but it is not scalable. Automation can help, but it is limited by the roles who are developing and using it. QA must be delegated and ingrained in every aspect of work.

QA performed early and throughout the solution delivery lifecycle (SDLC) improves the accuracy and effectiveness of downstream tests and reduces costs of fixing defects late in delivery. QA activities should be embedded in one’s job description.

QA is not a role. It is a mindset (way of working) that revolves around quality-first and proactive thinking. Good QA practices future-proofs solution investments by ensuring they are maintainable, scalable, and transferable.

QA is a shared responsibility. Your test plans and test cases are not static documents nor built in a single event. They are continuously updated and improved through feedback during the solution delivery process in collaboration with developers and other key stakeholders.

Start small to evaluate the fit and acceptance of new and modified roles, processes, and technologies. Aggressive initiatives and timelines can jeopardize success in big-bang deployments. Gradual and thoughtful adoption of QA ways of working helps your teams focus on practice fit rather than fighting the status quo. This approach must involve change tolerant teams, solutions, and cooperative stakeholders.

The value of QA stems from the assurance of sustainable and valuable solution delivery

What is quality and QA?

• Solution quality is the degree a system, feature, component, or process meets specified customer needs, customer expectations, and nonfunctional requirements. QA is a program of tasks and activities to ensure software quality priorities, standards and policies are met throughout the SDLC.
• Do not expect a universal definition of quality. Everyone will have a different understanding of what quality is and will structure people, processes, and technologies according to that interpretation.

What is the core of QA?

Verification

is evaluating work items to determine if they meet specified business and technical requirements.

Is the solution built right?

Validation

is evaluating the solution during and at the end of delivery to determine if they satisfy specified business and technical requirements.

Is the right solution built?

How is QA perceived in the organization?

• Efficient and effective QA practices are vital because solutions need to readily adjust to constantly evolving and changing business priorities and technologies without risking system stability and breaking business standards and expectations.
• However, investments in QA are often afterthoughts. QA is often viewed as a lower priority compared to other SDLC capabilities (e.g. design and coding) and is typically the first item cut when delivery timelines are under pressure.

Maximize the value you expect to gain from QA

Your QA program is not just about keeping pace with changes. QA is about setting a standard of software quality excellence aligned to stakeholder expectations and priorities while anticipating future challenges and opportunities.

Improved customer satisfaction

Solution issues are identified and addressed before they can negatively impact the customer. Preventative measures can then be implemented to maintain a consistent experience.

Enhanced security

QA enforces the right protocols and tactics are employed during the solution delivery. This standard is aligned to the organization’s security risk tolerance, latest trends, and industry regulations.

Business continuity

Good QA increases stakeholder confidence that the solution can reliably operate in sunny- day and rainy-day scenarios and meet defined service level agreements.

Increased resource utilization

QA practices streamline the SDLC process by reducing the time spent on fixing issues late in the lifecycle, ramping up resources unfamiliar with the solution, and paying down technical debt.

QA remains a challenge for many organizations

These challenges highlight critical gaps in our current approach, showcasing the necessity for a shift toward more integrated and automation-driven QA processes. This focus ensures that QA ultimately drives your competitive advantage.

Source: Katalon, 2023.

Increasing the QA budget does not guarantee success

Organizations are investing more into their QA practice.

Forty percent of large-scale companies are spending more than 25% of their budget on testing, with nearly 10% of enterprises spending more than 50% of their budget on testing (LambdaTest, 2023). This showcases how important quality is for all organizations.

But the reception and value of software products do not justify the money invested

78%: Average customer satisfaction score for the software industry.

Source: Fullview, 2023.

On-time, on-budget solutions do not indicate successful delivery practices.

90% of CIOs see at least some business frustration with IT’s failure to deliver value.

Source: CEO-CIO Alignment Diagnostics, Info-Tech Research Group, November 2022 to October 2023; N=115.

Design your QA approach

Stakeholders expect the speed and responsiveness of product delivery will not come at the expense of quality. A well-structured strategy streamlines delivery processes, upholds quality standards, and bolsters the solution delivery team’s reputation as a trusted partner.

Quality assurance is more than just software testing. Embrace a quality-first mindset that instills product quality accountability, fosters collaboration, and sets delivery expectations across all solution delivery lifecycle (SDLC) roles.

QA Strategy

Achieve a Quality-First Vision

• Establish clear targets to guide and measure effective QA.
• Create a unified definition of quality.
• Align to common quality standards.

Delegate QA Responsibilities

• Integrate QA early in the delivery cycle to identify critical issues sooner.
• Perform quality checks throughout the SDLC.

Instill Cross-Functional Accountability

• Include solution request accuracy and go-live sign-offs as stakeholder accountabilities.
• Empower SDLC teams to make local quality, low-risk decisions.

Build Your Toolbox

• Compile a comprehensive set of QA tactics, tools, methods, and standardized templates.
• Architect, rationalize, and monitor your QA tools and technologies.

Embrace industry good practices and leading-edge technology

• Adopt AI and automation capabilities in QA design, execution, and analysis.
• Incorporate iterative and collaboration practices (Agile, DevOps) into your QA practices.

Coach, Mentor & Support

• Guide SDLC teams and stakeholders with the priorities and tactics of the QA way of working.
• Facilitate knowledge sharing and collaboration across the different SDLC functions.

Purpose of QA Strategy

• QA Objectives and Metrics
• QA Standards and Definitions
• QA Roles and Responsibilities
• QA Communication and Reporting
• QA Process
• QA Tools and Technologies

Assess and strengthen your QA capabilities

Core Capabilities to Grow and Mature Your QA Practice

Vision and Buy-In

Clear direction and goals of the QA practice and ensuring the practice has the appropriate funding and stakeholder buy-in.

Execution Management

The tactics to manually and automatically execute the various QA activities and test types.

Planning

The creation of artifacts needed to define the scope of QA activities and for teams to confidently plan and commit to that scope.

Reporting and Communication

Preparing and delivering the outcomes of QA activities for the consumption of decision makers, SDLC teams, and dependent roles

Cross-Functional Collaboration

The collaboration among SDLC roles in QA activities and the involvement of QA in other SDLC activities.

Practice Management

Defined QA roles and responsibilities; processes, tools, and technology management; and tactics to support and improve the practice.

View the QA Current-State Assessment Tool

Your solution delivery lifecycle should embrace quality at its core

Connect all phases with a solution-centric approach that goes from the first idea all the way through to maintenance.

See our Evolve Your Software Development Lifecycle Into a Solution Delivery Lifecycle blueprint for more information

Explore the trends in the QA marketplace

1. AI and ML-Embedded QA Tools

   Embedding AI and ML in QA tools increased the efficiency, scope, and accuracy of test design and execution, enabling them to:
   • Create test cases from functional requirements and test scripts without code.
   • Analyze past testing activities to predict potential issues and defects of current delivery efforts and suggest root causes and solutions.
   • Provision accurate and realistic synthetic test data using production data to train ML models.
   See Leverage Gen AI to Improve Your Test Automation Strategy and Adopt Generative AI in Solution Delivery for more information.

2. Autonomous Testing

   Autonomous testing involves tests and other QA activities to be created, executed, and managed through intelligent algorithms without the need for human intervention. This capability enables the
   • Configuration of QA activities to new requirements, testing scenarios and observations of solution delivery activities.
   • Self-healing of test scenarios and scripts when issues occur.
   • Immediate feedback during any phase of the SDLC of potential risks or conflicts with quality standards and industry frameworks and regulations.
   See Tech Trends 2024 for more information on autonomous back office.

3. Scriptless Automated Testing

   Low- and no-code capabilities reduce or remove the technical skills traditionally needed to create, script, execute, and manage automated tests. This capability motivates the shifting of solution quality accountability earlier in the evolve-your-software-development-lifecycle-into-a-solution-delivery-lifecycle and enables the discovery of risks and defects before they cause negative impact and become expensive to fix. See Satisfy Digital End Users With Low- and No-Code for more information on low- and no-code.

4. End-to-End Testing

   The configuration and orchestration of automated tests to evaluate the functionality of the solution from start to finish under real user scenarios. End-to-end testing looks at:
   • Testing to ensure specific software layers or components work consistently and reliably across other parts of the software and system.
   • Testing to ensure specific functions work smoothly across the technical stack (from the user interface down to the infrastructure).
   This approach can involve the use of automated testing solutions alongside nontraditional testing solutions like robotic process automation. See Build a Winning Business Process Automation Playbook and Enhance Your Solution Architecture Practices to identify the impacts of your changes to your systems and users.

5. QAOps

   QAOps involves the embedding of QA procedures, automation, reporting, and technologies into the SDLC pipeline (BrowserStack, 2022). The goals are to guarantee high software quality consistently across teams and operationalizing QA-optimized, CI/CD processes for broader organizational adoption. QAOps shares many of the principles, behaviors, and best practices of DevOps and Agile methodologies. See Implement DevOps Practices That Work for more information.

6. QA Tool Ecosystem

   The team’s personal preferences for specific QA tools and technologies have been and are continually shifting away from the siloed, monolithic tooling and vendor stack that the industry standardized in the past. This demand pushed many vendors to position their solutions to build and strengthen relationships with third parties and deliver out-of-the-box plugins and customizable APIs. See Applications Priorities 2024 ass for more information on multisource ecosystems.

Extend the QA mindset beyond testing

Shift QA left and right

An emerging trend in QA is the adoption of shift-left and shift-right testing. Shift-left testing is a software testing approach that places strong emphasis on conducting testing activities earlier in the development process by shifting all testing activities to earlier development stages rather than leaving them until the very final stages (Katalon, 2023).

On the other hand, shift-right testing implies extending testing activities beyond the traditional development and release phases.

This involves performing testing activities in the production environment or closer to the end users after the software has been deployed.

QA involves testing across the SDLC

• Testing new requirements
• Testing new code
• Testing every build
• Testing every deployment
• Testing on production

Bridge your silos with DevOps

DevOps purposefully blurs the lines between these responsibilities, forcing collaboration. The developers start building the mindset of continually checking for errors in their code. The testers increase their responsibilities from validating the application to ensuring it is deployable at all times. They may even fix code as needed. All these pieces work together to ensure rapid delivery of features. The focus on the customer drives the work of the entire team.

Integrating QA into DevOps:

• Realign team structure
• Automate as much as possible
• Use metrics to track progress
• Run tests in parallel
• Have common set of processes & tools
• Continuous feedback
• Increasing viability
• Sufficient training

See Info-Tech’s Implement DevOps Practices That Work blueprint for more information.

Info-Tech’s methodology for building a software QA program

Blueprint deliverables

Each step of this blueprint is accompanied by supporting deliverables to help you accomplish your goals:

QA Current-State Assessment Tool

Assess the current state of QA in your organization at team and organizational level.

Test Plan Template

Complete description of the plans and scope of tests.

Test Case Template

Specify and communicate the specific conditions and scenarios that need to be validated and verified.

Key deliverable:

QA Strategy Template

A template to help you document a comprehensive description of the QA practices for your organization. It presents several activities required to validate and verify software solutions.

Blueprint benefits

Demonstrate the value QA brings to your organization

• Level set your QA expectations with stakeholders to ensure they are achievable and aligned to their strategic goals.
• Pinpoint and optimize the QA capabilities inhibiting your team from delivering solutions your customers need.
• Measure the effectiveness of your QA practice in the language your stakeholders understand and empathize with.
• Clearly illustrate your QA vision and optimization plan through a QA strategy and good communication.

Notable Impacts

• Clear understanding of the funding needed for the QA practice.
• Executive buy-in for broader QA improvement initiatives spanning across business and IT functions.
• SDLC functions motivated to collaborate, learn from each other, and adopt a quality-first mindset.

Consistent delivery of quality across your solution portfolio and teams

• Consolidate different and siloed perspectives and definitions of quality into a single interpretation and statement to rally behind.
• Leverage a common toolbox of QA tools, tactics, and templates that were optimized to meet your quality policies and standards.
• Embed and delegate QA tasks and activities throughout your SDLC roles and processes and empower team-level decision making where possible.
• Encourage the applying QA practices to the artifacts supporting the SDLC (verification) alongside the solution being delivered (validation).

Notable Impacts

• Organizational accountability of solution quality before, during, and after the solution delivery process.
• Common interpretation of quality attributes to ensure consistent compliance with your organization’s, industry’s, and regulator's policies and standards.
• Greater customer and stakeholder trust in the solution delivery team in delivering high-quality solution.

Measure the value of this blueprint

Executive Brief Case Study

INDUSTRY: Government
SOURCE: Info-Tech Research Group Workshop

Government Agency

A government agency worked with Info-Tech to develop a strategy to mature and scale their QA practice. The QA team identified several key QA objectives that they want to achieve through their practice:

• Ensure software products meet business, functional, and nonfunctional (including security, performance, integration, and regression) requirements.
• Build a disciplined and formal QA practice.
• Increase customer and stakeholder confidence, trust, and respect.

However, they recognized key challenges standing in their way, such as:

• Low QA resource capacity.
• Low availability of business subject-matter experts.
• Lack of automated testing and test automation tools.
• Very tight project timelines resulting in the cutting of QA activities.

See our sample workshop deliverable to know how an Info-Tech Quality Assurance Workshop helps improve your QA practice.

Results

By conducting the workshop, the organization was able to:

• Build a consensus of what QA means and list the necessary changes to be successful.
• Gauge the maturity and capability of the current QA practice to define a list of optimization initiatives and build a roadmap.
• Create the initial design of target QA roles and processes of the QA practice.
• Finalize the future state of QA roles, processes, tools, and tactics, including their implementation to other products and systems, with the existing test strategy.

Info-Tech offers various levels of support to best suit your needs

DIY Toolkit

“Our team has already made this critical project a priority, and we have the time and capability, but some guidance along the way would be helpful.”

Guided Implementation

“Our team knows that we need to fix a process, but we need assistance to determine where to focus. Some check-ins along the way would help keep us on track.”

Workshop

“We need to hit the ground running and get this project kicked off immediately. Our team has the ability to take this over once we get a framework and strategy in place.

Executive & Technical Counseling

“Our team and processes are maturing; however, to expedite the journey we’ll need a seasoned practitioner to coach and validate approaches, deliverables, and opportunities.”

Consulting

“Our team does not have the time or the knowledge to take this project on. We need assistance through the entirety of this project.”

Diagnostics and consistent frameworks are used throughout all five options.

Guided Implementation

What does a typical GI on this topic look like?

A Guided Implementation (GI) is a series of calls with an Info-Tech analyst to help implement our best practices in your organization.

A typical GI is 8 to 12 calls over the course of 4 to 6 months.