DEVOPSdigest

GenAI technologies continue to create rapid transformation across industries. My organization, Applause, surveyed more than 4,400 independent software developers, QA professionals and consumers globally for our third annual State of Digital Quality in AI report to explore the latest use cases, tools, challenges and user experiences with GenAI. The findings highlight that while investment and use of AI continue to climb, the adoption of essential quality assurance (QA) and testing practices for AI is not keeping pace.

Software Development and QA with AI

Leveraging AI throughout the software development lifecycle (SDLC) is bringing businesses a competitive advantage. Still, some organizations are slow to adopt this approach, and many are not altering testing practices to account for AI:

■ More than half of software professionals surveyed said GenAI tools improve productivity significantly, with 25% saying it brings a boost of 25-49%, and 27% saying it boosts productivity by 50-74%.

■ 23% of software professionals say their integrated development environment (IDE) lacks GenAI tools (such as GitHub Copilot) and 16% don't know if they have AI tools embedded in their IDE.

■ Red teaming, a best practice for mitigating the risks of bias, toxicity, and inaccuracy in AI, is only used by 33% of respondents.

■ The top AI testing practices involving people include prompt and response grading (61%), UX testing (57%), and accessibility testing (54%).

■ 41% of developers and QA professionals said they lean on domain experts for AI training.

While the productivity benefits that come from using AI are clear for the majority of software professionals, the best practices for testing AI are often not being incorporated in tandem.

AI Investment for Customer Experience

Organizations are investing in AI to improve customer experiences and reduce operational costs. Despite this, flaws and bugs are still reaching consumers:

■ More than 70% of developers and QA professionals said their organization is developing AI features and applications. Chatbots and customer support tools are the most popular AI applications being built at 55%, and 19% have started to build AI agents.

■ 65% of respondents reported issues using GenAI in the past three months. The top issues were that the AI responses:
- Lacked detail (40%)
- Misunderstood prompts (38%)
- Showed bias (35%)
- Showed hallucinations (32%)
- Were clearly incorrect (23%)
- Included offensive content (17%)

■ Only 20% of respondents said the GenAI tools they use understand their questions and deliver helpful responses every time.

While organizations are investing in AI with the goal of improving customer experiences, that goal is not always being met. Without incorporating proper testing techniques like red teaming and training models with diverse datasets, AI continues to yield flawed and inaccurate results. There is clearly plenty of room for improvement.

Additional AI Findings

Some additional interesting findings from this year's survey include:

■ The favorite AI tools from our 2024 survey are still the favorites in 2025, with 37% of respondents preferring GitHub Copilot and 34% preferring OpenAI Codex.

■ 78% of users want their AI tools to have multimodal functionality, or the ability to interpret multiple types of media — an increase of 16% from last year.

The results of this year's survey bring to light the disconnect between building and adopting AI applications and testing them. With so much investment and emphasis on leveraging AI to improve operational efficiency and enhance customer experience, it is critical to incorporate end-to-end testing best practices too. On top of the rise in investment and adoption, agentic AI is contributing to the technology's rapid evolution. Organizations that don't account and budget for adequate AI testing put their AI investments and brands at risk.