DEVOPSdigest

The Vellum 2025 State of AI Development report revealed key insights about the state of AI development heading into 2025. With insights from more than 1,250 developers, the results show how more companies can put their AI applications into production as models and tooling mature. The report sheds light on the progress and persistent challenges as organizations work to move AI from experimentation to production.

The Road to AI Production: What Is Blocking Applications?

The survey results paint a mixed picture of AI development progress. Only about a quarter (25.1%) of businesses had AI applications deployed in production. Other respondents shared they were stuck in pre-building (25%), proof-of-concept (21%), beta testing with users (14.1%) and talking to users and gathering requirements (7.9%).

Companies may have trouble crossing the finish line due to not having a good use case available or a lack of tooling expertise. This lack of technical knowledge can be holding back some companies from getting AI applications into production. Developers shared that the top challenge they were facing is managing AI hallucinations and prompts (57.4%).

A Team Effort

Developing with AI represents a new paradigm, where multiple parts of an organization are working together to build an AI application. Unlike traditional software development, AI requires a cross-functional approach due to the unpredictability of generative AI models and the increased need for domain knowledge. According to the report, product development teams (engineering, product and design), leadership and subject matter experts are all important players when developing AI technology.

Subject matter experts prove particularly crucial, as their domain knowledge helps ensure AI systems meet specific requirements and maintain reliability.

The Key to AI Success: AI Tooling

To tackle these challenges, developers can develop internal tooling or use a third-party solution to perform evaluations and test the accuracy of a generated response. While some companies choose to build their tools internally, others choose to leverage pre-existing platforms or frameworks. According to the report, the majority of teams choose to leverage internal tools (52.2%) rather than third-party resources (29.9%).

The report found that more than half (57.4%) of developers are performing evaluations on their AI and applications, with the majority of them conducting manual testing and reviews (75.6%). Despite increasing investments in automated evaluation platforms, most users are still manually checking for performing evaluations.

Even as more non-technical members are joining the AI development process, there is still a lack of technical expertise that is needed to properly evaluate an output. With the addition of proper tooling, developers will be better able to conduct meaningful evaluations more efficiently.

2025 and Beyond

When examining the impact of current AI implementations, competitive advantage and significant time and cost savings emerged as the primary benefits. Notably, nearly a quarter of respondents reported that their AI solutions have not yet delivered a measurable impact.

AI development isn't slowing down anytime soon with respondents sharing that they will be using technology to build more customer-facing use cases (58.8%) and more complex workflows (agentic) (55.2%). Companies are showing increased interest in developing more complex and agentic solutions, and this trend will likely continue into 2025 and beyond.

Using the right tooling to enable the right teams to develop AI solutions will be challenging, but if done right, it will be incredibly rewarding.