DEVOPSdigest

Solo.io announced the release of Gloo AI Gateway, which is designed to meet the emerging use case of accelerating AI innovation.

With Gloo AI Gateway, Solo.io is building on Gloo Gateway, which delivers an Envoy-based API gateway and ingress controller to facilitate and secure application traffic at the edge, to bring the same speed, security, and scalability to modern AI applications.

“AI Gateway is the most significant new trend in the API management and API gateway space,” said Idit Levine, CEO and founder of Solo.io. “As customers continue to adopt AI, we’re providing the ability to leverage Gloo AI Gateway to handle this additional new set of use cases in addition to their existing API management needs. With Gloo AI Gateway, we’re empowering application developers to move forward with their AI initiatives, but providing guardrails to ensure long-term success.”

Gloo AI Gateway optimizes Solo.io’s customers’ AI journey by delivering:

- Speed to deployment: Eliminates development friction, boilerplate code, and avoidable errors in applications consuming LLM APIs.

- Security and control: Protects applications, models, and data from inappropriate access and ensures safe use of AI with governance controls, auditability, and visibility into consumption.

- Scalability: Leverages advanced AI integration patterns for data augmentation and integration with cloud-native gateway capabilities to support high-volume, zero-downtime AI connectivity.

Key use cases of Gloo AI Gateway include:

- Multi-LLM provider support: Simplifies LLM access for consumers and creates centralized control, visibility, and governance across LLM providers.

- API key management: Securely stores LLM API keys as secrets and also generates API keys to map to one or more LLM provider secrets.

- Consumption control & visibility: Monitors and tracks LLM consumption efficiently with logging, analytics, and reporting features that ensure optimal resource utilization and cost efficiency.

- Prompt management: Streamlines LLM application integration with prompt templating and prompt enrichment and utilizes prompt guards and data exfiltration controls to reject inappropriate requests and sanitize LLM responses to ensure consistent governance and control.

- Retrieval augmented generation (RAG): Ensures LLM responses are grounded in accurate and relevant information, dynamically retrieved from external sources.

“At Solo.io, we are focused on meeting our customers at their ever-changing inflection point with their IT operation needs and providing them with the solutions they need to succeed,” Levine added. “Gloo AI Gateway is another example of how we’re fueling innovation for developers and enterprises around the world as AI becomes a mainstay in today’s application development.”