Ably Releases Kafka Connector 3.0
August 01, 2023

Ably, announced the release of version 3.0 of the Ably Kafka Connector that empowers developers to efficiently and reliably extend Kafka pipelines over the internet to millions of web and mobile users.

This latest version of the Ably Kafka Connector introduces a range of enhancements, including increased throughput and scalability, accreditation as a Confluent Cloud Custom Connector, and improved error handling and troubleshooting.

“Kafka is not designed for fine-grained data distribution over the internet to variable numbers of client devices,” said Matthew O’Riordan, CEO, Ably. “Building and maintaining an internet-facing messaging layer for realtime data distribution from Kafka to web, mobile, and IoT devices can be complex and resource-intensive. Using our Ably Kafka Connector, developers can effortlessly and securely distribute Kafka data to internet-connected clients via Ably, so they can focus more on their core engineering tasks.”

Key features of Ably Kafka Connector 3.0 include:

- Enhanced Throughput and Scalability: Ably Kafka Connector 3.0 leverages Dynamic Channel Configuration and REST support to provide superior throughput and scalability. With the ability to create an Ably channel (equivalent to Kafka topics) per user or session, developers can now easily distribute relevant information to clients connecting over the internet. Ably’s channels are optimized for cross-network communication that allows flexible message routing from Kafka topics, ensuring that each client receives only the messages pertinent to their needs. This scalable solution can handle millions of concurrent clients. Alongside Dynamic Channel Configuration, the Ably Kafka Connector 3.0 implements REST for record publication to Ably, leveraging parallel publishing through a thread pool mechanism to enhance throughput per sink task.

- Confluent Cloud Accreditation: Ably Kafka Connector is now certified as a Confluent Cloud Custom Connector. By using Ably as a plugin on Confluent Cloud, teams can quickly connect Ably to Kafka without code changes, ensure high availability and performance using logs and metrics to monitor the health of connectors and workers, and free developers from the operational burden of provisioning and managing low-level connector infrastructure.

- Improved Error Handling and Troubleshooting: Ably Kafka Connector 3.0 can now forward records to a dead letter queue (DLQ) topic using Kafka Connect dead letter queue support. Developers can easily debug and fix issues before they lead to system failures or missed data sets, whether it’s a permission-related error, unmapped data, expired credentials, or oversized records.

Ably remains committed to extending and improving the Ably Kafka Connector and welcomes feedback and feature requests from the developer community. The connector is available under the Apache 2 open source license.

Share this

Industry News

February 13, 2025

LaunchDarkly announced the private preview of Warehouse Native Experimentation, its Snowflake Native App, to offer Data Warehouse Native Experimentation.

February 13, 2025

SingleStore announced the launch of SingleStore Flow, a no-code solution designed to greatly simplify data migration and Change Data Capture (CDC).

February 13, 2025

ActiveState launched its Vulnerability Management as a Service (VMaas) offering to help organizations manage open source and accelerate secure software delivery.

February 12, 2025

Genkit for Node.js is now at version 1.0 and ready for production use.

February 12, 2025

JFrog signed a strategic collaboration agreement (SCA) with Amazon Web Services (AWS).

February 12, 2025

mabl launched of two new innovations, mabl Tools for Playwright and mabl GenAI Test Creation, expanding testing capabilities beyond the bounds of traditional QA teams.

February 11, 2025

Check Point® Software Technologies Ltd.(link is external) announced a strategic partnership with leading cloud security provider Wiz to address the growing challenges enterprises face securing hybrid cloud environments.

February 11, 2025

Jitterbit announced its latest AI-infused capabilities within the Harmony platform, advancing AI from low-code development to natural language processing (NLP).

February 11, 2025

Rancher Government Solutions (RGS) and Sequoia Holdings announced a strategic partnership to enhance software supply chain security, classified workload deployments, and Kubernetes management for the Department of Defense (DOD), Intelligence Community (IC), and federal civilian agencies.

February 10, 2025

Harness and Traceable have entered into a definitive merger agreement, creating an advanced AI-native DevSecOps platform.

February 10, 2025

Endor Labs announced a partnership with GitHub that makes it easier than ever for application security teams and developers to accurately identify and remediate the most serious security vulnerabilities—all without leaving GitHub.

February 07, 2025

Are you using OpenTelemetry? Are you planning to use it? Click here to take the OpenTelemetry survey(link is external).

February 06, 2025

GitHub announced a wave of new features and enhancements to GitHub Copilot to streamline coding tasks based on an organization’s specific ways of working.

February 06, 2025

Mirantis launched k0rdent, an open-source Distributed Container Management Environment (DCME) that provides a single control point for cloud native applications – on-premises, on public clouds, at the edge – on any infrastructure, anywhere.

February 06, 2025

Hitachi Vantara announced a new co-engineered solution with Cisco designed for Red Hat OpenShift, a hybrid cloud application platform powered by Kubernetes.