DEVOPSdigest

Ably, announced the release of version 3.0 of the Ably Kafka Connector that empowers developers to efficiently and reliably extend Kafka pipelines over the internet to millions of web and mobile users.

This latest version of the Ably Kafka Connector introduces a range of enhancements, including increased throughput and scalability, accreditation as a Confluent Cloud Custom Connector, and improved error handling and troubleshooting.

“Kafka is not designed for fine-grained data distribution over the internet to variable numbers of client devices,” said Matthew O’Riordan, CEO, Ably. “Building and maintaining an internet-facing messaging layer for realtime data distribution from Kafka to web, mobile, and IoT devices can be complex and resource-intensive. Using our Ably Kafka Connector, developers can effortlessly and securely distribute Kafka data to internet-connected clients via Ably, so they can focus more on their core engineering tasks.”

Key features of Ably Kafka Connector 3.0 include:

- Enhanced Throughput and Scalability: Ably Kafka Connector 3.0 leverages Dynamic Channel Configuration and REST support to provide superior throughput and scalability. With the ability to create an Ably channel (equivalent to Kafka topics) per user or session, developers can now easily distribute relevant information to clients connecting over the internet. Ably’s channels are optimized for cross-network communication that allows flexible message routing from Kafka topics, ensuring that each client receives only the messages pertinent to their needs. This scalable solution can handle millions of concurrent clients. Alongside Dynamic Channel Configuration, the Ably Kafka Connector 3.0 implements REST for record publication to Ably, leveraging parallel publishing through a thread pool mechanism to enhance throughput per sink task.

- Confluent Cloud Accreditation: Ably Kafka Connector is now certified as a Confluent Cloud Custom Connector. By using Ably as a plugin on Confluent Cloud, teams can quickly connect Ably to Kafka without code changes, ensure high availability and performance using logs and metrics to monitor the health of connectors and workers, and free developers from the operational burden of provisioning and managing low-level connector infrastructure.

- Improved Error Handling and Troubleshooting: Ably Kafka Connector 3.0 can now forward records to a dead letter queue (DLQ) topic using Kafka Connect dead letter queue support. Developers can easily debug and fix issues before they lead to system failures or missed data sets, whether it’s a permission-related error, unmapped data, expired credentials, or oversized records.

Ably remains committed to extending and improving the Ably Kafka Connector and welcomes feedback and feature requests from the developer community. The connector is available under the Apache 2 open source license.