DEVOPSdigest

Push Technology announced the launch of a new Kafka Adapter for their Diffusion Intelligent Data Mesh.

With the Diffusion Kafka Adapter, organizations can now efficiently and securely extend Kafka solutions over the Internet, streaming real-time data to millions of end-user applications. In addition, customers can easily manage the high-volume of data across geographically dispersed regions.

Kafka has proven capability for high-throughput data handling within the data center. When dealing with potentially hundreds of thousands of topics and concurrent connections in real-time mode, the qualities that make Kafka extremely powerful in the backend, fail to meet the unique requirements and challenges of edge networks. For example, Kafka performance suffers as the number of consumers increases. In addition, lack of topic granularity causes consumers to receive substantial amounts of excess data dramatically increasing cost. Diffusion with the Kafka Adapter handles these challenges.

The Diffusion Intelligent Data Mesh provides pub/sub functionality over last-mile network connections – web, mobile or satellite – reducing application code complexity and speeding time-to-market. The new Diffusion Kafka Adapter is fully hosted within the Diffusion Cloud infrastructure with an easy-to-configure user interface, for seamless integration with Kafka brokers. The Adapter automatically maps Kafka message types to JSON, allowing web, mobile, and IoT clients to securely consume data that is stored as rich data structures within Kafka.

The valuable customer benefits of the Diffusion Kafka Adapter are:

- Scalable Topic Models: Diffusion is purpose built to handle millions of unique topics, allowing data to be scaled independently from consumers. In-flight transformation rules allow event data to be disaggregated or remapped, enabling intelligent and efficient distribution of data, based upon specific consumer requirements, while throttling or delay controls are available as needed to facilitate the exposure of real-time data at different tiers of availability. Diffusion removes the overhead of scaling the consumer-side of real-time data, allowing organizations to pursue new business opportunities without worrying about operational complexities. The Diffusion Kafka Adapter exponentially scales Kafka performance by providing the ability to handle millions of topics and millions of concurrent connections.

- Security: Security can be a challenge on edge networks. With Diffusion, dynamic multi-tiered security access control is available out-of-the-box. A robust, pluggable authentication system makes it easy to combine standard security adapters (e.g. JWT, OAuth) with bespoke authentication strategies. Diffusion’s scalable Role-based Access Control system allows unique security policies to be assigned across hundreds of thousands of consumers and changes can be applied dynamically across all active connections and subscriptions.

- Efficient Data Movement: Diffusion is an intelligent solution for the challenge of optimizing real-time data distribution across unreliable networks, with adaptive flow-control, optional conflation of messages, and automatic delta streaming which can reduce bandwidth usage by up to 90%.

Sean Bowen, CEO of Push Technology, said: “Many organizations struggle to securely and cost-efficiently integrate Kafka into their operations as they build out highly-scalable, real-time web and mobile applications. Our Diffusion Intelligent Data Mesh and Kafka Adapter handle the complexities and address the challenges of Kafka data management and control over the network edge.”