StreamSets Adds New Features to DataOps Platform
September 11, 2018

StreamSets announced innovations that help companies efficiently build and continuously operate dataflows that span their data center and leading cloud platforms — AWS, Microsoft Azure and Google Cloud Platform.

New capabilities include data drift handling for cloud data stores for improved pipeline resiliency, continuous integration and delivery (CI/CD) automation that brings DevOps-style agility to dataflow pipelines, and the ability to centrally manage in-stream data protection policies for security and compliance.

These features build on StreamSets DataOps Platform’s rich catalog of cloud connectors, its cloud-native architecture for easy cross-platform deployment, and its ability to elastically scale dataflows via Kubernetes.

Features such as data drift handling and in-stream data protection are powered by StreamSets’ unique Intelligent Pipelines capability, which inspects and analyzes data in-flow, overcoming the lack of visibility common in traditional data integration and big data ingestion approaches.

A majority of StreamSets customers already use the StreamSets DataOps Platform for cloud dataflows, executing both “lift and shift” cloud migration projects that require peak throughput, and continuous real-time streaming of data.

“As our customers embark on their hybrid cloud journey, we see first-hand their struggle to orchestrate end-to-end management of data movement across a growing range of on-premises and cloud platforms,” said Arvind Prabhakar, CTO, StreamSets. “Our DataOps platform was architected as cloud-native from the start, allowing us to easily evolve with the market. Cloud drift-handling and CI/CD for dataflows are unique enhancements that help our customers on their journey from traditional to modern data integration based on DataOps.“

The expansion of data architectures into the cloud creates challenges for enterprises that still rely on traditional data integration software or single-purpose big data ingestion tools. Using these methods, pipelines take too long to build and deploy, and often rely on valuable, specialized developers. They are opaque, denying end-to-end visibility into pipeline performance to prevent failures or detect sensitive personal data in the dataflow. Finally, they are rigid, breaking whenever data drift occurs, such as when fields are added or changed or data platforms are upgraded.

With these new features, which began rolling out in late August, StreamSets DataOps Platform now offers:

- Development automation through a full-featured dataflow designer that includes “easy button” connectors for Amazon S3, Elastic MapReduce (EMR) and RedShift; Azure Data Lake Storage, HDInsight and Azure Databricks; Google DataProc and Snowflake

- Elastic scaling of cloud, multi-cloud and reverse hybrid cloud dataflows via Kubernetes

- New data drift handling, which automatically reflects updates to source schema in Amazon Athena, Azure SQL and Google BigQuery cloud data services

- A new CI/CD framework for automating frequent changes to dataflows through iterative design, test, validate and deployment steps

- New central governance of StreamSets Data Protector policies that detect and deal with sensitive data such as PII and PHI

Share this

Industry News

April 02, 2025

Kong announced the launch of the latest version of Kong AI Gateway, which introduces new features to provide the AI security and governance guardrails needed to make GenAI and Agentic AI production-ready.

April 02, 2025

Traefik Labs announced significant enhancements to its AI Gateway platform along with new developer tools designed to streamline enterprise AI adoption and API development.

April 02, 2025

Zencoder released its next-generation AI coding and unit testing agents, designed to accelerate software development for professional engineers.

April 02, 2025

Windsurf (formerly Codeium) and Netlify announced a new technology partnership that brings seamless, one-click deployment directly into the developer's integrated development environment (IDE.)

April 02, 2025

Opsera raised $20M in Series B funding.

April 02, 2025

The Cloud Native Computing Foundation® (CNCF®), which builds sustainable ecosystems for cloud native software, is making significant updates to its certification offerings.

April 01, 2025

The Cloud Native Computing Foundation® (CNCF®), which builds sustainable ecosystems for cloud native software, announced the Golden Kubestronaut program, a distinguished recognition for professionals who have demonstrated the highest level of expertise in Kubernetes, cloud native technologies, and Linux administration.

April 01, 2025

Red Hat announced new capabilities and enhancements for Red Hat Developer Hub, Red Hat’s enterprise-grade internal developer portal based on the Backstage project.

April 01, 2025

Platform9 announced that Private Cloud Director Community Edition is generally available.

March 31, 2025

Sonatype expanded support for software development in Rust via the Cargo registry to the entire Sonatype product suite.

March 31, 2025

CloudBolt Software announced its acquisition of StormForge, a provider of machine learning-powered Kubernetes resource optimization.

March 31, 2025

Mirantis announced the k0rdent Application Catalog – with 19 validated infrastructure and software integrations that empower platform engineers to accelerate the delivery of cloud-native and AI workloads wherever the\y need to be deployed.

March 31, 2025

Traefik Labs announced its Kubernetes-native API Management product suite is now available on the Oracle Cloud Marketplace.

March 27, 2025

webAI and MacStadium(link is external) announced a strategic partnership that will revolutionize the deployment of large-scale artificial intelligence models using Apple's cutting-edge silicon technology.

March 27, 2025

Development work on the Linux kernel — the core software that underpins the open source Linux operating system — has a new infrastructure partner in Akamai. The company's cloud computing service and content delivery network (CDN) will support kernel.org, the main distribution system for Linux kernel source code and the primary coordination vehicle for its global developer network.