DEVOPSdigest

In 1963, a team of talented engineers led by J.C.R. Licklider took on a challenge like no other before: to develop a technology that would allow a computer to be used by two or more persons simultaneously. With the challenge, they took a $2 million research grant, and thus commenced a journey, which paved the way to computer networks, virtualization technologies, and the Internet. But even four decades later, in the early 2000s, the world had still to discover one of the most impactful contributions of Licklider's team — Cloud computing.

At that point, the data still resided in on-premises databases, but the success of early Application Cloud giants like Salesforce, Workday, and Google signaled a shift in the established trend. Enterprise productivity, along with the data that fueled it, started moving to the shared services that lived outside the perimeter of the organization. It took companies like Amazon, Google and Microsoft another decade to reach the next critical milestone — the Infrastructure Cloud, which allowed organizations to not only run applications, but migrate their storage and compute resources to the cloud, recognizing massive gains in performance and efficiency. Despite these gains, organizations quickly discovered that having the means to store and process information wasn't enough to make it readily accessible. Data continued to suffer from the legacy of the on-premises days: endless siloes, inefficiencies, lack of interoperability and, as a result, poor security.

Fortunately, another turn of the decade has marked yet another milestone — a shift in data and cloud technology, which is often referred to as the Data Cloud. For the first time in history, in a matter of years, an entire ecosystem of cloud-based services appeared to store, process and analyze operational, business intelligence and other data. True to the diverse nature of data repositories, this ecosystem spans databases, data warehouses, and data pipelines. The best part? It works.

It may seem like the Data Cloud has appeared out of nowhere, but in reality, it's been a part of several large trends that the industry is going through:

■ Digital Transformation allows organizations to use digital technologies to create new customer experiences, business processes, or employee management to meet changing business, market and cultural requirements. The Data Cloud enables companies to efficiently capture all their diverse data, experiment with the various use cases, scale to accommodate their need to unlock the value from their data, and optimize the price for performance and value.

■ Data Democratization makes data available to a wide range of stakeholders in a business. It has been a key tenet of digital transformation, and helps unlock the value locked within proprietary data embedded in various pockets of the organization, which in turn helps improve both their top line and bottom line. Embracing the Data Cloud helps teams pick tools for managing, processing and analyzing data that make most sense to them.

■ Infrastructure as Code (IaC) allows developers to manage and configure infrastructure through text files in a human and machine-readable format. Easy to scale and prototype, IaC has seen a massive uptake as organizations embrace the cloud. Developers can now easily spin up Data Cloud services for their applications, and services to move data across from them to their data warehouse.

While still early in its growth, the Data Cloud has clearly proven its potential. From the beginning, the allure of the Data Cloud was in its ability to solve the siloing data problem. With its flexible architecture, it has quickly enabled teams to pick the technologies they most needed, providing easy access to the relevant data repositories, and eliminating the consistency and cost problems.

But quickly, other benefits of embracing the Data Cloud also became apparent:

■ The Data Cloud has brought the processing and business intelligence into the cloud, which has enabled virtually unlimited scale and performance, and unlocked many new use cases and opportunities.

■ The Data Cloud has enabled businesses to become data-driven, to make faster, better, decisions, and improve their customer reach, engagement and retention.

■ The Data Cloud has freed up a lot of internal resources normally required for infrastructure procurement, deployment and maintenance, given the as-a-service model of delivery. Those resources can now be used to continuously refine their products, unlock new opportunities and deliver deeper insights to all stakeholders.

Any type of cloud inherently introduces new challenges. Just like the application and infrastructure clouds, the Data Cloud brings with it security issues that have not been seen before. For example, one of the biggest promises of the Data Cloud is its flexibility and support for various applications, data sources, and third-party applications such as BI tools. With such flexibility, controlling access based on IPs becomes an insurmountable task, forcing organizations to find ways to integrate identity into the data source.

Further, governance and compliance becomes impossible to manage as services scale. Enforcing integrations with identity management for all data sources can easily turn into a compliance nightmare, and granting access to new users becomes such a burden that many people cut corners and resort to shared accounts and finding excuses to bypass the compliance requirements.

Finally, distributing data amongst numerous services inside a Data Cloud leads to a distributed governance model, which is a critical component in the cloud. Each tool requires different methods for managing identities and controlling permissions. They also each generate different logs and audit trails, complicating compliance and visibility.

Despite all the risks, it is clear that the promise of the data cloud far outweighs all the challenges. And as more organizations embrace the new ecosystem, more leaders start thinking about making their data not only more accessible, but also more secure. This thinking led to the emergence of a burgeoning cloud security industry in the past, and — if history is any indicator — we might be on the verge of seeing a brand new data cloud security category in the making.