Organizations, whether big or small, are embarking on a data lake strategy for centralized applications and applications coming together on a single central platform. With the huge amounts of data available in the market today, organizations have realized that they can derive value from it through data lakes.
Why are Enterprises Using Data Lakes?
Data lakes have become one of the most popular repositories used to store large amounts of data. The way data lakes are transforming business explains why 2024 will be the year of focus around data lakes as companies invest in analytics platforms and use data lakes to drive business innovation.
Take a look at the emerging data lake trends for 2024 that will affect organizations and governments adopting big data to taste success by discovering meaningful insights locked inside data.
Data Lake Cost Management
Trend #1: Implementing Financial Governance to Manage Data Lake Costs
To continuously monitor and safeguard themselves against unexpected spending, organizations require effective financial governance for data processing that can provide direct evidence of benefits received against spend. Financial governance ensures that wasteful spending is identified and eventually eliminated, therefore quantifying the return on investment of valuable spending.
Qubole provides powerful automation that allows administrators to control spending by optimizing resource consumption, using lower-priced resources, eliminating unnecessary resource consumption, and throttling queries based on monetary limits. Custom-configurable controls and insight into key sources of spend offer additional measures to oversee spending. Qubole also provides governance through intelligent automation capabilities like workload-aware autoscaling, intelligent spot management, heterogeneous cluster management, Qubole Cost Explorer, and automated cluster lifecycle management.
Organizations need powerful and reliable financial governance for data processing that can constantly monitor, safeguard against startling spending, and give direct proof of the benefits achieved. Financial governance guarantees that inefficient spending is identified and inevitably wiped out while measuring the ROI of important spend.
Data Lake Financial Governance with Qubole
Best Practice #1: Monitor, Manage, and Optimize Data Lake Costs with Qubole Cost Explorer
Qubole Cost Explorer helps organizations monitor, manage, and optimize big data costs by providing visibility of workloads at the user/department, job, cluster, and cluster instance levels. It also helps enterprises achieve a financial governance solution to monitor their workload expenditures using pre-built and customizable reports and visualizations.
With Cost Explorer, data teams can:
- Spend Tracking: Regularly track both Qubole and cloud spend on various assets
- Build ROI Analysis: Calculate the amounts spend on use cases to calculate the return on investments
- Monitor Showback: Showback costs incurred by various teams/business units
- Optimize Opportunities: Identify top spending assets to further reduce TCO
- Plan and Budget: Estimate the future spending based on past Job/User level costs
- Justify Business Cases: Show savings on operating costs to the finance team
Irrespective of the lifecycle stage or bursty nature of typical analytics and machine learning use cases, Qubole Cost Explorer provides data that can help you build the structure of an effective Financial Governance. On average, Qubole customers can onboard over 350 users in months and use an average of 1.5 million compute hours across multiple use cases.
Cluster Management
Best Practice #2: Automated Cluster Lifecycle Management and Workload Aware-Autoscaling
Qubole helps organizations save huge amounts of money with built-in platform capabilities and sustainable economics that allow your infrastructure to scale up or down as per one’s requirement automatically. Qubole provides automated platform management for the entire cluster lifecycle: configuration, provisioning, monitoring, scaling, optimization, and recovery. The platform maintains cluster health by automatically self-adjusting based on workload size as well as proactively monitoring cluster performance.
Qubole also eliminates resource waste by automating the provisioning and de-provisioning of clusters and automatically shutting down a cluster without risk of data loss when there are no active jobs. These decisions are based on granular-level details (like task progression) and occur autonomously in real time to avoid overpaying for computing, interrupting active jobs, or missing SLAs.
The top five ways in which organizations are leveraging Qubole’s cloud data platform facilities are:
- Isolate workloads in different environments and different clusters
- Isolate user workloads in different clusters
- Automatically stop clusters when not required
- Leverage a high percentage of Spot nodes with heterogeneous nodes in clusters
- Use auto-elasticity of cluster
Data Lake Governance and Security
Trend #2: Data Governance and Security Taking Center Stage in a Changing World
Data governance means setting internal standards of data policies to ensure that your data is secure, accurate, private, and usable while complying with external standards set by industry associations, government agencies, and other stakeholders.
As more and more organizations adopt data lakes, they require a single interface for configuration management, auditing, obtaining job reports, and exercising cost precisely- exactly what Qubole offers.
Best Practice #3: Open Source Flexibility Meets Enterprise-Grade Security with Qubole
At Qubole, We are constantly making additions to our Qubole Data Service (QDS) product to better meet the security, data governance, and compliance needs of our enterprise customers. In the past year, significant improvements have been made around network architecture, data encryption options, and internal operations.
- Network Architecture: Qubole now supports Hadoop, Spark, and Presto clusters in Amazon VPC private subnets. In AWS, the use of a Virtual Private Cloud (VPC) is increasingly gaining popularity, particularly when combined with private subnets. Clusters brought up in a private subnet are not reachable from the public internet. These clusters are thus more secure and less vulnerable to potential attacks.
- Encryption Options (HIPPA Compliance sing Encryption Option: There are three major areas where data encryption can be important:
- Data in transit: Qubole recently added support for Simple Authentication and Security Layer (SASL) for data in transit within Spark Clusters. These settings can be enabled easily as a Spark setting within the cluster settings. It also added early access for customers interested in SSL for data in transit for Presto clusters.
- Data in the object store: The object store (such as Amazon S3, Azure Blob Store) usually serves as the primary source of truth for data for our customers. Encryption for data at rest is generally up to the cloud provider. QDS provides the hooks to integrate with the encryption options provided in the object store itself.
- Temporary data in HDFS: Encryption for temporary data in HDFS is already available via a simple checkbox in the cluster settings.
- GDPR and CCPA frameworks with Delta Lake Tables and Apache Ranger.
Delta Lake Tables and Apache Ranger address distinct requirements of granular data access control and granular delete/merge/update respectively. Qubole Data Services helps organizations govern data in their data lakes across multiple engines. It also makes them future-proof for newer regulations while dealing with the massive volume and velocity of data.
As a built-in feature of Qubole data service, these open-source solutions are a part of everyday workflow instead of an afterthought point fix solution. Take a look at a few improvements that Qubole has made to Apache Ranger and Delta Lake Tables for their enterprise customers:
- Efficient updates, and deletes to data: Users can make inserts, updates, and deletes on transactional Tables—and query the same via Qubole Spark or Presto. Unlike the traditional approach which requires a rewrite of large amounts of data for even a few rows changed, Qubole writes only to changed rows thus providing faster rewrites, updates, and deletes.
- Direct writes to the final location on cloud storage: Open Source Apache Hive writes data to temporary locations first and renames it to the final location in a final commit step. Qubole writes directly to the final location and avoids the expensive rename step, thereby reducing the performance impact due to this impedance mismatch,
- Atomic operations to rename directories on cloud storage: When using the open-source version, directory renames being not atomic can make partial data visible in the destination directory, resulting in compactions (which perform a rename). Qubole provides atomic operation using a commit marker in the destination directory for the waiting reader.
- Single UI/Solution for multiple engines: Qubole provides a single and the same UI for using ACID capabilities using Delta Lake Tables or Apache Ranger across multiple engines. By making Apache Spark and delta.io part of Qubole Data Service, Qubole lets the organizations focus on building data pipelines, ad-hoc SQL queries, and ML workbenches at scale without performance impact.
- Use existing controls and infrastructure: With Qubole’s RBAC integrations of Active Directory, LDAP, and SAML2.0, organizations can leverage their existing RBAC solutions to manage user access to the data lakes.
- Hive Authorization: Hive authorization is one of the ways to authorize users from multiple accesses and privileges. Qubole provides SQL Standard-based authorization with some additional controls and differences from the open source. Qubole is the industry’s only vendor to support multiple open-source engines (Hive, Spark, and Presto) for Apache Ranger, the top framework for enabling, monitoring, and managing comprehensive data security. This support provides granular data access edges, within the Qubole data based on the user privileges platform. Qubole’s Hive authorization provides Qubole Hive users the ability to control granular access to Hive tables and columns. It also provides granular control over the type of privileges a Hive user can have over a Hive table.
Streaming Analytics
Trend #3: Enterprises Making More Informed Decisions with Real-Time Data
With Qubole Pipelines Service, businesses can complement their existing data lake with advanced features in a managed environment via the public cloud of their choice. This helps organizations to instantly capture streaming data from various sources, accelerate the development of streaming applications, and run highly reliable and observable production applications at the lowest cost.
The new features include:
- Accelerated Development Cycle: A pipeline can be developed within minutes without writing even a single line of code, thanks to the numerous built-in connectors, code generation wizard, dry run framework, and quick-start options that help accelerate development lifecycles.
- Robust and Cost-Efficient Stream Processing Engine: Leveraging Apache Spark Structured Streaming, Qubole added several enhancements including Rocksdb state storage, direct writes, and memory pressure scheduling, for reliably building and deploying long-running streaming applications.
- Comprehensive Operational Management: Qubole Pipelines Service includes a broad set of APIs and user interfaces for engineers to holistically manage the lifecycle of streaming applications and get continuous operational insights.
- Data Management and Consistency: Qubole’s new Pipelines service uses the ACID framework using Delta Lake tables to efficiently compact small files in the background while allowing concurrent read/write operations, without impacting performance.
Qubole Open Data Lake
The growth of data usage of ad-hoc data analytics, streaming analytics, and Machine Learning Models may be well understood, but what remains uncertain and thus completely unpredictable is when and how often a company’s needs for data processing will spike or fall, along with the costs. Therefore, organizations must rely on controls, automation, and intelligent policies to govern data processing and attain sustainable economics.
Qubole is helping organizations regain control of costs and succeed at their goals and initiatives without overpaying. The openness and data workload flexibility that Qubole offers while lowering cloud data lake costs by over 50 percent is unparalleled.
