Microsoft’s Azure Data Lake Store (ADLS) integration with Qubole Data Service (QDS) is a significant milestone in the journey we started when we launched QDS for Azure Blob Storage in 2017. With this integration, it will now be possible to run rich queries and derive deeper insights from your data in ADLS as well.
ADLS is an enterprise-grade hyper-scale repository for big data workloads. It enables you to capture and process data of any size, type, and ingestion speed in one single place. ADLS supports any application that uses the open-source Apache Hadoop Distributed File System (HDFS) standard. With its HDFS support, you can easily migrate your existing Hadoop and Spark datasets to the cloud without recreating your HDFS directory structure.
ADLS Vs Azure Blob Store
ADLS is storage optimized for big data workloads of all kinds – batch, interactive, and streaming. On the other hand, Azure Blob Store is a general-purpose object store that works well for various use cases and is not specially tuned for read/write accesses of big data workloads. With ADLS, there are no limits on the amount of data you can store, and it is optimized for high-throughput and Input/Output Operations Per Second (IOPS). ADLS also enforces HTTPS protocol for data transfer to and from the store, thereby enforcing better security.
For more details, click here.
Highlights Of ADLS And QDS Integration
- Configure QDS accounts with ADLS credentials for seamless and transparent access to ADLS on all (Hadoop, Spark, etc.) clusters in your account.
- Run Apache Hive, Hadoop, and Spark queries through the QDS platform, which can now access data in your ADLS.
- Migrate data from on-premise storage to ADLS using built-in native tools (in QDS) from a diverse set of storage solutions such as Azure SQL Service, Azure SQL Data Warehouse, Microsoft SQL Server, MySQL, and more.
- Migrate data from cloud object stores using distributed Hadoop (MapReduce) jobs from Azure Blob Storage to ADLS.
Getting Started
ADLS
Sign up for the Azure portal and create an ADLS account.
Click here for detailed steps.
Free QDS Business Edition on Azure
Visit https://azure.qubole.com and Sign up for free* QDS Business Edition on Azure
For detailed steps, click here.
Using ADLS with QDS
Read the Quick Start Guide here
*Qubole offers Qubole Data Service (QDS) Business Edition at no cost, but Qubole limits usage compute hours (AVMU for Azure) per month, which is approximately a $1000/month value. You must provide your Azure cloud account, and you are responsible for the infrastructure costs managed by Qubole on your behalf.
