In addition to the recent announcement of the integration of Microsoft’s Azure Data Lake Store (ADLS) with Qubole Data Service (QDS), Qubole now offers interactive querying capability on ADLS with Presto. This is a major benefit for businesses that want to do interactive queries against large data sets using the same Hive metastore leveraged by the ETL process on Hive and data science use cases on Spark.
ADLS is an enterprise-grade hyper-scale repository for big data workloads. It enables you to capture and process data of any size, type, and ingestion speed in one single place. ADLS implements the open-source Apache® Hadoop Distributed File System (HDFS) compatible interface. With its HDFS support, you can easily migrate your existing Hadoop and Spark datasets to the cloud without recreating your HDFS directory structure.
Presto is a distributed SQL engine, designed for running interactive analytics queries. Using a memory-oriented architecture in place of HDFS storage, Presto is able to rival the performance of traditional data warehouses but at a fraction of the cost, by relying on a horizontally scalable layer of memory-optimized compute nodes backed on to affordable and secure ADLS storage for your data lake. Presto can be used in place of other well-known interactive open-source query engines such as Impala, Hive, or traditional SQL data warehouses.
Analysts with existing skills can connect their preferred BI applications (e.g. Power BI, Tableau, Looker, Qlik, etc…) to the Qubole-managed Presto cluster via a common ODBC/JDBC driver. This makes adoption easy and maintains productivity.
ADLS vs Azure Blob Store
ADLS is storage optimized for big data workloads of all kinds – batch, interactive and streaming, and all types, both structured and unstructured. On the other hand, Azure Blob Store is a general-purpose object store that works well for a variety of use cases and is not specially tuned for read/write accesses of big data workloads. With ADLS there are no limits on the amount of data you can store and it is optimized for high-throughput and Input/Output Operations Per Second (IOPS). ADLS also enforces HTTPS protocol for data transfer to and from the store, thereby enforcing better security.
For more details, visit https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-comparison-with-blob-storage.
Highlights of Presto, ADLS, and QDS Integration
- Use Qubole to efficiently manage your Presto clusters by automatically taking care of provisioning and de-provisioning, along with dynamic auto-scaling the size of the cluster to handle the workload at any point in time.
- Connect your favorite BI application to a Qubole-managed Presto cluster.
- Configure QDS accounts with ADLS credentials for seamless and transparent access to ADLS on all (Hadoop, Spark, etc.) clusters in your account.
- Run Apache Hive, Hadoop, Spark, and Presto queries through the QDS platform which is now capable of accessing data in your ADLS.
- Migrate data from on-premise storage to ADLS using built-in native tools (in QDS) from a diverse set of storage solutions such as Azure SQL Service, Azure SQL Data Warehouse, Microsoft SQL Server, MySQL, and more.
- Migrate data from cloud object stores using distributed Hadoop (MapReduce) job from Azure Blob Storage to ADLS.
Getting Started
ADLS
Sign up for the Azure portal and create an ADLS account.
For detailed steps, visit https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-get-started-portal#create-an-azure-data-lake-store-account
Free QDS Business Edition on Azure
Sign up for free* QDS Business Edition on Azure by visiting https://azure.qubole.com.
For detailed steps, visit https://docs.qubole.com/en/latest/quick-start-guide/Azure-quick-start-guide/azure.html
Using Presto & ADLS with QDS
For detailed steps, visit https://docs.qubole.com/en/latest/quick-start-guide/Azure-quick-start-guide/azure.html#azure-getting-started-account.
*Qubole offers Qubole Data Service (QDS) Business Edition at no cost, but usage is limited by Qubole compute hours (AVMU for Azure) per month, which is approximately a $1000/month value. You must provide your own Azure cloud account and you are responsible for the infrastructure costs managed by Qubole on your behalf.
