Co-authored by Ashish Thusoo and Joydeep Sen Sarma.
2016 has been an eventful year for big data and the cloud. In the last twelve months, we’ve seen the shift from on-premise to cloud as the preferred deployment option for Big Data, the meteoric rise of Apache Spark across the enterprise, celebrated the ten-year anniversary of Hadoop, and witnessed a new generation of open-source analytics projects. Qubole also had an amazing year, achieving a number of milestones that are worth recapping.
The shift to cloud for big data has continued to gain momentum and that’s reflected in the growth of our customer base. Qubole now serves more than 150 companies, more than doubling our customer base from 2015. We’re seeing increasing adoption among large enterprises, adding companies like Warner Music Group, Turner Broadcasting, Activision Blizzard, NetApp, Lyft, Box, Amgen, Cimpress, Scripps, and many more as Qubole customers. Innovative mid-size companies like Los Angeles Lakers, Return Path, Traveloka, and Carbonite are also standardizing Qubole for their big data in the cloud. What’s perhaps most exciting is the explosive growth in usage. We ended 2016 helping our customers process over 650 petabytes of data every month! In 2017 we expect to exceed a run rate of over 1 exabyte a month – now that really puts the big in big data.
A major contributor to our growth was the industry shift away from on-premise and toward cloud as the preferred deployment model. Our Cloudera Migration Program, in partnership with WANdisco, has helped many companies gain the benefits of the cloud by easily shifting their workloads from on-premise Cloudera clusters to Qubole in the cloud.
All of this wouldn’t happen without significant progress on the product and innovation front. In 2016 we made big data even more automated and intelligent, extended our cloud platform support, and made meaningful contributions to the open-source community. Some of the more notable items include:
Automation/Intelligence
- Clusters on AWS can use heterogeneous machines types to improve cost and availability
- QDS now integrates Airflow as a Service for ETL pipeline management
Cloud platforms
- Upgraded QDS on Azure to take full advantage of the Azure Cloud
- Extended our market-leading support for AWS with a dedicated EU tier, enhanced security (VPC private subnets, IAM roles, and client-side encryption ), and the ability to subscribe to QDS through the Amazon Marketplace
Open-source
- Spark 2.0 Support in QDS: Data teams using QDS have started taking advantage of major Spark 2.0 updates including API stability, substantial performance improvements, support for SQL 2003, and enhancements related to Machine Learning including support for MLlib APIs in SparkR, and ML algorithms in DataFrames-based API.
- Officially released three innovative new open source projects: Rubix, Quark, and StreamX. Together, the three projects offer next-generation disk caching for cloud storage systems, cost-based SQL optimization, and ingestion services for streaming applications based on Kafka.
Partnerships are a key to Qubole’s success and we built some amazing relationships in 2016. We released a new version of QDS on Microsoft Azure. Launched our System Integrator/Consultants program. Lastly, we announced a partnership with IBM that will integrate the IBM Watson Data Science Experience with QDS. And we continue to work closely with Amazon to develop innovative features that really exploit everything the cloud can offer. We expect partners in 2017 to be even more integral to our success and we’re pleased to announce that Colin Savage has joined Qubole as our SVP of business development. Colin joins us from Splunk where he was instrumental in building their partnership program over the last 7 years.
As a company, we are in a very strong position. We announced a $30 million Series C round of financing, bringing Qubole’s total funding to $50 million. To help lead the company through this phase of hyper-growth, we added four new members to Qubole’s stellar leadership team in 2016, including David Hsieh as senior vice president of marketing, Ken Tamura as vice president of finance, Suresh Ramaswamy as head of web services division and Andrew Daniels as Qubole’s first-ever chief information security officer and VP of IT. We’re proud to say that the Qubole team has grown by more than 100% percent in the last year.
Of course, last but not least, Qubole has been honored with a number of industry accolades in 2016, just some of them including:
- CRN Big Data 100 2016
- Gartner Cool Vendor 2016
- Forbes’ Best Big Data and Analytics Companies to Work for in 2016
- Entrepreneur Top Company Cultures List 2016
- And many more…
We’re honored to have been recognized by so many organizations in the last year not just for Qubole’s advancements in big data analytics and scalability, but for our strong employee culture as well.
So what can we expect in 2017? Big data in the cloud will continue its explosive growth and enterprises will have to grapple with the complexity and challenges of scaling access to data and analytics for every department, function, and role. It’s not an easy task. We’ve begun to see a shift towards a new DataOps approach that is the linchpin to developing a self-service data infrastructure, enabling insights-based decisions for every team across the enterprise. We’ve spent time with data teams from Facebook, Uber, LinkedIn, and other big data pioneers, and captured the best practices that accelerate a company’s journey towards a
Qubole’s Year in Review: Big Data and the Cloud in 2016
– and we have some news. We’re publishing a book on DataOp this Spring, which will describe how to build a modern data platform and the processes, culture, skills, and best practices that enable 100s or 1000s of analytic self-service users in an enterprise.
We’ll be sponsoring the Data Platforms 2017 Conference which will take place from May 24 – 26, 2017, at the Wigwam Resort in Phoenix, AZ. This will be the only conference dedicated to making data teams better by learning from expert practitioners, networking with your data engineering peers, and learning actionable steps to create your next-generation big data platform.
We want to thank our employees, customers, partners, and investors for all the hard work, dedication, and support in making 2016 a milestone year for Qubole. With a strong tailwind behind us, we expect you’ll see even more exciting developments in 2017.
