QuData is full of valuable information just waiting to be extracted with the right approach. For most companies, that’s easier said than done. Cloud and data analytics both remain very buzzy concepts, tempting companies to dive in without any assurance their projects will yield valuable results. There have never been more tools available to more kinds of decision-makers than there are now, but on the other side of that coin, the spectre of complexity looms large. The complexity of data infrastructure is at an all-time high–a high that threatens to cripple most big data projects in the coming years.
Gartner predicts that 70% of big data projects will fail by 2018. This is not a happy picture being painted about a set of technologies we now have the potential to unlock vast amounts of actionable information. While the ecosystem of Hadoop tools continues to grow, the list of problems that can be addressed with big data grows as well. Meanwhile, the list of problems that can be practically addressed by actual big data projects undertaken by actual companies is stagnating in a mire of complexity and overburdened support teams. This is exactly the industry-wide challenge we sought to address at the Data Platforms 2017 conference.
The purpose of Data Platforms is to share knowledge on how to distill down the vast ecosystem of big data tools into real-world data platforms that work for real-world organizations and real problems. While many companies have faced challenges building data platforms, the ones that have done it well have done it very well. The LinkedIn(s), Facebook(s), and eBay(s) of the world offer evidence that the promise of big data can be delivered upon with the right approach. Qubole and Data Platforms are now working to bring that capability to every organization.
Watch Joydeep’s Presentation at Data Platforms:
Overcoming Complexity with Automation
While today’s data platforms are undoubtedly amazing feats of data engineering, it’s likely that we’ll look back at them as we do on early steam engines: beautiful, complex, delicate, labor-intensive, capable of sparking huge technological advances, but not practical in the long run. To get to the point where data platforms are truly powering the industry, we need to achieve the same hands-off reliability advancing combustion engine technology afforded the last great industrial revolution. While the knowledge sharing engendered by Data Platforms is essential in today’s environment, overcoming complexity and big data project failure will require a serious adoption of automation.
Providing the automation that will free up technical staff from routine maintenance tasks and make data accessible throughout organizations is one of the core purposes of Qubole, and ultimately of Data Platforms. The next generation of Qubole tools announced at Data Platforms 2017, is geared for those challenges with tools including alerts, insights, and recommendations for enterprise platforms, and AI-powered cloud agents that can begin taking mundane tasks off the plates of human staff.
In the spirit of the open-source technology that powers big data, by sharing methods and best practices with our colleagues in forums like Data Platforms we can bring up the success rate of big data projects. By leveraging the flexibility of the cloud and emerging automation technologies, we can build data platforms built to last.
