Big data has gone mainstream. The constant, exponential growth of volumes of structured and unstructured data has significantly increased the number of big data projects, especially over the last few years. Thanks to the increased availability of the open-source Hadoop analytics platform, and the growth of big data in the cloud services, big data’s barriers to entry are dropping constantly. Still, as large and small organizations adopt big data initiatives to capture, manage and derive value from massively large data sets, a large percentage of these big data projects are failing.
Big data flops can happen for a number of reasons. But industry experts agree that failures most likely occur due to poor project management. Yes, getting business insights from big data analytics is challenging. But more often than not, true project success depends upon the proper use of key management methodologies, from a project’s initiation to completion.
Best Practices for Managing Big Data
If your organization is looking to gain a competitive advantage from the actionable insights that big data analytics can provide, here are five key best practices for big data project management that can help to ensure success.
1. Begin with clear objectives
When the famed English adventurer George Mallory was asked, “Why do you want to climb Mt. Everest?” He is reported to have said, “Because it’s there.” That answer may work for mountaineers, but it doesn’t bode well as a reason for organizations to mount big data initiatives.
Of course, the data is there, and so are the means to mine it. But the bigger question for a big data project manager is “Why do we want to analyze our data in the first place?” The answer to that question is that big data analytics can help businesses find the solutions to problems—solutions that can drive profitability and competitive advantage. Managers that begin with well-defined objectives, having clearly identified what business problems they are looking to solve, stand a much better chance of achieving big data project success.
2. Get buy-in from all business stakeholders
Starting a big data project without the full support of all business stakeholders is a recipe for early failure. Once clear business objectives have been established, it’s important to effectively communicate those objectives to all key players to make sure that everyone is on the same page. Even then there can be resistance to the project. Research shows that many seasoned business leaders feel that gut instincts and real-world insight should take precedence over big data analytics in making business decisions. Consequently, leaders that harbor big data doubts may dismiss big data project insights that suggest a change or a different course of action for the company. These biases need to be addressed and reconciled so that big data projects become core strategies instead of peripheral, half-hearted undertakings destined to fail.
3. Make sure data is available to those who need it
A big benefit of getting stakeholder buy-in is that it can help to break down data silos. It’s not uncommon for each department within an organization—sales, marketing, HR, etc.—to have its own insight-rich data partitioned off in a silo where it can’t be readily accessed by other departments. In order for big data projects to thrive, data managers need to make sure that all organizational data is made available to those that need it when they need it. Only then can the critical patterns and relationships that lead to actionable business insights be found.
4. Build a skilled and knowledgeable team
Before tackling a big data project, it’s critical to assemble a team whose members have the knowledge and skills that will enable them to work with the data to produce the desired business results. While one solution is to hire data scientists, a better approach would be to first look at the skills that each existing team member brings to the table and then augment those skills with specific big data training. Many organizations will discover that there is a lot of crossover between the skills of a business analyst and a data scientist. And they can customize their training to fill in the important gaps between the two. While costing a lot less than hiring a data scientist, there is the added benefit of having team members with a business analyst mindset who know the right questions to ask of the data to find real business value. In addition, given the added time it takes to find and train a data scientist about the business, training an analyst to use big data can get a project off the ground faster.
5. Start small
With all the hoopla surrounding big data, there is a tendency for businesses to try to go big right off the bat. And so they invest in expensive technology to give them more storage and compute power without fully understanding what they are getting into. A better approach is to take advantage of open source big data technologies such as Hadoop—starting small to test the waters and then scaling up as clearer business objectives are established. Using big data in the cloud is one way for businesses to start small and start immediately instead of attempting a waterfall approach.
Are you interested in learning how real-world companies are managing big data? Watch this webinar to learn how DataXu was able to scale big data.
