It’s Friday and here are the latest updates on the big data industry. Before we dive in, I just wanted to share some thoughts. Big data is not a new concept and analysts, statisticians and data scientists have been involved with big data projects for decades. So why is it a big deal now to the point that Google is reporting record breaking searches. There are multiple reasons for this buzz and here are some.
(Note – Big data refers to data sets that cannot be managed, processed or analyzed using common analytics tools. The size of the data is one of the features of big data and is generally larger than Terabyes (TB))
#1. Social and mobile – Digital social media has given an opportunity to the users to share large-volume of data. The increase in user-generated content is the foundation of the rapid success of social platforms such as Google, Facebook, Twitter and LinkedIn. Could you imagine a world without any of these social places? It’s amazing how our lives have been changed for good by… guess who??? Yup it’s because of all of us. With a significant increase in social data, it becomes extremely important to understand and use data wisely for customer acquisition, retention, ROI and loyalty or even to solve complex human problems such as biogenetics or disease prevention. This requires special skill set and a need for next generation of analysts.
#2. Technological improvements – There has been a tremendous improvement in tools and technologies that makes it super easy to manage large volume of data. SAS, IBM and SAP are well known service providers and offer a variety of packages for solving big data challenges. SAS is clearly the leader in the data analysis, modeling, statistics and predictive analytics space along with IBM.
Source : Forrester Wave : Big Data Analytics Solutions, Q1 2013
SAP’s HANA is a challenger and offers in-memory data processing allowing significantly improved performance in data accessibility, processing and analysis. It supports both structured and unstructured data management giving SAP a unique advantage over its competitors.
#3. Cross-functional involvement – With advancement in data accessibility and processing big data projects are no longer purely owned by business intelligence or data warehousing teams. Marketing, product, finance and R&D teams are getting their hands dirty and are equally responsible for amplifying the big data buzz. The change allows program owners to be equal contributors in the outcome of the projects instead of relying to non business thinkers to drive business results. This is a tectonic shift in data ownership and is already changing how companies will conduct business in near future.
Big data trends this week
1. Hadoop community gets 50M in Fuding – Hortonworks (Apache Hadoop development, distribution and support provider) announced a new round of $50 million in fuding from Tenaya Capital and Draganeer Investment Group. Hortonwork’s vision is to make open source Apache Hadoop an enterprise ready platform. They plan to use the funds for adding engineering staff, infrastructure improvement and global sales team. Hortonworks was rumored to be acquired by IBM, but the new series funding gives them another opportunity to continue to strive independently in big data space.
2. Big data changing the world? – Enough has been said about the rise of big data economy and its advantages to businesses. I think its time for all of big data ninjas to prove that it really means big money or social improvement. Interestingly, there are several areas where big data initiatives are already looking promising.
a. Energy consumption – Opower is a customer engagement company striving for educating consumers and businesses on energy consumption using data analytics. They claim they have saved 2 billion kilowatts hour to date and counting using a patent-pending analytics engine.
b. Disease control – The normal procedure to understand the spread of any disease is collection and analysis of data from hospital and medical research institutions. The challenge is this process takes a long time due to manual collection routines and is not too effective in controlling the disease before it turns into an epidemic. Twitter has come to rescue in a big data analytics study of its kind. American Journal of Tropical Medicine and Hygiene was able to detect a cholera outbreak in Haiti post earthquake two weeks prior to the official reporting. AJTMH used HealthMap, Twitter data and other official data sources and saw a huge correlation between Twitter posts and official reports.
Source : The American Journal of Tropical Medicine and Hygiene
c. The next big TV Soap – Craking the code of producing a hit TV series has been an obsession of large networks for a long time. Even with the significant improvement in delivery mechanism (HDTV, Satellite, Mobile) there has been a limited success on the actionable analytics front…until now. Netflix and Amazon are using the data from their users to understand the viewership demand pattern. Both companies are trying to predict future TV series success based on past user experience and views. Netflix has already been betting on the data intelligence with the “House of Cards” and “Arrested Development” series while Amazon is using viewer data to understand which series it’s customer based would like to watch.
3. Hunk your Big data visualization – Splunk Inc., a popular data analytics company has announced the launch of their new application Hunk. Hunk allows Splunk Inc. to extend its data exploration, analysis and visualization to Hadoop databases. Users can run Hunk on their Hadoop projects without moving or migration any datasets. The system can be used to interact with Hadoop databases and create custom dashboards.