The trend has been clear for years… There is more data that humans can deal with. We must continually examine our approaches and do whatever it takes to make maximum use of our technology to digest and understand data.
There are no simple answers to overwhelming data challenges, but one thing is clear. The answer to our challenges is not “less data.” The reasons we have so much data is the continued drive to improve business outcomes in industry, the need to pursue scientific discovery in academia, and the need to improve government support to citizens and the mission of government. We need new concepts and technologies to deal with these massive amounts of new data. We need new ways of doing more analysis and gaining more understanding, and of course we need smart data solutions.
For years now people and our sensors and our computers have been generating more information than we can analyze. Breakthrough methodologies and new scalable data approaches based around Hadoop hold great promise to helping us make sense over increasingly large data sets. Sensemaking is key to success in most all other technology advancements. The ethical considerations around privacy and security over data will only grow.
A snapshot of the trend right now indicates:
- Estimates of 1 Trillion sensors by 2020
- Breakthrough methodologies and new scalable data approaches based around Hadoop hold great promise in sensemaking over large data sets.
- Sensemaking is key to success in most all other technology advancements.
- Ethical considerations around privacy and security over data will only grow.
- How ethical considerations are resolved will depend on culture, politics, other factors.
Open questions decision-makers should track include:
- Can there be widely accepted ethical frameworks for data and its use?
- Who will individual citizens trust to analyze their personal, home, auto data?
- Can behavioral analytics enhance service and security?
Books we appreciated for context around Big Data include:
- The Fourth Transformation: How Augmented Reality & Artificial Intelligence Will Change Everything
- The Inevitable: Understanding the 12 Technological Forces That Will Shape Our Future
- The Industries of the Future
- The Seventh Sense: Power, Fortune, and Survival in the Age of Networks
- The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies
- Rise of the Robots: Technology and the Threat of a Jobless Future
The way most enterprise technologists see these Big Data megatrend up close and personal is through the offerings of the companies providing big data solutions. For our reviews of the most disruptive technologies in the big data space see our Big Data Companies list.