What About My Big Data?
by Mike Harvey
Mon Jul 6, 2015
That's right. I have big data too. or at least I will soon. So now that all the hooplah over the last few years around HTML5 vs Native app is beginning to die down a bit (not entirely), what's the next big technology game changer? You probably won't be surprised to know that it is push technology. With the epidemic proliferation of mobile devices, we're all getting hooked on the real-time nature of the content to our phones and tablets and whatever. Where are opportunities lies here remains to be seen but we're starting to get a glimpse of where the traction is.
That's right. I have big data too. or at least I will soon.
To illustrate the emerging challenge, I would like to start out with a question. How many bits of information do you think you receive on a daily basis? 100? 1000? I haven't counted, though I wish I knew. One reason is there's no way to know since they come from everywhere. Do you have a weather app on your phone? Guess how many messages get pushed just for your weather app everyday. The answer is probably in thehundreds. Im not talking about just alerts, but all the changes to your local weather that occur in the course of a day get pushed to your device. Now multiply that by all other apps you have on your phone, your tablet, your PC, your TV, etc. and you start to get a picture of just how much information is sent to you every day. And therein lies the challenge for data providers - how do they get your attention? Assuming that this trend will only increase, the amount of data being streamed to just about every person on the planet will become epidemic and unsustainable. And that, my friends is what I call "pushing the limits of engagement". Yes the information boom is going to be a bigger crisis than global climate change. And for those reasons, the obvious critical technology is in information aggregation.
...by 2018 we will be saying three times as much per month as all of humanity before us ever said...ever.
What is Information Aggregation?
If you've been around the tech world at all, you have probably the term content aggregation. While content aggregation is a strategy to centralize content ( primarily web based), Information Aggregation goes a step further and encompasses all information that is available including notifications, email, text, posts, news, weather, etc. in it's rawest form and centralizing it. Let's take for example, Facebook. Did you ever notice the ads on the right are almost never ads you wouldn't be interested in seeing?. Facebook can target ads for you because they have aggregated information about all the "likes", groups, interests etc. you have submitted. From a target marketing standpoint, this kind of capability is extremely valuable. And think of the other big social sites, shopping sites like amazon, and others sites that are doing tthe exact same thing - aggregating information about you. I've heard people say "Amazon knows what I'm going to buy before I do". That is not far from the truth. And unless there's an information apocolypse, it's only going to better or worse depending on your point of view. But they are getting your attention, no?
What About Your Information?
Now for the billion dollar question - What about your information? I'm not sure about you, but keeping up with the information I am sent every day is a full time job. Not all of it is important. But how do I know unless I look at it? I have no way to filter it other than reading every linked in notification, every SMS message, every email from my 3 email accounts. Sure the phone helps some with the convenience of it. At least i can try to catch up while I'm in the waiting room at the dentist. But the information isn't going to reduce; it's going to increase. And therein lies the greatest opportunity for the next 5 years - Individual Data Aggregation or to coin an acronymn IDA. Suprisingly if you google it or search for it on Wikipedia, not much comes back related to what I would consider reasonably masquerading as a definition. That's why I know it will be huge.
Let's face it, we can barely keep up now. And the old addage, "What you don't know can't hurt you" may be true but it certainly can't help you. I for one was never comfortable with what I didn't know. If you are, your probably one of the lucky few, and this won't become a crisis for you. But read on anyway because there could be something valuable here that you may trip over in the future - like a tip on the next Google. But if you are like me, I absolutely don't mean to scare you, just to enlighten you that there is some hope blooming on the horizon.
The good news is that there is a growing interest from all companies in the big data arena which is still emerging, and the major players there are projecting uses for their "big data" solutions. Here a sample of related companies and information:
- Pentaho - A Comprehensive Platform for Data Integration & Business Analytics primarily focused in Financial, Health Care, Government, and Retail sectors
- Informatica - Focus is to help organizations of all sizes harness the power of timely, relevant, and trustworthy data for their competitive advantage. Areas of primary focus are Health Care and Finnancial.
- Process One - Leader in Instant Messaging and Push Solutions. MOst exciting is their upcoming beta releash of OpenPush which if it meets expectations, could be quite the open source shot in the arm for developers focused on moving data aggregation service to the masses.
- Tableau Software - Provide analytical tools and products that can access data froms a broad range of providers including Amazon Redshift, Google Analytics & BigQuery, Hadoop, HP Vertica, SAP, MS SRRS, and more.
This is by no means a comprehensive list. There are many players entering the space, which is encouraging though I predict it will be a few years before you and I have anything that will improve my day to day. But the solutions, players, and promises are encouraging. Will IDA happen? Absolutely - it's when remains to be seen. But the motivation is there for the Commercial/Financial, Health Care, and Government Sectors. It's only a matter of time before the technology trickles out to the mainstream users.
...the old addage, "What you don't know can't hurt you" may be true but it certainly can't help you.
Trending the Growth
Preface In order to understand just how much the information boom is going to grow, I went to noneother than Cisco for the answer. Because all of this information travels on networks (wired and wireless) but eventually end up on a wired network in a data center. And since Cisco has to plan for growth I figure they're probably the most objective source of real data (not marketing hype).
According to Cisco, monthly global mobile data traffic will surpass 15 exabytes by 2018. That's exabytes and that's monthly. Thats a staggering number considering that according to Wikipedia, 5 Exabytes represents all the words ever spoken by human beings. So in other words, by 2018 we will be saying three times as much per month as all of humntiy before us ever said. And that is just mobile data. And only 3.5 years away as of this writing. So buckle up.
I think it's safe to say that Big Data is only going to get bigger. And my big data is no exception. Our abilities to manage our own information are already becoming critical to our productivity and untimately our success. What's the killer app of the next three years? Why the MyDataAndStuff.com dashboard where I can see it all, manage it all, and control it all securely. And of course it will be a RIA!More Information and references
- Cisco Visual Networking Index: Global Mobile Data Traffic Forecast Update, 2013–2018
- Microsoft Research - (PASTE)
- ICWSM - Proposal for Personal Network Aggregation System for Real-time Communication Support
- InfoChimp - Streaming Data Aggregated Queries Real Time Dashboards/
- Infomatica Information Potential