Big Data? Or just More Data?
The term has been in use now for several years, but I hear it being used more and more lately, and the other day I heard someone use it in terms of they are working with big data, as they have 10 times more than they used to have.
Now this is very subjective, but my thought process to explain what ‘big data’ is has moved to the following definition, to be able to explain it in a way that is useful, wiki states:
Big data is a term for data sets that are so large or complex that traditional data processing application software is inadequate to deal with them.
The challenge with this definition is the definition of ‘traditional’ now becomes the roadblock.
So, I would suggest maybe you use a definition like myself.
If in the year 2010 you had a contact database for taking cars of clients CRM software, and you still record that data today, but you have a lot more of it, then this is not big data. If you are still generating the same reports coming to the same conclusions, and massaging the data in the same way, then you are not dealing with big data, if on the other hand, you are now running queries against the time slices that occur between each interaction, or looking to determine who is likely to buy based on behaviors of other clients, not be looking at them individually and making a decision that client of type X with the following criteria are likely to buy in the next 10 days, but an automated data processing suite, call it a neural network or an Artificial Intelligence has told you that clients X, Y and Z are likely to buy because they match a buying pattern from other clients but the system does not indicate why then this is a move towards big data.
We know use machine learning on our SFDC (salesforce) database to better predict likelihood and timing of purchase from our clients. Does it give us perfect results? not yet, we are currently adding more input criteria that we believe to be relevant, but it does help our sales team focus on likely outcomes.
If historically you have emailed data files through the corporate network, but now they must be moved by services like drop box, and they take 15 minutes to transfer, and you are starting to take advantage of new insights this is big data.
So, in summary, if you can process more data in a similar way that you did a few years ago, then you are probably suffering from more data but not getting the benefits of big data. Big Data means you must fundamentally change the way you process the data and the people involved with it (which should now be everyone), are analyzing in ways that were not possible a few years ago then you have entered the big data age.