Maybe my blogs are biased because the coffee group I hang out with is so far Left and so heavily populated with techies. I could be answering questions of no concern to real people.
The techies all seem to believe that robots and computers are going to eliminate all jobs and that this is a serious political problem. They also believe that ‘big data’ is going to revolutionize the way business is done.
On robots destroying the labor market, this argument was common in the 1840’s and every year since then. The cotton gin was villain of the day, after the sewing machine and the Jacquard loom.
It has never happened and it can’t happen because we modern commercial people keep inventing new tastes, desires and ‘wants’ for which we simultaneously create new jobs to satisfy our vast unfulfilled wish list. In rebuttal to the robots eating jobs, I ask my morning coffee group: ‘Where did all these yoga teachers come from?’ Answer: new tastes new jobs.
The ‘big data’ issue is one I can’t deal with easily because these people have no idea about statistics or how business really works.
I worked in bank marketing research for a decade and have been a consultant to many businesses for many more decades and an expert witness in related fields for even more decades.
I use statistical analysis all the time on data of all sorts.
Here is the reality. I can do research on an old telephone directory with a half million names, I can research the entire housing stock in the San Francisco Bay Area for the tax base and the utility using habits of millions of users over thousands of days.
In one study I personally did, I collected all the locations of stop signs in San Francisco and their density by block over a 45 year period.
How? By sampling.
As the sample population gets larger, if it is a random sample, it quickly comes to represent the total population. Regardless of the size of the total population. This can be true when the sample is only a few hundred, or a few thousand if one is looking for fine details about the total population.
So the size of the data batch has never been an issue. ‘Big’ data is a meaningless tirade. A sample can be a perfect representation for any data batch size.
As more business data is digitized, monitored and recorded, more can be done with the data and the analysis. Not because of the size of the data, but because of the automatic nature of its collection. If I could compare the flow of purchases between buffalo meat and beef in a few large stores in the S.F. Bay Area it would be fascinating to me. especially over the last five years. That would be 'granularity' in the data, not 'bigness'.
But the issue is data collection and the ease of collection, not the size of the data base.