all bits considered data to information to knowledge


The fine line between “Big Data” and “Big Brother”

The was never a lack of desire to collect as much data as possible on the part of business or governments; it was capabilities that always got in the way. With advent of "Big Data" technology the barrier had just been lowered.

Monitoring employees interactions in minute detail to analyze patterns, and get ideas on productivity improvements is not illegal per se...but it takes us one step further towards this proverbial slippery slope.

The recent article in The Wall Street Journal by Rachel Emma Silverman highlights the indisputable advantages but somehow glosses over the potential dangers:

As Big Data becomes a fixture of office life, companies are turning to tracking devices to gather real-time information on how teams of employees work and interact. Sensors, worn on lanyards or placed on office furniture, record how often staffers get up from their desks, consult other teams and hold meetings.

Businesses say the data offer otherwise hard-to-glean insights about how workers do their jobs, and are using the information to make changes large and small, ranging from the timing of coffee breaks to how work groups are composed, to spur collaboration and productivity.


[06.17.2013] Here's a blog post addressing the very same issues by Michael Walker, with benefit of hindsight after revelations on PRISM surveillance program:


Bringing spreadsheet to predictive modeling fight

The data is out there. Its tidal waves are sloshing around the world 24x7 only to be deposited in the depth of climate controlled server rooms. It keeps accumulating, and never goes away. These rich deposits of data are available for mining for those with right tools and knowledge of the terrain.

But the time is running out for the individual gold prospectors - their tiny pans (spreadsheets) are no match for dredges (BI suites) sifting through tons data day and night. The data mining focus is shifted from craftsmanship to industrial scale operations; data mining is becoming strategic advantage, a lever in negotiating deals and market positioning.

And then there is predictive modeling. Long domain of actuarial science and financial wizards-quants it is getting into businesses at all levels. If you are a small business owner negotiating health plan for your employees unless you are using some predictive modeling you are doing it based on guesswork and hunches, but your health insurer does not have to guess - it knows. It has models that analyze mounds of data in search for patterns, and then analyze this data across hundreds of different dimensions to come up with a pricing model for type of business you are operating, geographical location, demographics of your employees - in short, they know your business, and the call all the shots in the negotiations.

If you still using spreadsheets to analyze your business and negotiate deals with the partners it is time for an upgrade:  you are fighting predictive modeling cannons with a pen-knife.

Here’s how Pitney Bowes felt in 2001:

<Pitney Bowes> was renegotiating a contract with one of Pitney Bowes’ HMO vendors. After seven years of negotiating favorable rates based on actuarial data, which showed that Pitney Bowes employees tended to be younger (and thus healthier) than average, <they were> stymied. The HMO negotiator had data showing that even though Pitney Bowes employees were younger, they were sicker. And he was using that data to justify a rate increase.

"… every time I said something this guy had an answer. He must be doing something we’re not,’" <HR director>  recalls. That something turned out to be predictive modeling.”