all bits considered data to information to knowledge


Wisdom, the final frontier!

Rob Addy, bloggin' for Gartner:

Analytical prowess will be the battleground for service providers in 2013 and beyond. Are you ready to take the statistical fight to your competitors or will you be on the back foot when the time to run the numbers comes?  Ascending the knowledge pyramid from noise and misinformation to achieve wisdom is not easy.

How exactly the knowledge gets transformed into wisdom is beyond me... the qualitative quantum leaps occur at every step of the pyramid climbing: a rock might be a very large grain of sand but a planet is much more than a giant rock.


I know what you read last summer… And I know what you’re reading now

With proliferation of electronic reading devices we surrender many personal liberties we've taken for granted for so long: now it is possible not only to find what and when you bought a book but also whether you've read it, for how long, on what days of week, at what time, what drew your attention... As Wall Street Journal's article puts it "Your E-Book is reading You".

Convenience comes with many strings attached though. What would electronic equivalent of Bradbury's Farenheit 451 look like? The entire messy business of replacing hard-copy of newspapers and books detailed in Orwell's 1984 went away replaced by infinitely malleable bits and bytes. Nobody misses developing films - what about times when a photographic negative was a considered an irrefutable proof?

Personal reading experience becomes a raw material for data analysis, and I, for one, am rather uneasy with this brave new world. This adds to yet another piece of puzzle for constructing your personality on social networks, where people and organizations with BI savvy are mining your personal experiences in hopes to sell you ever more stuff (e.g. How Target Figured Out A Teen Girl Was Pregnant Before Her Father Did) - or for some other, not always as benign, reasons.



Enhancing analytics at employee store

The other day I was watching my kids trying on running/athletic/training shoes in the Nike Employee store (thanks to an acquaintance of mine!), and could not hep but wonder whether the company's missing an opportunity here. All ingredients were present:

  • a guest must register at the entrance with a valid ID.
  • a guest pass must be presented at the check out.

Granted, this gives Nike opportunity to track demographics and selection preferences (no doubt they are already doing this). But there could be more - with a little bit of RFID tracking they could also catch a decision making process. All they need to do is to add an RFID tag to each pair of shoes they selling (a dime or so per tag), and an RFID reader in every bench a customer takes the shoes to try them on (a bit more expensive but not prohibitively so).

Each pair of shoes within certain temporal proximity to the pair of shoes that has been actually bought could be considered as point in the selection process; accuracy could be increased by factoring in shoe size variation (e.g. size 8 and size 11 likely to be unrelated while sizes 10,10.5 and 11 would be), target gender etc.

Now the company has got not only the customer's age, gender and shoe preferences, they also have captured a record of the shoes that the customers looked at closely enough to be tried on but which were ultimately rejected... and this might help them to make some decisions of their own .


Lots of little brothers… all watching you

Predictive analytics at its best... and worst.  Charles Duhigg's article How Companies Learn Your Secrets published in New York Times opens a big can of worms here. The truth is that we are getting better and better with predictive analysis aided by ever powerful computers and software, and better mathematical models... and we are getting closer to the point where our secrets do not even have to be stolen as they could be inferred from mountains of tiny clues we left behind as we are going after our daily lives.

The key to make this happen, the facilitator is unique identifiers we acquire with our credit cards, loyalty cards and other numbers that could be used to track your activities. It has its uses - such as prevent fraud, prepare for an eventual disaster and so on.. But there is more insidious side to the predictive nalytics - instead of Big Brother watching we have hundreds of small ones actively engaged into collecting and trading our personally identifiable information - something we are only too happy to give away for a few pennies in discounts on overpriced merchandise.  So goes our privacy - not with a bang but with a whimper



Who mines the miners?

Organizations like to keep their cards close to the chest.  For a long time BI/analytics was all in-house affair: tools, skills and - especially! - data. The shift towards distributed computing models such SaaS and PaaS change everything.

The data needed for analysis might not be owned by the company; it might live - virtually - anywhere: public domain, subscription service, social networks such as Facebook, geographical data from Google Maps or Microsoft Earth. This is the secret ingredient for the analysis, and just as every true secret it hides in plain sight.

SAP has announced that its flagship analytics BI - Business Objects 4.1 - will have even tighter integration with Google Maps API, going beyond location services…

One can’t help but wonder  what data Google gets to keep for its own analytic endeavors as it tracks each call to its services.  Could it be that the corporate secrets are leaking out through usage patterns?


Bringing spreadsheet to predictive modeling fight

The data is out there. Its tidal waves are sloshing around the world 24x7 only to be deposited in the depth of climate controlled server rooms. It keeps accumulating, and never goes away. These rich deposits of data are available for mining for those with right tools and knowledge of the terrain.

But the time is running out for the individual gold prospectors - their tiny pans (spreadsheets) are no match for dredges (BI suites) sifting through tons data day and night. The data mining focus is shifted from craftsmanship to industrial scale operations; data mining is becoming strategic advantage, a lever in negotiating deals and market positioning.

And then there is predictive modeling. Long domain of actuarial science and financial wizards-quants it is getting into businesses at all levels. If you are a small business owner negotiating health plan for your employees unless you are using some predictive modeling you are doing it based on guesswork and hunches, but your health insurer does not have to guess - it knows. It has models that analyze mounds of data in search for patterns, and then analyze this data across hundreds of different dimensions to come up with a pricing model for type of business you are operating, geographical location, demographics of your employees - in short, they know your business, and the call all the shots in the negotiations.

If you still using spreadsheets to analyze your business and negotiate deals with the partners it is time for an upgrade:  you are fighting predictive modeling cannons with a pen-knife.

Here’s how Pitney Bowes felt in 2001:

<Pitney Bowes> was renegotiating a contract with one of Pitney Bowes’ HMO vendors. After seven years of negotiating favorable rates based on actuarial data, which showed that Pitney Bowes employees tended to be younger (and thus healthier) than average, <they were> stymied. The HMO negotiator had data showing that even though Pitney Bowes employees were younger, they were sicker. And he was using that data to justify a rate increase.

"… every time I said something this guy had an answer. He must be doing something we’re not,’" <HR director>  recalls. That something turned out to be predictive modeling.”