Can Virtual Reality Rescue Big Data?



The “Big Data” movement feels stalled.  A Google search for “Big Data is dead” produces a lot of, well, data about the topic.  What started as a noble idea, to capture all of the moving parts of our digital ecosystem and capture them as data points in a model to produce insights, turned into a movement.  Seemingly overnight “Data Scientists” were everywhere and it was the job of the moment.  But then that movement, and all of the promise it offered, turned into an IT hardware and software buying spree.  New staff showed up, with skill sets executives didn’t even understand, to manage this new team or division.  Words like regression and Hadoop were overheard in boardrooms when discussions of next years strategic plan were on the agenda.

We told ourselves this data was the promised land.  We simply needed to sit back, let the brains crunch some numbers with their shiny new toys, and the decisions would start to make themselves.  Heck, we could even fully automate the decision making we thought, which would leave us all more time to count all of the money we were going to make.

But then, like, nothing happened.

Well, not exactly.  We know something happened.  We continued to read articles about the folks at Netflix or Google and all the amazing things they built on top of their data and we felt a faint sense of envy.  An envy that would be stronger if we actually even understood what we should be envious of.  We produced stuff that was challenging to look at and harder to understand.  And the harder we worked at visualization, the more complicated the results and graphs would look, even if they were more colorful.

For the record, I am a optimistic and idealistic believer in data.  It is the new oil.  The opportunity is real and too good to pass up.  Regardless of the over hyped expectations of the past few years, our digital world does indeed generate massive oceans of data every second, and the truth is we ask some of the brightest minds our world has to offer spend countless hours working on it so that we can understand our world better.  We extract, wrangle, clean, analyze, crunch, correlate, visualize and perform many other tasks with respect to this data waiting for meaningful insights to emerge and propel our businesses to action.  We are tracking clicks, sales, downloads, views…..everything.  This data represents a rich multi dimensional look at all aspects of our natural world and lives.

Yet, I also know we force our data scientists to work in rudimentary, flat and two dimensional worlds.  They spend a lot of time staring at monitors, and that’s all we really have for them to work with.  We have lightening fast and sophisticated back end systems, and pair it with front end displays that were no different than the Cold War era displays of the 50’s and 60’s.

That is, until now.

Virtual Reality has the potential to help us construct living and breathing data environments we can walk, talk, and ultimately, work in.  We can unleash our scientists to work in an environment with the potential to match the depth inherent in their data.  The age of fully immersive and virtually real data environments has arrived.  Modern virtual reality display technology has put us on a collision course with massive data sets in a full 360 degrees and three dimensional space.  Ultimately, this could lead to more engaging data which yields more actionable insights. The challenge is to let the magic inherent in the technology lead to magical analysis and business outcomes.  In fact, improving the business outcomes might be harder than the technical challenges.

To be clear, this is exceptionally complex and truly a bleeding edge implementation of a leading edge technology.  A walk on the wild side indeed.  At the same time, the process is actually very simple if you are able to bridge the world of big data and 3D animation at a technical level.  With modern engines like Unreal and Unity, traditionally game development platforms but they also offer an integration point and development interface for 3D animations, many of the challenges boil down to being fairly similar to other large complex data visualization and technical projects.

The work to be done to produce such projects is a mad blend of art and science, a modern renaissance project of sorts that reflects back to a much earlier time when men like Da Vinci smashed together elements of physics, math, science, design, anatomy, and art all in the same project with exceptional precision.


Luckily, our modern age has folks up to the challenge.  Masters of Pie is a London based collection of Artists and Developers who “use cutting edge technology in meaningful ways”.  According to their site, they teamed up with their pals at Lumacode to form team LumaPie after being chosen to participate in the exciting Big Data VR Challenge, a joint initiative from the Wellcome Trust and Epic Games.


For the competition, Masters of Pie worked with data from ALSPAC, specifically the Longitudinal Study of Parents and Children on their ‘Children of the 90’s’ project. As explained by Masters of Pie, the study follows 14,000 children born in the 90’s and their parents and offspring until they reach the age of 70.  The data is a significant collection of characteristics including demographic elements like socioeconomic status, but also includes day to day data points on things like diet and health even including urine and blood sample results.

With the data loaded, the challenge was simple and complicated all at the same time.  They needed to understand the lives of these children at the most fundamental and basic levels possible.  And they were to use Virtual Reality technology, the most brilliant visual technologies our contemporary world has to offer. Their friends at Lumacode provided additional technical expertise, and they got to work.

While most folks would not admit it, that data would end up in Excel and bar charts at many places.  In the end, this is data representing life and activity.  It is layered, rich, has depth and meaning.  And yet, regardless of how much our systems that capture and clean the data costs, we put it in bar charts embedded in PowerPoint decks.  More than anything, folks like Masters of Pie are simply showing us there is a better way.  A modern age needs modern art and science to aid decision makers.  And it has been awhile since I have seen any PowerPoint I would call art.

Modern VR, for all the hope and glory, still has several limitations.  For starters, we are talking about huge amounts of data.  Simply building an environment to view it all can be a challenge.  This is where people like Masters of Pie start  innovating.

Most VR user interfaces today rely on some level of “gaze tracking”, which takes as an input the point a user is staring at.  Masters of Pie has created a system they refer to as “data-peeling”, where they take gaze tracking and trigger deeper layers of secondary information based on a surface layer of gaze tracking control.  This builds in a more manageable control scheme for the user and allows them to filter their field of vision.  Another innovation they leverage for greater user inputs is the Razor Hydra 3d mouse.  The speak highly of its intuitiveness and accuracy when working with different geometric planes in 3D space.

The critical move to make, and we see this movement begin to gain steam with projects like this, is to get to a place where the display technology adds more than just “ooohs” and “ahhhs”.  It should lead to results, which is where all of this should have led from the start.

As I said earlier, I believe in Big Data.  I have worked for years in some of the largest data sets in education and have been lucky enough to be part of teams that produced meaningful insight and innovation.  The opportunity is there and gets better each day as the size of our data continues to grow.  I am more convinced than ever our data can make our world a better place.

Mostly, its a battle against over hyped expectations.  The good news is, that is a battle that can be won.  All we need is for a few modern day Da Vinci’s to take on the challenge and deliver us art that is worthy of our science.

After all, it wouldn’t be an article about Big Data and VR if I didn’t add my own over hyped expectations to the mix.





Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s