Navigating Data & Analytics Technologies

During the Age of Discovery, explorers set out to find new lands and establish new trade routes for their countries. Facing terrible storms and fatal scurvy, perhaps the greatest danger came from the lack of reliable maps. Knowing how to return home safely was no trivial matter. With limited knowledge of treacherous oceanic phenomena and weather systems, cartographers would attribute sailors’ trials to monsters with which they decorated their work. (Interestingly, with one, maybe two exceptions, not “here be dragons”.)

At times it can feel like the landscape of data and analytics technology is fraught with similar obstacles. Even a simple question such as “how should I store my data?” or “what type of analytical tools should we use?” can lead to lengthy research and deliberations. Not only do organisations have many different core technologies to choose from — for one example, see the Apache foundation’s data storage projects — but these have been packaged and evolved by many solution providers. Navigating these options is not easy.

Is this explosion of solutions not a good thing? Certainly we are spoiled for choice when it comes to technology? While in some aspect the range of options provides benefits, it can also lead to three significant challenges.

The first one of complexity. While the idea of analytics is easy to comprehend, the range of solutions can impair decision makers’ judgment and draw the focus to technology, as opposed to the questions they should be answering.

Secondly, while cloud offerings can be more straightforward to deploy, many solutions still involve significant investment. This can lead to choice paralysis. “Before we go ahead, how do we know this is really the right decision?”

Finally, attracting talent becomes more difficult as an increasing number of specialised tools are implemented. You can find someone who knows Python, but will they be able to debug a production issue in your niche solution?

How can companies deal with these issues? Firstly, it is worth keeping in mind that the platform you use for analytics is only a part of the overall equation. Who you hire, for example, is far more important. Secondly, establishing a culture of experimentation and continuous learning will make it easier to rapidly evaluate options. Finally, it is worth understanding the technical concepts in enough detail (e.g. MapReduce) to understand which products are truly different, compared to those that offer variations of the same.

Understanding how to navigate this increasingly complex solution landscape involves simplifying as much as possible, and appreciating that whatever choice you make it is only a step— and not even the most important step— on the journey to unlocking the value of, and delivering impact with your data. Exploration always comes at a cost. Magellan, known for circumnavigating the globe, died en route (although he did discover a penguin, so there’s that).

— Ryan