Bad Questions and Sad Analysts

If there is one thing analysts dread hearing, it is “can you quickly pull some numbers on vague metric?” often followed by “it would be great if you can stick it in a chart showing how this is increasing.” Sometimes this will even lead to a request for a dashboard. Vague metrics often requested include brand health, return on investment (ROI), and engagement. Why are these bad? At first glance surely they seem like reasonable things to inquire about? As with many things in life, the problem is not what is being asked, but how it is being asked. Specifically, there are three problematic aspects to this question.

1. “Can you quickly” ⇒ Other than presuming that the analyst will drop everything to help, there is a more dangerous assumption here, namely that whatever data is required will be readily available. While it is easy for any analyst to dump a volume of data into a spreadsheet or CSV file, it is often unlikely to be correct. Unless the data being requested already lives in a neatly curated data set, the analyst will need to head out into the digital wilds to coax and corral the required facts and figures. This is likely to involve additional work from a data quality and architecture standpoint.
2. “pull some numbers” ⇒ This phrase has probably alienated more analysts than most others. The entire point of analytics is to move from data to insight. It is painfully ironic that the people who are most often stuck at the first link on the value chain are precisely those who should be helping an organisation move along it. Asking an analyst to spit out reams of data is akin to using a helicopter to go shopping at the local grocery store. While possible — landing zone permitting — it is a waste of its power of flight!
3. “on vague metric” ⇒ Some metrics are well defined, e.g. “the number of sales in our store today” or “the amount of money we spent on TV ads last month”. Other metrics are not, and these vague metrics can cause a lot of frustration unless clearly defined and agreed with stakeholders. While brand health, the ROI of company investments, or the level of engagement of customers with content are quantifiable, it is fiendishly difficult to capture these in a single metric without extensive work beforehand. For example, what constitutes engagement? Is it views, visits, dwell time, likes, comments, shares, some combination of these? If the latter, how do you weight them? etc. One number score are never one minute scores.

The good news is that asking better questions and creating happy analysts in the process is not too difficult. Firstly, acknowledge that data nowadays is often disparate, complex, and disorganised. Finding the right data takes time. Secondly, realise that most analysts did not get advanced degrees to mindlessly retrieve numbers. Ask questions that they can solve: What product should we launch next? Is Company A or Company B a better fit as a partner?How do we cut waste in our supply chain? Finally, in cases where a simple metric is required, be specific. Ensure that everyone agrees on what the metric means, how it is calculated, and why it matters to the organisation.

People need autonomy, mastery, and purpose to be fulfilled in their jobs. By asking better questions you can ensure that the analysts in your business will feel more autonomy, get to flex their mastery, and develop a robust understanding of their purpose. Just do not ask them to “pull the numbers”.

— Ryan