I’m currently in the queue for Oasis tickets. Rather than mindlessly watching the counter of people in the queue ahead of me fall (currently 184,984 people), I thought I’d start writing a blog article. Unfortunately, attempting to write a blog whilst hungry is perhaps a suboptimal approach for mustering a bit of creativity. So instead my thoughts have gone towards lunch, a burger, maybe a hanger steak, or what about rib eye steak. Ultimately, all these things are derived from different cuts of beef. If you ordered a burger and got a steak, you might be somewhat confused, right, even if ultimately they are all made of beef. It’s not only that the cuts are different, they are also likely to be cooked and prepared in very different ways as well.
It’s kind of like that when it comes to data. You might have precisely the same dataset, but precisely how you slice and dice it will give you very different results. The first thing to consider is how you judge what a “good” dataset is? Let’s think about macroeconomic forecasting data, of the type that Turnleaf Analytics produces, primary for inflation, but also various growth proxies like ISM manufacturing. The simplest way to assess how “good” this forecast dataset is to calculate the absolute mean error, between our forecast and where the actual number ends up in the future. You can do the same thing with other forecast sources, and then compare, and pick the best one.
In practice, you might actually combine our forecasts, with other ones, such as your own forecast. For example, if both sets forecasts you are examining are quite accurate, and the errors are not correlated, you will likely observe that their combination produces an even smaller error. Furthermore, we need to consider that it is likely that we will combine not only different forecasts together, but many different types of datasets. This combination of datasets is likely to give us different and likely better results, compared to looking at one dataset in isolation. Given there are so many datasets out there, it is likely that the combination of datasets selected can differ significantly from team to team. On top of that the data analysis which will be performed will likely be somewhat different, with different objectives.
It doesn’t necessarily need to be the mean absolute error in our forecast case. For example, the final objective could be to create a trading signal, whether to buy or sell an asset. Here, we are concerned more with maximising the profitability of those trades, which is a different objective function to minimising the mean absolute error. Of course, we might conjecture that these various errors are connected. However, when we are combining many datasets together the relationship between them can become more complicated. In particular, what matters is how orthogonal our forecasts are to the other datasets, and that will in turn help us in getting to our objective function.
So in summary when trying to understand how useful a dataset is, you might well come up with a different view from other folks, because you will likely combine it with different datasets, and of course the analysis you perform on it can be different. With all these degrees of freedom, it is perhaps not surprising.
Just for the record, I did get to the front of the Oasis queue, and then the website crashed… Oh well! With additional data (ie. the priority ballot), I might well have a better result. Better data, helps with better decision making! Now, where’s that hanger steak I was craving just before lunch…