[006] 模型在生产上无法发挥功效，该何去何从？

“作者从数据的角度去分析了模型失败的原因，归纳成了3点：模型运行速度、训练数据与现实数据分布的差异以及变量特征的不可用。文章正文分别就这3点进行了具体错误行为的罗列以及提出了该如何避免的建议，整篇文章的篇幅不长，但内容还是比较不错，是一次不错的自我反思。

The reasons for this are often down to three major factors:

• The model cannot infer fast enough, be scaled enough or the cost of hardware is too expensive
• Real-life performance is not the same as development estimates
• The data is not available

So, what do I mean by that?

Covering each one in turn we have:

Model Speed

Sometimes we can get a model that while it performs well in testing just cannot scale up to production. If your model is 100% accurate at predicting hurricanes 48 hours in advance, it will not be of any use if it takes 49 hours to make a prediction.

This can be affected by things such as the hardware cost. The costs of the additional complexity (or simply IT costs) need to be outweighed by the gains the model gives you.

You can also find that the time to take the algorithm and deploy it means it's outdated. A famous example was Netflix’s $1million Algorithm Prize. Even though the new algorithm gave the desired accuracy, it just was not the right solution anymore when it came round for deployment.

Real-Life Performance

This can happen where the model in development has [“missed a trick”](https://idioms.thefreedictionary.com/missed+a+trick#:~:text=missed a trick)-,miss a trick,meaning that one is opportunistic.). Often this can be tracked down to the data being used for development and training not being representative of the real world.

I have seen cases where a company has developed a “simulation” of their business to generate their machine learning data and another that forced all its employees to fill out a survey to gather data for a “likelihood to buy” model. Both of these ended as you would expect with the data having absolutely no correlation to real-life and a proper data requirements and collection program having to be instigated so a new appropriate algorithm could be feature engineered and trained.

Data Availability

This is an odd one and one that is often missed but easily avoided with some pre-work, so I shall talk about it the most.

I find that many people rush into designing and training models without setting the foundations right. As for the previous ones, a bit of pre-work in setting out what you want to achieve, what a success looks like and how it will be used (aka. requirements gathering) can substantially reduce your risks.

What happens here is that when it comes around to deploying the model, the data being used is not actually available in the production environment. You see it most often when custom data collection has been started but a sustained collection method is not maintained.

For example, a snack vending machine is machine learning enabled. It can tell you how much stock it needs per day and with that, you can plan your logistics and refilling timetables across your entire network. However, one of the key components to the model is it knowing how many customers it has missed. Now while during collection they had someone monitoring an existing machine to see how many people wanted to use it, in real-life the machine has no way of knowing if someone wanted to use it and just walked past when they saw it was empty. In real-list deployment the vending machines would request less and less stock over time. But why was this?

What was happening is that because the missed customers could not be recorded the model was always being told the missed demand was always zero. This meant the model never upped its stock to compensate but would slowly over time have its stock reduced. This logically makes sense because they could sell out their stock and it would register as a perfect prediction as it had no “missed sales”.

We all know what was happening was that it was missing them, it just did not realise, but because it was a key variable it causes a malfunction in its operation.

This would have been avoided with some pre-thought. By mapping what data your model needed and working back to how it was collected you could have determined there was no mechanism to collect it. It would then give the data scientists the time to either:

• Request a suitable sensor was installed
• Remove the feature from their training

In training the model, they could have run the data through a simulation of the devices sensors to convert it into the expected signal during deployment. This would have led to less embarrassment later and a better model.

If you think about it, this one case actually covers all the major points I gave out earlier. Funny that…

Summary

In this article I have quickly laid out some of the major pitfalls that I see when having to recover a failed machine learning project.

Almost all these failures happen by an eagerness to start a “data science journey” but insufficient attention was taken to map out the requirements of the task.

As the old adage goes:

““measure twice, cut once”（三思而后行）