How would you measure the success of a ML project?

 You can work haphazardly or you can work efficiently. Finding a good model for a complex problem like brain voxel prediction is fundamentally a Search Problem. There is another important caveat, due to the No Free Lunch Theorem, models that work well for one brain area might fail in other brain areas. I read a paper where they did voxel-wise encoding with low-level Gabor encoders and more high-level semantic encoders. Different area voxel activations were predicted well by models with different inductive biases (priors). 

The Algonauts project provides ROI's for voxels. There are primary visual areas and areas that activate in response to facial features, for example. Part of the purpose of statistics is to aggregate data, so for each model that you train, you could create contrast maps between predicted and actual brain activations and average them within a ROI. You could still display the contrast map, but also show the metrics for specific ROI's. This brings me to the next distinction.

The challenge of the Algonauts project has two main principal components: Semantic and Computational. The Semantic aspect is gathering information about a specific ROI to try to understand what it is computing and this leads to intelligently (not blindly) selecting an encoder model for this model. For example, a model that has been trained to recognize emotions in images with faces would be useful for ROI's that respond to face images. The Semantic dimension requires human insight and contextual knowledge. It's related to Theory-- the Theory of Brain Function. When a model gives a particular result, you might as the question: "Why did I get this result?".

The Computational aspect is more practical. It's about coding, libraries, analytics, visualization, pipelines, machine learning experiment tracking. These tools help to answer question, "What is computed in this brain area?". 

How to model the time spent on these two dimensions? Perhaps you should always work on what is your weakest link and try to strengthen it, because if one part isn't working, everything will come down like a chain of dominos.

In this case, my weakest link is the practical side. I need to learn MLFlow and Streamlit to build analytics and experiment tracking. So great:-) I will focus on that! I will keep blogging about my thoughts and the  things that I learn:-)

I need a visual feedback mechanism. Feedback is a fundamental mechanism behind efficient model search.

This post is the s**t https://towardsdatascience.com/5-tips-for-mlflow-experiment-tracking-c70ae117b03f