MLFlow considerations

What are the things that I need to keep track of in my Algonauts experiment?

Use one representative validation set (if I use a face detector, I would first need to limit the set of images to ones with faces and then generate a training set and a validation set).

1. Model that I am using as a feature extractor (I plan to try out not only AlexNet, but also  other pre-trained models: face detectors, face emotion detectors, sentiment detectors, Visual Transformer) -- keep only the name, in order to not save the weights of the model for every round

2. The layer of the model in a particular regression experiment (I suppose I would need to search over all the layers to find selective cells)

3. The parameters of the regression model-- if I use ridge, regularization factor etc.

4. Prediction accuracy for each voxel-- i.e. around 20000 values for each subject-- it's a brain map. Use variance explained on the validation set. 

5. Log the plot of the prediction accuracy brain map on the validation set as an artifact.

6. Compute the average variance explained for the ROI's and log it, make a bar plot out of it and log it as an artifact as well. 

7. For each ROI, log the distribution of variances explained of the voxels and make a histogram plot out of it (log the hist plot as well). 

The idea is to figure out which models and which features of the models give the best variance explained for a particular voxel and for a particular ROI. These are different levels of granularity. We can then ask the question, are the variances explained heterogeneous in the ROIs. What if we use multiple models or features to predict the voxels in the ROI's?

And what about PCA on model parameters? (see https://www.youtube.com/watch?v=jobQmEJpbhY&ab_channel=CognitiveComputationalNeuroscience)

Okay, these are just the model based feature extractor predictions. What about the components added to the stimulus information by inputs from other areas? How can we include those? These brain areas are also processing stimulus information, so is it appropriate to use orthogonal projections in PCA to explore these effects? How do I design experiments to relate these two sources of information to each other?

I will do divide and conquer. First focus on stimulus derived components of activation and then think about adding the influence of other brain areas.