MacKay

 So I am still working through the first lecture and the first part of David MacKay's Information Theory course and book (available for free online).

Right, so he starts with a repetition code encoder. This is surprisingly important. I mean, it sounds trivial. It's adding redundancy in the most trivial way possible, you simply repeat the bits before transmitting them. But neural systems do it. EnsemblePursuit was an algorithm (check my github) for finding repetition codes in calcium imaging recordings. If the channel is noisy, repetition codes reduce the probability of decoding error when a majority vote decoder is used over the bits.

Usually we go in the other direction. We try to find coordinate transforms that reduce redundancy. A repetition code seems like a strange idea. Exercise: Design a R3 repetition matrix for a vector of bits:-) Hint: You are duplicating the row space, not the column space. How does this affect the sizes of the nullspaces? The column spaces of the transfer matrix remain independent. I mean what would happen if you multiplied it by the transpose and took the PC's (subtr the mean etc)? Can we figure out the PC's without computation? What do these things mean?

I was stuck today. I kept thinking if I can go in the reverse direction of PCA. I squish my data into coordinates along PCs, Vs. But given a prediction on PC's can I expand it back into the dimensionality of my data. And yes I can! I  just multiply the u's by the coordinates and sum together the pieces:-)

This stuff is at the basis of coding and decoding. And statistics is involved!

This course from Harvard: Linear Algebra, Probability and Statistics is Amazing!

https://people.math.harvard.edu/~knill/teaching/math19b_2011/handouts.html

Paradox from the book:  Since data often come with noise, one can simplify their model and reduce the amount of information to describe them. 

So we add redundancy so that we can later remove it to get rid of the noise?