Overfitting means that you are fitting both the predictable component of the system and the noisy part. It is useful to think of the the data as being generated from a predictable system with some element of noise added in.In a chemistry application this might mean that you can explain a chemical reaction with 90% accuracy, and the 10% of unexplainable variation is due to human measurement error, imprecise tools, and microscopic imperfections or dirt on the test tube. No matter how well we understand chemical processes, we cannot predict with 100% accuracy every time.

An equity trend prediction system might break a price series down this way:Here we are assuming that our system is only fundamentally capable of predicting big trends and not small price movements. For example it might be trained on EOD data so intraday movement is unexplainable.

Overfitting occurs when our learning algorithm fits the noisy portion of the data rather than the trend. It's especially hard to deal with in a trading system because price movements seem to be primarily noise. The algorithm is basically distracted from the trend by all the random movement.

Here are two models an algorithm might learn:
We do not trust the model on the right because in the future it may wiggle anywhere.

There are a few ways to avoid using an overfitting model. The first way is to realize that overfitting occurs when the algorithm "memorizes" the training sample in its parameters. So by limiting the number of parameters it will help prevent the problem. In an SVM this means keeping C lower or, in the case of a polynomial kernel, keeping the degree to something reasonable (which depends on the system being modelled). In a neural network this means limiting the number of hidden layers and nodes. Each neuron's weight helps learn the training data, with too many neurons, the data is essentially memorize. Ernie Chan recently wrote about a parameterless system.

Another way to avoid overfitting is to cross validate the model. Train it with part of your historical data and then test in on the unused part. Repeat so that all the data is used as test data in one of the iterations of cross validation.Using cross validation you can get a good idea of what to expect when you apply your model to unseen data. If the model is overfitting, performance during cross validation will be poor and possibly inconsistent. The disadvantage is that cross validation takes computation time since it basically re-learns the model the number of times you cross-validate (e.g. 4X in the diagram above). With neural nets it can take a very long time.

There are multiple ways to think about overfitting and they each help intuition. I feel like I should say more about splitting the data into noisy and explainable components but I hope it will be clear enough already. Just realize that what is noise to one model may be explainable by another and vice-versa. Combine two models and you may be able to expand the explainable portion and shrink the noisy part. But that's another topic.

Anyway please comment if something is unclear or you have something to add. School resumes in a week so my schedule will be impacted soon.