Comments on Max Dama on Automated Trading: Overfitting II: Out-of-Sample Testing

Carla, Overfitting would occur when you tweak par...

2009-07-26T10:26:12.259-07:00

Carla,

Overfitting would occur when you tweak parameters to make the backtest look better. Once you start modifying the optimization period you will overfit at least a little. But just because it looks good doesn't mean you overfit.

The line between fitting the model and overfitting is impossible to see clearly. Just try to get a feel for the signal/noise tradeoff and then play it safe.

If you'd like to get others' opinion on the software, you can write something up and I'll post it on your behalf (citing you of course) or you could try posting on a forum like elitetrader.com or quantnet.org.

By the way, your site looks very good.

Regards,
Max

I am not in any competition, just someone trying t...

2009-07-26T10:17:15.691-07:00

I am not in any competition, just someone trying to save a life insurance policy that I have borrowed against from tanking. I get lots of advice from people who want me to cash out and buy their annuity. I just want to make this policy grow. at 8+%.
I have a limited universe of about 12 funds. If I sell the policy, I will owe $100000 in taxes and be a pauper for the rest of my life. I have been doing my own trading for 20 years, through a lot of harrowing times. Any thoughts, advice, ideas, help out there?
Besides what the 5 insurance salesmen already told me?

Carla

Would anyone like to concur on Optimal Trader? I h...

2009-07-26T10:04:21.884-07:00

Would anyone like to concur on Optimal Trader? I have been looking at it for about a month. I think I understand the model fairly well. I have not traded on it yet because I don't understand how to set the optimization period.

Carla

Hi, I have subscribed to Optimal Trader. I think ...

2009-07-26T10:01:03.950-07:00

Hi,

I have subscribed to Optimal Trader. I think I have figured out everything, except how not to overfit. I am supposed to look for peaks, and make sure that the buy signals don't come only in the dips and the sell signals don't come only in the peak. It seems to me that the signals are somewhat random. I guess that means that mean that I need to shorten the optimization period?

Erk,I think fears of having an idea stolen and arb...

2009-01-11T19:53:00.000-08:00

Erk,

I think fears of having an idea stolen and arbitraged away are overblown. Some strategies capitalize on very specific arbitrage opportunities that no one else has noticed, but this SVM approach is very general. It could apply to any tick size, any instrument, etc.

My email is maxdama at berkeley.edu
It's good to know that the notes make sense to at least one other person.

Regards,
Max

Yeah, I also thought the same when I first saw you...

2009-01-11T18:57:00.000-08:00

Yeah, I also thought the same when I first saw your plan few days ago. Fortunately we are trading on different markets using different vehicles. It is good to see that someone else is using a similar reasoning after all. I really enjoyed your blog btw. I have a couple of different variations of 'SVM on time series' papers that you haven't post here. I will send them sometime tomorrow, but first I need to finish my code now. ;)
cheers.

Erk

It sounds like we are taking very similar approach...

2009-01-11T18:20:00.000-08:00

It sounds like we are taking very similar approaches Erk.

Max

Hi,At the moment I am just optimizing two paramete...

2009-01-11T12:31:00.000-08:00

Hi,

At the moment I am just optimizing two parameters of a RBF C-SVM for each classifier. I have around 10 of them. The test data size is fixed and decided by a simple heuristic(should be big enough), so no optimization there. It is in the todo list with lots of other things ;)

Best,

Erk

erk,How big is your out of sample set and how many...

2009-01-11T10:30:00.000-08:00

erk,

How big is your out of sample set and how many pairs of parameters do you test? The out of sample set should roughly be larger than the number of parameter pairs. You should plot a surface of performance with each parameter pair to determine if there might be overfitting, in which case it will not be smooth.

Regards,
Max

Hi Max,Reading your last comment I felt the urge t...

2009-01-11T10:14:00.000-08:00

Hi Max,

Reading your last comment I felt the urge to share that I am also having similar difficulties at the moment. I am also an attendant of IB Olympiad this year and utilizing a support vector machine approach. I am trying to use SVMs for classification purposes though and it seems that parameter optimization for out of sample set yields overfitting for this set. To combine it with a boosting approach again does not help too much for generalization. Anyway, good luck in the competition. Cheers.