It's informative to see the history of automated trading to realize that it hasn't been around for very long so applying cutting edge machine learning could still give an edge.
All of these are by David Leinweber, who was one of the first to use AI methods for trading. Dr. Leinweber also founded the Center for Innovative Financial Technology (CIFT) at Berkeley's Haas School of Business, which provides many automated trading related resources for students. Here's an excerpt of his biography, Nerds on Wall Street.
This article from the Journal of Portfolio Management describes how one of the first AI-based systems worked, complete with pictures. "Intelligence amplification" did not end up trumping fully automated systems but I think it's one of a few paths which trading systems could have gone which may still be fruitful in the future.
This next article covers the algo execution arms race. Obviously no one wants to get gamed when they are transacting a large volume.
Finally this Google Tech Talk entertainingly describes the electronification of markets:
http://dl.dropbox.com/u/39904/Blog%20Datasets/Sys3/sys3.m
instead of:
http://dl-client.getdropbox.com/u/39904/Blog%20Datasets/Sys3/sys3.m
The pattern to get the new links from the old ones is to just delete "-client" and "get". Hopefully they will restore the old links since I had posted files on many pages.
This script, sys3.m, is what I run to execute all the IB commands. It defines the futures contracts, turns them into IBContract objects, connects to TWS, downloads historical data, and then processes the data into usable periodic return format. This is not intended to trade real time so do not expect that functionality; however adding that yourself would not be difficult with this as an example.
Here are all the files you need (if something's missing, leave a comment please):
sys3.m (same as above)
connecttws.m
loaddatatws2.m
preprocess.m
twsevent.m (very important)
To test it, download the files into the current Matlab directory, run Matlab, run and login to TWS and wait for it to fully load, then at the prompt type > sys3 and hit enter. Wait until it has said "end" seven times- it's currently configured to download historical data for seven futures, 'VIX' 'ZB' 'YC' 'YM' 'GE' 'ES' 'HE' - "end" signals each data stream download finishing. Historical data from IB will then be in the variable "returndata", ordered by security definitions, so you can play around with it. Look through the code to see what else is going on.
If it keeps saying "waiting", that means there is a problem receiving the data so you probably need to cut it off. On my computer it does not say waiting more than a few times, if any. For those who are interested, it waits for each data stream to arrive before requesting new ones because it is received asynchroniously and this is the simplest way to keep straight which is which.
In sys3.m you can see that I had to tediously enter the contract details for each future from IB's website by hand. I have another function which does this automatically, but it has not been integrated into the main script yet. After running the sys3 script, run this one- reqdet.m (abbrev. for "requestdetails.m"). In the variable "secTest", in Matlab, you will find fully completed contract information for the securities. This only requires specifying the symbols for each security you want in a list and the expiration data for all of them. reqdet.m will make it much easier to download data from a huge list of futures.
I could use help from anyone else working on Matlab/IB in answering two questions:
1) How do you find out which futures/expiries have decent volume? Futures are relatively new to me and compared to stocks, there is no "volume" number.
2) How do you create a stream of historical data that goes back a long time, over multiple expiration dates? I've looked up "continuous contract" and it sounds like what I want, but the implementation is not simple because contracts don't all have the same expiration dates. Perhaps you know of another data source that provides this already (does IB?)?
Also, please share any scripts and funtions you have created to do similar things. I would especially like to see alternative event handler functions. I welcome questions/comments too.
I previously explained why market returns should be lognormally distributed with positive daily expectation (not continuous). However, imagine a security that is artificially constructed to make daily returns opposite of what it is based on. Then it should have negative daily expectation. This is the Ultrashort Financial Sector ETF, SKF, with a dash of leverage added in.
If you look at the chart of UYG and SKF over one year:
it is obvious that the mean of the two return paths is way less than 0%. UYG is the 2x leveraged financial sector ETF and SKF is the inverse of that. SKF is at +32.99% and UYG is at -89.54% from where they were a year ago. Intuitively you probably expect that the sum of the returns is equal to 0%. If not, it should be possible to constuct a pairs trading strategy which shorts both of them and makes a high return, e.g. -1*(32.99-89.54)/2= 28.75% annualized, with very, very low risk (since they move opposite each other daily).
First of all, how are SKF's returns engineered?
Some kind of swap- basically a bet on the direction of the financial sector index put out by Dow Jones.
However I find it more intuitive put another way. Consider this thought experiment:
You think the financial sector is going to drop more over the next 3 days- how will you replicate the daily returns of UYG, inverted? For the sake of example, imagine the price of UYG is currently at $100. Your first inclination is to short UYG and then just stay in that position for 3 days. The first day it falls 10%, you now have $10/10% of available, uninvested capital. The next day it is down another 10%, i.e. -9$ = ($100-$10)*-10%. But this only translates to you having made 9%! Next day, another 10% i.e. $8.1 = ($90-$9)*10%. Now it's way off, only 8.1% when you aimed for 10%. The total is 127.1
Obviously the problem is that you had uninvested profits sitting on the sideline at the beginning of each day. If you cover the short at the end of the first day and then use all your money, $110, to open a new short position on UYG for the next day, when it falls 10% on day 2 you will make 11$. And the next day, covering and reinvesting in a similar fashion you will be up to $133.1 total. The trick is compounding the short position by reinvesting. It's very, very risky because you essentially buy high and sell low to to match the daily returns (remember- buy low and sell high is supposed to be how to make money).
Take a look at p. 18-20 (20-22 of the pdf doc) of Statement of Additional Information for Proshares Trust. The colored tables show exactly how volatility and expected return interact, which I explored in the previous note. It's quite well hidden, even the watered down version has only a tiny little link embedded on the SKF product webpage:
Another "problem" with SKF is its excessive leverage. Using data for the year up to 2/9/09, this Excel sheet shows that the optimal leverage would be .569 . Anything less than one means it's overleveraged. I used Excel's 'solver' add-in to find how much leverage maximized ending wealth, but feel free to test different numbers, including less than 0, de-inverting it. The cell you modify is in orange and the effect on final price can be seen in blue. (fyi spreadsheet methodology: leveraged returns are the daily closing price ratios, minus one, times the leverage multiplier. Finally this is turned back into a stream of prices, with the oldest price on 2/12/08 being the basis- the formulas are simple) Basically SKF is inappropriate for anyone who wants to hold it for a long time because it goes over the optimal Kelly leverage.
I like the Ultrashorts because I'm too young to open my own margin account, but it's hard to look past the steady historical downtrend of the Ultrashort ETFs. However they make for interesting studies in financial engineering and position sizing and probability. I doubt most investors understand exactly what they are getting. I've been doing quite a bit of trading (compared to fundamental long-only "investing") to enjoy the volatility of the past year and the Ultrashorts can be profitable. It's nice having a short position that cannot lose over 100% no matter what, unlike a normal short.
Unfortunately it doesn't look like I've found an arbitrage opportunity. Poor performance is just the result of uncommon volatility. Please leave a comment if you have any ideas related to this or anything else - I always may have missed something.
Recently I've been curious about volatility and leverage. Here's the problem: let's say you make or lose 10% of your total wealth based on the outcome of a coin flip. You make this bet a few times in a row. If you win then lose, you have 100*(1+10%) = $110 then 110*(1-10%) = $99. Likewise if you lose then win, 100*(1-10%) = $90 then 90*(1+10%) = $99. The more you play this game the more money you will lose, assuming the coin is fair. Basically you lose more when you're up and win less when you're down- both are bad.
This is one way of seeing why daily stock market returns must follow the log-normal distribution, which is right-skewed instead of a normal distribution. (in contrast to this Dutch book approach, you can also understand it as continuous returns are normally distributed, but daily/periodic are log-normal). If returns were symmetric, like the coin flipping game above, stocks would be pulled toward zero over time.
And this would increase with volatility. For example, lever the coin flipping game 2x, so it's 20% depending on if you win or lose (even without financing costs). 100*(1.2)*(.8) = $96 = 100*(.8)*(1.2) so now you end up with even less. If the game is positive expectation (such as +15%, -5%), i.e. you have an alpha generating strategy in market terminology, then leverage is a tradeoff between this downward pull of volatility and the upward pull of positive expected value.
It's obvious that 0% leverage give $0 downward pull, we showed 10$ is $1 down, 20% is $4 down, a quick calc shows 100*1.3*.7 = $91 i.e. $9 down, etc. This simple plot illustrates the exponential trend of losses with respect to leverage/volatility:
Fortunately the effect of levering up positive expectation is usually higher than the downward pull of volatility. The losses above are in terms of $ but since the starting amount is 100, in terms of percents it would be the same. The game starts out at 10% volatility so 20% corresponds to 2:1 leverage, 30% 3:1, 10% 1:1 (no leverage), and 0% volatility means all cash/out of the market. If you have theoretical alpha of say 10%, 1:1 leverage will pull it down 1%. 2:1 leverage will pull it down 4%, but also double it to 20% for a net of +7% [=(20-10)-(4-1)]. 3:1 increases the volatility to 30% as above decreasing the return 9% but it raised it to 30% theoretical for a net of +5% [=(30-20)-(9-4)].
You can see in the example above that leverage helps but the benefit decreases (+7%, then +5%, then ...). The alpha rises linearly with respect to leverage but returns are eaten away polynomially (degree 2) with respect to leverage on the bottom by volatility. Eventually the polynomial curve must cross the linear one and increasing leverage will then harm returns. The Kelly formula essentially finds this optimal crossover point. This is a toy example but perfectly applicable to trading.
In the next note I will analyze the ultrashort ETF, SKF. It's very interesting because it has to replicate the returns of the financial sector, leveraged 2x but inversely. So if the financial sector, XLF, goes up 3%, SKF has to go down 6% etc. The really intruiging thing to remember is that XLF's returns must be right skewed, log-normal, or else it would steadily fall by te effect of volatility. However SKF is -1*XLF meaning that it is left skewed and should therefore be negative expectation, and furthermore suffer extra volatility problems because it is double leveraged. My question is how can an creation like SKF be possible in a no-arbitrage environment if it always loses money. And if it is possible, where does the money go that it's losing... {can I have it?}
