UPDATE:
It looks like the service I'm using as my public server broke all their links today. The new link are of this form:http://dl.dropbox.com/u/39904/Blog%20Datasets/Sys3/sys3.m
instead of:
http://dl-client.getdropbox.com/u/39904/Blog%20Datasets/Sys3/sys3.m
The pattern to get the new links from the old ones is to just delete "-client" and "get". Hopefully they will restore the old links since I had posted files on many pages.
This script, sys3.m, is what I run to execute all the IB commands. It defines the futures contracts, turns them into IBContract objects, connects to TWS, downloads historical data, and then processes the data into usable periodic return format. This is not intended to trade real time so do not expect that functionality; however adding that yourself would not be difficult with this as an example.
Here are all the files you need (if something's missing, leave a comment please):
sys3.m (same as above)
connecttws.m
loaddatatws2.m
preprocess.m
twsevent.m (very important)
To test it, download the files into the current Matlab directory, run Matlab, run and login to TWS and wait for it to fully load, then at the prompt type > sys3 and hit enter. Wait until it has said "end" seven times- it's currently configured to download historical data for seven futures, 'VIX' 'ZB' 'YC' 'YM' 'GE' 'ES' 'HE' - "end" signals each data stream download finishing. Historical data from IB will then be in the variable "returndata", ordered by security definitions, so you can play around with it. Look through the code to see what else is going on.
If it keeps saying "waiting", that means there is a problem receiving the data so you probably need to cut it off. On my computer it does not say waiting more than a few times, if any. For those who are interested, it waits for each data stream to arrive before requesting new ones because it is received asynchroniously and this is the simplest way to keep straight which is which.
In sys3.m you can see that I had to tediously enter the contract details for each future from IB's website by hand. I have another function which does this automatically, but it has not been integrated into the main script yet. After running the sys3 script, run this one- reqdet.m (abbrev. for "requestdetails.m"). In the variable "secTest", in Matlab, you will find fully completed contract information for the securities. This only requires specifying the symbols for each security you want in a list and the expiration data for all of them. reqdet.m will make it much easier to download data from a huge list of futures.
I could use help from anyone else working on Matlab/IB in answering two questions:
1) How do you find out which futures/expiries have decent volume? Futures are relatively new to me and compared to stocks, there is no "volume" number.
2) How do you create a stream of historical data that goes back a long time, over multiple expiration dates? I've looked up "continuous contract" and it sounds like what I want, but the implementation is not simple because contracts don't all have the same expiration dates. Perhaps you know of another data source that provides this already (does IB?)?
Also, please share any scripts and funtions you have created to do similar things. I would especially like to see alternative event handler functions. I welcome questions/comments too.
36 comments:
Hello Max,
My good friend Jeff Ryan has written an R package that allows you to interact with Interactive Brokers' TWS.
IBrokers is on googlecode and please bear in mind that it's still in (pre) alpha stage.
Best,
Josh
Hi max,
Yes, the work that Josh and Jeff has done is quite nice as is R :-)
1) "Decent volume" is in the eye of the beholder ;-) You might try some kind of moving average of volume to "see" what is "normal"... Volume data is available for futures from IB. Why you aren't seeing it is another question.
2) You would buy it from a market data provider or record it yourself 8^) Why does it matter that different future contracts expire on different timeframes? Each future has a predictable expiration schedule.
Since you went with matlab It has become difficult for me to follow along ;-(
Cordially,
-Digital Dude-
"I don't know, I can imagine quite a bit..." -Han Solo-
Hi max,
Yes, the work that Josh and Jeff has done is quite nice as is R :-)
1) "Decent volume" is in the eye of the beholder ;-) You might try some kind of moving average of volume to "see" what is "normal"... Volume data is available for futures from IB. Why you aren't seeing it is another question.
2) You would buy it from a market data provider or record it yourself 8^) Why does it matter that different future contracts expire on different timeframes? Each future has a predictable expiration schedule.
Since you went with matlab It has become difficult for me to follow along ;-(
Cordially,
-Digital Dude-
"I don't know, I can imagine quite a bit..." -Han Solo-
Josh,
I'm looking forward to trying out IBrokers, thanks for bringing it up. I would like to contribute if I find features I need that are not implemented.
Digital,
I would like to automate downloading data, so different expiration dates are a problem because it adds something that needs to be entered by hand.
By decent volume, I mean that even though some futures are listed on exchanges, the data is patchy because so few trades are happening. "Volume" would be an simple way to screen these contracts out beforehand.
Regards,
Max
A nice set of back-adjusted futures data here. Can't help on the continuous algo.
Cheers,
Shaun
Nice set of back-adjust data here: http://www.tradingblox.com/tradingblox/free-historical-data.htm
-Shaun
Thanks Shaun, looks good.
Max
Hey Chintan,
Nice to hear from you again.
The SVM system is actually working well for the IB contest. So I guess that's a yes on ML algorithms. You can always control overfitting by using regularization (to penalize "complex" models) or by using crossvalidation.
Regards,
Max
Chintan,
In response to your deleted comment (all comments are saved in my email when they're submitted):
There has actually been less work on ML applied to traditional "pattern" features than pure price streams and other indicators. I think it's a promising approach if you can find a way to generalize patterns. Here's a paper that took a very creative approach to this problem, which changed my thinking on the topic and on interpreting number sequences in general. I don't know of any other applicable papers.
I am enjoying the IB contest and I have no complaints. People can game any system just like Madoff, Stanford, and Satyam (closer to home). It's nice just having the data.
Regards,
Max
Now a days, world is full of jerks but my heart says that you are very good human being.God Bless You.(I know you are atheist but I ain't. ;-) )
Chintan,
I'm always curious to see what you might come up with.
Thanks for the compliment.
Regards,
Max
Max - I use CSI Data www.CSIData.com for my data provider. The software allows you to automatically DL data for a user defined # of closest contracts. I pull data for the closest 10. I just set it up to pull down data at 8:15pm EST using windows scheduler. CSI is the best source I have found. Also, www.tradingblox.com has some free data available on their site that comes from CSI.
Watch out what you get with continuous contracts. There are different ways to construct them, some of which will give you dramatically different results.
Eric
Thanks Eric
Hi max
I have downloaded your sample files and wanted to ask a few clarification questions.
1) Am I correct in my understanding that IBevents is not complete, ie code would be needed in for each case eg 'tickSize'?
2)To see the methods of d in the IBevents file does methods(d) work?
3) I am trying to simply store realtime tick data (price, volume, time) in a matlab matrix or set of arrays so that i can run some matlab functions on it. How would i be able to store the data that is coming back from tws into matlab?
I am not new to matlab but have not warked with java and structs before so i would appreciate your help
Thanks
TC
TC,
1) Yes, IBevents is not complete. If you expand it, please tell me.
2) You can use methods(tws) to see what API methods there are. (Assuming that you named the ActiveX connection variable tws like in my code)
3) I haven't written a function to save the incoming data. You will need to keep in mind that the data can come in asynchronously when you are writing this extension to TWSevent.
Regards,
Max
Hi Max,
You can calculate a 'continuation price' for a sequence of individual futures by calculating their returns and then backcast from the price level of the future of interest (Jun09 say), then not long before the expiry of the March contract switch to using its returns to extend the price level back through time.
You can use returns or arithmetic changes is you prefer.
I hope this makes sense.
Re: volumes & futures you may want to consider open interest instead, but I'm not sure where to find it in the IB API
Thanks for your site, its very cool.
Craig
Thanks Craig, that's more or less what I've been thinking.
Hi Max,
Thanks for this post it was quite informative in terms of linking Matlab to IB on a PC. Unfortunately, the connection setup you've created uses activex controls which (to my knowledge) do not port to MacOSX.
I am trying to determine whether or not you know of any other methods of linking IB to MatLab (other than MatLab2IB) on a Mac.
Assuming there does not exist a method (short of coding it) and given that there is now more literature online (Rmetrics conference) featuring IBrokers and R do you think it is worth the investment to buy Matlab2IB? (on a student's budget).
Thanks in advance,
Sean
Sean,
It should be possible to use the Java API through Matlab. Matlab can interface with Java. If you get it working, please tell me.
Regards,
Max
Thanks Max, I will look into this and keep you posted.
Hi Max,
Thanks for the code.
But I couldn't run it.It gives me this error...
Next valid order id: 1
Unknown event: openOrderEnd
Market data farm connection is OK:ibdemo
??? Undefined function or variable 'today'.
Error in ==> loaddatatws2 at 9
tws.reqHistoricalDataEx(c, securities(c), ...
Error in ==> sys3 at 50
loaddatatws2
Could you please take a look.
Thanks for your help.
VK
VK,
Try replacing the function today() in the code with now()
Regards,
Max
Really great info here! Thanks!! If you are still looking for a way to find out which contracts have volume in IB, you can use the 'Market Scanner' (I think it is under 'tools' tab or something like that) Then you can filter it for futures contracts and sort for volume.
When are working with live data streams in matlab, how do make your code process it in real time? Does it do it automatically or are their special functions? For example, if I want to "buy if the price is > $10 IF Vol is < 3000k", how does my code know that these criteria are met? Do I need a timer to make it check periodically, or is it 'live'?
Thanks AC
Thanks AC I didn't know about the scanner.
To run in real time, you subscribe to a real time data feed through the API. Then it sends you price updates as often as it can. As the price updates arrive, they're processed by your eventhandler file. The eventhandler file will run the system logic and then submit orders.
Regards,
Max
Since I can't find much about this with google, I must be making it more difficult than it is. For some reason I can't get my head wrapped around it. Just to clarify: If I want to collect tick prices in a matrix as they come in, then use this in Simulink, wouldn't I need a trigger to make it append the matrix when it receives a new data value? Or if I want to use the data for an indicator, like a moving average, how does it know to recalculate when a new data point arrives? And is a matrix, in fact, the best way to feed the data to Simulink? How about charting? How would the chart know to update itself? I am in way over my head!! LOL
It seems like gathering live data in matlab would be a common task, but I have had a REALLY hard time finding anything about it on the net. There is the DAQ toolbox, but that seems to be for hardware. Any suggestions?
Thanks!
AC (I presume),
I agree it's unintuitive. The thing to understand is that what looks real time on a computer is actually happening at 30fps or whatever. Like in a video game, the whole screen is redrawn 100% every fraction of a second. Nothing is "updated", it's just totally redone.
Matlab is the same way. When every new data point arrives, you re-run your whole program. The arrival of any message from TWS causes the eventhandler function to run. You modify the eventhandler to have it run your program.
The eventhandler stores all messages in the expanding cell array names eventdata. This cell array will contain confirmation and error messages in addition to data requests so it requires a some user-defined logic to extract the messages (in this case data) that you want.
Regards,
Max
"You modify the eventhandler to have it run your program."
Ahhh. And the lights come on. It makes sense now!
Thanks!!!!!
You're welcome!
Hi Max,
Great stuff, thanks for sharing all this. But the links like sys3.m are broken today. I'm also getting asked about drop box by the next page, do I need to use this to download your files?
Also - do you get notified of comments on ancient posts or do I need to get your attention some other way?
Thanks,
AC (a different AC)
AC,
It looks like the service I'm using as my public server broke all their links today. The new link is:
http://dl.dropbox.com/u/39904/Blog%20Datasets/Sys3/sys3.m
instead of:
http://dl-client.getdropbox.com/u/39904/Blog%20Datasets/Sys3/sys3.m
That's annoying, but do you see the pattern so you can still get them? Just delete "-client" and "get" Hopefully they will restore the old links.
Regards,
Max
Thanks, got them! The formatting was lost but this gives me a good start.
AC
Post a Comment