In case the code below is too mangled, I uploaded the txt file version of the following note here.

One of the main challenges for beginners with Interactive Brokers' API is that it's asynchronous- you say reqHistData and then nothing happens, and then some fractions of a second later a totally different method gets fired with the data you requested as an argument. For many systems, this slight speed optimization is unnecessary. It only helps when you're requesting many different things all at once. For the speed increase, the API is significantly less beginner-friendly.

I'll explain how to make the API synchronized so that when you call an API function, it returns what you requested directly, just like a normal function. After googling it, in the technical terminology, it looks like I'm converting the API from "non-blocking asynchronous" to "blocking synchronous". You'll need IbPy if you don't already have it.

This is inappropriate for systems that rapidly react to streams of intraday data, but is useful for requesting historical data, portfolio information, executions, the time, and the next valid id.

Let's review how the API works:
The two key parts are:
1) EClientSocket - sends requests
2) EWrapper - receives requested values

After requesting something through the EClientSocket, you need to wait for a bit until all the requested values have hit the EWrapper.
In general to syncronize things we'll follow this pattern:
def syncronized_request(x):
myEClientSocket.request_x
wait
return myEWrapper.x

'request_x' is a usual EClientSocket method which you can look up in the API doc.
'wait' is defined like this-
def wait():
inc_secs = .2 # a fifth of a second
while not myEWrapper.isDone:
sleep(inc_secs)
myEWrapper.isDone = False
'myEWrapper.x' pulls the requested data out of the local variable 'x' where it has been stored in your overwridden EWrapper class.

First import some libraries:
Boilerplate Code (1)
Then create your event listener subclass:
Boilerplate Code (2)

Then instantiate the requester and receiver:
myEWrapper = syncEWrapper()
myEClientSocket = EClientSocket(myEWrapper)

Now we can write the two simplest syncronized API functions:
def getTime():
myEClientSocket.reqCurrentTime()
wait()
return myEWrapper.time

def getData(contract):
myEClientSocket.reqRealTimeBars(tickerId=1,
contract=contract,
barSize=5,
whatToShow='MIDPOINT',
useRTH=1)
wait()
myEClientSocket.cancelRealTimeBars(tickerId=1)
return myEWrapper.bar

Notice that we already added the variables 'time' and 'bar' to myEWrapper

Let's add two more functions to hide the implementation:
def connect(clientId):
myEClientSocket.eConnect('localhost', 7496, clientId)

def disconnect():
myEClientSocket.eDisconnect()

Run Boilerplate Code (3) to define my simplifying etf contract factory

Now we can use the API easily, just call:
spy = contract_etf('SPY')
connect(1)
getTime()
getData(spy)
sleep(1)
getData(spy)
disconnect()
exit()

Now that you see it works, you can look back and see how it worked
Feel free to reuse or modify my code

------------------------
Boilerplate Code (1)
from ib.ext.EWrapper import EWrapper
from ib.ext.EClientSocket import EClientSocket
import datetime as dt
from time import sleep

Boilerplate Code (3)
import ib.ext
def contract_etf(sym):
c = ib.ext.Contract.Contract()
c.m_symbol = sym.upper()
c.m_secType = 'STK'
c.m_exchange = 'SMART'
c.m_currency = 'USD'
return c
Boilerplate Code (2)
Subclassing EWrapper is the most intimidating part at first. Here's my subclass of it, which you can just copy:
class syncEWrapper(EWrapper):
## variables to store requested data
isDone = False
time = None
bar = None
## overwridden listener functions
def author():
return 'Max F. Dama'
def tickPrice(self, tickerId, field, price, canAutoExecute):
raise NotImplementedError()
def tickSize(self, tickerId, field, size):
raise NotImplementedError()
def tickOptionComputation(self, tickerId,
field,
impliedVol,
delta,
modelPrice,
pvDividend):
raise NotImplementedError()
def tickGeneric(self, tickerId, tickType, value):
raise NotImplementedError()
def tickString(self, tickerId, tickType, value):
raise NotImplementedError()
def tickEFP(self, tickerId,
tickType,
basisPoints,
formattedBasisPoints,
impliedFuture,
holdDays,
futureExpiry,
dividendImpact,
dividendsToExpiry):
raise NotImplementedError()
def orderStatus(self, orderId,
status,
filled,
remaining,
avgFillPrice,
permId,
parentId,
lastFillPrice,
clientId,
whyHeld):
raise NotImplementedError()
def openOrder(self, orderId, contract, order, orderState):
raise NotImplementedError()
def updateAccountValue(self, key, value, currency, accountName):
raise NotImplementedError()
def updatePortfolio(self, contract,
position,
marketPrice,
marketValue,
averageCost,
unrealizedPNL,
realizedPNL,
accountName):
raise NotImplementedError()
def updateAccountTime(self, timeStamp):
raise NotImplementedError()
def nextValidId(self, orderId):
pass
def contractDetails(self, reqId, contractDetails):
raise NotImplementedError()
def bondContractDetails(self, reqId, contractDetails):
raise NotImplementedError()
def contractDetailsEnd(self, reqId):
raise NotImplementedError()
def execDetails(self, orderId, contract, execution):
raise NotImplementedError()
def updateMktDepth(self, tickerId,
position,
operation,
side,
price,
size):
raise NotImplementedError()
def updateMktDepthL2(self, tickerId,
position,
marketMaker,
operation,
side,
price,
size):
raise NotImplementedError()
def updateNewsBulletin(self, msgId, msgType, message, origExchange):
raise NotImplementedError()
def managedAccounts(self, accountsList):
raise NotImplementedError()
def receiveFA(self, faDataType, xml):
raise NotImplementedError()
def historicalData(self, reqId,
date,
open,
high,
low,
close,
volume,
count,
WAP,
hasGaps):
raise NotImplementedError()
def scannerParameters(self, xml):
raise NotImplementedError()
def scannerData(self, reqId,
rank,
contractDetails,
distance,
benchmark,
projection,
legsStr):
raise NotImplementedError()
def scannerDataEnd(self, reqId):
raise NotImplementedError()
def realtimeBar(self, reqId,
time,
open,
high,
low,
close,
volume,
wap,
count):
self.bar = (time, open, high, low, close, volume)
self.isDone = True
def currentTime(self, time):
self.time = dt.datetime.fromtimestamp(time)
self.isDone = True
def fundamentalData(self, reqId, data):
raise NotImplementedError()
def error(self,*err):
print err



Here is a video of me executing the code so you can see how if all the jumping around was confusing.

I previously wrote a note on calculating the optimal amount to leverage your portfolio to maximize compound growth using your trading strategy's historical returns.


I tried to extend it to use the actual data rather than a histogram approximation. So I just ran it with all the historical returns (not binned) assigning uniform probability to each historical return observation. It gave nearly the same results as the histogram approximation, but it was more accurate in cases where the binning had been done heavy-handedly. To figure out why this was the case, I looked back at the expected log growth formula, sum(p_i*ln(1+f*r_i)). From that it became obvious, as perhaps it should have been from the start, that just using point masses was fine. If you have multiple events with the same return you can add the probabilities if you like by expanding the sum and refactoring terms.

So now in my Kelly fraction computing code the histogram approximation (empirical density optimization) is optional.

I also improved the optimization routine so that it no longer uses brute force. The details are not important, but the basic idea is that now it does hill-climbing, and uses information about the steepness of the optimization surface at each point to jump faster.

Thirdly, I added a bit to simulate the effect of adding a stop-loss. It aggregates the probabilities of all the returns which were below the stop at the stop percentage itself. That's how it works. Easily abusable, but possibly an interesting way of bending the empirical distribution. It assumes your stop is based on a percent rather than dollar loss amount, etc.

Here is the code. First copy the kelly, empirical_distribution, and expectation functions into the command line. Then run the example code in the comment at the top. The example illustrates a nice advantage of this method of optimizing leverage rather than the analytic way which assumes Gaussian returns. You'll need the widely used numpy, matplotlib, and scipy libraries, which I believe give you access to pylab as well. If not you need that too. Notice that you can suppress the extra output by changing the disp=1 and full_output=1 arguments to 0 in scipy.optimize.fmin_cg.

The one issue it has is that sometimes the optimal leverage will be at a point that would make one of the (1+f*r_i) quantities in the sum above less than 0, which then causes the log(1+f*r_i) to be undefined, so the whole sum (i.e. objective function) is undefined, and it breaks the optimizer (it will just hang forever as far as I can tell). For example in the case of a Gaussian with mean 0.05, sd 0.10, the optimal leverage will be .05/(.1^2)=5, but of course the return values that have been observed could be, say -25% (i.e the support of the Gaussian is over (-inf,inf)), so 5x leverage pushes it under -100% and causes the undefined error. I'm not sure how to correctly handle this, or what the right answer really is.

The next step is to add a parameter that allows you to include transaction costs and releveraging frequency in the optimization since higher transaction costs result in a lower optimal leverage. This is because if you use leverage then any change in your holding's prices will require you to releverage. For example: you have 3.0 leverage and $100 in your own cash ie a $300 portfolio. Then the price goes up 100%. You now have a $600 portfolio, but $400 is your cash now. So your leverage is only 1.5. Assuming the Kelly fraction is 3.0, you need to buy more shares, which incurs a cost. Etc. The difficulty in computing effects like these on the optimal leverage are what causes people to fall back on heuristics like half-Kelly (well, that and black swan events).

I'd like to hear anything you know about this topic if you have anything to share.

Here's some interesting information shared by RTS:


I wonder about other non-US exchanges.

I've been using Python with IbPy, rather than Matlab. The code I post will most likely be in Python going forward. I like it because it's more natural, ex. default and variable arguments are concise, no semicolons, dictionaries; it's open source so there are more libraries and it's free (Matlab requires a "license server" for instances running in parallel, for example); and it's easier to organize code with better package management, simple classes, and files are more flexible.

Anyway, I like to know what's going on with my system when I'm away. The system writes out a log of errors, trades, finished processes, etc, and also emails me important events like Trader Workstation shutting down, and executed trades. These go to my Blackberry wherever I am. To send emails from/to a Gmail account, which I like to use because it has a lot of storage and can be searched easily, I had to write a little script (code at the bottom) to add to the Python logging library since Gmail uses a special kind of authentication not supported by the standard library.

Here's how I use it:
import logging
logging.config.fileConfig("/yourdirectorystructure/logging.conf")
logging.debug('Live trading online')
logging.critical(strategy_name+' '+signal+' '+position_size+' '+etf)
What that says is: load Python's logging library (which does most of the work for me); then configure it according to my preferences in the file logging.conf; then as an example send a debug-level message 'Live trading loop starting'; and then a critical-level message with the strategy name, buy/sell signal, position size, and ETF name. These two messages are taken out of context from one of my strategies, so the variables don't reference anything here. The first is called when I turn on the system, the second is called whenever a trade is executed.

It looks like this in the inbox:

logging.debug(msg) sends a debug(low)-level message to be logged, in my case it prints to a console window and writes to a file. logging.critical(msg) sends a critical(high)-level message, which goes to the console and file, and also gets sent to my email address.

It used to be frustrating to wake up in the morning and see that my system had crashed at some unknown point in the night for some unknown reason, but now I know what's going on, when it's happening, and why.

Python's logging library isn't documented very well, especially the email handler and file configuration, so I thought sharing these two parts would be useful. Here's my configuration file, from above (modify it to set the directory where you want to store logs), and the custom Gmail email handler (modify it for your own username/password). The overall logging system works well and adds the minimum amount of superfluous code.

I'm interested in knowing how others have approached the problem of monitoring and logging.

It's rare for a gap to open between two nearly identical [platinum] ETFs/ETNs. But it still happens and it was tradeable too, I did. Visual evidence against efficient markets:


A friend of mine recently started using Interactive Brokers through Python on Linux, and referenced the installation guide in my previous note. He wrote a much better, clearer set of instructions. Python is a really good language to code in and Linux is easier to work in and has less overhead processes than Windows.

percent sign (%) lines are comments, other lines are to be entered in a terminal.

%-------------------------------------------------------------------------
%---
%1. Run Ubuntu 9.10 Installer
%---

%---
%2. Update ubuntu 9.10 with the latest everything
%---

sudo apt-get update

%there were 159 MB of updates on 12-17-2009

% If you have any problems while running the update, run this to resume the
% update (important if you are downloading a few hundred MB of updates).

sudo dpkg --configure -a

%install flash

sudo apt-get install flashplugin-nonfree


%---
%3. Download trader workstation
%---
%URL:
%http://interactivebrokers.com/en/control/systemstandalone.php?os=unix&ib_entity=llc
%
% Or you can just run the following command
%

sudo wget http://www.interactivebrokers.com/download/unixmacosx.jar

%run this in your home dir or your downloads dir and it will download the file

%---
%4.Install Python, using the following commands
%---

sudo apt-get build-dep python-numpy python python-scipy python-gtk2
python-imaging ipython

%this is about 326 MB as of 12-17-2009

sudo apt-get install python-numpy python-profiler python-gtk2
python-scipy python-matplotlib python-matplotlib-doc ipython pychecker
pylint pyflakes python-doc python-examples python-imaging
python-scientific python-serial python-docutils python-glade2

%this is about 80 MB as of 12-17-2009

%---
%5.Install and Configure Java and then Install and Configure Trader
Workstation (TWS)
%---

%install the java runtime using the following commands:

sudo apt-get install sun-java6-jre sun-java6-plugin sun-java6-fonts

%install the Java Dev Kit will give you the jar command necessary to install TWS

sudo apt-get install sun-java6-jdk

%test the java installation with this command

java -version

%make a directory for IB TWS

mkdir /home/YOUR_USERNAME/IBJts


%move the IB download from the download directory to the home directory

cp ./unixmaxosx.jar /home/YOUR_USERNAME/

%extract the TWS files

jar xf unixmacosx.jar

%install TWS

java -cp jts.jar:pluginsupport.jar:riskfeed.jar:hsqldb.jar:jcommon-1.0.12.jar:jfreechart-1.0.9.jar:jhall.jar:other.jar:rss.jar
-Xmx256M jclient.LoginFrame .


%---
%6.Install IBpy
%---

%download ibpy wget IbPy-0.7.6-9.51 (current as of writing 12-17-2009)

wget http://ibpy.googlecode.com/files/IbPy-0.7.6-9.51.tar.gz

%extract

tar xzf IbPy-0.7.6-9.51.tar.gz

%go to the directory

cd IbPy-0.7.6-9.51

%setup

sudo python setup.py install

%run TWS

java -cp jts.jar:pluginsupport.jar:riskfeed.jar:hsqldb.jar:jcommon-1.0.12.jar:jfreechart-1.0.9.jar:jhall.jar:other.jar:rss.jar
-Xmx256M jclient.LoginFrame .


%run IBpy demo

python IbPy-0.7.6-9.51/demo/example_opt