Comments on Max Dama on Automated Trading: Monitoring a System

JB, Take a look at this source code to use matplo...

2010-02-02T08:13:58.817-08:00

JB,

Take a look at this source code to use matplotlib.finance : http://matplotlib.sourceforge.net/users/screenshots.html#financial-charts. It has a very easy to use function to do that, it's what I use for EOD data and I've never had a problem.

Yes, I use sql.

Regards,
Max

Do you use data only from IB? Wondering if you'v...

2010-02-02T05:07:49.966-08:00

Do you use data only from IB?

Wondering if you've used any available python scripts to get data from Yahoo finance? I'm migrating from Matlab to Python myself and trying to determine if I should write the replacement for the Matlab Data Toolbox myself. I don't use IB.

Secondly, do you use sql to persist the time series data?

Thanks.

Anon, Yes, I have no problem with it. The one st...

2010-01-25T23:00:14.184-08:00

Anon,

Yes, I have no problem with it.

The one strategy I have deployed at the moment has no challenges related to exposure since its holding period is low and doesn't use leverage.

I validate with backtesting and common sense. Some backtests are debatable, some are quite compelling.

Regards,
Max

Max, do you actually let your system make automate...

2010-01-25T18:03:26.511-08:00

Max, do you actually let your system make automated trades with 'real money' or are you paper trading?

I'd be very interested to hear how comfortable you are doing the real thing?

What you do to validate your algos and control your exposure?

thanks

Paul, R isn't a full programming language. The I...

2010-01-25T12:06:00.295-08:00

Paul,

R isn't a full programming language. The IB API conversion isn't complete either.

Emerson,

Thanks for the interesting suggestions, they are new to me.

Regards,
Max

As a sysadmin who is starting to get into automate...

2010-01-25T08:58:39.798-08:00

As a sysadmin who is starting to get into automated trading I have more relevant exp. on the monitoring than the trading :-)

For more advanced monitoring of multiple systems (or many software components) a tool like Nagios is useful. It can be setup fairly easily to watch your database, cpu, ram, disk, and any daemon process you may have developed. One nice feature is that you can write your own plug-ins in any language that will run on Unix. Very customizable for advanced alerting. (Send some alerts to mail, or others to a pager all dependent on time of day as an example.)

If you have a very large/complex environment the ultimate tool is CFengine. It is used by IBM to provide the 'healing' in self-healing servers and by Google for extensive automation of their environment. Advanced, but useful for creating an environment that will attempt to fix problems before notifying you. (Great in a 24x7 shop)

I have used both extensively in large web and compute clusters.

Scripts/wrappers are fine if you have a few items to monitor, but as soon as you have an environment with many moving parts (10+) it makes sense to migrate to Nagios.

For my own environments I use Nagios to watch the 'prod' and 'dev' Mysql instances, TWS, disk space, network connectivity, etc.

Another great tool if you deal with large volumes of log files is ''splunk'. Best thought of as 'Google for syslog'.

Hope that is useful to somebody, somewhere. :-)

Paul, I've not used this myself, but it appears y...

2010-01-25T06:51:00.033-08:00

Paul,

I've not used this myself, but it appears you can have both:

http://rpy.sourceforge.net/

Mat

I recall you used to do R coding? Are you not usi...

2010-01-25T06:39:41.880-08:00

I recall you used to do R coding? Are you not using it anymore?

I'm trying to decide should I learn R or Python for my automated system and quant analysis to replace Matlab at home. I guess you prefer Python, Max?

Mat, That's a really nice example. I use a functio...

2010-01-25T06:04:14.507-08:00

Mat,
That's a really nice example. I use a function based on that to chart PnL and other timeseries. I don't use scikits.timeseries, if that's what you're referring to, because sql handles time series indexing well. With sql I can just use datetime throughout rather than scikits.timeseries Date's too.

Jim,
I haven't used them.

David,
Thanks for sharing, that sounds like a good design. Which hedge fund do you work at? I'm trying to figure out where I should intern this summer.

Regards,
Max

We use Python at the hedge fund I work at. All of ...

2010-01-25T05:16:00.037-08:00

We use Python at the hedge fund I work at. All of our automated tasks are Python scripts and we use a wrapper script to launch and monitor them.

Each task uses the Python logging module to log whatever is appropriate for the job and the wrapper script captures this output in both a file and in a database table.

When the task has completed (success or failure) the entire log is emailed to the development team (we just use the smtplib module because internally we don't need to do special authentication).

The wrapper script also allows us to capture stdout and stderr in cases where we are calling a non-Python executable where we cannot control it's output logging.

Max, Sorry for being a little off-topic, but have...

2010-01-24T20:47:24.133-08:00

Max,

Sorry for being a little off-topic, but have you ever tried Matlab alternatives like Octave or Scilab? I've read that those packages are sometimes not 100% compatible with Matlab projects but I was wondering if you ever tried them.
Thanks for sharing.

Jim

Python is amazing. Check out numpy, matplotlib and...

2010-01-24T19:58:06.776-08:00

Python is amazing. Check out numpy, matplotlib and the timeseries stuff. It's very handy for crunching numbers and graphing.

Here's a simple example

Mat