Algorithmic Trading: A Practitioner’s Guide

You had my upvote at Lisp :P

It probably has so many upvotes because it's quite a novel topic for HN. You might not appreciate it as much, being in the industry, but algorithmic trading is a field most of us haven't explored. Most of the content on HN is focused on web design to be honest, so things outside of that can get quite a lot of traction.

Do you have some recommendations on good books on trading systems? I work in the energy trading world, and in general the systems there are behind the financial world by ~10 years.

I hope to learn more about this topic!

How complicated were your algo trading strategies? And have you thought of translating these to non-crypto?

Did you measure your returns in USD or ETH/BTC?

I've come across many profitable traders in Crypto who only measure in USD and when you compare their results to just buying and holding ETH/BTC at the start of the cycle they come out behind. Everyone feels like a great trader in a bull market.

It works, but do not expect huge returns.

I have worked on multiple strategies. - Strategies using technical indicators do work, but you have to reasonable. If you find these giving higher than expected returns, or too many consecutive wins - take the money. Stop live trading and continue dummy trading - eventually there is a point where you can start live trading again. The thresholds will be determined from backtests.

- Statistical strategies work for swing trading. However, these are very difficult to figure out. Need a lot of data (for backtesting). These work consistently over a longer period and might look loss making over a small period.

- Scalping within 5-10 minutes works pretty consistently, signals based on options data.

- Understand and realise the law of large numbers, and use that to your advantage.

- Give importance to understanding the concept of Time. There is a whole lot of weird and scammy pseudo-science around it. Therefore do not blindly rely on one theory.

You're way, way better off finding a small outfit of motivated people to join if you want to work in this field.

Success rate for newly formed hedge funds is probably something like 50% over a 5 year horizon. For solo traders, it's probably closer to 1%.

There is WAY too much stuff you need to be on top of, if you want to be successful, than a single person can manage. It's about as feasible as starting a one man (coal) mining operation, where you must dig the tunnel, extract the coal, process it, sell it, manage the business operation and retain your sanity. You need at least a handful of people to be effective, all with different specialties.

...not to mention that if you're successful as a solo trader, you could be making 10x more with proper financial backing. If your strategy is returning 25% annually on a personal 1 mil investment, there's a good chance it'll still be returning >20% on a 10 mil investment (once you get to 100 mil or into the billions, your returns start to diminish due to market impact). Find a good backer, they'll literally give you money and a support team for a piece of the profit.

I've been attempting my own strategies for around four years now, after being inspired by a data mining Master's course I was taking. It has been a long and hard road for me, and I have mostly been trying to find correlations up into this year. As a one man show with a wife and kids, my project time starts at 10pm and sometimes goes until 3 to 4am. Then I wake up at 7 and take the kids to school and work my day job. I can usually do this two or three times a week, but lately I've been going to bed early as it's been wearing me out. I've had so many 4am nights that it's probably been a bit detrimental to my health at this point.

I'm not doing any order book or HFT stuff. It's mostly mean reversion < 15 minute trades. I have a considerable amount of options trading experience and that's what my main strategy revolves around. I don't have a killer strategy that some of these algorithmic firms are capable of, but the current one is promising. I also have a fairly rudimentary form of backtesting, which is a Jupyter Notebook crunching data I've collected over the years. I use some ML python libraries as a bonus, but I haven't created any strong enough attributes yet to achieve a high amount of predictive accuracy in that sense. My strategies mostly revolve around simple statistics.

My stack is a Linux VM with MySQL, running a suite of python scripts that collect data in real time and make the automated trades through TD Ameritrade's API.

By the way, I wouldn't consider myself smart or gifted - my strength relies solely on my determination. If you were to ask me if it's worth it, I would say "mostly." At the very least I've come away with a much higher understanding of the equity/option markets, and I know much more about MySQL, Python and data mining. At this point though I'd take a job with an algorithmic firm in a heartbeat.

> Anyone here making real money from algo trading?

No, citadel securities just had a record year internalizing retail flow.

I started off doing it on my own back in 2008 with nothing more than an Interactive Brokers account. Then I went all in and started a company to do it professionally around 2012.

As far as strategies go, I've said it here before, but all my strategies are quite simple and straight forward. The difficulty is almost always the execution. Almost all of my strategies are arbitrage or market making, and some of them trade off of events, like earnings, interest rate announcements, the weekly petroleum status report, and things you'd find on an economic calendar.

None of my strategies are speculative, that is I have no idea what company will perform well over a long period of time and in fact in many cases I don't even know what the company I'm trading even does. I also don't make use of any so called technical indicators, like relative strength, or fibonacci this/fibonacci that.

For the vast majority of my activity, my algos only enter into a trade if it's guaranteed to make money.

To give a vague idea of what the process looks like, every strategy starts from the premise that the market is perfectly efficient and there is no opportunity to make money. We then construct a model of what a perfectly efficient market should look like. We then backtest this model to determine whether the market is actually perfectly efficient. In most cases it will turn out to be either very close to efficient, or inefficient but in a way that cannot be profited from after taking into account fees, latency, and other factors. But sometimes you find areas where the model predicts certain behavior and the backtesting shows that the real world doesn't follow that model. In that case we then proceed to investigate very carefully what is going on. Did we make a faulty assumption? Is our model not seeing the whole picture so that there are factors we did not take into account? This happens A LOT and in fact we are very skeptical when our backtesting deviates from our model. But if after scrutiny we find no reason to doubt out model, then go from backtesting to running it in a simulator, and then from simulator to running it live on a very small scale, and then over time increasing the scale.

Our backtesting and simulating is incredibly sophisticated and precise. Almost all publicly available backtesters just look at trade activity or very low resolution data, and never accounts for things like market impact. Our backtester takes into account individual orders and simulates the entire order book, taking into account where our order would be placed within the order book. We also simulate potential market impact as well, which requires us to run simulations in parallel with different potential market impacts so we can see what the worst case scenarios are and there's a host of other factors a good simulator and backtester will take into account.

A colleague of mine does doing spread trading around commodities. He has 2 investors who put the money up, so I think you need a decent bit of leverage to actually cover fees and produce a return...

Managed to stay even for 3 months. Arcane stuff is what I got but I find it sensible.

>Isn’t it futile as an individual?

HFT is, more or less... but algo trading in general can be as simple as executing strategies you would otherwise manually perform. I believe there's still enough alpha out there.

Algo trading != HFT. HFT is a type of algorithmic trading strategies, categorized by trading interval. In the same categorization there are mid-freq strategies as well. The term "high frequency" is rather subjective, it depends on what alpha you're targeting after.

HFTs is about using c++, putting your box as close to the market computer as possible and building out a private optical fiber from chicago to new york to lower your latency vs competition... algo trading can be anything from putting your personal strategy into code to using machine learning to discover trade signals etc

HFT is mostly arbitrage, that's why they need the speed to reduce the risk. The rest of algo trading (and quant trading in general) tries to guess which way the market and individual securities will go based on market indicators and work over longer time horizons than HF traders work in. A simple example is that in low interest rate stocks will tend to outperform bonds, so the trade is to go long stocks and short bonds. Of course we expect that simple trades like this quickly be seen to be profitable and do more people do it and so the benefit of doing the trade versus just going long the market (beta) should eventually go down. So quants are always looking for other, more complicated trades as their current ones become less profitable.

HFT exploits very short lived movements. If you had some algorithm to predict long-term trends, then you could beat HFT without relying on low latency.

Back in the day I was day trading a stock at a gig that provides high speed internet and professional trading software. Many of us did have medium to long term profitability, so I guess it's the same for algo trading.

Someone who merely bought and held tech stocks, like Apple & Nvidia, beat virtually all funds since 2009. There are always ways to make money even when your competitors have such advanced tools. The world of finance is big enough that there are opportunities for players of all sizes and resources. Look how badly AQR has done despite hiring from such a qualified talent pool.

For an individual - HFT isn't all that accessible. Fees gonna kill ya. But yeah - you just zoom out a little bit and work from there.

You might have multiple strategies in the same symbol, one intended to send out post-only orders and one intended to send out crossing orders. Niche functionality like this can help achieve that separation of concerns.

Crossing orders are considered liquidity taking rather than providing since they're interacting with another market maker's resting (providing) orders.

> I'd argue that an order that'd immediately be filled does provide liquidity to the market overall.

No, it takes liquidity, by definition.

I would argue that it provides liquidity solely to whoever placed that resting order. But it takes it from the rest of the market.

The article isn't about discretionary trading. It's about algorithmic trading, which is a bit like the word 'hacking' in that it means one thing to practitioners and another to the general public.

To practitioners it refers to the use of algorithms to trade large quantities of stocks or futures or whatever. The goal is to reduce trading costs by executing small trades at the right time. This is distinct from and rather more common than quant trading where an algorithm actually decides what bet to make.

Your story doesn't make any sense. Was this guy trading his own account, or was he employed as a proprietary trader, or was he running his own little investment fund using another company's platform? In any case, a legitimate trader won't be "locked out".

Are you perhaps referring to a locked market?

https://www.investopedia.com/terms/l/lockedmarket.asp

"Algorithmic trading" in the sense described in this article only makes sense if you have a customer who wants you to execute large trades and pay you a commission.

You can try some trading using algorithms to identify profitable trading opportunities. That would normally fall under "prop trading" instead. There are definitely people doing this in ways achievable by a home hobbyist, but don't expect any low hanging fruit. Find some niche in some less liquid instruments. Don't look for anything that relies on being fast - someone else will be there who measures latency in nanoseconds.

I spent a good few months building a system for fun/interest. Darwinex has an interesting business model - check it out - but it shows leaderboards of supposed profit-generating trading systems.

Thing is, one will be hard pressed to find solid, relevant, detailed and current information about legitimately profitable trading strategy. Once a strategy becomes widely used, the 'market inefficiencies' being exploited cease to be readily available, so there is a strong incentive to keep a good strategy private. I vaguely remember reading about HFT firms obscuring their trading activity for this reason. Additionally you will find the online world absolutely saturated with the grifter types. So good info is hard to come by. Be prepared to learn at least the fundamentals of both trading and statistics, do lots of testing, and basically figure it out on your own because nobody successful (and smart) is sharing their reliable, profitable strategies (that doesn't mean you can't learn from them - just don't copy paste).

It is a very interesting domain though and I thoroughly enjoyed learning about it all and building mine.

>How would I implement algorithmic trading at home?

I made a python framework to try to help with this: https://github.com/robswc/stratis

However, it became hard to maintain an opensource version. I would say its still not a bad way to get a head start (bias there, ofc)

I will say, its not something that can be done "on the side." I originally made a decent amount of $ by getting lucky in crypto. I figured I'd "just become a algo trader" and it was much more difficult than I could have imagined.

I actually _just_ started a series on how to build an equity trading system from scratch. I planned to put part 1 out later this week but eh, I'll post it now, it touches on it better than my comment here can :)

https://robswc.substack.com/p/building-an-equity-trading-sys...

* the post assumes you already have alpha (a profitable strategy)

> Has anyone done this successfully?

Regarding this, I actually originally talked a bit with the (now infamous) SBF of FTX about this. I originally wanted to join Alameda Research... but at the time, I didn't want to move across the world. For years I regretted that decision (not so much now, lol) I started my own firm though and I have made a decent amount of $. Truth be told though, knowing what I know now, I might have put more energy into a start up. I enjoy the challenges of algotrading but doing some consulting work, I think I enjoy "building" more than running statistical tests, cleaning data, etc etc.

To sum up this comment though, _if_ you had a profitable strategy and _if_ you built a system that could reliably execute trades, you certainly could be successful. It is very difficult though.

For starters, open a brokerage account at a broker that provides an API. There are many currently available including Ally, Tradier, Alpaca, Think or Swim, Interactive Brokers etc. Although, I think Think or Swim is currently not providing new keys due to their merger with Schwab.

Once you got that part setup, try to write a simple program that buys or sells an equity from the command line .

From their own you can start exploring trading algorithms. Some are pretty basic, like when the price crosses above a 200 DMA, then buy a stock as that usually signifies an upward run, while selling if the price falls below the 200DMA as that is bearish.

Then you can read some books on price action, fundamental or technical analysis and build your own algos.

There are platforms like alpaca.markets that deal with all the infra-aspects and allow you to concentrate on the algorithm part.

There is litteraly 0 chance for you to be profitable in the medium to long term if you don't have years of experience in the industry.

Anyone claiming that is not the case is just misinterpreting his PnL.

The internet is full of people thinking they are market wizards just because they cannot residualize their idiosyncratic returns properly against beta, sector, country, size properly.

First, become a PDT (25k min to pattern day trade), meaning you can't execute more than 4 trades in a 5 business day time-period. So if you are serious, you have to get to that first... then realize that whatever broker you are using, they probably have an API.

I do raw api calls, but you have to be so very careful. You can do something wrong and totally, absolutely jack yourself up, so this way is only for the very brave, but I think it offers good opportunities to understand the markets better if you can stomach the extreme risk profile.

Check out https://algotrading101.com/learn/ for a good set of practical articles to give you some context. I wish I could still point to Quantopian, which is where I started my learning process on this, but it died.

I'd like to know how an individual can do this, 1) And avoid some code bug causing them to lose all their money, and 2) Not get lose all their money to some company's API fees...