Get the top HN stories in your inbox every day.
ye-olde-sysrq
kasey_junk
Not only are the platforms table stakes they are the more straightforward part to build. Even at the bleeding edge of latency you can usually work your way to the limits of your platform budget without having to find anything novel.
The strategies though are where the discovery is. There are a few strategies that are well known and still profitable but those are largely consolidated to the biggest firms. For everything else it’s a discovery process. And done strategies are only profitable for very short regimes.
I miss it sometimes too, but so much has been consolidated it’s largely a big firm world now.
ye-olde-sysrq
"they are the more straightforward part to build"
More straightforward, heh, sure. But still damnably complicated. Which just goes to show how much money and how much engineering talent is invested in this world that these things are so taken for granted.
ilyt
So many intelligent people applied for something that has no value whatsoever to humanity regardless of how you look at it.
DrBazza
> it’s largely a big firm world now.
Largely, though I receive 1 or 2 job specs every week for start ups with the keywords 'hft' and 'low latency'. Admittedly there's going to be duplication there if you read them closely.
I think it's a bit of a myth that (ignoring FPGAs) that writing a low-latency software trading system is a time/cost expensive process. Anecdata = I worked at two firms where we did a rewrite from scratch with teams of 5-6 people and traded in the market within 3 months. I'd argue a senior dev that's been around the block a few times could achieve similar when you remove corporate politics, and bikeshedding over design.
The big firm part is paying for multiple quants at $200k++ to come up with strategies and historic market data access for trading models. Small firms are getting backing as long as the co-founders are 'ex-CxO from MegaCorp'.
dm3
> I think it's a bit of a myth that (ignoring FPGAs) that writing a low-latency software trading system is a time/cost expensive process.
This depends a lot on the complexity of the trading system and the trading venue specifics. A system to trade single stocks or futures can be built, certified and running in 3 months. A system for options market making will take a lot longer.
undefined
skummetmaelk
Seems more pointless than crypto to be honest.
fnordpiglet
HFT provides a great deal of liquidity and efficient pricing in markets that are adapted to it. It provides a real service, allowing people to transact without using a large bank or broker efficiently on an open market. The fact you can click buy and it buys on almost any stock is likely due to a HFT on the other side. That may not mean much to you directly but it does provide a lot of utility in markets. The biggest gripe people bring about HFT is it during high volatility HFTs usually pull out of the market at a time when people really value the increased liquidity. I think some of the more advanced HFT firms though have moved into longer time range trades which helps provide more liquidity in those markets.
shric
My understanding was that HFTs make the most profits during periods of highest volatility. Why do they pull out?
Roark66
It may seem so at the first glance, but such trading if carried out by many independent providers actually increases the liquidity if the market for everyone else.
Likewise with crypto. Personally I don't think that if you're calling an API over an Internet it matters if your trading bot is written in go or python (mine was in python). Use the language you're most comfortable in. The network and trade submission/execution at your broker will be 10x slower than your bot anyway. Unless the size of your operation approaches the size where you can get direct market access which seems to be reserved for big forms only.
Solvency
"increases the liquidity if the market for everyone else"
That's just it. The whole premise is pretty absurd. The market, the actors, everything. It's so far removed from literally anything remotely human. It's the financial equivalent of an infinite sea of AI bots producing CVs and research papers which are only being evaluated and read by other bots.
If you step away from it all for a second, what the hell is the endgame of this whole hustle.
nvarsj
It's quite a statement. You're almost saying capitalism and efficient markets are pointless. Maybe they are, but I think it's nothing like crypto.
In the old days before HFT, you weren't sure you'd get the best price. You'd have to rely on a broker to make sure that happens, but as a retail trader you generally got a worse price/out of date price.
Nowadays with HFT you can get pretty much the best price anywhere. Those <1ms HFT traders make that happen. It's the efficient market hypothesis in effect, made possible by HFT.
Now is it pointless to shave off even more ns in the all out war to grab a piece of the order flow? For retail and institutional investors, at a certain point, yes it's completely pointless.
But it's also just pure capitalism at work. HFT firms compete against each other, and the competition is about speed to provide the best price and volume. If you try to regulate with something like enforced delays, then what do you compete on instead?
Ecoste
> You're almost saying capitalism and efficient markets are pointless.
Capitalism necessitates efficient markets (and efficient markets necessitate HFTs) so any criticism of HFTs is a direct criticism of capitalism as well. I mean this isn't really a problem that's specific to HFTs -- there are just a lot of jobs that we can perceive as providing no value or even negative value (jobs that are possible specifically due to capitalism e.g. payday loans, 2008 style trading, certain scams or predatory practices etc.)
I think we can all agree that this is a flaw, and we're not criticizing capitalism to replace it with something else, but rather just recognizing this as a problem. At the end of the day, a useless job is a useless job even if it exists solely because of capitalism.
tesdinger
> efficient markets are pointless
You are profiting off workers as a middle man in the economy by doing HFT, and trying the justify it by some vague concept of the "correct price". You are producing nothing of value, merely taking away value before someone else notices it is there.
undefined
semilattice
>You're almost saying capitalism and efficient markets are pointless. Maybe they are, but I think it's nothing like crypto.
People are saying this because, HFT sounds similar to 'crypto mining'. That's people with best infrastructure, the 'big-guys' -- win. While leaving out the retail investors as broiler chicken, pumped with 'drugs' (by influencers) to spend more on imaginary assets, so that they can be used for 'food' by these 'big-guys'.
There are different influencers for retail investors vs crypto. In retail investing there are promises of 'retirement paradise', actual tax deductions, the Jim Cramer-like people (at least what I heard in US)
For crypto investing the influencer are different, the geography is wider. A promise to participate in markets if you do not live the country that has adopted US/UK-based financial services.
- - - By the way, I think the markets will still have liquidity if there is a rule to wait, for say, 30 min before a stock that was just recently bought -- can be sold (unless by a clear fat finger mistake)
This rule will cause the HFTs to stop existing in the current form.
musicale
Nonsense - what could be more beneficial to society than providing imaginary "liquidity" by interposing yourself between legitimate buyers and sellers?
It's a bit like stock brokers - and why wouldn't we want stock brokers to operate at drastically faster-than-human timescales, because we all know the value of a company changes every nanosecond! And "flash crashes" create opportunities for investors to make huge amounts of money!
And just as crypto has poured money into GPU companies (providing opportunities for enterprising secondary market resellers of same) HFT has poured money into networking companies.
Short of creating the great firewall or helping governments slurp up all the traffic on the internet, what could be a more beneficial application of network technology?
HFT (like finance in general) has also gobbled up lots of tech grads, making it easier to find jobs at tech companies.
nostrebored
> Nonsense - what could be more beneficial to society than providing imaginary "liquidity" by interposing yourself between legitimate buyers and sellers?
Now when I say the same thing about index funds people get all huffy
alphanullmeric
And like crypto, feel free to ignore it instead of making things you don’t like illegal.
ciwolsey
Dont make scams illegal just because you don't like them!
tesdinger
I hope someone steals from you and police ignores you.
wwilim
Sample size one. Contracting for a trading shop for 2 years taught me that it's not technical aptitude that keeps the business rolling.
mr_o47
If you don’t mind me asking how did you land a contract gig at trading company
blibble
however ineptitude very quickly brings the entire thing down
_zkyx
YES, 100%! I honestly, cannot agree more. I've read pretty much every book that I could get my hands on and no one talks about the platform. This makes so much sense. Thank you for your perspective. It seems like so much of this stuff is silod off into each company and no one is talking about it.
HFTs are definitely playing a completely different game. I was reading about the exchange architectures and how things are actually wired. I'm getting my data from SIPs while HTFs are directly connected to the exchanges [1]. I'm transacting in seconds and they, like you said, are transacting in microseconds, so there is no comparison. Which, in a way is actually nice in that I'm not really competing with them. Or, maybe I am but I can still make some money. haha.
Cheers and thanks for the awesome comment!
[1] https://www.researchgate.net/figure/Latencies-in-the-Electro...
nurettin
I'm in the medium frequency trading world. (calculate lots of market conditions, send at most a few orders a minute) I'd like to understand the HFT world a little better.
Is it basically "do what MFT does, but faster", or is there any specific advantage like getting into an order queue with priority?
mmasu
There’s a very nice book about this subject called “Dark Pools”. I found it very enjoyable, you certainly seem to have the knowledge to enjoy it even more :-)
anon-3988
Is there a market for s much “slower HFT”. There’s a very wide gap between day traders and ns traders after all.
bryancoxwell
Fascinating. Thanks for sharing.
_zkyx
Happy to answer any questions about this. It's been a side project that turned into a full blown obsession. There is nothing too secret about the system since it's more about having a solid platform that you can plug your strategies into. I'd probably even open source it but I'd have to clean up all my hacks :)
antiviral
You may be interested in this: https://www.techtrader.ai/#wall
Excerpt from his site:
"Tech Trader is a fully autonomous trading system live with no human intervention or updates, now for over 10 years. It is unique from conventional algorithmic systems, not only because it actually is fully automated, but because it takes a "human" approach to markets. It is not quant. It is not stat-arb. It is not high frequency. It is a program that looks at stocks the same way a person does but with the cold discipline and infinite attention span of a machine. It is analogous to having a thousand independent traders each focusing on a single stock, as opposed to a single quant manager trying to make sense of a thousand datapoints. A person doesn't think through stats, correlations, or complex math models when trading, and neither does Tech Trader. Tech Trader leverages technology to do what human traders do at scale rather than approach markets from the point of view of an academic, mathematician, or scientist.
Since its launch in Dec 2012, Tech Trader has been trading live capital completely on its own, fully automated in the truest sense with no human input, no tweaking, no updates. It is, for all intents and purposes, an autonomous hedge fund, one of the first to truly trade unsupervised for years on end. Whereas many "automated" or "AI" funds may have a hundred scientists providing the actual intelligence behind the curtain, the creator of Tech Trader consists of just one person - a self-taught individual going by the gaming moniker pftq, who created the system at age 21 and has long moved on to other interests. "
dist-epoch
No broker API change in 10 years? No bugs?
I call bullshit. Reads more like a scam for trading signals.
antiviral
I don't run that site myself, but a quick search on sec.gov shows it's a registered hedge fund:
https://www.sec.gov/Archives/edgar/data/1653903/000165390318...
I can't say any more about its validity. Ask your financial planner if techtrader is right for you :)
_zkyx
Thanks, I'll check it out!
daniel-cussen
[dead]
k9jd883jb
Awesome write up. I have a similar project in Go myself, although I just use minute bar data instead of realtime ticks.
Can you share your approach for plugging in various strategies? I quickly learned that having a pluggable strategy system is tricky as it could span across multiple layers of the system.
Also, with backtesting, are you storing and replaying all the quote/tick data? or just using the historical aggregates?
_zkyx
I wish this was better, but honestly, I'm just hard coding them right into the BUY loop. So, I need to restart the app anytime I want to change something. That's why I build this logic to dump and reload the state into a gob file (go memory dump essentially). Ideally, you'd have some type of format to write out your algorithm, and then have a way of hot loading it or something. But, I don't change these enough to really do that yet.
For back testing, I download all raw trades and quotes, and put them into 1 file sorted by time (1 file per day). Then, I compress them using lz4. This allows me to sort of replay the entire market and build up all my intraday backtesting from the source. This took me a long time to figure out and build but has been so worth it. So, I have an off-line script that basically, loops through these files, and replays the market, and makes simulated trades, and then spits out what would have happened. There is a GUI for that too so you can go in an explore the trades and see what triggered the buy and sell. I have seen nothing that goes backtesting for intraday like this.
This is super inefficient but I'm just building the aggregates on the fly. I could probably cache them somewhere but it takes maybe 4-5 minutes to replay a days worth of trades/quotes and build all this so I haven't bothered yet.
slashdev
Fascinating. Have you considered open sourcing this without your algorithms ( empty buy/ sell loops)?
I’d like to play with something like this but I couldn’t be bothered to build it all from scratch.
Shocka1
I've been algo trading as a side project for a few years now, but all through TD Ameritrade. I get all my quotes and whatnot with them, and it's free with my accounts. I also get other data elsewhere, which is free as well.
I was curious in getting a websocket setup with Polygon after seeing your post, but I noticed it's $29 for first package with the websocket feature. Then I noticed that the package with real-time data is the Advanced package at $200 a month. I'll admit, real-time with unlimited API calls sounds pretty sweet, as my strats rely on to the second data. However, I don't know if I can justify $200 as my algos run no more than 78% win rates with a normal bankroll funding it all. I'm looking to save wherever I can while I build these things and get a passive income stream rolling in.
I would love to know which package you were using, as I didn't see it in the time I quickly read your post? Also, any pro/cons to that specific package? Any and all other details are welcome, and if you would rather respond to my email, it's in my profile. Thanks!
kirse
Do you have an email? I've been building a similar system for awhile except in F#, would love to connect.
undefined
programtrader
[dead]
TacticalCoder
You connect to IB's TWS API... If you execute your trade there only, wouldn't it be an option to fetch the price in real-time from the IB API and not having to use polygon.io at all?
I have no idea: does IB send the price feed in real time? (they certainly send the data in real-time to TWS as it constantly updates right? But is the order book available through their API?)
Basically and even though I know this was published on polygon.io's blog, would that work by only using IB / TWS's API?
_zkyx
Yeah, I initially tried that actually. They have very low resolution data and do not provide raw trades/quotes (something like a message every 250ms), you cannot watch the entire market so you're stuck with like 100 tickers [1] vs 5500 tickers, and fetching historical data is very cumbersome. You can basically pull down pre-aggregated candlestick data and I wanted to base things off the raw trades/quotes. I actually probably spend a month or two trying to make this work. So, that's why I'm using polygon.io in that you can stream trades/quotes for the entire stock market in real-time, without any caps, which is amazing!
[1] https://www.interactivebrokers.com/en/?f=%2Fen%2Fgeneral%2Fe...
programtrader
Congrats on the post. Very inspiring.
I've been going through this journey myself. I started learning on Tradingview, then bought Build Alpha to discover how to test strategies. I chose Portfolio 123 for my automated factor trading but had been working towards creating a program/basket trading system like yours that can act on intraday data.
I moved to long-term investment until I could build a simulator capable of verifying the correctness of my investment strategies using fuzzy testing ideas stolen from TiggerBettle.
I have almost two years of polygon quotes and trades for the whole market captured with a monotonic timestamp to be able to replay the data –and test the handling of polygon socket glitches.
I'm focusing initially on capturing the data in a way that allows fast replay and aggregation, similar to what Kafka can do with topics but in-process using zig and custom memory-mapped data structures. My idea is to be able to generate signals like VIX (once I add options data), ETFs, and indexes and hopefully be faster at doing so than others :), please HN folks, call me out here if I'm being too naive.
This has been a three-year learning process for me. I have been a retail investor for +10 years, but over the last three years, I've gone deep into learning algo-trading, drank del Prado Kool-aid, and read numerous trading and investment books.
I'm now focusing on my technical chops to build the engine to build order books for individual stocks, baskets, and indexes with realistic market prices. I aim to develop a system that can get as close to the market price in the next dollar bar as possible.
This has been a very lonely journey, and after reading the responses to this post, I'd love to connect with others on a similar path. Sending you an email!
czbond
@WestCoastJustin
I've been really wanting to use Go, but as you say, much of the community is Python due to the data analysis strengths. To the detriment of the other things Python does do poorly.
Can you give some thoughts with your experimentation on the following from a Go perspective.
1. Supported TA libraries in Go. I'm familiar with TAlib (python), bloom, etc. - certain forks tailored to real time rather than historical (eg: no re-compute on ticks)
2. Data storage (article mentioned you're all in memory). I've been using S3 & ArticDB
3. If your in-data memory is treating you well for multiple TA calculations (example: in Python, you can compute & save pickled dataframes - and re-read those over longer time periods)
_zkyx
> Supported TA libraries in Go. I'm familiar with TAlib (python), bloom, etc. - certain forks tailored to real time rather than historical (eg: no re-compute on ticks)
I've been basically, just manually coding the algorithms from python into Go. ChatGPT is amazing at this. I really only just about 4 so it was a one time thing.
> Data storage (article mentioned you're all in memory). I've been using S3 & ArticDB
Yeah, I ran into issues and then was like what would be the fastest, then just went in-memory. I download all raw trades/quotes each night and store then into gob+lz4 compressed files. Then for backtesting and stuff I can load these in and build the aggrogate bars on the fly.
> If your in-data memory is treating you well for multiple TA calculations (example: in Python, you can compute & save pickled dataframes - and re-read those over longer time periods)
Yeah, I have a historical lookup table that I build nightly too. This gives me a reference point when I'm doing % change calculations and stuff. I should probably have mentioned that.
iakh
I'd be careful using ChatGPT for this. I tried the same recently to bootstrap spot yields from par, which is a fairly common code problem with hundreds of examples online. ChatGPT couldn't do it. It produced code that looked right, but would fail my tests abysmally. I ended up writing it by hand. I hope you are validating their code with known data.
pdimitar
Not a complete answer but I quite liked https://github.com/markcheno/go-talib for technical indicators.
eshnil
Why not switch to Mojo lang for this? It's Python-compatible with Golang like performance from what I hear.
czbond
@eshnil (& others familiar with Mojo)
I had not looked at Mojo... thanks for the pointer. In the past, I've had issues with library compatibility with compiled derivative languages of interpreted ones (eg: Crystal of Ruby, etc, etc).
Know if Mojo directly uses existing Python ecosystem?
I've been using Polars, etc.
silisili
I'm excited for Mojo, but it'll be years before it's ready for prime time as a general purpose language. It's not even available to download yet AFAIK...
jjtheblunt
It's not available yet outside of hosted environments provided by Modular.
bberrry
Hilarious how you suggest someone use a technology that isn't even publicly available, showing how extremely shallow your knowledge is.
_zkyx
Haven't looked. Honestly, I'm so invested in this now it would be a pain to change anything.
happytiger
How do you relate with tradingview?
I've used ATR bots for years, and would love to hear your thoughts on how what you're building delineates itself as a distinct product strategy beyond just the programming language GO.
I've used wonderbit, zigz, and some of the others.
Cool project.
pyrrhotech
One of the most misunderstood concepts about algorithmic trading is that speed is not a critical factor for the majority of systems. My systems like https://grizzlybulls.com/models/vix-ta-macro-mp-extreme have been crushing the market with live trading for 3+ years and yet trade on a frequency of only once ever 18 trading days on average (some a little more often, some even less), and are set up to only generate signals around hourly breakpoints.
The last 18 months have been weaker than the first given the enormous structural shift in the market in this high inflation and rapidly rising interest rates environment, but we've still managed to deliver a return of +14.11% since the site launched in Jan 2022 compared to -7.83% for the SPX. We've managed to do it without any use of leverage and also with lower drawdowns as well of -16.48% vs. -27.57% for the SPX over that time frame.
WA
Your chart including what looks like backtesting data starting 2009 looks a bit disingenuous. It’s a totally random starting point. It’s obviously not your actual performance, if you say you started 3 years ago. It’s made to look like your model actually outperformed the market significantly.
Kudos to you if you really sit on an untapped gold mine, but imho, there are some red flags that makes me not buy in.
pyrrhotech
Thanks for the feedback. I try to call out in several places in the app that 2009 - 2020 is backtest only, but perhaps I need to make it more clear. No one should ever expect a model to trade as well in live trading as it backtests, and that's been true of nearly all our models (save the anomaly of TA-MR-Basic). However, there's more than enough room in the returns and drawdowns to underperform the backtest while still producing significant alpha, which is what we've seen with 4/7 models and especially with the top two.
However, the April 2009 start date is not actually random--it's the first start date for which intraday futures data is available for more than just front month contract. Several derivative indicators of the VIX futures curve are the most foundational to all the VIX-based models, and they simply cannot be processed without it. The VIX futures were only created in 2004, and I've scoured the internet for intraday data for more than just front month (can't create the curve if you only have front month data), and the earliest it can be found is April 2009.
WA
Thanks for the explanation, I appreciate it.
Brystephor
How much external data is used for these models or human input/fine tuning on a regular basis?
pyrrhotech
The models vary greatly in the number of external data sources they pull from to compute signals. The simplest ones like https://grizzlybulls.com/models/ta-mr-basic and https://grizzlybulls.com/models/ta-trend-basic only use a technical analysis calculations on the raw price and volume data of SPX on a few different timeframes. The most complex pulls data from dozens of external sources as it incorporates macroeconomics, monetary policy, various market sentiment sources, yield curve, valuation data, fundamentals, earnings estimates, etc.
It's also worth noting that every additional data source adds some risk of that data source being down or publishing inaccurate data during real-time signal calculation which can cause inaccurate signals, so in order to justify that risk, the external source must meaningfully contribute to alpha or better risk-adjusted returns.
None of them involve a human element in real-time. However, they are occasionally updated as new data comes in, but any updates only apply going forward so as to preserve the live trading history accurately (live trading start date varies by model from mid 2020 to jan 2022 with 2009 - 2020 being purely backtest for all models).
MuffinFlavored
> The simplest ones like https://grizzlybulls.com/models/ta-mr-basic and https://grizzlybulls.com/models/ta-trend-basic only use a technical analysis calculations on the raw price and volume data of SPX on a few different timeframes.
Am I right to doubt that something this simple generates any alpha whatsoever?
aantix
For those that are interested in algorithmic trading - take a look at Collective2. Where engineers make available their buy/sell signals for a subscription fee.
It's like the minor leagues for algorithmic trading. It's fascinating.
The system keeps track of the gains/losses, so no cheating on the reporting.
You can authorize Collective2 to access your Interactive Brokers account, so that the trade signals are managed on your behalf.
It's been around for at least a decade, so you can see some longstanding performance numbers, but most systems just don't last that long.
https://collective2.com/leader-board
None of the leaders have been there for very long. One to two years. Showing that most system's alpha disappears fairly rapidly.
The equity curves are sporadic ~a few of the trades accounting for a majority of the gains.
asavinov
When developing an automatic trading system the following aspects are important:
- Data feeds and data ingestion. It can be a fairly independent component which collects data from different sources (might be even discussion forums) making it available to other components in some uniform format
- Feature generation. The source data is rarely used in its original form for decision making and having good (informative) features is frequently the primary factor of success. Moving averages is an example but nowadays this will hardly work
- Signal generation. Here some logic should be applied in order to emit discrete decisions and such models are heavily parameterized with thresholds.
- Real trading and order management as well as coordination of all activities.
The article sheds some light on the technological aspects and the general pipeline used to process the data and manage orders. Although it might be interesting indeed, I would expect more details about how to scale the solution and how to implement it asynchronously. Especially if it uses Go which has a special construct for that purpose - channels.
I understand that it is not the focus of the article, but having some general information about its trading logic and how to plug new and parameterize existing strategies would help. Some links at the end are quite interesting for me because I am developing an intelligent trading bot based on ML and feature engineering (https://github.com/asavinov/intelligent-trading-bot) for which such articles might be quite important
_zkyx
Hey, thanks. Yeah, I agree with you. That's an oversight on my end. I'll tell you here though.
I'm just using go routines and channels to talk between them and then a giant mutex for locking. That's basically it. So, as new data comes in, it builds aggregates (tick based candlesticks) as needed, this then triggers the the BUY logic loop on that new data, if something is detected, that triggers a IB API order. It is dead simple and nothing complex in here. I've had upwards of 100 positions being tracked at anyone time and seems to just work. So, I haven't messed around with complex async logic too much.
I'm actually just hard coding the parameters right into the BUY loop. This probably sounds crazy but for a small setup like this they don't change that much. So, I can run some trades, tweak things, restart, and then test some more. I imagine if you were doing that in an enterprise setting you've have some formal language and hot loading and stuff. But, for me hard coding seems to work well enough.
antiviral
Also, what's your approach to risk management?
For example:
1. Do you have anything that limits the size of a single trade or position?
2. How do you measure and manage overall volatility/risk/VAR to your portfolio?
3. What kind of safeguards do you have to avoid catastrophic bugs? (For reference, see Knight capital and how a single bug brought down the entire company: https://www.henricodolfing.com/2019/06/project-failure-case-...) Given how fast your system trades, I imagine it must be difficult to visually spot these errors. (You did mention paper trading, but I wonder if you have anything else you want to mention)
Thanks so much for sharing your knowledge publicly. It's very much appreciated!
_zkyx
Here's some simple rules. I basically just read about these and then stole the ideas. I wish I had data to back this up and it seems to be working.
- No single bet can be more than 5% of all money. The small bet sizes are what really saves your bacon in that even if a few lose 10+% you're still fine overall. - I'm also limiting the amount of shares I bet and try to keep it in the low hundreds so that I get really fast fills. - I have something that stops everything if I lose more than 1k in a day.
I don't do anything to measure overall risk or volatility. Almost everything I'm in is highly volatile. I'm basically betting as things go up and then try to cash out. Sometimes you hit the top.
Yeah, I inspect all the trades at the end of the day, well and during the day, and try to feed anything new back into the system. This is 100% manual. But, like if a single trade loses $100 or something definitely I'm in there looking at what happened.
asavinov
Does it work synchronously or asynchronously? For example, if you integrate WebSockets then you could act as soon as you get new data. If it works synchronously then you regularly request data (say, every second) and then act.
If you process many positions and then choose top 5 candidates then do you choose one of them for buying or you can allocate resources between them (depending on some score)?
_zkyx
Both? I'm getting a constant stream of data via the websocket. As data comes in it gets added to the large in-memory object, and then once I have X number of trades, I'll go and build the candlestick bar. As soon as that bar builds, it triggers the BUY loop to inspect it, and see if it matches what I'm looking for. If it does, it send a buy order out to the IB API. I'm not ranking anything. I'm just looking to see if that matches and then buy. So, I have something that tracks the total money I have, and if I have any free money (and we don't already own that stock), it makes a bet. That's the logic. I wish I had some ranking logic but that's what it's doing.
That logic is happening for 5500+ stocks all in real-time which is pretty insane. But, it works really really well. Go is amazing. At market open and close there is like 60k+ events per seconds across trades and quotes. So, that loop is processing like 60k events at times, and building each stock out, and then looking to see if we should buy/sell.
RomanPushkin
Go itself, or any other language won't give you too much of an advantage. What gives you advantage is trading algo, which is always hard to find. I've spent months on figuring out the best parameters for trading. Ended up this working only on historical data, while in reality it was totally different.
I could use Visual Basic, and it would be better than Go, Rust, or whatever it is out there, given the algo and strategy are flawless. Language is just a tool. It's great you used Go, but I think the title is a bit misleading - people think of it as some kind of advantage. It isn't.
And for HFT trading a language with Garbage Collector is not a great choice IMO.
asavinov
> What gives you advantage is trading algo, which is always hard to find.
At the end it is necessary to make a decision whether to buy or sell (and how much), which will compete with other decisions made based on some logic. Developing such a logic (strategy) manually is of course quite difficult. I developed an intelligent trading bot which derives its trading strategy from historic data:
https://github.com/asavinov/intelligent-trading-bot
Currently it works for cryptocurrencies but can be applied to other markets:
https://t.me/intelligent_trading_signals
> I've spent months on figuring out the best parameters for trading. Ended up this working only on historical data, while in reality it was totally different.
It is a typical situation. The whole problem is to develop a strategy which works for future (unseen) data. Even backtesting algorithms should be designed in such a way that future experience (data) does not leak to the past.
_zkyx
Hey, I write this. I did program this in Go. The reason is that it's my go to programming language. Using it is actually a disadvantage in that the industry uses C++ and Python but that's just what I know how to program in. I wasn't trying to be misleading. Go does work really well for taking in data, doing something with it, and then calling a remote API. So, it actually works really well but if you wanted to get a job based off this it probably wouldn't help you.
hu3
Go is great for network heavy apps like algo trading!
I used Go to write trading algos that would find small windows of triangle arbitrages in crypto exchanges. Made me some money but the risk of a big loss made me stop pursing crypto trade and it required too much time and attention. It's a full time job from my experience. Reasons I could lose big at any given time if I scaled up the stakes:
- Exchanges temporarily pausing some specific crypto trading for N reasons (happened very often) while I'm in the middle of the arbitrage.
- Getting caught in the middle of a pump and dump event (also frequent)
- Any algo mistake that would perform excessives trades in succession would incur huge losses because crypto exchanges charge %.
Also, most cryptos have too low volatility. Every time I tried to scale my bot would start interferring oo much with the market. And it wasn't even much money.
tomcam
It was my (admittedly ignorant) impression that C++ was preferred over garbage-collected languages because you'd be more likely to avoid GC pauses. Yes, of course I know that Go's is super efficient, but if you're doing a million trades a day even a tiny percentage of slowdowns that would be otherwise preventable could be a career-limiting move. Am I just behind the times?
smabie
It's a full time job bc your competitors are full time quant traders :)
moneywoes
Have you tried non crypto markets
maest
You can make GC languages work for HFT if you do zero allocation code.
There are places that use Java and just preallocate all the memory they need at startup. Jane Street famously uses OCAML
c7DJTLrn
>And for HFT trading a language with Garbage Collector is not a great choice IMO.
Depends. I've heard of places that use Java by creating a huge heap, minimising allocations to almost zero, and just restarting the app periodically before it can fill up. You can achieve very good performance doing this and you don't have to worry about memory bugs.
IshKebab
You overfitted. That can happen due to human optimisation as well as computer optimisation. After you have experimented on some historical data for a while it becomes less useful.
mhh__
Or the model was crap.
cachvico
A language should be considered with its ecosystem, which is why the author started delegating parts to Python by the end.
Therefore the title was actually on point.
dataviz1000
I've been building a bot with Typescript because it is what I know and Python because of the tools available.
It is an overwhelming lonely endeavor. With all my other projects, I've always worked on teams, although they have always been very small teams and most of my work was autonomous -- still there was the occasional meeting and stand-ups. I've been working on this for six months and thought about bringing a friend onboard for no other reason than to not be alone.
dzink
Count me in the group of solo algo trading developers. Maybe we should connect and chat.
antiviral
This is why I miss Quantopian. They had a great discussion forum, an ongoing competition, and annual conferences.
I would be interested in some kind of regular meetup as well to discuss new techniques and to analyze publicly disclosed strategies, etc.
I recently posted this article on HN... There is so much more knowledge to share:
Def_Os
There are dozens of us! I'd be open to a chat channel or something.
MuffinFlavored
I wonder how many people are actually willing to share "secret" details versus how many actually have anything to share that is worth anything.
We all know how to load up OHLCV data and do basic math on it. Where does that gain us any edge, you know?
Shocka1
Myself included. One issue I've run into with the public Discords is there are just too many people and therefore too much noise. A smaller non-public group would be ideal.
Aken
This article is well-timed as I was about to use options monitoring as an excuse to get into some other technologies. Probably not going to make a ton of progress as a side project, but I'm always up for a chat!
IshKebab
How can you hope to compete with the pros? You're not going to be faster than them and unless you have several PhDs in maths it's unlikely you'll come up with a better algorithm.
The only way I can think of is to get data that they can't.
kevstev
Not everything has to be super fast. I was using quantopian to trade based mostly off fundamentals data on a monthly rebalance basis successfully. At least enough to beat the sp500 for a few years- I stopped because I went to a hedge fund that prevented me from trading that frequently and in most small caps. If you want an idea take a look at the piotroski f score and factor based investing. Aqr for example is still largely factor based.
usefulcat
There are plenty of strategies that can be profitable on a small scale but which just don't work as you scale up the capital or leverage. Such strategies can be simultaneously profitable for a small operation and not worth the bother for most larger trading shops.
IshKebab
That doesn't make any sense. Can you give an example?
tdiff
I find this article to be absolutely pointless and clickbaity. It can be boiled down to: "you can use poligon.io for market data but algotrading is difficult anyway so there is not much to share yet".
_zkyx
Hey, I'm sorry you feel that way and I wrote it. Personally, I wanted to share the high-level structure of how you'd build your own system. I would have loved to have seen something like this when I went down this rabbit hole in that I needed to figure it all out myself. What do you think would have made it better? I'd be happy to roll that back in.
silentwanderer
FWIW I thought it was a really good overview of the project structure and I appreciated the lessons learned as well.
MuffinFlavored
lol just admit you work for a niche SaaS in Polygon and you wanted exposure
dzink
I love finding other people who enjoy the pure feedback loop of code, strategy, and money that trading provides for me. I’ve been working on timing and correlations between stocks and indexes as a source of alpha and having some success. Now trying to automate the process and absolutely loving the work along the way. Not trying to play market maker and not jumping between stocks. Instead honing in on an accurate model that monetizes a few really well. Some tools I find useful: TradingView and Pinescript indicators and strategies, excel models with exported data for backtesting, Python and Go for the backend machine learning, ChatGPT for faster iteration on new code. Would love to talk shop if you guys are interested: trading @ dianazink.com
TechBro8615
Nice writeup. I appreciate the concrete discussion and especially the screenshots. I understand not discussing strategies, but can you elaborate on whether your trading is entirely based on technical analysis, or do you use external (or "alt") data feeds? In other words, is this a mostly closed system with inputs from Polygon and outputs to the IB API, or do you have a more extensive setup of custom data feeds of news sites, twitter, reddit, etc? If so, I'd be curious what challenges you face balancing those with historical backtesting, when some data sources might only have partial historical coverage.
_zkyx
All my data is from polygon.io and nothing external. I have been exploring using lookup tables of pre-computed historical values to do things like anomaly detection (ie, is this normal activity for this stock). A good example would be BXRX today [1]. I have been looking at options trading activity too and trying to use that as a signal.
blobbers
Regarding open sourcing: don't be afraid to show your flaws; it doesn't have to be lonely if it is open source.
Obviously whatever trading bot you're running separate from the actual trading engine itself is somewhat proprietary, but it would be great for the community to get more of this type of software in the hands of other hackers.
Quantopian / Robinhood tried and failed, and the numerous clones since then have been somewhat sub par.
JacKTrocinskI
You might find https://www.quantconnect.com/ interesting! They offer a platform for quickly developing and backtesting trading strategies. They have a good community and overall do a great job. Give it a shot!
sneak
Slightly related: when running 3 screens like that, put the middle (primary) one in landscape, so the array is like a big H.
It works better for videos and code, IMO. You still have two huge portrait fields for documentation/webpages/etc, or 8 portrait-oriented quarters.
I usually run 8 portrait-shaped windows quartered on the sides, then two side by side on the middle, which are each approximately square.
I don't like my editor window getting too tall.
ilyt
Eh, honestly only thing horizontal is good for is media, any documents/code is better in narrower vertical columns.
Honestly my perfect screen for work would probably had something like 1:1 ratio, so I can have 2 nice columns of code that are tall enough that can be split horizontally if needed while still being useful.
_zkyx
Yeah, personally, it's been a game changer just being able to load so much more into your field of view. Having two side-by-side makes it really easy to look at other parts of your code and make changes. I'm loving it! I'll give the H method a try for a few days and see how it goes.
Get the top HN stories in your inbox every day.
I worked in HFT for a while. I find the whole space fascinating. I'm glad OP found similar thrills.
"This aspect, the platform itself, seems to be often overlooked in most discussions. Many conversations revolve around strategies (mean reversion, trend following, linear regression, etc.), and backtesting, without fully addressing the practical mechanics or logistics of strategy implementation, particularly in the context of live, intraday trading."
I'm glad you had fun, OP, but also I think I can shed some light on why most people discuss strategy.
Trading is a perfect storm of ridiculously high tech, ridiculously complicated, ridiculously regulated (Not over-regulated, mind you, this isn't a value judgement. But the amount of regulation is extremely high.), and ridiculously competitive.
But that said, it's the last bit that drives it all. Since it's so competitive, even though building an order entry system, and a risk system, and a position-tracking system, etc is a huge accomplishment (again, congrats OP!), it's table-stakes to even dip your toes in the pool here. Trading shops can attract top talent and robust, bespoke trading systems are basically cost of entry.
So people talk about strategy because everyone already has the table-stakes stuff and are now trying to make money with it.
It doesn't help, too, that lots of market participants aren't even playing the same game. In HFT, we operated on trades with alphas that lasted a few seconds, where races to entry/exit were battled in shaving nanoseconds off FPGAs being able to shoot out orders and microseconds off wireless networks flying market data around new jersey. Meanwhile, banks are more concerned with elections and geopolitics than they are about the weather in Carteret. (Rain = no microwave network for the day). And then there's a million strategies in the middle with alphas that last from hours to weeks.
So it makes it really hard to even speak the same language to each other when talking in common forums.
It's a fun world. I miss it sometimes.