Get the top HN stories in your inbox every day.
xrd
shepherdjerred
I felt this a year or two back, but today I've had as good of an experience on Docker w/ arm64 as I do w/ x86_64. I use arm64 Docker a lot since I work on a M1 MacBook.
I usually stick to the common base images, e.g. ubuntu, alpine, nodejs, golang, etc. and install based off of that. Also, I rarely write Dockerfiles these days and instead use Earthly [0], which is a tool that really shines as a CI/make alternative, but it incidentally also has a nicer syntax which makes it easier to write multi-platform Docker images.
What images or other problems have you ran into on arm64?
[0]: https://earthly.dev/
xrd
For example gitlab. The latest arm image, as far as I could tell, isn't the same as the most recent x86. And, iirc, it was from some other person, not gitlab. It's often hard to tell what you are getting when you run an image, because docker pull can pull an image that isn't a multi platform build. I've had issues where the SSL certificates don't work, and I'm assuming it is because the stack could listen on 443, but the full ssl when running on arm didn't work. I'm not sure if that was because it is emulating using Rosetta or whether the software inside the container built correctly but isn't actually running on the arm platform correctly or what. It just feels like the wild west with arm images right now. I'm sure it will get better but it is still a minority platform and that comes with those issues.
And, this might just be exposing my ignorance. Until recently I hadn't needed to use arm but now with macos it's gotten more interesting and more complicated.
yjftsjthsd-h
> And, iirc, it was from some other person, not gitlab
That would have to be a different image, then?
shepherdjerred
> The latest arm image, as far as I could tell, isn't the same as the most recent x86. And, iirc, it was from some other person, not gitlab.
Ah, that's fair. If you're running software packaged by others, that might be less well-covered because you'll have to wait until all of the vendors you care about add in that supprort.
If you're developing software in arm64 Docker, I think that case is pretty good today.
jeroenhd
You can inspect the layers of a Docker image. Tools like dive[0] provide a quick and easy way to navigate through the different components your image of choice is made up of.
In terms of functionality once the container is running, you'll have to put some amount of trust into the project maintainers, no more or less than the trust you need om amd64. For containers repackaged by third parties that's quite a pain, but in most cases you can get by just fine with the official container.
If your container of choice has been made by someone real fancy, you may be able to get reproducible builds for all the files inside the container. That would verify that the source and the binary match (though container metadata may not, so a direct image compare would be challenging).
fzeindl
> never having to understand the myriad packaging issues with whatever language the service is written in, python, nodejs, etc)
How do you fix issues with the docker images if you don't understand them?
jrockway
This sounds about right to me. At work, we make a rather complex stack that uses quite a few third-party containers. When we wanted to do arm64 support a couple years ago, most of these dependencies did not support arm64, so we had to build and publish the containers ourselves. (We already sort of had to do this anyway, because customers ran into Docker rate limiting issues, and images from our account aren't rate limited because we pay them not to. But when we only supported amd64, we just re-tagged and pushed.)
As an aside, some comments in this thread say "just look at the layers", but that's the wrong level of abstraction for multi-arch images. In the past, when you ran "docker pull ..." you were looking for an Image Manifest: https://github.com/opencontainers/image-spec/blob/main/manif.... But now in the world of multi-arch, you are getting an Image Index first: https://github.com/opencontainers/image-spec/blob/main/image...
znpy
I’m not excited that much yet, because except for dumb single board computers, i still can’t get a proper arm system to run at home.
My home server (a repurposed fujitsu esprimo q920) is still intel based and it doesn’t seem to be anything available with comparable performance and connectivity. And I’m not even considering the price point.
Basically: arm cpus don’t play any significant role in my everyday computing life.
At work, I’ve been migrating all of our infra to graviton and we realised substantial savings… but then again, I don’t pay the cloud bills and my salary is still the same, so meh.
acdha
I don’t have too many gaps but also don’t use that many different base containers for security and reliability reasons. As you mentioned, I feel like in a decade the current experience of running random code from strangers all over the internet with no more protection than Docker Desktop provides is gong to sound similar to 1970s swingers’ accounts of unprotected orgies sound to all of us who grew up after HIV, etc. where people will kind of accept that it happened but be amazed that everyone was so reckless.
morrbo
You're not wrong...but it will get there and get better. I believe asahi will be a driving force behind it, and arm in general being more widely used for non mobile device stuff....however (despite using fedora on arm64 as a daily driver) I firmly believe we're going to be 6-12 months absolute minimum until arm docker is "alright" (I'm also broadly including fully user transparent x86 emulation into this sweeping statement with no basis lol)
bsnnkv
The friction with using Docker across arm and x86 was one of the big reasons that I ended up learning NixOS. Now, all the services on my personal remote box and all my one-man-SaaS services run on NixOS + systemd services and my life is so much easier and less stressful.
throwaway81523
TFA describes the E3-1230 as an 8 core server when it is actually a 4 core server with 8 threads. That means the ARM vs x86 per-core performance comparisons are off by a factor of 2. I stopped reading when I noticed that. For cheap sustained compute, it's hard to beat a Hetzner auction dedi.
novakwok
Thanks for pointing out, I've made some updates on blog post to make the description more accurate.
ksec
This has always been the case with vCPU. But many didn't know vCPU in many cases means thread and not Core.
tmikaeld
Great article, thanks for sharing!
We're using Hetzners new ARM servers ourselves, to convert images to WebP (Yes, your company name is really confusing!) and they perform almost as good as the Hetzner AMD instances.
But since they're so much cheaper, we can easily fire up many of them and use a load-balancer in front, saving a ton of money compared to dedicated servers.
novakwok
[Yes, your company name is really confusing]
LOL, we're not a company, we are just a small team of three individuals(Nova Kwok,Benny Think and Tuki Deng).
[convert images to WebP]
May I ask your use case on this? (As we've recently launched a product called WebP Cloud might fit this need. (And we're actively seeking seed users.))
WebP Cloud documentation here: https://docs.webp.se/webp-cloud/
tmikaeld
Alright, they way it was mentioned in the article made it sound like a business, sorry about that.
Your service looks great, but we long since concluded that using an API for image conversion would be many times more expensive than using our own setup. And we also have mixed in fetches from external sources, storage in S3, Cloudflare workers and generative AI mixed in the bag - no single service supports all that yet (hint).
adventured
> they way it was mentioned in the article made it sound like a business, sorry about that
It is a business. They're selling a service. I don't know why they're protesting at the notion of being a company, they're de facto a business (selling service behind a brand, which they're openly promoting to sell more services).
dsign
Ha! Another one :-) !
We created a company that does something similar[^1]. The tech was great and the company is profitable, but the market is really, really tough, with incumbents (read: existent CDNs) playing all sort of "standard business practices"[^2] to keep customers in their more expensive business. And yes, in this line of business you really want the cheapest hardware.
[^1]: Support for transcoding images to WebP, AVIF, JpegXL, and selecting on the flight the best format for serving individual images in a website. Company (ShimmerCat AB, a Swedish registered company) is currently for sale; contact the CEO if you want a bargain[^3], last time I heard ask price was X0 000 USD, with X less than 9. I'm not part of the company in any capacity any longer.
[^2]: Read: standard dirty tricks to suppress the competition.
[^3]: Who is the CEO is public in the Swedish registry of companies.
mpweiher
Since moving to Apple Silicon, I've been wanting more ARM options in the cloud. Although it is possible to host x86_64 VMs, having fewer differences is obviously better.
I've been using Oracle's free tier for a while, and it's been OK. Performance-wise, my Objective-S and libµhttpd based web-server appears to be doing around 1800 requests per second, and held up fine to a HN hug of death.
Hetzner was far, far easier to set up, both from their console and via the API. Performance was comparable.
shepherdjerred
AWS has great support for arm64 instances
mduggles
I’ve been migrating workloads away from x86 and towards ARM on AWS and GCP since they’ve been available. This review does a great job of kinda giving you an idea of what you are gonna get as a platform, but if you are interested I strongly recommend the experience on any cloud provider.
While there was some work to benchmark and validate, the cost savings have been non-trivial. Plus this change happened as we were all switching to the M series Macs so ironically now our entire chain end to end is off x86.
aidos
For us it was driven in the other direction. With the introduction of the M1s we knew that we’d be on arm locally soon enough. There was a bit of work in the transition but things have improved since then. Definitely happy running on all arm now though.
Hamuko
I just refer to ARM Lambda runners as a free 20% discount since it makes absolutely no difference in runtime but costs less.
I'd also run ARM database instances but I think those are still not really that readily available.
Arnavion
Alas all my stuff is in Azure, and I'm still waiting for them to offer smaller VM sizes comparable to their existing B line. I currently use a B1s (1 CPU 1GiB) that comes to ~$5/mo while the cheapest ARM VM would be ~$25/mo (2 CPU 4GiB).
dijit
I was keen on migrating to ARM, but there seems to be no benefits from doing so on GCP; I'm open to be wrong here.
From what I understand they're using Ampere Altra, which have single thread performance similar to Skylake; but the cost is equivelant or worse than the x86 e2 series.
e2-standard-4: USD 97.84/mo
t2a-standard-4: USD 112.42/mo
(sustained use discounts apply to neither).
EDIT: I see you're in Denmark and are operations focused. I am too operations focused and just across the bridge in Malmö, maybe we could hang out.
mduggles
Yeah sorry I should have been more clear. Currently the ARM instances in GCP when you use them as spot basically never get interrupted. We’re big into GKE so use them as a preferred node group for interruptible pods. I assume due to the pricing you mentioned usage is very low.
So basically any background jobs or big batch processing jobs that required a lot of CPU time. We have multi-arch container builds so if we can’t scale out the ARM node group not a problem, go back to x86. But it was worth the optimizing to get effectively always available spot instances.
Yeah always open to meet up with folks. I’m on mastodon at matdevdug@c.im.
re-thc
The real hidden gems of GCP are the 90% off spot instances in a lot of regions for e.g. N2D.
ARM makes 0 sense on GCP if you can use those.
johncolanduoni
T2A vCPUs are full cores though right? While E2 and most other instances are hyperthreads.
bennythink
Actually I’m in Sweden. Of course we could hang out in sometime. Just cross the bridge. Here’s my email emeries-atolls.0w@icloud.com
brian_cunnie
I like the article, but I wish there had been an "Abstract" or "Executive Summary" at the top so that I'd be spared having to read the entire article to find out the results. I'd like to have seen something along the lines of the following:
"We found Hetzner's ARM64 offering, specifically the CAX21 with 4 cores, 8GB at $8.40/month, to be a performant and cost-effective alternative to x86_64-based solutions."
langsoul-com
Also add, based on tests arm performed 8% worse than amd64, but this is offset by the 14% savings.
petercooper
Also, notably, the team who did the benchmarking were impressed enough to have actually switched entirely to said CAX instances for their app.
novakwok
Good idea! I've added some TL;DRs at the beginning of the article.
daneel_w
According to Oracle's documentation their Arm servers are not virtualized cores but instead actual on-core tenancy, referred to as OCPU instead of conventional vCPU.
https://blogs.oracle.com/cloud-infrastructure/post/vcpu-and-...
undefined
nezirus
Interesting read. I'd like to know more about alpine problems (even just to confirm my bias against it, unless space savings are the most important thing).
For me, Hetzner is mostly baremetal provider. They have dedicated RX line, and if you have base load, a couple of those could run it all (use hetzner cloud instances for scalling and failover)
novakwok
[I'd like to know more about alpine problems]
Sure, and we're planning to share another post later on the whole procedure of our migration from AMD64 to ARM64, and in that post we'll include more details about Clickhouse's problem if we can definitively establish that the problem was caused by alpine. (After this incident I personally have bias against alpine images too
Comparing alpine and non-alpine images on DockerHub:
https://hub.docker.com/layers/clickhouse/clickhouse-server/2...
https://hub.docker.com/layers/clickhouse/clickhouse-server/2...
There is just ~66MB(255MB vs 321MB) of size difference, my personal advice after this to to avoid alpine images in production as much as possible :P
FlyingSnake
This is a great article and it’s nice to see we’ve lots of alternatives to run ARM servers.
I ran the now defunct Scaleway ARM server mentioned in the article for several years. For €2,99 it was a surprisingly useful machine. I ran several projects (.net core) on it and it was quite good for those simple workloads. I looked for alternatives for a while but nothing turned up until Apple restarted the ARM revolution with M1.
EVa5I7bHFq9mnYK
I've been using a cax41 (16 cores) instance for numerical computations recently. Geekbench scores are 774/10221, costs $0.04 hourly ($27 monthly). Perfectly stable. No throttling (probably not that popular yet hehe). For my specific program it's 10% slower than my laptop's 11980HK processor (8 threads, 16 hyperthreads).
nikita2206
I’m always so taken aback when I compare VM prices from Hetzner/OVH and AWS/GCP.
Similarly sized machine in AWS seems to be around $300 monthly, that’s 10x cost.
jeroenhd
Amazon/Google has fallbacks across regions, several layers of data storage redundancy, high-speed. highly configurable software based networking and so much more.
Hetzner/OVH has machines with almost no failover, with no extra availability zones, with no backup guarantees, very little in the way of custom networking, and doesn't integrate with dev tools quite as much.
They're different products. For most people, going Amazon/Google makes no sense. However, if you absolutely MUST keep your data available after or during a fire [0] and keep your systems running during datacenter downtime, you're better off with AWS/GCP/Azure. SLAs with many nines can't afford cheap servers, and that's where the big cloud providers make a lot of money.
Up until recently I saw a lot of people and companies move back from the cloud to self-managed dedicated hardware in data centers. All most companies need is half a rack in two places and a competent sysadmin team, but externalizing the risks is often attractive because disasters and bad failovers do happen sometimes.
[0]: https://www.datacenterdynamics.com/en/opinions/ovhclouds-dat...
nikita2206
Absolutely no arguing that AWS adds more value.
Another thing about AWS/GCP, they are also good at locking you in. For example you want to shift some workloads to Hetzner while leaving others in AWS, you will get a bill for egress out of AWS.
EVa5I7bHFq9mnYK
>> many nines can't afford cheap servers
I wouldn't say Ampere Altras are cheaper or worse servers than AWS's Gravitons. And many nines is a fiction anyway. For example, Google Maps had two prolonged downtimes in 2022.
undefined
throwaway2990
> Hetzner CAX11, with a virtualized ARM64 processor, 42 cores, 4GB memory, priced at $4.91 USD per month, referred to as CAX11 for simplicity.
Haha I wish it was a 42 core for $4.91
Small typo for them to fix.
novakwok
Thanks for pointing out, now fixed!
qingcharles
Wow, this is timely. I just bought their cheapest one last night (about $4/mo) to play with and performance test it for ASP.Net Core, vs. their x86 boxes.
I tried to be ultra cheap and not buy a v4 IP but it appears Microsoft doesn't have v6 IPs on all their download servers which is causing me pain.
Get the top HN stories in your inbox every day.
I'm excited about arm in more places but my experience with arm and docker isn't as easy as i expected
Is it just me? When I've started using arm more, I've noticed that docker images are often incomplete or behind the x86 release cycle.
I love the ease of wiring docker images together for all my services (corollary: never having to understand the myriad packaging issues with whatever language the service is written in, python, nodejs, etc).
But when I'm using an arm image, often it is not the same version as the latest on x86, or even worse, is packaged by someone random on the internet. If I were to install the JavaScript service myself, I could audit it (not that I ever do!) by looking into the package.json file, or reviewing the source code, or whatever. There is a clear path to reviewing it. But with a docker image from the Internet, I'm not sure how I would assert it is a well behaved service. Docker itself gives me some guarantees, but it still feels less straightforward.
I've packaged things for an arm container myself and it isn't always exactly the same as for x86.
Is this just me? Am I doing it wrong on arm?