Get the top HN stories in your inbox every day.
kdrag0n
l-albertovich
I've been looking into some of the options for containerized development environments and what I observed was that with lima even when using virtiofs it slowed down to a crawl when building a project from a shared directory so I installed orbstack and gave it a try and I was very pleasantly surprised to see that it was super fast, just as fast as if I was running on a local copy.
Needless to say orbstack earned its place in my setup so thanks a lot!
Edit: sorry, I wasn't clear, what I'm using in orbstack isn't a container instance but a full VM. I don't expect to find a huge difference with containers but I figured I'd make it clear so I don't cause any confusions.
bakhtiari_rev
I would definitely buy OrbStack if it become non free.
the only thing that i'm afraid of is this:
- i work in an enterprise company, and they don't want to pay for docker. as a software engineer i want to be able to buy OrbStack pro license with my own money and be able to use it both in company or my personal environment. is it possible? or your future licensing plan would prevent me to use it in professional environment and only allow me to use it in personal environment?
kdrag0n
Yes, you'll be able to buy a commercial/business use license yourself.
bakhtiari_rev
Your answer is still vague. So suppose we have this 3 hypothetical plans:
- Professional : XXX $ - Business YYY $ - Enterprise ZZZ $
which XXX << YYY << ZZZ
I'll repeat my scenario: - I'm a Backend Developer - I don't have any share in the company. - I don't have any management position in the company. - I'm not forcing any one in the company to use OrbStack to progress my mission in the company. - I'm solely using it inside own device provided by the company. i don't have it installed in my colleagues devices. - this company device is only in my own control and is not shared with anybody else in the company. - this company device is not used in any automation CI/CD pipeline in the company. - I'm willing to use it as a docker GUI for the docker images hosted in companies account in Google cloud. - I'm also sometimes/occasionally, willing to use it in my personal toy projects using the same company laptop in my spare time using the same instance of OrbStack license.
based on these are you implying that i'm not allowed to use `professional` license and i need to use `business` license?
Or you are telling that it's okey for me to use my personal `professional` license in this scenarios?
stefandesu
> OrbStack (https://orbstack.dev), which is a Docker Desktop and WSL alternative
Wait why have I never heard of this? I just tried it out and it seems to work amazingly well.
drcongo
I've been using OrbStack for a few weeks now and, while initially skeptical that I would in future consider paying for it, now I think I would. I've found it to be less troublesome than Docker desktop, has some prettier terminal output, and it starts up and quits so fast. These might seem like small quality of life improvements, but added together it's just a lot nicer to use. Looking forward to seeing where it goes next.
kdrag0n
Glad to hear that!
transpute
> Apple doesn't allow third-party VMMs to set the necessary CPU flags for Rosetta.
Would that change with Asahi Linux host OS and macOS guest?
kdrag0n
I can't speak for the feasibility of running a macOS guest on Asahi, but yes, Linux could be in control of the guest's CPU flags. Same goes for the m1n1 hypervisor.
I can't find the exact post but I remember marcan saying that he doesn't want to poke the bear of using Apple's custom ISA extensions on Linux. So even though it's possible, I'm not sure whether it'll actually be done.
jsd1982
Been using vftool to host docker instead of Docker.app. It's been alright so far, got rosetta set up inside to run x86 images. All it takes is a little startup script and some up front work to prepare a disk image with Ubuntu server.
Every once in a while, the guest Ubuntu kernel will oops about smp related things. Every time it boots its clock is reset almost a month in the past and I have to force sntp to correct it. After sleeping the host, the guest clock will not advance so it'll be behind several hours.
I'm not sure where to assign these issues: Ubuntu kernel? virtualization framework? vftool? bad configuration?
moondev
Does vftool use https://developer.apple.com/documentation/virtualization or https://developer.apple.com/documentation/hypervisor ?
https://github.com/evansm7/vftool appears to indicate the former, but I thought the latter was required for rosetta so interested to try this.
EDIT - looks like I had this backwards, vftool indeed supports https://developer.apple.com/documentation/virtualization which is the newer framework that supports Rosetta. The older framework is https://developer.apple.com/documentation/hypervisor
jsd1982
Right, it uses Virtualization Framework. I forked vftool [0] and added direct Rosetta support which was trivial to do according to the Apple documentation; just had to convert Swift example code to Obj-C.
Have had a PR open for several months with no attention from the author. I almost forgot about it until seeing this HN headline.
[0] https://github.com/JamesDunne/vftool-rosetta
The Rosetta support is surprisingly trivial to use. The virtualization framework exposes a mountable volume containing a single `rosetta` executable. In the Linux guest, you just register binfmt_misc support that recognizes an x86 ELF image and point it at the `rosetta` binary. It works for docker too so you can run arm64 and amd64 images.
_joel
Has there been a significant development? Been using it with https://mac.getutm.app/ for a while
jasoneckert
I've also been using the Apple Virtualization option with Linux VMs in UTM and have had zero issues whatsoever. The biggest benefit to doing so in my opinion isn't necessarily the noticeably better performance, but the RAM usage. I've noticed that Linux VMs in UTM that run using the Apple Virtualization Framework use significantly less RAM compared to otherwise because the framework provides dynamic memory allocation. This makes it very well suited to running Linux VMs on systems that have less memory (e.g., an M1 MacBook with only 8GB of RAM).
The only thing I'd like to see added to the framework is the ability for nested virtualization (which is now available in the M2 chips, but isn't built into M1 chips).
moondev
> The only thing I'd like to see added to the framework is the ability for nested virtualization (which is now available in the M2 chips, but isn't built into M1 chips).
Is there an example of this working in any context? Until then I don't see how it can be considered available.
EDIT - Confirmed below it does work in Linux on metal - Props to the Asahi team!
oneplane
Yep, works in M2 in asahi linux, but I'm not sure Apple has it enabled in their framework.
WesolyKubeczek
Somehow I cannot get a Fedora 38 (or for that matter, Rocky/Alma 9) installer to boot with this option, it all works fine on QEMU, though.
There could be knobs to twist but UTM doesn’t expose any, so…
john_alan
Isn’t it really hard to use though? Like you need decomposed Linux bits.
undefined
bippingchip
+1 for UTM. I use it to run an arm Linux distro that uses Rosetta for x86 binary only programs. Works like a charm and blazingly fast!
satysin
Perhaps I am reading what you wrote wrong but are you saying it can use Rosetta within an ARM Linux VM to run x86 applications in the VM?
hsn915
UTM is great in theory but I can't seem to be able to get directory sharing to "just work" and it's very frustrating.
serhack_
Under the roots, utm can choose to use Virtualization.framework or QEMU (IIRC there should be a setting for that). It seems that Apple is actively working on this framework.
aa-jv
At first I thought this was being done to pump us up for new Mac Pro hardware that might be on the horizon, but then I wasn't sure, but now I'm also still not sure it isn't .. ;)
But, regardless, the subject of using a Mac to host a plethora of different platforms (in my case, for build-server duties) is indeed a fascinating subject. If I can get my target/builds set up on the next Mac Pro, whatever that is going to be, I'll be quite happy to do so .. especially if it isn't just doing MacOS/iOS builds, but whatever else my framework of choice (JUCE/Tracktion) supports (Android, Linux .. Windows .. etc.)
detourdog
I think this is being done because the Mach microkernel was designed to run multiple OS's at the same time. I think Apple finally has enough developers to start pushing into new areas such as a framework to advantage Mach.
saagarjha
Virtualization is, of course, based on a very standard KVM-like API. There's not much Mach in its design at all.
aa-jv
"Being done", I'm not sure anything is happening here other than a happy user pointing out that Apple have some advances in this area, but .. your point is that this has been there all along.
Alas, if only there were the hardware at scale which supported Apples' Mach implementation, at favourable economies.
One way this is favourable to this dev is if Apple finally give me a machine which I can use to Build All the Things™, including iOS and MacOS and Windows and Linux and Android, oh my!, virtually and/or with x86/ARM cores onboard, oh yeah!, without having to resort to the typically draconian bollocks on other platforms, or rent someone elses hardware, or whatever.
A new Mac Pro that can be loaded to the gills and host several competing OS's at once, for the purpose of software development? Sign me up, because I'm tired of build-box'ing rigs and fighting that darn license ..
ch_123
Mach was designed to run multiple OS interfaces in the same way that Windows NT was - multiple compatibility layers on top of a common kernel (e.g. WSL1). What is being discussed here is virtualization, for which Mach is no more suited than Linux, Windows, Sys V Unix etc.
wslh
I understand that you can use it with https://developer.apple.com/documentation/virtualization/run... ? What is the difference or improvement about using UTM (macOS newbie here :-) )
In the UTM page it says "Under the hood of UTM is QEMU, a decades old..." so are we talking about the same thing as Apple Virtualization Framework. I am confused qbout the QEMU mention.
bonzini
There are two frameworks, Hypervisor.framework which is a low level abstraction of the processor's virtuazionation functionality, and Virtualization.framework which is basically a QEMU implementation.
wslh
I am confused by the different comments here: does UTM only uses QEMU or it also uses AVF if the chip architecture of the host and guest is the same?
Edit: I think the catch is that QEMU uses AVF if available [1].
[1] https://medium.com/code-uncomplicated/virtual-machines-on-ma...
ewuhic
How does one actually enable AVF on UTM?
I've got some Ubuntu VMs set up already, and as per [0] I cannot find an option called "Use Apple Virtualization" neither in UTM, nor in the VM settings.
I am on Intel Mac.
[0] https://www.jamesarmes.com/2022/12/linux-mac-utm.html#once-m...
c-hendricks
I think it's only available when creating a VM, you cannot change after the VM has been created.
timsutton
Shameless plug for a couple blog posts I have written as explorations of the Virtualization Framework:
- Introduction when Monterey was released: https://macops.ca/macos-monterey-apple-silicon-vms/
- New features added in Ventura: https://macops.ca/virtualization-updates-2022-06/
john_alan
Cool! What’s the difference between Virtualisation.framework and Hypervisor.framework?
sweetjuly
Hypervisor.framework provides only the bare necessities: the ability to manage vCPUs, stage 2 page tables, and handle hypervisor exits. It does not do any sort of virtualization except for the CPU itself (which, on ARMv8, is very little due to hardware providing good primitives already). Virtualization.framework is full operating system virtualization suitable for running modern Linux and macOS guests.
vbezhenar
AFAIK Virtualization.framework is built on top of Hypervisor.framework. Former is high-level one allowing you to create Linux VM with few hundreds lines of code. Latter allows you to run your whatever you want with few tens of thousands lines of code, but absolute freedom when it comes to drivers and every low level detail.
john_alan
Very cool thanks!
infosecb
VirtualBuddy is a great open-source project that takes advantage of this framework: https://github.com/insidegui/VirtualBuddy
jbergstroem
Orbstack is a relatively new player that offers both VMs and containerization on top of the virtualization framework: https://orbstack.dev
psygn89
Side note: It's free for now but will be a paid service later.
eslaught
It's kind of wild that Apple provides code samples you can use to directly create your own VMs. Has ended up being the most straightforward way to run virtualized macOS on a macOS host that I've found, and comes with the nice benefit that your can read the code yourself and fiddle with the knobs.
https://developer.apple.com/documentation/virtualization/run...
w10-1
I can't upvote this enough: just take the sample code and build on it.
I tired of evaluating and learning various wrappers and helpful virtualization applications. Being able to configure directly in a normal language is great, and it eliminates software supply chain concerns.
Plus, I'm finding programs are running faster in my virtualized ubuntu than on macOS "bare metal" (still on Intel, where big memory is cheap)
ayewo
> Plus, I'm finding programs are running faster in my virtualized ubuntu than on macOS "bare metal" (still on Intel, where big memory is cheap)
Can you explain more? Sounds like you are saying your Ubuntu (guest) VM reacts faster to your actions than the underlying (host) macOS?
kdrag0n
It's possible because of differences in performance/efficiency core scheduling and QoS: https://eclecticlight.co/2022/07/18/virtualisation-on-apple-...
antrod
Please God let them use this to virtualize macOS inside iPadOS. Not as a daily driver but for the occasional use of a full desktop OS (when plugged into a keyboard/mouse). Then I could stop traveling with two devices once and for all.
bwilliams
If we can’t have native macOS on the iPad this would be the next best thing. Imagine having Linux or macOS on the iPad that is treated like any other app (sans memory limits and etc.). You could have a full blown desktop at the ready when iPadOS isn’t up to the task.
The iPad could finally utilize the m1 processor to its fullest.
moondev
FWIW this is how it works on ChromeOS today. From the Linux terminal app (crostini lxc container) qemu launches macOS, Windows and Linux vms. They even support nested virtualization on my Framework Chromebook - so vms can run inside those vms.
I agree if Apple implemented virtualization on iPad/iPhone this same way it would be a huge for unlocking their full usefulness and capabilities while still maintaining host isolation/sandbox.
pjmlp
The irony of a monolithic kernel to use containers and VMs inside VMs all over the place.
matthew-wegner
I think more likely is that xrOS will use this to virtualize both iPadOS and macOS. Initial rumors are that it can run iPad apps out of the box...
_s
https://developer.apple.com/videos/play/wwdc2022/10002/
^^ For the curious ones :)
More or less it's an abstraction on top of Hypervisor to make VMs much simpler in Swift.
From what I remember, Docker tried to use this but suffered greatly on disk performance.
serhack_
I'm posting this because I've just found it. I'm trying to boot a custom toy kernel that I'm developing (with virtio support). Wish me good luck!
OSDEV rocks!
serhack_
Just a question: would anyone be interested in a good blog post explaining my journey with OSDEV and Virtualization.framework ?
aa-jv
Don't ask, do. This is a fascinating subject.
I'd be quite happy to see how a fat MacOS machine is being used as a build-server for multiple generations of MacOS versions, as well as how it could be used (via Linux VM's) to also build for Android, Linux, Windows, etc.
Essentially, this subject is all about the build server destination; there is an award for how many different targets you can get from the same code-base, and being able to load up a fat Mac that approaches the nexus, is certainly of interest to us developers.
VM's are key tools in modern build environments. Having a fat Mac running multiple VM's to solve the target/build infrastructure problem is fascinating on all fronts..
sroussey
Would be great to run GitHub actions locally that use whichever VM is required.
mike_hearn
Sure, the collective internet programming community can always use good blog posts, especially on relatively obscure topics!
mactunes
Absolutely, that would be great!
serhack_
yay! Understood
rtpg
Most definitely
andyjohnson0
Yes!
sweetjuly
Fair warning, you might be better suited by QEMU since debug support in Virtualization.framework is not great. QEMU has support for debug stubs out of the box. QEMU does support Hypervisor.framework so your performance should still be good through it, assuming you're using a matching architecture as your host (i.e. don't build an x64 kernel for your ARM Mac)
xrd
I tried a bunch of different options and landed on multipass.
I'm curious whether someone has found a good option for running x86 images on m1 chips.
I'm generally happy with the multipass VMs I'm creating but they are arm architecture so then if i use docker images inside them I'm limited to arm images. Maybe I just haven't explored how to run x86 correctly within multipass?
I'm finding that usually arm images are a step behind the x86 images if they are official, or they are built by someone outside the organization, so I'm never sure I'm using a safe image.
Do any of the options here make it performant to run x86 on apple silicon? Some interesting comments on Rosetta that I don't fully understand.
jlnthws
The key is to ensure your x86 code is run with Rosetta instead of being emulated (with qemu for instance). By upgrading to Ventura and using the latest Docker Desktop which allows Rosetta translation, I was able to run x86 containers 6 times faster than before. Last time I checked, Multipass had no options for Rosetta translation yet (it's planned though).
Keep in mind Rosetta has a few limitations like it can't translate AVX instructions.
joshcsimmons
I've had the best experience using Parallels. Not sure what exactly goes on under the hood but x86 performance on my M1 Mac is snappier than what I've experienced in VirtualBox (have never tried Multipass).
Regardless, performance on my beefy M1 Max MacBook Pro is pitiful compared to running natively on an x86 system.
xrd
You pay for parallels, right? I always try to find something that's fully open so I'm able to control my destiny fully. That's a bit ironic trying to run on apple hardware, but if the entire stack above the bottom layer is open then I can easily switch hardware of course.
greggsy
That’s noble, but open source doesn’t always deliver the same benefits of COTS products, such as stability and support. That trumps ideology for many people and businesses.
eyberg
Translating x86 -> arm or vice-versa is always going to be slow. No amount of software will ever overcome that. If you are using arm you really need to embrace it. Running x86 on arm is only a hack - whether it's apple or some random github - it will only ever be a hack. It is one thing to dev on arm and deploy to x86 but expecting x86 to perform remotely close to the same exp on arm isn't something that's going to happen.
Having said that..
If you really want to run x86 images on arm, and while we're definitely not 1:1 with docker (and don't want to be) we're interested in people's experiences with https://ops.city as there is native hardware accelerated arm support and x86 on arm. (I'm with the company behind it.) It's definitely going to be more performant than a docker image since docker/k8s not only rely on a full blown linux but also abuse iptables, et al. It also has support for bridging (outside of the vm) and 9pfs so that would help alleviate your pain.
vbezhenar
Virtualization Framework is awesome and I'm slowly building my own VirtualBox-like app, learning Cocoa programming. Hopefully will release it some day.
One big thing that's missing is better storage support. Right now it's just disk image and that's about it. No way to implement something like qcow2 with snapshots and whatnot. It's missing a lot of opportunity for VMs.
Another big thing that's missing is USB pass-through.
Other than that: it does absolutely everything I need. It even provides some kind of hook for network which should allow to build userspace router with any network architecture (didn't try it myself, though, but I think it should work).
I was able to implement thing that I missed for years on M1 mac. Basically I run Fedora VM, I installed Rosetta helper inside, and docker. And then automagically my x86 images started to work with reasonable speed. I don't know if Docker Desktop supports it already, I don't want to touch it with a longest stick in the world. But for myself I solved this issue.
It was rock stable in my experience and just worked. qemu on the other hand managed to crash my entire macOS (hopefully that bug was fixed).
Another thing that I liked about it: protocols are "modern" and open-source. Almost every driver is virtio. No ancient cruft. No proprietary things. Eventually other operating systems could be run on this solution.
daneel_w
I suppose this is something else than the Hypervisor framework (https://developer.apple.com/documentation/hypervisor) - its successor or a convenient complement? Does anyone know if there's a convenient command line utility for wrangling VMs built on this framework?
sigmaris
This is higher-level than the Hypervisor framework; this Virtualization framework providers an entire VM with virtio peripherals including a display.
https://github.com/lima-vm/lima can use Virtualization framework for creating VMs, there is also https://github.com/gyf304/vmcli as a very simple CLI utility for running VMs, though it's not very actively maintained.
Get the top HN stories in your inbox every day.
I use Virtualization.framework for OrbStack (https://orbstack.dev), which is a Docker Desktop and WSL alternative.
It's great overall, and fairly convenient, but it has a fair share of bugs and limitations. The Virtualization service crashes with various combinations of Linux kernel version, macOS version, and architecture (ARM/Intel), so making it stable in each setup takes quite a bit of work. Device support is limited (no USB, etc.), and workarounds have other limitations. Rosetta is really buggy.
The popular belief is that Virtualization.framework is inherently faster/lighter than other VMMs like QEMU, but I haven't found evidence to back it up. I spent a day prototyping a custom VMM (with an open-source base) and was able to get dynamic memory allocation working [1], as well as faster file sharing in some cases. There's a lot of other things I could do better (faster, simpler, more reliable) with a custom, tightly-integrated replacement.
But unfortunately, that's not an option. Apple doesn't allow third-party VMMs to set the necessary CPU flags for Rosetta. There are too many users relying on Rosetta for fast x86 emulation for me to ditch it, despite all its bugs.
So yes, I still use Virtualization.framework, but only because I have no choice.
Happy to answer other questions about the framework!
[1] https://twitter.com/kdrag0n/status/1645645284721184768