John Carmack's arguments against building a custom XR OS at Meta

I just realized that you might not know what a "bootcamper" is. Facebook's hiring process generally goes like this:

- you're interviewed with a random team and evaluated if you'd be a good fit for the company.

- you are hired and go through a multi-week "bootcamp" to learn FB's vocabulary, processes, and tech stack, fixing some real bugs and implementing some real (but minor) features in the process.

- upon completing the bootcamp you seek a team that is of interest to you and if interest is mutual, you join the team. If you can't find a team after X weeks, you part ways with the company.

Knowledge of C (XROS was written in C and during the interviews the candidate rather uncommonly wasn't given a choice of programming language) and general understanding of how a computer works at a low level. Knowing the purpose of "volatile", understanding cache lines, mapping virtual memory to physical memory, DMA, this kind of thing.

I think everyone had a degree but looking at my degree (applied math) in particular nothing that I had learned at the uni was immediately useful and I think there isn't really anything that would prevent a smart person with a GED and some history of, say, Linux kernel contributions from succeeding on a team like this. Except may be a degree is needed for H1B visa for those who need it.

If it's so bloated then just start cutting

Whatever expertise you need to prune a working system is less than the expertise you'll need to create a whole new one and then also prune it as it grows old

Perverse incentives are everywhere...

Laughed out loud

I may be the only person who ever understood every detail of C++, starting with the preprocessor. I can make that claim because I'm the only person who ever implemented all of it. (You cannot really know a language until you've implemented it.) I gave up on that in the 2000's. Modern C++ is simply terrifying in its complexity.

(I'm not including the C++ Standard Library, as I didn't implement it.)

This is a pretty standard document length. Modern microcontrollers have similar lengths (e.g. ATSAMD51 is ~2000 pages). Some of it is not software related, things like pin outs and electrical and mechanical descriptions.

It does take a huge amount of work to write and maintain. Typically the authors are not technical, so it also relies on the designers being available to answer questions as well. Then there’s a choice of how it’s written: narrative and potentially imprecise but readable, or terse and precise but hard to read. There’s both styles in the same document, terse for register descriptions.

Look up the Texas Instruments am3358. It's a tiny SOC, it was used in the beaglebone black. Its technical reference manual[1] is over 5000 pages, and it details all peripherals, all of the interconnects and every single register in the system. This, by contrast, is really just an overview.

Regards to (1), if you don't publish this information you're not selling a CPU, you're selling a very expensive chunk of sand. There is simply no way that a customer can guess at what your implementation looks like. Additionally, Intel barely has IP in the traditional sense. They hold patents, but their only real competitor in making x86 processors, AMD, has a long-standing mutual non-enforcement agreement wrt patents.

Regards to (2), I'm guessing a majority of this PDF can be generated sort of like you generate API documentation from doxygen comments.

[1]: https://www.ti.com/lit/ug/spruh73q/spruh73q.pdf?ts=175651560...

> Real question: Why do you think Intel does this?

I'm not sure large traditional silicon vendors like Intel, TI, et al re-evaluate the documentation requirements (and costs) on a chip by chip basis. It's probably done by chip class and for companies who've been selling chips by the millions over many decades to industries as diverse as defense, aerospace, automotive, etc there are classes of chips where robust, complete documentation is not only expected but often a required part of the RFP, compliance or conformance processes.

While this level of effort probably isn't needed for every chip in that class, it could be hard to reliably predict when a general purpose chip is still in the design phase which customers may be interested in it during its life (which for some of these chips might be decades). Many chips which conform to MIL-SPEC or other similar standards which can require extensive documentation are simply enhanced versions of standard chips, so the docs exist anyway. Finally, there's the organizational capabilities and culture aspect. Once the org needs to maintain the systemic ability to generate serious documentation at scale, you end up with a lot of managers and staff who think this way.

For datasheets that's normal. Might even be leaning towards smaller than average for the device in question.

For comparison, a data sheet for a single transistor can be around 12 to 30 pages. A data sheet for a tiny microcontroller is probably a few hundred pages.

I once wrote a driver for a flash chip and that had a data sheet of around 80 pages.

In terms of (2), I wonder if it's even possible to write a driver without such a document. In the end, the vendor is on the foot for the driver for major platforms (let's assume Linux) - if they can write a Linux driver without a similar spec to this doc, then the doc probably doesn't need to exist since the business wins from hobbyist drivers will be low. If they can't though, then it's just a matter of formatting an internal document for public consumption - the doc itself has to be maintained anyways so the cost seems lower and maybe reasonable. I have a feeling the doc is necessary but I am not specialized in the field.

Assumptions, fair or not, about (1) seems more likely somehow.

Probably CPU vendor culture? I forgot how large Intel's manual set is, but ARM's was ~11k pages the last time I checked. Intel's was smaller, but not that much smaller, certainly within an order of magnitude.

Good christ this is my current work laptop. It...mostly doesn't work. Plug in a USB camera and it'll just go. Several drivers, userspace utilities and other daemons and sometimes gstreamer works, but does Zoom work? Who knows!

I mean when the hobby OS wants to shut down, it can power the machine it's running on down. Not unlike what would happen if you clicked power off on your desktop OS menu.

Getting it to work on GCP meant properly driving something called the Intel PIIX4 controller which was emulated into the VM.

Separately from the OS being able to turn itself off, the OS needs to process a signal received by the hypervisor on this controller to support the hypervisor gracefully shutting it down. Otherwise GCP will wait 90 seconds after it has sent the shut down signal to give up and terminate the VM itself.

The problem I was trying to solve was (a) OS can shut itself down in GCP (b) restarts in GCP from the GCP console would be instant, rather than take 90+ seconds

> You are kinda wasting your time, the only way to make it work is to make the hobby OS Linux.

Not the parent, but of course they're wasting their time... That's the point of a hobby OS.

I'm working on a hobby OS, and I have no illusions that it's most likely fewer than 10 people will ever run it, and less than 100 will hear about it, but it lets me explore some interesting (to me) ideas, and forces me to learn a little more about random pieces of computing. If I ran on GCP, I'd want the reboot button to work. That sounds useful.

On the topic, I don't see why anyone would want to build a general purpose OS. There's enough already and even with the shrinking of hardware variety, there's a lot of stuff to support to make a general purpose OS work on enough hardware for people to consider using it. You can take Linux or a BSD and hack it up pretty good to explore a lot of OS ideas. Chances are you're going to borrow some of their drivers anyway, and then you'll end up with at least some similarity... may as well start there and save a lot of time. (My hobby OS has a custom kernel and custom drivers, but I only support a bare minimum of devices... (pc) console i/o, one real NIC, and virtio-net... that's all I need; I might add support for more NICs and more consoles later)

I write documentation as I'm writing the code. In my opinion, the code is only as good as its documentation -- they're two parts of the same thing. It's mostly comments at the top of files, and sometimes a markdown file in the same directory.

This way, good documentation is priced into my estimate for the project. I don't have a work item "spend a few days documenting." Nope, if I'm doing a foo then that includes documenting a foo at the same time.

With documentation one of the major hurdles is the maintainance. It is caring for a set of documents, created by people with different specializations, that describe the artefact from specific perspectives, but need to be kept in sync with the active creation and evolution of the artefact itself.

This is not impossible, but the effort and costs required are substantial and often lose out on a priority basis to just fixing or improving the product itself.

    > Look at Apple's documentation

You make a good argument, but let me take the other side:

What you describe is probably necessary for getting _fast_ Linux compatibility. However, if you are willing to take the overhead of a few layers of indirection, you can probably sandbox the Linux land somewhere, and not have it impact the rest of your design much.

Most hardware access doesn't have to be particularly efficient. And, yes, for the few pieces of hardware that you do want to support efficiently (eg your storage devices or networking, whatever you want to concentrate on in your design) these you can handle natively.

Btw, I would suggest that most people these days should write their toy operating systems to run as a VM on a hypervisor like Xen or similar. The surface to the outside world is smaller that way.

If you're going this route, I have found netBSD a better option for this sort of thing.

It has a rump kernel architecture which makes reusing the drivers almost trivial compared to reusing linus drivers with a new kernel.

  > you're never going to get there off of recycling linux's device drivers because that forces you to design your new OS as a linux clone in which cade you definitely did not need to write an entire new kernel from scratch.

probably it would have to be very general in design, but something along the lines of driverkit or iokit might work?

Is this the old 'an OS is just a bag of buggy device drivers' argument?

    > custom Wifi board

I don't like Meta, but there used to be a time where big corp used to spend 30% of its budget on R&D. It's how we got all the toys we have now, R&D labs of big Bell and others.

So please don't mock the spend. Big spends fail sometimes, and at least people were paid to do the work.

I don't know how you get paid for it, but if you want to write your own kernel, I'd start with an osdev tutorial. started with this one [1], but this one [2] has a promissing name... and I haven't really looked around.

It helps to have a concept to guide you too, but you can certainly make some progress on the basics before you figure out what you really want to do.

[1] https://wiki.osdev.org/User:Zesterer/Bare_Bones

[2] https://osdev.wiki/wiki/Multiboot1_Bare_Bones

Asahi Linux.

Isn't that what low level does on his YouTube channel teach people to reverse engineer stuff?

Impressive. JC is always one of the engineers I look up to and read up to when depressed.

John Carmack, David Cutler, Tom West, Cameron Zwarich, etc. There are about maybe 50 of them.

The quality you can achieve with simple painted textures and computed lightmaps never ceases to impress.

At that time, Rage was delayed forever I consider it vaporware, falling the same category was Half-Life 3 or Duke Nukem Forever.

Still, I saw this demo at that time and I felt it was impressive considering the toy level performance of 2010's smartphone.

> were arguing against fucking John Carmack

I am sure Carmack himself encourages debates and discussions. Lionizing one person can't be expected of every employee (unless that person is also the founder or the company is tiny).

Damn, that's medieval. Anyone should be able to challenge anyone regardless of status.

Ugh. Can we as an industry stop blowing people up like this? It’s a clear sign that the community is filled with people with very little experience.

I remember this guy wanted $20 million to build AGI a year ago (did he get that money?), and people here thought he would go into isolation for a few weeks and come out with AGI because he made some games like that. It’s just embarrassing as a community.

I disagree that you should just defer - but it’s sad that politics was obviously consuming and inhibiting his ability to help the product.

Can’t really imagine a better person to argue against?

In my experience the one big problem on the Quest 3 is the user interface. I am still puzzled why they made a floating taskbar with tiny buttons that you have to hit with VR controllers. I have good eyes, decent hand-eye coordination and don't have shaky hands, yet I manage to hit a button at first try maybe 40% of the time. They made a cut-down 2D desktop interface that makes up a small fraction of the field of view for a VR device and called it a day, and then put the user into some virtual room with zero interactable elements.

Meta Quest 3 feels like sci-fi tech with badly executed UI design from the 90s.

Would love to hear Carmack's thoughts on render cost...

I'm sure that's true but I've worked with a lot of engineers that are of this caliber and as long as you can form a coherent logical explanation they will bend they're way more open than you expect. But you got to put in the work to make that argument. They won't take it on faith

If you want to build 3d glasses probably you should fpcus on 3d glassess and software for it.

Not some new OS so you can start making the glasses 2 years later.

Just like when you want to bake a cake, you dont start by designing an oven, or creating an universe from scratch

What was your experience working with him?

Seriously? Have you never had a person more powerful than you tell you that you’re wrong when they in fact are wrong? Often in corporate environments the answer to a “what to do next” question isn’t easily provable, and people who take advantage of this can make life really suck.

of course he has more power, but at this point, he's earned it.

and also, it wasn't enough to "win" against a den-of-wolves place filled with power-players like meta.

Are you arguing that everyone's power is equal? What an asinine position.

Meta's AR/VR division has burned a huge amount of money and years of time, with relatively little to show for it. Now it seems to be on the verge of being cancelled or slashed back, and in response people are saying that this proves VR, something Carmack champions, is commercially untenable or even that Carmack himself is partly responsible for the failed initiative. I don't even entirely agree with him on the question of whether anyone should try developing a new OS, but he's been proven absolutely right that there was no room for him to be that complacent about the use of Meta's resources.

There were almost no kids on the XROS team. The bulk of the team were E6s with graying hairs, multiple kids, and very impressive history of work on other well-known operating systems -- and most of them wrote a lot of code. This was the senior-est team I ever was a member of. Also, the most enjoyable interview process I've ever been through, no bullshit whatsoever and a rare case that I actually had to implement the exact thing that I was asked about during the interview (took me 3 weeks compared to 20 minutes during the loop, go figure).

XROS was an org that hired for specific specialist positions (as opposed to the usual "get hired into FB, go through the bootcamp, and find your place within the company"). At one point we got two separate requests from the recruiting execs: - Your tech screen pass rate is way too low compared to other teams at FB. Please consider making your tech screen easier to expand the pool of candidates. - Your interview-to-offer rate is way too low compared to other teams at FB. Please consider making your tech screen more difficult to reduce time that engineers spend on interviewing and writing feedback.

Anyway, IMO it was a very strong team in a very wrong environment. Most of the folks on the team hated the Facebook culture, despised the PSC process (despite having no problems with delivering impact in a greenfield project), had very little respect for non-technical managers coming from FB proper (the XROS team saw themselves as part of Oculus), and the majority I believe fled to other companies as soon as the project was scrapped. The pay was good however, and the work was very interesting. My overall impression was that most people on the team saw XROS as a journey, not a destination, and it was one of the reasons why it was destined to never ship.

No, no. If you want your VR apps running in two years on something that looks like an OS, just build an app that runs on an existing one.

If you want to be the dominant player in the market in 10-15 years, build the OS and keep funding it.

>Is he now more concerned with protecting shareholder value? Who cares.

It doesn't sound like he's concerned with waste. It sounds like it's a typical Carmack argument - distilled and hyper logical, and his conclusion is more to do with the pointlessness of it. He actually concedes the point that the project may have been highly efficient (which it may or may not have been, he was steelmanning).

His main points seemed to be:

If every cycle matters and efficiency is paramount, just make the project monolithic C++ code. If every cycle matters, that is somewhat incompatible with general purpose OSs, and if it doesn't, the existing landscape is more than good enough. Presumably, he's calling out the absurdity of counter-arguments which are being unrealistic about the objectives of creating a new general purpose OS, while also focusing on extreme efficiency. He states that the requirements to fully achieve these objectives would require a "monastic coding enclave" like Plan 9 OS, and it wasn't realistic even with the high talent in Meta.

And that plays into the second point, which seems to essentially be "new OSs aren't a draw for developers, they are a burden". This is painfully obvious when looking at the history of OSs and software, and it's the obvious reason why "let the kids build their operating system ffs" should result in a reflexive "noooo..." from the greybeards. The deeper point though is that if A. is achieved, the B. Burden on devs will be even more onerous. Therefore unless the entire project is committed to truly moving crowds to new paradigms (good luck, literally billions have been lost here), just use the proven, faily high performance options that have widespread support.

The conclusion is "on balance, it's a bad idea." He's arguing it sharply (although I understand a Carmack steelman is intimidating to attack), but in essence it's a fairly banal and conservative conclusion, backed with strong precedent.

This is how megacorps die. You’re describing Intel-level complacency.

Professional engineers cannot be made immune to criticism.

> This is Meta. Let the kids build their operating system ffs.

the problem was, it was holding back products. Because if youre going to make your own OS, it changes what chips you put in. If you don't know what chipset you're going to have, you don;t know what your pixel budget is, you can't plan features.

It takes about 2 years to get hardware out the door, and another 1.5 years to iron out the bugs and get a "finished" product.

A junior engineer can make an API that can basically live forever as tech debt.

(Especially if it's an "API" persisted to disk)

I can’t say much without it being clear who it was, but a critical low level thing.

And the Sr was generally a very nice person who did not give much weight to levels, willing to engage with anyone.

It's insane how VR has succeeded beyond most people's wildest dreams on the hardware front (all that hardware that goes into a VR headset either sounded like science fiction or seemed like would be exotic stuff costing tens of thousands of dollars), and the software also had standout successes, but it kinda just petered out both in the entertainment and professional realms.

>That monolithic C++ embedded application is going to be fundamentally built out of a scheduler, IO and driver interfaces, and a shell. That's the only sane way to do something like this. And that's an operating system.

Exactly! I picture the choice being grandfathering in compatibility with existing OSes (having the promised performance of their product in fact indirectly modulated by the output of all other teams of world's smartest throughout computing history and present day), vs wringing another OS-sized piece of C++ tech debt upon unsuspecting humanity. In which case I am thankful to Carmack for making the call.

I can understand how "what you're doing is fundamentally pointless" is something they can only afford to hear from someone who already has their degree of magnitude of fuck-you money. Furthermore in a VC-shaped culture it can also be a statement that's to many people fundamentally incomprehensible

I'm not sure if you are trying to be /s, but yeah that's basically what I'm trying to say. Definitely better than working on those recommendation systems.

Damn, I'd pay to work in some serious OS/Compiler teams, but hey why should they hire me? Oh well...Yeah I'm doing a bit of projects on my side but man I'm so burnt out by my 9am-5pm $$ job + 5pm-10pm kid job that I barely have any large chunk of time to work on those.

Carmack saw it as a waste of time. Is this really what we are doing now? Justifying that my waste of company resources is no less inefficient than the others?

One could argue that if it’s not in your swim lane, you just let it fail. And if you aren’t that person’s manager, you tell them the code or design that you are reviewing and thus the gatekeeper is not adequate. Politely. You said your part and no need to get yourself in trouble. Document and move on. If the company won’t listen then you move on. No need to turn it into a HR issue.

Complaining is always an option. The problem is that HR actually takes the complaint seriously.

Just because something isn't an HR violation doesn't mean it's not wrong, rude, or unprofessional. Society is not a computer program. Being tactful is important to well adjusted people.

yeah and Doom Guy takes a lot of issues with Masters of Doom. You get the impression that MoD was looking to create a McCartney vs Lennon story and stretched the truth to do so (there are several factual errors in the book).

In Doom Guy, though, Romero says that after he left iD, he heard from others still working there that the company had become something of a dictatorship under Carmack, and that within X months (I forget how many), half the company had quit. Romero also qualifies several times that they were all in their early/mid-20s and didn't have the requisite life experience to be handling business situations well

Accessing hardware directky via /dev/mem is literally the original kernel bypass strategy, before we got the UIO and VFIO APIs to do it in a blessed way.

I don't care who agreed to what, it's bad engineering practice to take a working successfully launched product and throw out its entire working software stack no matter how inelegant it seems. To what end? What did Fuchsia offer? When it finally shipped -- what, 2, 3 years late? --- custmers couldn't even tell it happened.

And in order to make it happen it also required writing the already-launched HTML-based UI in Flutter/Dart. Again ... why? What for? There wasn't even a working "native" Flutter at the time, despite promises, and there certainly wasn't a working accessibility stack -- no screen reader, no magnification, nothing -- so that all had to be kludged in. It was everything wrong with the "rewrites considered harmful" distilled.

Not to mention terrible for morale, execution, planning, budget, customer satisfaction.

I was just a lowly SWE 3 "IC" just in the trenches, not nearly as "important" as all that, so my opinion mattered not at all. But to me it violated every sound engineering / project planning principle I'd learned in the 15 years of my career up to that point. Just another event that led to me becoming quite cynical about the ability of leadership at Google to actually manage anything of significant complexity that wasn't ads/search related.

Again, Fuchsia .. very neat. But it didn't belong there.

I think what op was thinking of was early harmonyos, seen people confusing those a lot. Harmony now ofc isn’t what

If it's even a rumor then I started it, I just can't imagine Fuchsia serves any other purpose. I don't even usually give Google a lot of credit, but I just can't imagine they made something this useless and misunderstood the feasibility of such an OS this badly. It would be like Hewlett-Packard in the early 2000s levels of incompetence.

Microsoft used to legit do this in the 90s. Recruit bus factor 1 employees from competitors by offering them large salaries.

It was much easier to cripple your competition back when there were several orders of magnitude less software engineers in the world.

Isn't this how folks use Linux today? It's the only tool they know and don't understand the tradeoffs, hurting the product.

It should be fine to give your opinion on efforts.

Carmack had no direct say over research AFAIK.

That’s not how big companies work.

But the name Carmack carries some clout and people listen to him (too) closely because of his reputation alone. This is soft power that automatically comes with responsibility.

This assumes that you would be sincerely listened to, which you wouldn't in a case like this. Higher ups in large organizations don't have the bandwidth to listen to everybody.

Your sub's officers also need to constantly be aware of what to communicate to whom and in which language. Your superiors certainly kept you in the dark about a ton of concerns that were on their plate because simply mentioning them to subordinates would have been too distracting.

It's been the case since the PS3: https://www.psdevwiki.com/ps5/Hypervisor

I think the context is that foveated rendering ties sensor input (measuring gaze direction) to the rendering pipeline in a way that requires very low latency. Past a certain point reducing latency requires optimizations that break normal abstractions made by user land, so you end up with something more custom. I'm not sure why that would require a whole new OS, the obvious path would be to put the latency-sensitive code onto dedicated hardware and leave the rest managed by Linux. If a bunch of smart people thought XROS was a good idea there's probably something there though, even if it didn't pan out.

No it isn't

It is a lot less if you are aiming to support a small set of platforms, don't need general driver support for everything possible accessory and peripheral under the sun, and if your file system usage is limited.

If you are building for a single abstraction, code gets much simpler, instead of building a platform that multiple abstractions can then be built on top of.

If you are China, the vendors are you and money is treated differently than in the west. Balance sheet will accommodate a project like that easily, especially if it decouples them from the US. They’ve already got their own software ecosystem which most people don’t hear about or heard once or twice, and it’s running their tech scene.