Get the top HN stories in your inbox every day.
orf
dralley
> a standard, absolutely gigantic C/C++ project that re-invents absolutely everything including an event loop, http server, XML parser, a C++ tool to generate C code to create a “str hash” implementation, a custom lexer for WQL and a parser for a “mof” file format.
This is the part that gets missed when people talk about "dependency inflation" re: Rust. Absolutely, it is a problem, but most C codebases of a sufficient size and vintage are vendoring some absolutely insane hidden "dependencies" that, on average, probably get a lot less testing and attention than the average package on crates.io.
There are definitely risks with both approaches.
capableweb
Not sure why what you're saying would be more true for C/Rust/JavaScript or any other language. The perverse idea of adding dependencies without actually reading through all the code you pull in exists in basically all language ecosystems, Rust included.
jstanley
In C it is much more inconvenient to include external dependencies, so it's much more common for people to roll their own implementations rather than using a tried-and-tested one.
gsam
CIM/WBEM goes all the way back to 1996. They essentially wanted a management infrastructure on all kinds of devices (including different architectures, so actually C made sense then), but that also notably included remote access. At the time, SOAP was still popular, so here we are with a rather silly transport protocol and all kinds of overhead reinventing things like SSH. However, the overall goal still makes sense, it was essentially a way of 'object'-ifying everything from logs to other metrics. This fit in with the overall mode of thinking in MS with DCOM and COM (and registry), and structured configuration/management. I'm sure it's paid massive dividends on Azure Linux infrastructure. For highly structured objects, SOAP and XML aren't a terrible fit, but I doubt many people would do the same thing again today.
Honestly, they just needed to rewrite it in a safer stack. However, that still may not have saved them from all these vulnerabilities, given the scope of what they're implementing as remote management protocols. The relative scrutiny, fuzzing and manpower just hasn't been there, especially when it's obfuscated by various layers.
onionisafruit
Not to take away from the rest of what you said, but I don’t think SOAP was _still_ popular in 1996. I don’t think it had become popular yet. I don’t think I even heard of SOAP before 1999 or 2000. I’m not a trend setter or anything, but if it was popular, I probably would have at least heard of it.
homarp
https://www.xml.com/pub/a/ws/2001/04/04/soap.html says SOAP was started in 98
gsam
That's fair, I was more speaking about XML and its use as a form of binary transport. Things like WS-Management and explicit SOAP obviously came a little bit later, and SOAP-like technologies were popularized for more general use in the 2000s. I think it's fair to say my experiences in general lean more towards observing standards groups.
jerf
Greenspun's law, "Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp." is really just a special case of a general principle. There's a lot of cases where if you use the wrong tool and/or don't know the right libraries, or don't or can't use them, or just won't, that you'll reinvent something that already exists, badly.
Cyph0n
Looks like it’s mostly C, and not great C at that… In my humble opinion, this is a scary looking project.
randomfool
Specifically- it appears to be mostly C99, lots of macros, lots of ugly buffer manipulation. But it's tooled for PREFast so you know it's secure! (https://github.com/microsoft/omi/blob/4ce2cf1cb0aa656b8eb934...) (j/k... I haven't seen that in ages). Guessing a port of MS's internal WMI codebase?
This codebase just smells of rot all over.
pjmlp
That is why there is "Modern C", "Modern C++" as talked at conferences, blog posts and endless over here, and then real life code that we actually find out in production.
pjmlp
In typical Microsoft fashion, regardless of how much victories .NET and now Rust dev units achieve, the OS ones keep pushing for C and C++ no matter what.
That is also why we now have WinRT, WinUI, Sphere OS, Azure RTOS, regardless of what Microsoft Security advises.
the-dude
But does it receive email?
jeffbee
The "fix" doesn't include any tests that would indicate that it actually works. Doesn't seem like a very healthy project.
reilly3000
After the JEDI black eye and slow attrition of frustrated developers/CIOs, this feels like a huge blow. Cloud is all about trust. They broke trust by installing the agent without disclosures, and shattered it with this vulnerability. Their customers running Linux vms have a pretty simple migration path to other cloud/hosting vendors, and most people have learned that only having one approved vendor for cloud just doesn’t work. Q4 and 2022 could be real rough for MSFT at least within the scope of affected customers. Moreover the torture that is Teams has got to have consequences as millions have had to suffer, and Windows 11 is a dumpster waiting to catch fire. My Threadripper 1950X can’t run Win11, and Edge is so difficult to detangle it’s practically antitrust-worthy. Oh, and setting up hundreds of thousands of developers to unknowingly commit licensing violations via Copilot…
Whatever magic Nadella brought to revive the company is starting to wear off, at least from where I’m sitting.
908B64B197
> JEDI black eye
How much of that is due to the new administration not liking the previous one?
> slow attrition of frustrated developers/CIOs
Citation Needed
> Moreover the torture that is Teams has got to have consequences as millions have had to suffer
Then why not use one of the 10 competitors for chat apps?
reilly3000
>> JEDI black eye
>How much of that is due to the new administration not liking the previous one?
"According to a New York Times report, after President Joe Biden took over this year, his administration examined the status of the contract and came to two conclusions– that the legal challenges could continue to stall JEDI for several years, and that the technological concept had already become outdated.
The DoD will now have a new system called the Joint Warfighter Cloud Capability (JWCC), in which both Amazon and Microsoft are expected to win contracts, and possibly more cloud players. Unlike the Trump administration, which wanted a single cloud provider, the Biden administration will be dividing the contract between multiple companies, allowing the US military to not get locked into a single vendor." [1]
It sounds as if there was a political component, but mostly that they didn't care for the deal. It was a bad look, but there is no evidence there was failure to perform on Microsoft's part.
> slow attrition of frustrated developers/CIOs That was speaking from pure anecdotal evidence on my part. A friend who ran cloud migrations for SMBs at an MSP share a huge amount of frustration with instability, incomplete services, and poor support. I've also heard from others in the cloud architecture community about concerns about products being less feature-complete, as well as concerns with Cosmos DB specifically. Personally I had some negative experiences with Azure 2015-17 but it seems like quality and UX has improved significantly since then.
>> Moreover the torture that is Teams has got to have consequences as millions have had to suffer
>Then why not use one of the 10 competitors for chat apps?
Many orgs are tied to O365 and have made the decision to centralize services around it. Adding a new app like Slack would require enterprise services to support SAML for user provisioning and new tooling for management. Most complaints I've come across center around rampant UI glitches and concerns with video conference quality relative to other options. My son was forced to use it during the pandemic and had plenty of his own complaints about it, along with those of his teachers. Here's a thread from other user reports[2].
None of this is to that that MS hasn't had wins or happy customers; any large company will have its share of blemishes. Azure's growth story has been really strong, but it seems to be slowing down and investors are taking note[3]. I'm grouchy about some issues with Windows recently and took this opportunity to compile all of the negative stuff I've been seeing about them, but that isn't to say that they are irredeemable - just not invulnerable.
1. https://indianexpress.com/article/explained/biden-government...
2. https://www.reddit.com/r/MicrosoftTeams/comments/mcx4oy/why_...
3. https://www.bloomberg.com/news/articles/2021-07-27/microsoft...
mdriley
isoprophlex
This can't be real, I thought. This is just a stub in a random commit somewhere.
But no. Same code is on master branch atm.
What is this, a joke?
"Never attribute to humor, that which is adequately explained by incompetence"
watermelon0
At least it's fast. :D
jamesfinlayson
And properly SAL annotated too.
haimez
An encryption scheme with truly memcpy-like performance characteristics.
cube00
Let's hope memcpy hasn't been #defined
dspillett
I'm going to give some benefit of the doubt and assume¹ that is some sort of stub for dev/test, replaced by one of a selection of proper symmetric encryption options in any production use. Amusing anyway.
¹ Anyone with more spare time than I want to look deeper to confirm or contradict?
formerly_proven
It's used there: https://github.com/microsoft/omi/blob/e4d72481fa2f805148c9c8...
Also note that neither EncryptData nor DecryptData have any way of passing a key in their API. So it's very unlikely that these implementations could be conditionally compiled in place of a real implementation (but they're the only definition of these functions in the repository anyway).
mulander
It used to have actual Windows specific crypto code[1] which has been removed in the linked commit.
I assume this has been ported from Windows and later never implemented the ripped out components. That said, I don't know the windows API so apart from confirming that they exist in Windows docs[2] I can't assess how valid their usage was.
[1] - https://github.com/microsoft/omi/commit/edbe231042173018c529...
[2] - https://docs.microsoft.com/en-us/windows/win32/api/dpapi/nf-...
dspillett
> neither EncryptData nor DecryptData have any way of passing a key
There could have been other smells involved, like the key being held in some more global scope instead of being passed in via the call stack.
smcleod
It blows my mind how easy it is to get root with this - "Simply remove the auth header and you are root."
reilly3000
That the kind of thing you get with homegrown web servers and corporate deadlines. I see how a QA person wouldn’t test that case, it’s almost a base assumption that the header would be mandatory.
flyinglizard
This is the second Azure disclosure from the guys at Wiz, following the one with Cosmos DB few weeks ago. Now, it’s much more interesting when you take into account that Wiz was founded by the guys who sold a cyber security company, Adallom, to Microsoft and then served in senior roles around Microsoft cloud security post acquisition. Assaf Rappaport, Wiz CEO, served as fhe GM of the Cloud Security Group at Microsoft for five years [0].
I wonder what the people at Microsoft are thinking of this situation.
jcims
I'm sure its awkward but they appear to be an equal opportunity reporter - https://www.wiz.io/blog/black-hat-2021-aws-cross-account-vul...
Their product is pretty interesting. I did a demo a while back and their approach of building a large graph and doing a good amount of reachability (network and entitelement) leads to some useful signal. It's not entirely novel, JupiterOne has been doing something in the same vein for years now but it seems to work well. Still feels a bit rough around the edges but i thought it was interesting and could work well in situations where simple checkbox-style config analysis falls short.
keiko-z
I was able to leverage JupiterOne to identify the misconfiguration mentioned in the Wiz article across all my AWS accounts in a single query. Pretty nifty.
They shared that query and a bunch more in this blog: https://try.jupiterone.com/my-bucket-my-data-or-is-it
kerng
Interesting. Would be curious to know what parts they worked at. GM seems high enough to know about "areas to look at".
Still doesn't excuse having such bugs in the first place though.
paulryanrogers
What's the implication? That faults were included or neglected to be sussed out by their next company?
flyinglizard
Absolutely not. But it’s still kinda weird. Like Facebooks CISO leaving their post to found a company which promotes itself with Facebook CVEs. To quote Matt Levine, “I don’t know”.
lloydgrossman
> Now, it’s much more interesting when you take into account that Wiz was founded by the guys who sold a cyber security company
security people sell security company and create a security company. color me shocked.
yellow_lead
Nice out of context quote.
asien
No surprise to be honest.
What rather surprises me are the comments claiming MSFT is suddenly going to go bankrupt because there is a pre-install spyware.
Have you ever worked in a Fortune 500 ?
Do you know how hard it is in those company to get anything done that has not been budgeted and planned years in advance ?
Do you seriously believe Fortune 500 using MSFT are going to suddenly migrate ALL their Azure workload somewhere else like some kind of startups run by 3 devs ?
Yes Azure VMs have spywares builtin , but last time I checked all Europeans companies that are using American Cloud fall under the « Cloud Act » legislation which « legally » requires cloud vendor to hand over ALL the data the company is currently hosting.
I’ve worked in some of the largest insurances in Europe they LOVE Microsoft and Azure , even if there is this spyware issue , they will pretend it doesn’t exist and do business as usual.
Im pretty sure everyone will forget about this news in 1 month or so.
baybal2
I am using Azure on one project as per client's requirement.
One thing I can say, Microsoft hasn't changed a bit. It's a Win98 experience in the cloud.
Now we are fighting Microsoft silently blocking entire ASN of Airtel in Nigeria, and:
1. First, obviously pretending having no idea what are we talking about, and bullshitting us
2. Then not acknowledging. "We checked it, everything is fine"
3. When faced with traceroute - "Maybe it's a third party network provider"
4. When faced with WHOIS record - "I don't know what really to do"
5. "When faced with "this is the last day we are using you" they finally escalated it, and then "It's our DDOS team doing, we have no bearing on them"
6. So, you see it's their official policy to bullshit clients. They clearly knew something from the start, but tried every diversion, thinking they are talking to some $10 per hour anykey man.
Soon found out that Azure silently blocks a big chunk of third world ISPs, and other small datacentre providers without an option to appeal.
skim_milk
I had to tell one of my clients to disable IPv6 on his machine because our Azure Web App wasn't reachable half the time. And he lives in the UK! Support wouldn't do anything after I had to remote in to his computer and run traceroute etc. to figure out it was 100% a problem with Azure itself.
pineconewarrior
Okay, I need to know how you convinced a support agent to let you remote their machine. I could solve so many problems this way.
skim_milk
That was awful wording on my part, I meant to say that I had to remote into my client's machine to disable IPv6 because he couldn't connect to the Azure datacenters we were using half the time except when using IPv4. Literally portal.azure.com and everything else on Azure wouldn't respond until I got my client to disable IPv6.
gibs0ns
This essentially echos my experience anytime I have had to deal with Microsoft support. It's unfathomable how bad their support is and the difficulty to get anything done even when presenting clear evidence/facts.
corty
Microsoft doesn't cater to the more knowledgeable admin population I guess. They are used to the point-and-click crowd mindlessly accepting every answer as gospel. Because usually you can't prove them wrong anyways, everything being opaque and closed in Microsoft-land. And because of that, MS/Windows admins never learned how to get to the very bottom of things.
ficklepickle
I just spent the day making an azure app and taking packet dumps so I can prove it is their fault. Traffic to one of our vendors goes into a black hole, but only from the US South East region.
908B64B197
> Soon found out that Azure silently blocks a big chunk of third world ISPs
Maybe the best recourse here is to get rid of the spam coming out of this ASN.
baybal2
Following this logic, the first whom they need to block should be American ISPs.
North America is the biggest source of spam mail, and DDOS by far.
908B64B197
There's a signal to noise ratio to consider here.
DrAwesome
While I think this article is unnecessarily critical of the Azure OMI agent, this is a very "What the heck, Microsoft!?" moment for me. Of all the pieces of Azure infrastructure, the OMI agent is absolutely something I expect to be well-tested and secure.
I recognize that bugs happen, but allowing a remote client to execute commands as root by simply removing the authorization header should have been caught by automated testing.
icecap12
Wiz is definitely trying to drum up some excitement around their new business with all these recent disclosures. Related but slightly off topic, we pay an industry analyst for high level market research and he's been screaming from the building tops about Wiz.io for the last 6 months. I'm honestly tired of hearing about it, he must get a cut of the sale. I wonder how long Wiz will keep this up; until they get sold presumably, then they'll go the way of the other big cyber firms and tone down the big hacks.
shir1
If you want to see what all the hype is about - I can try arrange a demo meeting for you ;) shir@wiz.io
bostik
Peeking in from the sidelines, as someone who probably will want to arrange a demo too, here's a thing I've been scratching my head with. Your online pitch contains a logical puzzle I haven't been able to decipher.
Patch management and vulnerability checks, right. (VM) OS configuration checks, sure, why not. "No agents or sidecars to deploy" ... hang on, how does that combine with the other two? In order to check for installed/missing patches for on-system software you have to have some kind of access to the underlying system.
Surely you are not snapshotting root volumes for your analysis needs?
0x500x79
I believe that Wiz is snapshotting root volumes for their analysis.
flyinglizard
Very slim chance it’ll get sold. It’s going for an IPO to play alongside other cyber security companies like Sentinel One. The founders are already rich enough from previous ventures.
throwaway984393
Wait, is Azure not patching this on customer systems? They're leaving it up to customers to patch a hole they themselves introduced?
azurezyq
GCE's counterpart doesn't seem to have a public endpoint and its functionality seems make sense: https://github.com/GoogleCloudPlatform/guest-agent
I have to say the problem is not oss, not agents, but Microsoft.
throwdbaaway
https://github.com/Azure/WALinuxAgent - I think this is the equivalence of GCP guest-agent, serving similar functionalities, and is pre-installed on all official images, otherwise basic things like authentication and image baking will break.
By setting the provisionVMAgent property to false when creating a virtual machine, WALinuxAgent should run with all extensions disabled, and I think that's as minimal as a Linux VM can go on Azure.
This property, however, can't be set via https://github.com/ansible-collections/azure, which is of course another lovely OSS project by Microsoft. I didn't bother to send a PR.
The OMI agent seems to be a different beast that is way more obnoxious. The closest thing on GCP is probably the collectd agent and the fluentd agent installed for Stackdriver Monitoring and Stackdriver Logging? Plus whatever OS config to enable unattended upgrades.
I just learnt from this HN thread about the SSM agent on AWS. That one does seem equally obnoxious as the OMI agent.
EDIT: Looks like collectd and fluentd have been replaced by https://github.com/GoogleCloudPlatform/ops-agent
dialogbox
And it is written in Go. I think it is generally a safer choice.
throwdbaaway
https://github.com/GoogleCloudPlatform/compute-image-package... - it used to be just one project with Python, C++, and shell scripts. The Python bits got rewritten into Go, while the others got split into their own repositories.
reilly3000
That immediately makes me wonder about AWS SSM, which I spotted preinstalled on a Ubuntu 18.04 AMI the other day. I think it can’t be accessed outside of the VPC but it’s worth further research. Agents are scary.
Get the top HN stories in your inbox every day.
The apparent fix: https://github.com/microsoft/omi/commit/4ce2cf1cb0aa656b8eb9...
Looks like a standard, absolutely gigantic C/C++ project that re-invents absolutely everything including an event loop, http server, XML parser, a C++ tool to generate C code to create a “str hash” implementation, a custom lexer for WQL and a parser for a “mof” file format.
Still not clear on what it does. Seems safe.