Get the top HN stories in your inbox every day.
maxbond
fragmede
In Mr. Zatko's previous appearance before Congress* in 1998, he was addressed by his hacker nom-de-plume, Mudge. Perhaps they see fit to continue this practice.
maxbond
That's certainly true, I've seen this testimony before and were that the only thing that was off about the document I wouldn't have made my comment. Any one of these elements isn't sufficient to justify suspicion of anything, it is the preponderance of them.
The use of the Mudge is certainly the weakest of the indicators I've presented; what do you think of the strongest of them, the unusual use of emphasis? What do you think of the three of them taken together? If you were the author of this document and you just found out that the public would never see it, only officials at the SEC et al - would you feel like this document was likely to be effective for that audience? Is there truly nothing about it you'd change in that circumstance?
cbdc_watcher
Why should the style matter more than the content. If our gov agencies are as sensitive to style of communication as you suggest then I would say they are wasting our tax dollars with their extreme focus on aesthetics and spurious semantics.
fragmede
Personally, I think Mudge wrote, hoping that this document was going to go before Congress and wrote for them, and that possibly that he would have to go before them. Thus the (over) use of bold emphasis is for a broad audience of which certain members won't (or can't) read as critically as others.
I'd write like there's no way this doesn't leak to the public eventually. It's Twitter we're talking about. Juicy gossip is the point of the platform so of course it's going to get spread. I don't know that it undercuts its message though.
peteradio
Is there some other legitimate example affidavit which to compare? Is SEC bound by a non-superlative clause? Perhaps the drafting lawyers demanded mandate of flair.
maxbond
To clarify this isn't an affidavit, I just meant that there is clearly a stylistic choice here to have a certain form. Here's an arbitrary example of a criminal complaint (just because it was the first court document I found):
https://www.gwern.net/docs/darknet-markets/2013-ulbricht-com...
The way things are presented in sections with numbered subsections, the way some things in the beginning appear to be redacted, are clearly meant to convey the authority of an official document submitted to a legal body.
I'm sure this doesn't violate any sort of law and that you are permitted to write a letter like this and submit it to regulators. What I'm suggesting is that the stylistic choices they make are indicative of the audience they are writing for, and that it is different from their stated intentions. And that suggests to me that this document has an agenda and is willing to accomplish it through deceptive means (those being, lying about the purpose and audience of the document to enhance it's perceived authority by giving the general public the impression they are the fly on the wall of an official proceeding rather than reading a document for their own consumption that may have no bearing on the outcome of this dispute), which is my personal standard for calling something propaganda.
choppaface
Page 40 has a redaction over the circumstances under which Mudge was terminated. The intro mentions a Board Member who silenced Mudge and the reactions are done in two blocks so you can infer who that person was. This whole report is a clown show.
This is Elon’s attempt to burn US taxpayer money helping launder him out of having to pay ~$10b to walk away from Twitter. It sounds like Mudge has his own litigation regarding severance. So it was a no-brained to join Musk’s mercenaries.
This issue is not worth taxpayer time. You want less regulation? Then stop crying wolf to Congress, and more importantly, stop just doing dumb stuff like buying a company you can’t actually run.
m00x
That's a rather large claim without any substantial evidence.
This is HN and it only serves as a court of opinions, and Elon isn't popular right now, but let's keep ourselves intellectually honest.
Twitter has had several security issues, including Saudi spies[1], a large-scale hack of several celebrities[2], and even recent ones[3]. Twitter is one of the largest tech companies and a very important town square.
Mudge was hired to do a job by Dorsey, and the newly appointed execs made it very clear that they don't give a shit about security, when the lack of safety measures has been demonstrated to be dangerous to the public.
[1] https://techcrunch.com/2022/08/09/twitter-spy-convicted-saud... [2] https://www.thewrap.com/teen-mastermind-behind-elon-musk-joe... [3] https://www.hackread.com/twitter-data-breach-accounts-sold-h...
blitzar
> That's a rather large claim without any substantial evidence.
> Twitter is one of the largest tech companies and a very important town square.
94th on the chart [1] - as for a town square I respectfully disagree, its a website - 12th according to rankings. Just nudges infront of the other famous town square, pornhub and a little further behind, linkedin. Half the 'traffic' of reddit, and 1/3 of apple somehow, not to mention the giant in the room, facebook. Surely if there is a town square that is it ... i doubt many would even agree with that.
> the newly appointed execs made it very clear that they don't give a shit about security
Allegedly
> when the lack of safety measures has been demonstrated to be dangerous to the public.
I am yet to see said demonstration. Are we worried that XYZ blue checkmark twitter users public tweets are not encrypted at rest on their servers? Or that the entire corpus of the library of twitter might be lost in an catastrophic event?
[1] https://companiesmarketcap.com/tech/largest-tech-companies-b...
[2] https://www.semrush.com/blog/most-visited-websites/ (oh how i miss alexa)
flomo
Twitter is under a FTC consent degree, which they allegedly have been grossly violating, so the taxpayers are already up in their business.
undefined
aaaaaaaaata
Implying it can be "properly" run seems mildly disingenuous in the context of your otherwise fair comment.
maxbond
They didn't imply that there was a proper way to run Twitter (they stated Musk was unfit to run Twitter without addressing whether anyone was more fit), and if it did it wouldn't have been disingenuous. It is not disingenuous to believe there is a proper way to run Twitter, that's just an opinion you may disagree with.
noptd
Is the style more important than the content?
Seems like he wrote it with multiple audiences in mind and I don't see how that detracts from his claims or message.
maxbond
Style is context which should be considered to understand content, and without considering it our understanding of the content may be incomplete (as I argue is the case here). I make no claim that either is more important.
I would claim they did not write for more than one audience. I'd refer you to my other comments in this thread for that argument.
undefined
cozos
What is the proper way for emphasis if not with *bold*?
maxbond
Emphasize however you want, I'm not the style police, I'm just arguing the choice to use emphasis rather than plainly state the facts is incongruous with the stated intention of the document and is an indicator that it has a different intention. I'm not saying they did it _wrong_ I'm saying that we can infer something from the choice _to do it_.
a_puppy
Wow, this is a wild read. Some of the most shocking parts:
- Lack of development and testing environments; engineers build, deploy, and test code directly on the production environment
- >50% of employees having access to the live production environment and sensitive user data (getting _worse_ over time)
- Lack of logging of what people did with their production environment access
- 30% of employees' systems had disabled software updates
- Twitter "has never held proper licenses to the data sets and/or software" they used for some ML models
- "The majority of the systems in the data centers were running out of date software no longer supported by vendors"
- Misleading the board (e.g. trumpeting "we have endpoint monitoring software on 92% of employee systems!" but neglecting to mention that endpoint monitoring software reported 30% of employees' systems had disabled software updates)
- Misleading the FTC (e.g. implying that data was deleted when users closed their accounts, when in fact it was not)
majormajor
Paragraph 15 is amusing in that it undermines their weird attempt to connect to current affairs without much basis - it comes right out and says "executives are incentivized to avoid counting spam bots as mDAU" yet the greater thrust of the section is that Musk's dispute of the 5% number - a number based on "mDAU" - is generally "correct." That seems extremely twisted: Musk is claiming Twitter claimed something they didn't, Mudge's claim here is that Twitter execs are highly incentivized to be honest about that number they actually claim. In terms of "total" bot accounts... it's a free service with open signup on the fucking internet. You aren't gonna crack down on that effectively without draconian measures that few people really want. Twitter "intentionally prioritized" growing the base of users they were confident in showing ads to since that's the core of their business? Yawn. Company focuses on what actually makes them money, seems responsible of them to their stakeholders! If the bot "problem" got bad enough that they couldn't monetize, they'd be incentived to fix it; they shouldn't necessarily spend millions on trying to fix it just because people complain. There are bots on Twitter, news at 11, this supports the idea that Twitter wasn't lying in their financial reporting. So why are they leading with this BS part?
The other sections are much more interesting.
danielmarkbruce
The first 20 pages are basically garbage - as you say, Twitter made it clear what they were measuring and communicated it.
He's basically saying "the execs didn't prioritize what I thought was important! It's illegal!"
random314
The fact that the document began with 20 pages of word twisting to pretend that Parag lied to Elon, while no such thing happened is really bizarre.
I thought the whistle blowing was about state actors being granted 100% access to user data. Instead it dedicates the first 20 pages whinging about a non issue. I lost any motivation to read the docs any further.
He also complains about data center ops quality at Twitter. While Mudge pretended this is extremely unusual, I have never worked at any fortune 500 that would meet his criteria of fault tolerance, and I have worked at several. With original architects disappearing and employee churn this is impossible. And why does he think this needs to be whistle blown? He sounds really immature.
Is there any thing of interest beyond the elon musk drama?
danielmarkbruce
Exactly. It's a bunch of stock standard issues.
nrmitchi
When this story first started, there were obviously comments of "this seems timed to support Musk's argument", and counter points of "this would have been in progress long before Musk's offer and has nothing to do with it".
There being an entire section titled "Lying about Bots to Elon Musk" makes this entire document seem flimsy at best.
It's entire possible to make whistleblower compaints about security issues at Twitter without trying to testify to an impending lawsuit.
dang
Recent and related:
--- edit: one thing I regret is not changing the cnn.com URL to the original WaPo reports a few days ago, since they were obviously much better - so I'll add them here:
https://www.washingtonpost.com/technology/interactive/2022/t...
https://www.washingtonpost.com/technology/2022/08/23/peiter-...
---
Twitter CEO Parag Agrawal on whistleblower story - https://news.ycombinator.com/item?id=32565019 - Aug 2022 (82 comments)
Twitter’s former security chief says company lied about bots and safety - https://news.ycombinator.com/item?id=32564630 - Aug 2022 (2 comments)
Ex-Twitter exec blows the whistle, alleging reckless cybersecurity policies - https://news.ycombinator.com/item?id=32562815 - Aug 2022 (597 comments)
Not so recent, but related:
Twitter shakes up its security team - https://news.ycombinator.com/item?id=30026171 - Jan 2022 (110 comments)
Twitter names famed hacker 'Mudge' as head of security - https://news.ycombinator.com/item?id=25115754 - Nov 2020 (172 comments)
perihelions
Also related (thread on the departure of Mudge from Twitter)
https://news.ycombinator.com/item?id=30026171 ("Twitter shakes up its security team")
dang
Thanks! I've added that one.
schainks
Top reply to Parag’s letter is interesting: https://twitter.com/DrTechlash/status/1562211536826863618
Almost makes it look like the same PR people work for both Twitter and FB. Or, these PR tactics are google-able and everyone rips the same template now.
choppaface
Relevant: dang is a YC employee and not a neutral aggregator. For a balanced perspective, it behooves the reader to consider sources that do not have financial conflicts of interest in the industry being covered.
nrmitchi
You're on Hacker News. This entire website has a "financial conflict of interest in the industry being covered", given that the "industry being covered" in this story is "security at one of the largest tech companies in the world".
Further, all dang is doing here is referencing other HN submissions that relate to the same topic. Are you claiming that he's picking and chosing related links based on some form of bias?
grzm
'dang does nothing to hide his affiliation and is widely known on HN as its primary moderator/curator. One of the tasks he takes upon himself is to provide additional links relevant to the submission as he's done here. If you've got additional links you think are relevant, I'm sure he'd be happy for you to include them.
bushbaba
I really hope that's not true. Ooph. "Twitter data centers were fragile, and Twitter lacked plans and processes to “cold boot.” That meant that if all the centers went offline simultaneously, even briefly, Twitter was unsure if they could bring the service back up. Downtime estimates ranged from weeks of round-the-clock work, to permanent irreparable failure"
dehrmann
I suspect a lot of services are in the same state. As things get large, systems become more complex and this is a lot harder to test.
chickin
How many large tech companies do you think can do that? Each service wants to benefit from the use of other services. Ensuring there aren't dependency loops in this complex graph sounds tricky.
ryan_lane
You have to have enough people to "be that asshole" that tells people in their design docs that they are creating a dependency loop and need to design things to avoid it. I did that for quite a while at Lyft, and I'm hoping that people continued doing that after I stopped. My guess, though, is that I and others have missed things and that even somewhere that was careful to avoid it, there's still some places that could make cold boots difficult.
The best way to avoid this problem is to have a secondary datacenter that is hot, and to never let both fail. Of course, this is what twitter is doing. Designing a DR plan where you can lose multiple primary/secondary datacenters is hard enough that basically no one does it.
This is something that really doesn't belong in the whistleblower complaints because it wasn't under his responsibilities, twitter wasn't lying to anyone about having properly implemented DR plans, and twitter's DR plan isn't out of the ordinary across the industry.
danielmarkbruce
Google wasn't 100% certain they could cold boot around 2015. Not sure if it's still true. It's crazy difficult to figure out if you can at some point.
yuhong
Part of the point of SSD based search engines is they are much easier to cold boot as well.
buildbot
If that happens, can the rest of the fucking internet cold boot?
na85
I mean all you really need is DNS. If the root servers come back up I don't see why the rest couldn't.
danielmarkbruce
half joking - wouldn't surprise me if many DNS services have a dependency on s3...
danielmarkbruce
Has anyone worked with him? This report makes him sound immature, unprofessional, and out of his depth with respect to legal matters. Is that an unfair characterization?
void-star
I know Peter Zatko (mudge) professionaly (having spoken at computer security conferences and worked at more than one company that also worked with him), and personally (to an extent, hung out and talked at aforementioned security conferences, talked on irc And email in the good old days of #hack and bugtraq, and have several friends in common, was extremely impressed talking to him about how to disclose a bug I and the l0pht had independently discovered back in the day), and reputationally (the guy demanded, rightfully, a ton of respect amongst peers back then and now). I have a very hard time with this characterization.
calgaryeng
More (content inside the brackets than) out!
imron
But at least the most important part is right (the brackets are correctly balanced.
plugin-baby
Demanded or commanded?
danielmarkbruce
Fair enough, good insight
mrex
It seems extremely unfair. Rather than focusing on the content, you're criticizing tone. It comes off as a structured tactic, given that famous image of Mudge with unkempt hair in a suit sitting before Congress, intended to draw attention away from his words and onto his rough edges. It's an ad hominem attack of a particularly cheap variety.
What sounds immature, unprofessional, and out of depth is Agrawal's mass e-mail tarnishing Zatko's reputation, a tactic that your post seems to repeat from a different angle.
I'd rather have the theory of general relativity written in crayon, than the most beautiful calligraphic illustration of nonsense. Wouldn't you?
danielmarkbruce
I'm criticizing the content. The first 20 pages are... bad.
I should have used the word "seem" rather than "sound" - "sound" does make it seem like I'm criticizing tone.
mrex
>I'm criticizing the content.
With all due respect, no you aren't. There isn't any criticism of the actual content in your post. It's all critiquing the tone - what style he used when writing, who you think he thinks his audience is, how informal his phrasing is. None of that has ANYTHING to do with content, it's all about form.
Aissen
Interesting tidbits to HN users, who like to complain about new Twitter accounts being disabled and required to add a phone number: this is called "ROPO" internally, and multiple executives actually want that to be disabled. But Mudge asked for research that proved it was one of the most efficient anti-spam measure they have at their disposal.
ratsmack
I read most of the document, and even if a fraction of it is true, Twitter will have a lot of explaining to do in front of the SEC. It sounds like it would be easy to find criminal intent.
blitzar
> 32. Unfortunately, as detailed in the rest of this disclosure, Agrawal's misrepresentations about spam bots are just the tip of the iceberg.
If the smoking gun is the bot section, then there is nothing here at all.
SilverBirch
I think a lot of this is... bad. But not bad bad. It's bad in the sense of "This company is rubbish and Mudge clearly fought with Agrawal" but not bad in the sense of "This company is clearly doing illegal things".
For example, Mudge claiming he was ordered not to present a report to the board. He might not be happy about that, but that's perfectly fine, it's not down to him to chose what the CEO decides should be presented to the board, you can have perfectly reasonable disagreements about what's appropriate and at the end of the day the call is down to the CEO. The fact that there are other cases where board members intervene to tell Mudge this reinforces that.
Or the claim that the CEO instructed Mudge to send the board documents they both knew were misleading. This is an explosive claim, but it seems highly unlikely that Mudge can prove Agrawal knew the documents to be false, and misleading is impossible to know, because he can't know the context they were presented in. I think it's just highly unlikely that Twitter's CEO is so incompetent that he's just moustache twirling and lying to everyone, it seems highly likely this will come down to Agrawal having a different opinion or interpretation of the facts.
What really undermines the claim is when we get to this section:
>Agrawal’s tweet was a lie. In fact, Agrawal knows very well that Twitter executives are not incentivized to accurately “detect” or report total spam bots on the platform.
Mudge is massively over-reaching here. At best, an argument can be made that at some point there are some perverse incentives where allowing spam bots could inflate numbers to make the company look successful. But even if that argument were convincing, which it isn't, Agrawal clearly doesn't believe it. It's trivial to make Agrawal's argument here.
That's why this looks like big claims, but unsupported claims. Because where it's clear that people can reasonably believe what they say but disagree with each other, Mudge claims one side must be lying. What could easily presented as openness, honesty and transparency about the challenges the company faces("Mudge asked the Head of Site... what the underlying spam bot numbers were. Their response was "We don't really know.”) - Mudge basslessly claims this is essentially proves they were acting in bad faith.
This all looks designed to be explosive on first sight, but not actually correct in the detail.
infamia
> At best, an argument can be made that at some point there are some perverse incentives where allowing spam bots could inflate numbers to make the company look successful. But even if that argument were convincing, which it isn't, Agrawal clearly doesn't believe it.
It is clear to me that Argrawal is trying to obfuscate and confuse. Every time he is asked about the total number of bots on Twitter, Argrawal subtlety changes the subject to mDAU. He knows this is not the same as total users, but he chooses to change the subject and pretend he's answering the question. This is very shady in my book, and his attempt at subterfuge and lack of transparency is very troubling. I've seen other CEOs do similar things when they don't want to talk about something potentially damaging. The fact that Argrawal artlessly tried to smear Mudge earlier in the week does not make me trust that he has good and forthright intentions. Argawal seems very shady, and I wonder how you can trust him?
SilverBirch
It's not some subtle sleight of hand, the number of bots on twitter is not what is claimed in the legally required filings. We can argue till the cows come home about how many bots they are, but it's not relevant to the legal wranglings that are going on around Musk buying twitter. Musk is saying "The bot number is higher than you're saying" and Agrawal is saying "I've never made any claim on the bot number", and Musk might be right that there are lots of bots, but if he is right, it just highlights how dumb it was to make an unsolicited bid to buy the company with no due diligence.
And this isn't just some arbitrary thing that Agrawal has made up either, there's one very specific reason why you would care about mDAUs not bots. mDAUs are the number of people you can advertise to, and Twitter's revenue is advertising. It's like saying Twitter has an army of 10,000 cats in it's basement. It might be true, but it's not relevant to the value of the business.
danielmarkbruce
It's not shady because it's the thing they report to shareholders. He knows it isn't the same thing, but so do the people asking the question.
He isn't obliged to engage on every question in the exact way the questioner asks.
peteradio
> it's not down to him to chose what the CEO decides should be presented to the board,
Isn't it? Does the CEO get to carte blanche decide to hide security holes from the board?
SilverBirch
In practical terms pretty much yes. The CEO gets pretty wide latitude. Let's start from a pretty standard premise - there are security holes in every company. Clearly we're not going to disclose every single one to the board.
So it's up to the CEO to make the judgement on when something rises to the importance of needing to notify the board or needs input from the board. The board is there for strategy, not for operations. So as long as there's nothing relevant to the high level strategy, there's no real reason to tell them about individual operational issues.
An obvious example here, is that Mudge cites that Agrawal tells the board that they have end point management, but Mudge knows the end point management shows lots of devices aren't secure. The end points being insecure isn't a problem to bring to the board, as long as you have a strategy to rectify it- which presumably they do, since that's why they've instituted end point management in the first place. And remember, the board knows that they've got security issues and they're addressing them, because the board approved them hiring Mudge as head of security in the first place. So the question is why the Head of Security thinks he needs to go and advise the board of all of the operational issues he's tackling. They know there are issues, that's why he's employed.
peteradio
A strategy like "manage endpoints" would not be satisfied if people are actively able to avoid it, no? Wouldn't that mean the board's strategy is being ignored and evidence of it concealed by the CEO?
undefined
Get the top HN stories in your inbox every day.
This document feels out of place to me. It's addressed to various regulators, and at a surface level it has the form of an affidavit or other legal document. But the language is far too familiar to be writing for that audience - they refer to Mudge consistently as Mudge rather than, say, "Mr. Zatko"; they make liberal use of superlatives; they add emphasis using bold text. This is kind of like writing a cover letter where you explain why the job would be great for you rather than why you would be great for the job, it's going to sound wrong to your intended audience, and it will degrade your credibility in their eyes (rightly or wrongly). This isn't the style you would adopt if you wanted such austere organizations as the SEC, DOJ, and they even mention _Congress_ to take you seriously.
This leads me to believe that this is a document for public consumption adopting the aesthetics of a letter of concern sent to regulators, and that this document is being submitted to the court of public opinion. I don't doubt that Twitter executives are borderline fraudulent and may have crossed the line into outright fraud - I'd be unsurprised to learn that about any group of executives at any large company. But this document has more the feeling of propaganda than a serious appeal to regulators.