March 20 ChatGPT outage: Here’s what happened

They are not suing anybody, just because it’s open source doesn’t mean talking about bugs is taboo. It reads to me like raw information, very good. They said they contacted authors helping with upstream bug fix. It’s great example how to deal with this kind of problems if anything to me.

Shifting the blame to open source isn't a good look. I like what Bryan Cantrill had to say about it: https://twitter.com/bcantrill/status/1638707484620902401

They're not blaming anyone. Objectively there was a bug in that library that caused the problem.

If someone asked ChatGPT to generate some code, then copy and pasted it mindlessly into their project, I wonder what OpenAI would think about the claim:

"The bug was a result of faulty code produced by ChatGPT."

Using an open source library is like copy and pasting code into your project. You assume responsibility for any pitfalls.

According to this is not fixed yet... https://github.com/redis/redis-py/issues/2624

I don't recall anyone advocating omitting the name log4j when that security bug dropped in 2021. How is that situation materially different from this? redis-py fixed the behavior so it's definitely not the case of working-as-intended.

To be clear, I would be up in arms if OpenAI was trying to hold redis potentially legally responsible.

Damn open source! Those guys aren't living to their paychecks, I foresee staff cuts...

I understand and agree that framing open source as the issue is ridiculous. They may be redeemed though through the likely scenario that they didn't write this - GPT probably did. Perhaps their GPT doesn't like open source. :D

Staffing an actual 24x7 rotation of SREs costs about a million dollars a year in base salary as a floor and there are few SREs for hire. A metrics-based monitor probably would have triggered on the increased error rate but it wouldn’t have been immediately obvious that there was also a leaking cache. The most plausible way to detect the problem from the user perspective would be a synthetic test running some affected workflow, built to check that the data coming back matches specific, expected strings (not just well-formed). All possible but none of this sounds easy to me. Absolutely none of this is plausible when your startup business is at the top of the news cycle every single day for the past several months.

> […] it was because they're based in San Francisco. Do they really not have a 24/7 SRE on-call rotation?

OpenAI is hiring Site Reliability Engineers (SRE) in case you, or anyone you know, is interested in working for them: https://openai.com/careers/it-engineer-sre . Unfortunately, the job is an onsite role that requires 5 days a week in their San Francisco office, so they do not appear to be planning to have a 24/7 on-call rotation any time soon.

Too bad because I could support them in APAC (from Japan).

Over 10 years of industry experience, if anyone is interested.

nobody qualified wants the 24/7 SRE job unless it pays an enormous amount of money. i wouldn't do it for less than 500 grand cash. getting woken up at 3am constantly or working 3rd shift is the kind of thing you do with a specific monetary goal in mind (i.e., early retirement) or else it's absolute hell.

combine that with ludicrous requirements (the same as a senior software engineer) and you get gaps in coverage. ask yourself what senior software engineer on earth would tolerate getting called CONSTANTLY at 3am, or working 3rd shift.

the vast majority of computer systems just simply aren't as important as hospitals or nuclear power plants.

You don't need 24/7 SREs, you could do it with 24/7 first-line customer support staff monitoring Twitter, Reddit, and official lines of comms that have the ability to page the regular engineering team.

That's a lot easier to hire, and lower cost. More training required of what is worth waking people up over; way less in terms of how to fix database/cache bugs.

Probably because they launched ChatGPT as an experiment and didn't think it would blow up, needing full time SRE etc. I don't think it was designed for scale and reliability when they launched.

Do events like this cause them to lose enough revenue that it would make sense to hire a bunch of SRE's?

Just add metrics for the number of times "ChatGPT" and "OpenAI" appeared in tweets, reddit posts, and HN comments in the last (rolling) five minutes, put them on a dashboard alongside all your other monitoring, and have a threshold where they page the oncall to review what's being said. It doesn't even have to be an SRE in this case; it could be just about anyone.

did you reach out via https://openai.com/security.txt?

You won't fire off a quick email nor warn others because there's no bug bounty?

The explanation at the time was that unavailable chat data (due to, e.g. high load) resulted in a null input sometimes being presented to the chat summary system, which in turn caused the system to hallucinate believable chat titles. It's possible that they misdiagnosed the issue or that both bugs were present and they caught the benign one before the serious one.

same to my. actually only the summary of the history was from a different user. the content itself was mine.

It’s a bit unclear if this means that 1.2% of all chatGPT Plus subscribers were active during that 9-hour window

.... "I am asking for this ticket to be re-opened, since I can still reproduce the problem in the latest 4.5.3. version"

The PR: https://github.com/redis/redis-py/pull/2641

According to the latest comments there, the bug is only partially fixed.

> "If a request is canceled after the request is pushed onto the incoming queue, but before the response popped from the outgoing queue, we see our bug: the connection thus becomes corrupted and the next response that’s dequeued for an unrelated request can receive data left behind in the connection."

The OpenAI API was incredibly slow and lots of requests probably got cancelled (I certainly was doing that) for some days. I imagine someone could write a whole blog post about how that worked, it would be interesting reading.

Was this written by ChatGPT? Maybe it found the bug as well, who knows.

… in this case this variant seems more appropriate:

  There are 3 hard problems in Computer Science:
  1. naming things
  2. cache invalidation
  3. 4. off-by-one errors
  concurrency

their description of the problem seemed kind of obtuse, in practice, these connection-pool related issues have to do with 1. request is interrupted 2. exception is thrown 3. catch exception, return connection to pool, move on. The thing that has to be implemented is 2a. clean up the state of the connection when the interrupted exception is caught, then return to the pool.

that is, this seems like a very basic programming mistake and not some deep issue in Redis. the strange way it was described makes it seem like they're trying to conceal that a bit.

I think most app using redis-py rarely cancel async redis command. Python async web frameworks is gaining popularity, but the majority of people using python for their web application is not using an async framework. And of those people that do use them, not many of them canceling async redis requests often enough to trigger the bug.

There is a 1 year old autoclosed issue which is very similar to OpenAI's issue: https://github.com/redis/redis-py/issues/2028

1. Redis can handle a lot more connections, more quickly, than a database can. 2. It's still faster than a database, especially a database that's busy.

#2 is an interesting point. When you benchmark, the normal process is to just set up a database then run a shitload of queries against it. I don't think a lot of people put actual production load on the database then run the same set of queries against it...usually because you don't have a production load in the prototyping phase.

However, load does make a difference. It made more of a difference in the HDD era, but it still makes a difference today.

I mean, redis is a cache, and you do need to ensure that stuff works if your purge redis (ie: be sure the rebuild process works), etc, etc.

But just because it's old doesn't mean it's bad. OS/390 and AS/400 boxes are still out there doing their jobs.

I'm confused on why the need to complicate something as seemingly-straightforward as a KV store into a series of queues that can get all mixed up. I asked ChatGPT to explain it though, and it sounds like the justification for its existence is that it doesn't "block the event loop" while a request is "waiting for a response from Redis."

Last time I checked, Redis doesn't take that long to provide a response. And if your Redis servers actually are that overloaded that you're seeing latency in your requests, it seems like simple key-based sharding would allow horizontally scaling your Redis cluster.

Disclaimer: I am probably less smart than most people who work at OpenAI so I'm sure I'm missing some details. Also this is apparently a Python thing and I don't know it beyond surface familiarity.

Yes! I have been spending the last couple months pulling out completely unnecessary redis caching from some of our internal web servers.

The only loss here is network latency which negligible when you're colocated in AWS.

Postgres's caches end up pulling a lot more weight too when you're not only hitting the db on a cache miss from the web server.

For caching the query results you get from your database. Also it's easier to spin up Redis and replicate it closer to your user than doing that with your main database. From my experience anyway.

Better concurrency (10k vs ~200 max connections compared to postgres). ~20x faster than Postgres at Key-value read/write operations. (mostly) single threaded, so atomicity is achieved without the synchronicity overhead found in RDBMS.

Thus, it's much cheaper to run at massive scale like OpenAI's for certain workloads, including KV caching

also:

- robust, flexible data structures and atomic APIs to manipulate them are available out-of-the box

- large and supportive community + tooling

My redis clusters are 10x more cost effective than my postgresdb in handling load.

For caching somewhat larger objects based on ETag?

People know it, that's all.

That sounds more like a hindsight thing. In most systems authorization doesn't happen at the storage layer. Most queries fetch data by an identifier which is only assumed to be valid based on authorization that typically happens at the edge and then everything below relies on that result.

It's not the safest design but I wouldn't say the client should be expected to implement it. That security concern is at the application layer and the actual needs of the implementation can be wildly different depending on the application. You can imagine use cases for redis where this isn't even relevant, like if it's being used to store price data for stocks that update every 30 seconds. There's no private data involved there. It's out of scope for a storage client to implement.