The Pumpkin Eclipse

You could but a base level firmware on ROM, with a hardware trigger, and all that does on boot is listen and receive a signed firmware to write to the system. It needs a way to be triggered through hardware examining traffic and that also needs to require the seen command be signed. That recovery boot system needs to be as simple and minimal as possibly so you can have good assurance that there aren't problems with it, and should be written in the safest language you can get away with. Guard that signing key with your life, and lock it away for a rainy day, only to be used if much of your fleet of devices is hosed entirely. It should not be the same as a firmware signing key which needs to be pulled out and used sometimes.

I think that could work, to a degree. There's always the risk that your recovery mechanism itself it exploited, so you need to make it as small and hardened a target as possible and reduce its complexity to the bare minimum. That doesn't solve the problem, which might be inherently unsolvable, but it may reduce that likelihood of it to levels where it's not a problem until long past the lifecycle of the devices.

the bootloader installs the firmware. if you corrupt the bootloader, it can't install anything anymore. you'd need to physically access the chip to use an external flashing device. Some devices have non-writable bootloaders. They have an internal fuse that blows after the first write, so the chip's bootloader is locked. That means you can always flash a new firmware, but you can't fix any bugs in the bootloader.

It's an interesting challenge because the device is nominally "under ISP control" but any device located in a customer's home is under the physical control of the customer. The mistrust between the ISP and the customer leads to "trusted" devices where the firmware, including the backup, can be overwritten by the ISP, but then cannot recover if it gets corrupted. And believe me, the corrupt firmware scenario happens a lot due to incompetence.

This is getting attention because it wasn't incompetence this time.

But how does blank, unprovisioned equipment discover a path to its provisioning server? Especially in light of the new "trusted" push, this is an arms race in a market segment such as routers where there isn't any money for high end solutions - only the cheapest option is even considered.

tl;dr: a social and economic problem, likely can't be fixed with a purely technical solution

Generally the way this works is you have two partitions in your flash chip. One contains the current firmware and the second is a place to drop new firmware. Then the bootloader twiddles a bit somewhere and boots to one partition or the other. There's really nothing stopping you from wiping the previous partition once you're done.

I think some routers still have a single flash partition and the update process here is a lot more hairy and will obviously not retain the previous version after an update.

Apart from attacks like this, there's absolutely no reason to have a protected read only copy of the factory firmware. 99.9999% all you would ever need to do to recover from a bad flash is to just fail back to the previous image.

A proper read only factory image would require an extra ROM chip to store it, as well as extra bootloader complexity required to load from ROM or copy to flash on failure. It's just barely expensive enough at scale to not be worth it for an extremely rare event.

> Sure, disabling firmware updates would have prevented this attack, but it would also prevent security fixes that keep the routers from being turned into a botnet.

But a switch on the route: Flip the switch the router reboots to a known safe OS, that downloads, verifies, and updates the firmware. Then it waits for you to flip the switch back before it will behave as a router again.

Unless attackers manage to steal key-signing codes, and also intercept and redirect traffic to their webserver to send a fake firmware, this seems secure to me. Only downside I'm seeing is that it would be impossible to put in a custom firmware. Maybe add a USB-key firmware option?

> It seems like a misdesign if it's possible to destroy all of the firmware, including the backup.

Humor me; how would that work? If anything, I'd expect it to be easier to overwrite the inactive slot (assuming an A/B setup, ideally with read-only root). If you really wanted, you could have a separate chip that was read-only enforced by hardware, and I've seen that done for really low level firmware (ex. Chromebook boot firmware) but it's usually really limited precisely because the inability to update it means you get stuck with any bugs so it's usually only used to boot to the real (rw) storage.

Eventually in the satellite world, card emulators took over and only the receiver was a vector of attack, but then the receivers started getting simulated too.

The nice thing about emulators is that you could intercept calls that you wanted and send your own response while still taking any and all updates. Hard to break when you have more control than they do.

> Some USB drives (Kanguru) and SSD enclosures (ElecGear M.2 2230 NVME) have firmware and physical switch to block writes, useful to boot custom "live ISOs" that run from RAM.

For the rest of us, there's Ventoy. https://www.ventoy.net/

Funny thing about directv is that because they allowed for many manufacturers to build receivers, directv had little control over the receiver firmware, so these counter-counter measures weren’t necessary at the receiver level.

Other providers that rolled out their own receivers had high control over the receiver firmware and once users figured out how to protect their cards, the receivers became an effective attack vector for the lazy.

But that’s where a lot of the public knowledge about JTAGs really started coming to light. Awfully nice of them to put in a cutout at the bottom of the receiver.

When your provider cuts you off, that’s when you know that your provider has a legit upgrade you need to take. Take the update and then lock stuff up again.

Of course, I don’t think you’re supposed to make mods to your vendor provided equipment…

In the satellite world, this would happen too: old firmware would be cut off. That’s when you go legit for a while with your sub’d card, take the update, and watch your sub’d channels until the new update could be reverse engineered. And probably have some heroes learn the hard way of taking the update and having some negative impacts that are harder to reverse.

Do ISPs and modem vendors roll their own OTA infrastructure and signing key management, or contract it out?

Of course a backbone provider can directly inspect the source and destination IP addresses of any traffic transiting its network. How could it be otherwise? Thats not fingerprinting, it’s just pulling fields out of a struct.

Tor does defeat this though. Rather than seeing the true destination of your traffic they see that of a Tor exit node.

The physical servers do not matter. Someone owns the physical cable.

> They have to parse the TCP headers in hardware anyway

Backbone routers have no need to implement stateful TCP inspection or deal with the transport layer for TCP, dealing with IP is enough.

Is that actually feasible with their budget?

If we are generous and assume there a zettabyte of data a year that they want to store.

At consumer prices, you would have to pay $10B per year just buying hard drives yet alone the operational costs/redundancy.

The budget for all of the US intelligence services is ~$65B. I think if they wanted to actually do what you are describing it would be the single biggest intelligence expense they have and I don't see how you hide that.

Yup. Pretty much all ISPs collect sflow/netflow from their devices to be able to debug problems or detect ddos.

These aren't likely 'top flows', since the C&C data will probably only be a few kilobytes.

So to capture this, you at a minimum need to be logging every TCP connection's SRC IP and DST IP.

And they seem pretty confident in their worldwide map and fairly exact counts, so I would guess they must have probes covering most of the world doing this monitoring, and it likely isn't just 1-in-a-million sampling either...

What do you mean by just run exit nodes? The linked section says that just running exit nodes allows the exit node to steal data sent over plain HTTP. Is that actually a problem? Who's using plain HTTP? The linked section says just running exit nodes doesn't allow deanonymization.

the only problem with tor are exit nodes. they should not exist.

both for safety and ending abusers.

"They can man-in-the-middle the traffic" could be interpreted as them having to actively do something to become the man in the middle, when they already are.

It's likely they just do sampling (think netflow) to get some statistics over the data that's already transiting their network.

The interface to DSL or coax only has to be a layer 2 bridge. You can put many modems into bridge mode so they don't do any layer 3 (IP) at all. For fiber, if you don't use PON at least, even a standard SFP(+) will do.

You can tell I didn't read the article :)

OpenWrt works for some modems pretty fine. It's not straight forward as the VDSL firmware can't often be distributed but poeple use it on avm Fritzbox devices. Also LTE devices are supported. Not sure about cable modems, probably not. It's probably involved and not straight forward so for most users, even technical ones it's no alternative.

The article says that the "modems" affected are the Sagemcom F5380 and ActionTec T3200 which from a quick search looks like full fledged CPEs aka routers with a web interface and NAT, WIFI and all the stuff. They also write about Censys and banners so it looks like they had their web interface exposed to the Internet.

When I hear people say they use OpenWRT I assume they have their modem in bridge mode so that it doesn't even have an IP. OpenWRT would save you in that case.

Ah, that's what I get for not reading the article.

It's huge shame Pascal basically stopped building those boards since AMD and Intel wouldn't play ball. I'd really like to have something like an APU with 10G connectivity with an x86 processor that was not built and designed in China running open firmware. With PC Engines gone now, I think you're basically out of luck.

The PC Engines boards are great for that. I've got mine running OPNSense (FreeBSD) and it's not required any fuss.

It does include information which the original article specifically excluded from mentioning: the ISP involved.

"Windstream" is mentioned in the first paragraph of the Ars article, while the Lumen post makes references to "a rural ISP" throughout the post.

It's a HUNSN RJ36. It came preloaded with pfSense, as many of them do, but I immediately made a full disk backup and then wiped and installed with a Linux distro because, well, "This is Linux. I know this." You're going to find a lot of people who strongly prefer one over the other, and you may find you prefer pfSense over a "do-everything-yourself" Linux distro if you give it a shot. There are also Linux distros that are targeted for network appliances, and setting them up (correctly) can be easier if the distro is built for the task.

There are quite a few machines in this category, and what's in stock at any given time tends to rotate relatively quickly. I think the one I bought might still be available, but you will want to check to see if there is something with specs that will work better for your use case.