Get the top HN stories in your inbox every day.
mattbee
This is why my old hosting company changed its auto disconnection code to work at 11am on a Monday morning, never at a weekend.
That way it's the bill-payers that notice and get in touch, not the poor tech on the out-of-hours rota.
Also, it only firewalled the customer's public IPs from the Internet - it didn't stop servers. That way when the account is settled there was often nothing for the techs to do.
Also it was only ever done after the bank reconciliation was marked as complete, just in case...
We relied on goodwill and recommendation of engineers for our growth, so were keen not to make their lives harder for someone else's bureaucratic (or cynical) mistake.
czbond
The method you describe seems like the proper way to achieve this thorny issue.
kyriakos
That's how hetzner handles late payments and only does this after 2 warning emails.
bingo-bongo
That may be true for billing, but Hetzner killed all access to my dedicated server on a weekday afternoon and when I reached out to them within 30 min, the response was “it was done by the networking department, I can’t do anything about it, but they’ll look into it tomorrow..” .. :| wtf?! not okay Hetzner! (My “mistake” was a interface mis-configuration, nothing abusive or otherwise problematic)
Gibbon1
In the normal BnB world anyone with a account gets a phone call before they get cut off.
hazbo
A few years ago I was working at a startup and we had just started moving some stuff over from AWS to GCP. Things were going pretty good with GCP until one day they pulled the plug on everything with no explanation. It turns out that our company credit card details had been fraudulently used without our knowledge. The criminal had decided to use the card to buy Google Ad Words or something like that - this is the same card we use on our GCP billing account. Anyway Google just took prod down with no notice on grounds of fraud. Could happen to anyone.
theshrike79
Never mess with Ad Words, that makes the Alphabet Beast angry every time.
I've heard the exact same Ad Words story a few times now, someone used their CC or had at one point in the distant past used with Ad Words and the system flagged it. Everything gone with no warning, no explanation and no way to complain. Even getting to the front page of HN didn't work =)
paulgb
We were lucky enough that getting to the front page of HN did work for us[1], but it's worth noting that it doesn't even have to be your card! Simply having a card from an issuer (in our case a legit, YC-backed, US-based startup bank) that got flagged was enough to have our account suspended and appeals denied.
mcv
Sounds like credit cards are the wrong payment system for GCP. Do they support direct bank transactions? They definitely should, if they don't trust credit cards (which I can understand).
fortran77
But the person you're replying to didn't use ad words! Someone stole his CC number and used adwords.
theshrike79
Alphabet doesn't care, as far as they are concerned, someone messed with their Holy Money Machine and doesn't deserve to exist.
samstave
Someone needs to pipe ad-words into DALLE and see what the ad-words-as-prompts-as-an-AI-service pumps out. Im out of credits at the moment, else I'd do it.
samstave
OK - I am a fully paid person on DALLE and MIDJOURNEY. Someone give me some google adwords prompts to pipe to these "things" (scary fn things - artists are dead)
staticassertion
I hear way too many stories like this about Google products. Whether it's Youtube, AdWords, GCP, or anything else - you get blackholed by them and there's zero recourse.
I just can't trust Google for something like running production services.
hbn
Several years back I wanted to try my hand at making an Android app and putting it in the Play Store. I knew ahead of time from reading all the horror stories that it's best to make a second dedicated developer Google account because once you turn an account into a developer account you're basically painting a target on your back to be randomly deleted by the powers that be. Additionally, not sure if this is still the case but at the time once you converted to a developer account you could never change your account's country again.
rurp
The last time I tried to make a new Google account it required a valid phone number, so this might require getting a second phone line these days. I wouldn't trust Google to only disable one account if there is a related one using the same number.
stuff4ben
"can't trust Google"
There, fix that for you. If it's mission-critical, there are better customer-focused solutions out there than Google. Despite their massive brain power, they just can't seem to get dealing with users correctly.
hinkley
I recall there was a time in my life where I thought that knowing exactly why a project was failing would make me feel better, but it ended up just making me feel worse. Being right while a project crumbles around you is not an accomplishment. Fix what you can, and move on when they get tired of fixing things.
I don't understand why so many people find comfort in being able to blame a vendor for problems that could potentially end in mass layoffs.
alyandon
they just can't seem to get dealing with users correctly
upupandup
Thing is they have products that we use personally and all of those are linked to GCP. So if we do a chargeback on GCP billing that was by error, nobody to respond, they have the potential to impact ALL of our personal Google accounts.
Doesn't get more dystopian than this. Fuck Google.
hinkley
Google seems to be a giant case study in "why spend ten minutes doing something when I can spend a week automating it instead?"
How do you have so many employees and yet everything important is run by unsupervised computers?
belter
Sounds like those Whiteboard interviews need to incorporate Graph Deterministic Algorithms for Highly-Dynamic Networks...
gumby
Need another Leetcode section on heartlessly screwing your SaaS customers.
HeavyStorm
Wasn't there an algorithm for that?
cft
For Google a customer is a data point in their machine learning set (hopefully weighted by revenue). The decisions are made by a ML model. If you treat your relationship with Google as such from the beginning, you can extract some value if you use Google for near zero value use cases - spam/marketing emails, disposable non-real time computations, Adwords/AdSense on spammy sites.
Not sure if having an Android phone is safe enough for critical use, because it's entirely tied to a Google account.
aaaaaaaaata
GrapheneOS.org, use sandboxed webapps &/or profiles.
cft
Does it have push notifications (Google Firebase)?
le-mark
This, either be big enough for a dedicated account representative (do they even offer that?) or use for noncritical workloads.
mhoad
Does this not cover everything people need right here?
$29/m + 3% of monthly spend https://cloud.google.com/support/docs/standard
Or alternatively $500/m + 3% of monthly spend https://cloud.google.com/support/docs/enhanced
badpun
Credit cards in general sound like a shitshow that maybe the consumer have the stomach for, but it should never be used for critical business processes. Doesn't Google accept more reliable forms of payment?
luka-birsa
A suggestion for engineers that lack business experience. As soon as your project grows beyond the non-hobby scale and starts making real money, invest some time to reach out to company providing the services that keep it alive and find an account manager that will take care of you.
We're a Google customer and of course we had our fair share of issues (btw: it's the same with Azure, based on our experience), and we always escalate to our account manager. That usually starts turning the wheels much faster.
Also, do not be afraid to ask your account manager for additional discounts. Everybody in sales has some wiggle room to get you a better deal. With Google their account manager actually suggested that we work through a reseller and we're getting % off the list prices.
Dealing with Google, Amazon, etc.. is not like dealing with code. It's not only transactional and you need to invest some time in the relationship even if you're the customer.
jnsaff2
I found the Azure reps to be completely useless for any of the actual problems.
I worked for couple of really big enterprises, one of which had such a high commitment that Azure sent two engineers to sit with the teams working on their cloud. Highest level Enterprise support .. and yet for actual non-obvious problems it took them a month to reply with: can't help you with this, you must be doing it wrong.
This was techsupport tho, I'm sure the billing for multi-million commitments was just fine and dandy including wining and dining.
One other company was a big AWS user (and some Office365 so there were Azure pitches too) and GCP tried to get in with the business as well. When we went to their offices the level of patronizing smug that came from their reps was astonishing, so off-putting that I refused all further interaction with them as I wanted to barf.
dharmab
I worked for one of Azure's biggest customers and our account management teams were fantastic. A few individuals were some of my favorite people I've worked with. That's where all the talent in the Azure support org goes- promoted to the biggest contracts.
SoftTalker
Yes the big fish get attention and the small fish get ignored. Welcome to the B2B universe. If you're a small business relative to the scope of your counterparty, you generally won't get good service or have any leverage.
Seek out resellers or providers who are closer to yourself in size. You want it to hurt a bit if they lose your business. Of course you can go too far the other way as well, if you're a large client of a small company you'll probably get good service but they will be living parasitically off of you which isn't healthy and they may not be able to respond quickly enough if your needs grow.
Gibbon1
You comment reminded me of what my dad said about service contracts from IBM back in the 1970's. They promise they'll get a tech on site in 24 hours. They don't promise he'll know how to do anything.
nXqd
just like any other reps from Microsoft services, just completely useless.
notyourday
I have dealt with Google and AWS under 4 different companies that spent between low double digit to high triple digit a month.
General view:
* AWS - Great customer service. Steak house level customer service when moved from cc billing to contract billing.
* GCP - terrible service regardless of spend and regardless of dynamic or contract billing. AM do not know products. AM's are equivalent of BOA branch employees that solve a problem by calling the same number the customer calls and give the phone to the customer who came to see them in their branch, except that the customer service phone is the same opaque process the customer is subjected to. The only GCP product that had good customer service was the original App Engine, supported out of Germany ( I think ). We could get in touch with engineers responsible for it over video chat.
MOARDONGZPLZ
Same experiences here. AWS was awesome. They did crazy things I wouldn’t expect like spend days in person and join us at equinix colo to get the direct connect working. We didn’t even spend that much relatively, under $100k/mo.
upupandup
THIS. I love how AWS AM go out of their way to deliver, where as Google's AM simply start to ghost you when the conversation gets to heated or not contributing to their bonus cheques.
As I've said and others have experienced, GCP's AM are killing their business and we've built up everything around AWS. There's no need to even learn Terraform, all of what we do are automated anyhow without it. CDK is enough.
Screw Google, screw Azure.
aseipp
At a prior gig we were using their FPGA instances and had a way of causing hard faults on the system that required reboots, with specific FPGA bitstreams. Their systems monitored this and when we reached out about it they scheduled at least 2-3 different calls with the engineers/PIs who were directly responsible so we could coordinate with them and help pin down the issue to get things working. We were literally renting by the hour; I don't even think we had an account manager, with variable costs; maybe a few hundred to 3k a month?
imroot
Ironically enough, the last time I was "Courted" by google cloud to move some of our infra there (after it went public that we had raised a round), they were up front about how their "account management sucks" and that we'd have a better "customer experience" if we went through a reseller.
That...always sticks out to me, but, when dealing with the Google Education folks, there were some things that even resellers couldn't do, and I had to email people directly (via LinkedIn stalking) only to have my issue disappear a few days later with no reply.
upupandup
Thing is parent's advice is horrible because the account manager at google simply started to ignore my billing issues. Like I could reach out to no one, the account manager realized I was going to leave and stopped responding.
So the "make sure you invest in an account manager" is not really an option at Google. AWS does this very proactively.
Unfortunately after 5 years, I can still see Google Cloud screwing over their customers like this. They seem determined to send more customers to AWS.
bastardoperator
This is an easy way to accelerate the sale without having to do any of the leg work. The same thing happens when people use procurement vendors like SHI.
As a sales person I'm thinking sweet, I get to send a PO to SHI for example, I don't have to negotiate price, volume discounts, or go back and forth with legal departments depending on whos paper we used. You just need to approve and I get paid, probably quicker than if I used an internal deal desk.
samstave
TOTALLY.
-
I had amazing AWS reps, that I had a good personal relationship. Request your rep to visit your office, and have lunch.
Seriously.
The best reps I had from AWS moved to GCP, and while I couldnt move my loads from AWS to GCP, I still had a good relationship with them. One of the best things an account rep can do is understand your business, and, in my case, actually pushed for changes in the system to our benefit based on how we were using the system.
They have evolved a lot since that occurred, but a good rep is going to seriously push for your success. (They want you to be successful and increase your use! :-) and thats a good ting)
So, make requests for your reps to meet with you face to face.
dheera
Someone should invent a sue-as-a-service ("SaaS") where you can just sue companies in a couple clicks if they shutdown your services or have bad account management that leads to downtime.
Considering the problems are repetetetive, the legal documents can easily be templated and reused and the victim would not have to waste much time or effort in court. Scale the lawyers just like we scale cloud instances.
Maybe even have a way to programmatically sue based on automated downtime metrics.
anonymousiam
The reason this hasn't been done is that the ToS for all enterprise-scale cloud services would prevent success, either by requiring arbitration vs. litigation, or via a disclaimer of liability.
dheera
If it's so easy to prevent someone from ligitating why don't all small businesses do this? For example, why don't restaurants have you sign a ToS on the menu saying that by ordering you agree to not ligitate?
brightball
This is sound advice in my experience. The last company I worked with who was utilizing Google Cloud had a great experience, an account manager and a reseller who offered training as well.
patchtopic
after Azure sunk my employers project with no recourse or discussion allowed, I always advise everyone I meet to run far away from Azure if they value a reliable service.
exikyut
"No discussion allowed" sounds impressive, almost like legal impetus was involved. What parts of the story can be shared?
filoleg
I think when the parent commend said "no discussion allowed", it was in the context of "no recourse", not that they got some sort of a pseudo-gag-order.
I.e., basically meaning that Azure refused having any further discussions between then and the parent user, as attempts to resolve that specific case. Not that they dont't allow the parent comment user to talk about it with others on HN (or elsewhere).
patchtopic
I dug into the archives, every attempt to discuss or resolve the issue got shut down with the boilerplate text of:
=============================
Greetings for the day!
I have received an update from Account Research Team: As part of our strong commitment to the protection of our customers and our interest in preserving the quality and integrity of the Azure Marketplace, we perform supplementary reviews of accounts which may exhibit irregular or suspicious activity. Your account was selected for one of these reviews and after careful consideration, this account will remain closed.
Please understand that we keep security checks like these in place in order to protect the quality and integrity of the Azure Marketplace
=============================================================
The ridiculous thing.. my employer makes huge use of Office365? We signed up for Azure via a slightly different path with no issues.. but I know this could be pure chance it just happens to work as the service could be teetering on a precipice of arbitrary policy application and the rug could be pulled from under it any second with the same infuratingly cheery account deletion..
pdimitar
Pretty sure your parent poster just meant "they cut you off and you can't do absolutely anything about it".
mylons
lol. trying to get an actual human being at google to talk to you is an insane task if you’re not a substantial sized account.
intellectronica
Generally, all cloud providers have two ways of engaging with and billing customers. One is using a credit card, which is very easy for getting started. The other one is having an established relationship with a sales / account team and setting up billing by invoice. Smaller companies sometimes don't understand that and stick with credit card billing for too long. Once you move to production and have your own customers that you can't let down, you should really make the extra effort and move to invoice billing (with all the upfront overhead that entails).
baobabKoodaa
Ah yes, it was the customer's fault. Stupid customer using method recommended by service provider.
doix
It's like looking both ways when crossing the road despite the light telling you that you can go. Sure you have right of way and if a car hits you, it's their fault. But at the end of the day, you are the one getting hit by the car.
Google is at fault for how they handled it, but the parents comment is still good advice for anyone reading this thread.
bastawhiz
How are you supposed to trust a company who told you they wouldn't screw you because of their own error, then screwed you anyway? Having a relationship with them doesn't matter if they still can't be trusted to do business. The type of payment method you use is not a factor that plays into whether you can trust your vendor.
latexr
According to the OP, Google assured them a suspension wouldn’t happen then did it anyway.
In the car analogy it’s akin to looking the driver in the eyes and them signalling to you it’s safe to cross, only to hit the gas and run you over as you do. No amount of looking both ways before crossing would have saved you.
jquery
It sounds more like they looked both ways, they saw a car, they made eye contact and the driver gestured for them to cross, then when they began to cross the driver ran them over. And now they’re getting advice to never cross streets as a pedestrian. Sound advice can also be victim blaming.
Hnrobert42
I don’t take GP’s comment that way. I take it as a suggestion for how to avoid issues. I try not to be quick to judge when someone is offering helpful information.
helsinkiandrew
Hang on - what would you do if you were running a business with hundreds of thousands of users and someone doesn't pay? Was it an accident? has the customer gone bust, or died, or are they trying to defraud you and get a month or two of free service.
Credit cards are exceptionally finicky, very susceptible to fraud, and payment can often fail. They're great for one off purchases and services that don't matter if they don't go through - but not suitable unless you're going to check that payments are made, which the OP seems not to have done. Perhaps GCP notifications should be better (I've experience the similar scenario with AWS and everything got fixed within a day) - but surely its up to the customer to make sure they pay.
Beltalowda
I actually had some payment troubles with Linode a few months back; I had cancelled one of my bank accounts and forgot the CC there was tied to Linode, and because credit cards are not very common where I live and it can take a bit to request one so I couldn't "just" pay. Oops!
100% a mistake on my part of course; I emailed Linode, explained the above, and they said "sure, is 3 weeks time enough to sort it out? I've delayed any action on your account until such-and-such date but let us know if you need more time".
I've been a Linode customer for several years, paying about $100/month. Just a guy running some small things, hardly a big customer.
If Linode can do it, then surely Google can. Yes, you will probably have the occasional non-payment, but is kicking your people off your platform for a simple administrative mistake/mixup that much better? Accepting the occasional non-payment is better than kicking off people at the drop of a hat. I've since paid several more $100/month to Linode and everyone won in this situation: Linode because they kept a customer who will keep paying, and me because I could get my stuff sorted without having to worry about having my services cancelled.
Google's policy is simply short-sighted on every level, and harms Google's business interests too. The reason they can do things like this is because they have money to spare. Every business that's not swimming in cash like Scrooge McDuck treats their customers like this; the "big tech" company are the exceptions.
nosianu
Your rhetorical questions don't match OPs story. According to OP they did pay, manually, not even by credit card.
> however we went ahead and made a manual payment covering all the outstanding amount + extra
(the entire last paragraph of OPs text has those and more details)
I suggest not using and then arguing with and against fantasized stuff when we are talking about a concrete problem for which we have a concrete description, no?
Given that OP paid, of what use is your objection about non-paying customers here?
You also omit this important part:
> They ... assured the project will not be suspended.
sbuk
It's simple - you give them notice during local working hours (i.e. not the weekend), that way the customer has time to rectify, and if they don't you can take action and have proof of warning them of the consequences of ignoring the notification. If they are having payment issues, it affords both parties time to discuss the issues and see what can reasonably done. It's called customer service - something that Google clearly struggles with.
bastawhiz
I run a business with thousands of users. When payment fails, I try again eight times over two weeks. You get notices each time. There's no surprises. There's no excuse for shutting someone off abruptly for credit card issues.
the__alchemist
I strongly disagree. The host must reach out to the customer, person-to-person, before shutting off a service.
undefined
soheil
This is the line where HN becomes not so useful. Maybe try to imagine there are other professions beyond engineering that have existed in parallel that have their way doing things oh I don't know like following standards and best practices in accounting.
jasonlotito
If you are providing someone a service, and you rely on a third party to provide that service, and that third party screws you over, then by extension, you are screwing over your customers.
That is your fault, and your customers have every right to blame you. You are responsible for the due diligence on which providers you use.
Blaming a 3rd party might absolve you of some of the blame, but your still not providing the service you agreed to provide.
And before you try to suggest anything else, the condition here was "having customers you can't let down."
You have customers you can't let down. That's the context.
undefined
michaelcampbell
I'll counterargue with a saying from the driving context; "The graveyard is full of people who were `right`."
klohto
What is this fud? Both AWS and Azure won't suspend your account for several months, even with charges failing. Especially not if you're communicating with them. It's incredibly easy to get a hold of human support in case of the other clouds.
jaywalk
My company isn't very big (we spend in the very low five-figures with Azure per month) and yet we were assigned an "Azure Success Manager" (or something close to that) who I have a standing monthly phone call with, and can reach out to at any time to oversee any issues that may come up. We actually did have a billing issue come up a while ago (it was Microsoft's fault) and although our account wasn't in any real danger of being shut off, she stayed on top of the case and made sure it got resolved quickly.
wongarsu
> we spend in the very low five-figures with Azure per month
That's over $100,000 a year, at that point having a Key Account Manger (or Success Manager or whatever) to talk to should be the expectation.
htrp
That's because Microsoft understands and hires humans for customer support.....
RA_Fisher
That’s awesome customer service! Thanks for sharing.
nucleardog
Had AWS payment issues at one company spending ~$20k/mo. Had some technical issues trying to catch the bill up. Contacted support, obviously concerned. It was entirely a non-event to them. Basically just said as long as you're in contact with us, we're not gonna shut anything off don't worry about it. And... they didn't.
At other companies where we've had an active and involved account rep literally just shot off one email and got back a "no worries, I put a block on the account so it can't be suspended without my approval".
I know in the past I've had personal accounts go a couple months in the red when a card expired and I wasn't staying on top of the emails.
These sorts of things are really only an issue with one specific provider.
CSDude
I forgot to pay 200$ Azure bill on a test account for 3 months because I missed the mails. They just waived that one after 3 months without me even requesting and I've been paying since. Google is just hostile to everyone by default.
JacobiX
We use Azure and Linode for all of our cloud services. The number one reason: we can contact human support at any time and we have a designated customer success manager. In many situation this proved to be critical, especially for resolving paiement issues, etc
laegooose
yes, in my experience, AWS keeps running everything as is when there is a problem with the payment method or a disagreement about bill
andix
Ofc they don’t. Because they charge A LOT for the service. Even if the chance of the customer paying is just 30% it’s probably still a profit (it’s mostly licensing costs you pay with cloud bills, infrastructure costs are maybe 20% of it; hardware, electricity, cooling). And on the way they can charge a lot of late fees, that raises profit even more.
belter
These type of stories, that appear regularly on HN, seem in the vast majority, related to only one specific provider.
b112
Google is just horrible, when it comes to any form of support, paid or not. They make paypal look helpful by comparison.
I really can't understand why anyone would use any of their services. Support counts. It's one of the first things I look at, when evaluating any external partner. What does it cost, how long does it take for it to happen, etc?
Whatever anyone else says in this thread, that's what you need to determine for yourself. Whatever company, what is the support channel like, how responsive is it?
If you don't know? If you can't get immediate emergency response and support, and I mean immediate, then why are you even hosting there?
markbnj
> Google is just horrible, when it comes to any form of support, paid or not. They make paypal look helpful by comparison.
We have been running production on Google Cloud for six years. We are not a large customer: our spend is in the low five figures. We pay for a middle-tier support plan and support has been excellent during this entire period. We very rarely need to take advantage of it, since the underlying systems are extremely stable and almost never cause us any issues (compute engine, GKE, GCS, Memorystore, Firebase, CloudSQL, etc). Just thought casual readers of this thread could use a counterpoint.
_fat_santa
> I really can't understand why anyone would use any of their services.
I can speak to this because a few months ago when I started a project, I went with Firebase. At the time the rationale was, I need something more complex than Netlify/Vercel, but not AWS that will suck up all my time figuring out IAM roles and other dumb stuff.
It helped speed things up in the prototyping stage, but I will likely move my infra over to AWS soon.
vbezhenar
Google tech is the best. Google invented k8s, their k8s cloud is the best in the world. There's nothing similar to GKE Autopilot. Similar things could be said about some other services. People want the best cloud and they hope that those issues won't affect them.
RajT88
A fellow I know used to do contract work with GCP. Same horror stories.
gnfargbl
It would be nice if Google actually told their customers that this is an option. As a result of your post, I realise that I quality for invoiced billing and have just applied for it. Thanks!
If I had an "established relationship with a sales / account team" then yeah, perhaps I would have been informed. Unfortunately my experience is that unless you're spending above some relatively high threshold (>$10k/mo? >$100k/mo? >$1m/mo?) then Google don't prioritise your business.
TomSwirly
> Smaller companies sometimes don't understand that [...] you should really make the extra effort
Let me get this straight. This company does everything they were supposed to, paid on time, and Google's own error cuts them off, and you are blaming them?!?
intellectronica
You don't know that. Maybe that's the case (I'm sure it happens sometimes), but most likely it isn't.
Cederfjard
I think it’s fair not to take any given story at face value, without ”receipts” to back it up. You however seem to me to go in the other direction and barely want to give OP the benefit of the doubt. What details would you like to see in order to be convinced?
From your profile I clicked through to your LinkedIn via your website, and I think it would’ve been becoming of you to disclose that you were a senior engineering manager at Google Cloud up until earlier this year, given the context of the discussion and your stance on it.
rfrey
If you assume, without evidence, that an HN user is lying if they share a story outside your experience -- there is little reason to use HN.
dvfjsdhgfv
I'm very happy Hetzner has all these options built in from the very start. There's no need to make any "extra effort". Actually, most of my servers are on yearly billing to make things even easier for everybody.
madeofpalk
It's not really the payment method that's key here, it's having a relationship with sales / account manager.
smileybarry
Right, but Hetzner also just doesn’t auto-charge you (from my experience, at least). You get invoiced and pay it manually (there’s a link in the invoice), then the invoice is settled immediately in your account panel. I’ve even been late once by forgetting about it and they didn’t cut me off in any way.
tmpz22
Can we talk about how google monetizes (weaponizes) this sales relationship? They force it on you as "common sense" but it makes them A LOT of money.
Ultimately its as bad as car dealerships to require a special connection to get basic tasks completed.
FBISurveillance
Except finance.
At certain scale (in my case, 5 figures/mo spend with a vendor and up), paying upfront is not something finance usually likes and NET-60/NET-90 is the norm.
dvfjsdhgfv
Actually, it was exactly finance in the case of one of my clients. "Dear Mr dvfjsdhgfv, we're receiving invoices from this Hetzner company for some ridiculously small amounts each month which adds to our overhead, could you kindly ask them to bill us yearly? Thanks!"
cosmodisk
Of course they don't like it, but that's why CTOs and the likes should be talking about these things in management meetings and not just fairy dust future plans.
junon
Someone just the other day said Hetzner did the exact same thing though...
dvfjsdhgfv
I'm not a blind fan of Hetzner and I'm aware of their limitations, but billing works extremely well for me, and they're very patient with me when I forget to pay an invoice (sending friendly reminders and not threatening to cut me off right away).
However, I remember a case when I was the culprit - spam was being sent from one of my servers (one of the users had a common username and password, something I should have never allowed) and Hetzner notified me immediately, but they didn't block the server in any way (something I'd expect). They gave me some window for explanation/solving the problem. I was very happy with how they handled the issue - but of course others may have had different experiences.
harha
If you’re dealing with customers that way you should have a big red warning sign everywhere to not run anything in production under any circumstances.
Otherwise don’t take their money and advertise that everything will be easier and more reliable
thedougd
Cloud providers have a minimum spend for invoice billing. For GCP you need to spend at least $2500 a month for 3 months to qualify.
PaywallBuster
Not exclusive to GCP.
Backblaze has previously locked our account on a saturday.
They claim we were putting too much pressure on their cluster.
They could've locked the 1! token that was causing 90% of the load, but they locked the whole account which was used by 10 different apps for a veriety of purposes causing unecessary downtime in some production workloads.
Also the workload has been quite consistent for days/weeks, so they could've detected it earlier and/or address it in a better way.
moonchrome
I don't see this as related - your account triggered some abuse detection - did you ever go to them and validate what you're doing is covered by the terms you're paying for ?
I mean it sounds like a dick move by Backblaze, but comparing it to what OP is describing is downplaying how bad this looks for GCP IMO.
Going through with official contact at GCP, being assured he did everything right and that it was taken care of - and still getting shut down. In my book that's just inexcusable - if you can't count on that kind of support to work then how can you rely on that platform for anything ?
dchia
If you are paying for the load/pressure, what's the problem?
jeroenhd
Backblaze is very cheap and has several "don't overload our system" clauses in their terms and conditions.
They're cheap and reliable, that generally comes at the risk of early termination. I don't think you can have a cheap, reliable, unlimited service; pick two at most.
jsiepkes
Isn't everything metered with Backblaze (ie. no unlimited)? I think only things such as delete aren't metered. But in order to delete something you must have first uploaded it (which is metered).
If everything is metered it seems kind of odd to complain about a lot of activity. Since it's literally the thing you make money with.
smileybarry
They probably meant their B2 storage service, which is like S3 and bills for storage, egress, some operations, etc. That’s one service I wouldn’t expect to have some cancellation for overload in place. You’re using a token anyway, why not just rate-limit you instead?
jquery
Indeed. If a customer tries to backup a petabyte of data on the base plan (because it’s “unlimited”) and they get the boot, they really shouldn’t be that surprised. Wouldn’t really compare it to OP’s situation, where they got an assurance in writing they were in good standing, and didn’t appear to be blocked for violating ToS.
dman
Whoa - how much traffic were you generating?
alphaomegacode
For a company that operates at such massive scale for their own services, Google has some glaring oversights when it comes to enterprise services.
I have a friend who volunteers as a webmaster for a charity org and they kept getting notices about their impending shut down of G Suite and the need to move to Google Workspace. The problem seemed to be a credit card and yet, when my friend was showing me on his laptop, there was no place for them to enter credit card information. They ended up losing all their email and Drive & Docs access for days and couldn't even get an email or phone address - paid or not - to address the issue.
After getting burned at a mid-sized business I used to work at with Angular 1.x and Google Glass, I learned that Google apparently sees business services as experiments rather than long-term investments.
christophilus
Was Angular 1.x an official Google thing? I always thought it was just a Google employee’s project.
alphaomegacode
I'm not sure if it was ever an official thing but I remember accessing their pages and speaking to their reps at conferences regarding Angular.
For example, if you look at the Angular 1.2 docs, look at the bottom of the page here and you'll see Google included their copyright on those docs https://code.angularjs.org/1.2.27/docs/guide/ie
skybrian
I doubt it was ever an official Google product because open source packages generally aren't. But there was a team at Google working on it when Angular 1.x was big.
jamespwilliams
The same is true of DigitalOcean, fwiw. Shutting down machines used in production with no notice, outside working hours. I don’t know why these companies don’t care - they’re massively shooting themselves in their feet. Incidents like this turn customers off for life.
jabiko
I actually managed to get my account suspended one day due to nonpayment. It was just a personal toy project and it was completely my fault: I was automatically filtering the mails into a folder that I didn't read until it was too late.
If you look at the sheer amount of emails they sent me before suspending the account, Digital Ocean did everything right: https://i.imgur.com/ah13bEA.png
They gave lots of warnings and also sent an email with the date the account was going to be closed a week in advance.
dkersten
I was using Digitalocean for something last year and for reasons didn’t have the money to pay for the bill when it was due, they always have one month of warnings. Ie if I paid one month late, everything was ok and nothing was ever shut down and they sent multiple emails during that time. I knew exactly where I stood, what I owed, when I had to pay by, when they would shut down services etc. That seemed reasonable to me.
terom
Just like you setup multiple instances across different availability zones for redundancy, you also need multiple billing contacts in case one of them fails and causes an outage.
Not sure how to mitigate these risks for providers that only support one "project owner" with a single billing address :/
Xylakant
Use a “synthetic” project owner with an email address that forwards to the group.
sh4rks
Is it just me, or does imgur never load on Firefox mobile?
chucksmash
I noticed something similar five weeks ago. Imgur would get stuck on the loading animation if I was in private mode but work otherwise.
It seems to work—at this moment, anyway—even in private mode for me now though. ¯\_(ツ)_/¯
__gtux
It loads fine for me in FF mobile.
kdumont
We went through a similar event last week. Droplets shut down and account locked without warning and because of “violation of the DigitalOcean user agreement”. Luckily, it was on a sandbox environment we use for testing before taking anything to production. All we had done was set up a simple keycloak server.
The real kicker is that they refused to tell us what terms we had allegedly violated because it’s “critical to maintaining our security systems.”
And they had “just conducted a manual review” and confirmed the suspension.
Then, after escalating further and a few choice words about how much I’ve spent on their platform, sure enough they conducted another “manual review” and found no issues.
I’ve really tried to avoid moving to AWS over the years, but I’m really reconsidering.
magicalhippo
> outside working hours
Working hours are relative. Our support routinely gets awoken at like 3am because someone in Singapore, where one of our larger customers has outsourced their IT to, just started their day and decided to follow up on a case.
Though I'll agree that weekends should be considered off-limits regardless.
qabqabaca
> Working hours are relative
Sorry but for me this doesn't cut it when you have staff and availability zones in every continent on the planet. It isn't some mom and pop shop in one town. It shouldn't be much to ask a company of Google's size and breadth to consider your local timezone for urgent communications.
magicalhippo
It was more a comment on how it's difficult to pin down working hours in this international day and age.
I mean we're based in Norway, but what if we outsourced our IT to Chile? How would Google or similar know that they need to contact our IT guys during Chilean working hours, and not Norwegian working hours? If it's an invoice that's not paid, well that's done here in Norway though, so definitely Norwegian working hours there...
Sohcahtoa82
Years ago, I used a cheap droplet on DO as an IRC bouncer to hide my home IP address. Someone came into a channel I was an op in and started spamming racial slurs, so I did what a normal op does and banned them.
They proceeded to DDoS my droplet.
Only a couple minutes later, DO responded by disconnecting my droplet from the Internet for 3 hours, supposedly to protect other customers. Thanks, DO, you let the DDoSer win. My droplet was handling the attack just fine, it just made my IRC connection a little laggy.
As a $5/month customer, I was just small potatoes, for sure. But I had ideas for products in the future, and I decided that I would not use DO for them, knowing that some skiddie could just take it offline for 3 hours by just packet flooding it for a couple minutes.
Croftengea
Care to share more info please? Was it because card didn't go through? I've got some important stuff there.
jiggawatts
It has been covered here on HN several times, but to summarize:
Budget cloud hosting on a credit card allows scammers to use stolen CC numbers to spin up VMs to mine crypto.
Essentially the hosting company is offering a cash equivalent for unreliable credit.
This kind of abuse occurs at enormous scale, because it can be worth it for an attacker to use botnets to throw tens of thousands of credit cards at hosting companies. If you never actually expect to pay any money (especially your own!), then even thirty minutes of free compute is worthwhile to chase. There have been similar attacks against GitHub Actions, because they're also free and general-purpose.
This is also why the free-tier, dev-only, education, or similar MSDN or VS Subscriber type cloud subscriptions never permit the use of GPU instances. Too tempting a target!
The large providers like Azure and AWS are basically printing money, so they don't care about the small-fry scammers. They're too busy trying to hold on to the firehose of cash.
Smaller, budget providers like Digital Ocean are running too lean to tolerate scammers, but also can't afford to have humans in the loop from the sheer scale of the attacks.
So they use heuristics that are 99.99% accurate (or whatever), which means one in ten thousand legit customers gets axed along with the scammers. Oops.
Essentially, as a customer of these small providers you are taking on some of their risk in exchange for the discount over the big-name vendors. This is especially true if you pay them with a credit card, irrespective of past payment history. Like I said, they simply can't afford to keep a human in the loop.[1]
An example that came up here was a startup that had an account "ticking along" with a handful of dev stuff, went "live" with a big launch, and so almost all of their servers was loaded to nearly 100% capacity thanks to efficient containerisation and auto-scale.
Good job devs... except that looks identical to a hacked account that has suddenly started to mine crypto on every machine.
Axed.
[1] It's even more complex than you think. Employees can and have been bribed to let the scammers through! The employees themselves might be mining crypto. The scammers can trick them, lie, beg, or just figure out the system from repeat conversations. The only recourse is to take the human out of the loop entirely, nothing else works for them. If you get to be the 0.01%, you generally have no recourse.
Croftengea
Thanks for this very detailed explanation, much appreciated!
michelb
I'd like some more info as well. Only case I know was from someone who didn't see the emails about issues with their payment.
rendaw
At a previous employer we were hosting with DO via credit balance. Our finance department couldn't get things together and kept forgetting (almost every month) to add new credits to the account. Despite this DO never stopped or terminated our servers, and we were often days to weeks behind on the payments.
mro_name
they are big and you are small. It's like walking over ants.
arvindamirtaa
Ants pay the bill!
tinus_hn
Ants that don’t cause trouble pay the bill. Ants that cause trouble are a net loss.
Customer service representation is one of the biggest expenses these companies have. That’s why they go to great lengths to make sure you can’t contact them and why they just close your account with no recourse as soon as you start causing trouble.
BrainVirus
they’re massively shooting themselves in their feet
It doesn't matter. Google operates at scale. As long as they save more money by not bothering to handle such incidents better, they will see it as a perfectly good business policy.
Incidents like this turn customers off for life.
Small customers are completely interchangeable for larger service providers. You can easily get new customers by throwing around some free credits to college kids and posting a few tech ads masqueraded as tutorials.
danielx
It sucks that they gave you conflicting information that resulted in this but I wouldn't trust having my cloud infrastructure payments be handled using a single credit card.
If you have a spend of over $2500/mo you can apply for invoiced billing directly from Google or if you have less than that you can contact a GCP reseller partner to get invoiced by them.
Some GCP reseller partners also have a close relationship with Google and the contacts that come with that relationship.
booi
$2500 is honestly not that much. Google cloud dropped the ball on this and shouldn’t need to be handled with special relationships
Spare_account
All of the relationships I have with suppliers are handled by an account manager at the supplier who I can call to get things sorted out. This is normal.
Why should a business relationship with Google (or any other cloud service) be any different? I don't use GCP but if I was going to start, I would ensure that I have a person to call or its a non-starter.
remus
> Why should a business relationship with Google (or any other cloud service) be any different?
Depends on the size of the account surely? Maybe there's some support person to talk to, but there's plenty of services I subscribe to fo $10-100/month and I dont have an account manager for any of them.
jen20
> Why should a business relationship with Google (or any other cloud service) be any different?
Because google choose to make this as difficult as possible. A couple of years ago I worked on a third party IAAS database cloud product (think Atlas or Elastic Cloud, but for a different database), and while Azure had the worst tech overall, Google were ultimately the hardest to deal with by a VERY long way because they don’t acknowledge the existence of people as anything other than ad targets.
nikanj
Because the account manager would be a human. Google never hires a human if it can use shitty AI instead.
rroot
What is much and what is not is relative. $2500 breaks the bank for tiny startup and is a rounding error for Google.
upupandup
Not much for you but a lot for others. This type of arrogance really doesn't belong in this space. Maybe you spend $25,000? $250,000? peanuts to large companies so its okay that Google cloud drops the ball on you?
vasco
This is the only right answer with large cloud providers. Get a human to be involved with your account and do invoice payment as soon as they allow it.
gareth_untether
Isn't this type of service supposed to be automated? Why is having a secure service about who you know?
mro_name
because it's about trust.
Axsuul
What's the benefit of invoiced billing?
thematrixturtle
You have a person to reach out to if things go pear shaped, and N(weeks|months) to sort out payments issues instead of having everything get suspended by robots.
Cantinflas
What about telling those robots not to suspend everything and wait for N(weeks|months) or human intervention?
Dave3of5
This is what OP should really have done. All of the big cloud providers allow this. We had the same thing with AWS. As soon as you contact your account manager with things like this shit gets done quick. If you are on Pay as you go with a CC your messages just fall into a black hole.
julianwachholz
> "We're fully redundant across AZs, regions, and even cloud providers!" crows the engineer with a single corporate credit card backing the entire house of cards.
https://twitter.com/QuinnyPig/status/1288275701389389825?t=F...
pontifier
Murfie died because of Google cloud costs.
It's a bit of a long story, but from what I was able to piece together, a Google cloud service provider recommended some changes to the way the storage was done to save money. Mostly, because of those changes they got hit by a surprise $30,000 bill for data egress. It was basically the last straw for a company that wasn't profitable due mostly to cloud costs. It killed the company. That's why I had to step in to try to save them.
unity1001
> data egress
Google, AWS are still charging people mid-2000s bandwidth prices. Its outrageous. Digitalocean's egress price is 1/10th of Google's cheaper egress pricing. Same with Hetzner. And with Hetzner dedicated servers? Its unmetered.
mrtksn
If I recall correctly, there was a story on HN about Hetzner threatening to shut down an "unmetered" server for using too much bandwidth.
here it is: https://news.ycombinator.com/item?id=32242987
unity1001
From the discussion:
> - The link says "my friend got this", so even the person who authored that post isn't claiming to have received this notice. This is a second hand report, at best. For all we know, their friend received this notice years ago, and mentioned it to the OP, who didn't realize it was old news.
There have been people confirming in recent HN discussions that they were using full 1gbps of their dedicated box for a while since Hetzner declared their true-unmetered policy a year or so ago.
nwsm
Does the customer data have a proper home yet or is it still sitting in the broken-into shipping containers?
hbn
According to the owner's twitter, he's been repeatedly getting robbed.
pontifier
Containers are repaired and under surveillance.
Infernal
As I read through this incredulously, I was relieved to slowly realize Murfie was a company, and not a person.
thatwasunusual
Murfie was the virtual Muppet character who only existed in the cloud. :)
exikyut
More details that I found about what Murfie is, and the rescue operation: https://www.theverge.com/2020/2/5/21121594/crossies-murfie-m...
What a fascinating concept.
bilalq
I wonder what it would take to bring a culture change to Google so they can finally start having some empathy for customers and stop the casual cruelty. A new CEO?
emsy
Regulation, full stop. The reason Big tech makes so much profits is because they have no accountability and at their scale, social shaming doesn’t work.
rglullis
Right. Heavily-regulated industries like banking, auto and healthcare are known for being so much better at treating its customers like human beings...
belorn
Unregulated banking tend to have a bit more ponzi schemes, pump and dump, and money laundering than regulated banking.
Unregulated auto tend to have a bit more lemon markets than regulated auto.
Unregulated healthcare tend to have a bit more snake oil, blood letting and cocaine than regulated healthcare.
Unregulated food sector tend to have a bit more Listeria and Salmonella then regulated food sector.
No regulation are not know for being that nice either, which is to say that some regulation is good and some regulation is too much, but one can't dismiss regulation as just being bad.
antihero
In these industries the regulation is the only thing preventing them being unfathomably worse.
harry8
Yup. Too big to fail sucks just as hard in banking.
Google needs to be broken up as badly as banks do. It's a battle that keeps on coming up and here it is again.
iakov
I haven't heard stories about bank's AI deciding to close a person's account, take all their life savings, and refuse any explanation, contact or cooperation. Maybe regulation has to do something with it?
nightfly
You think those industries suck now, imagine what they're like without regulation holding their feet on the path a bit.
Sander_Marechal
They are, in Europe, where we have decent regulation and consumer protection with teeth.
unity1001
Heavily regulated industries ANYWHERE in the world except the US work perfectly fine. Its because regulations are watered down in the US that they don't.
josephg
I'm a fan of regulating big tech, but in this case reputation will also make a big difference. I know lots of people who actively avoid google's cloud products because they don't trust google not to pull stunts like this.
There have been too many stories on HN and twitter over the years of business critical google accounts being suspended for no reason or google cloud products massively increasing their price overnight.
I don't trust google as a provider for business-critical infrastructure. I'm far from alone.
karamanolev
The cost to provide proper support for the 99.9% is much much larger than the benefit of bringing on the possibly 0.1% who don't trust Google and avoid their products. This minority cannot be the solution to the problem.
christophilus
I'm not opposed to regulation, but in this case, another solution already exists. That is, competitive services that pride themselves on great customer service and satisfaction. Apple kind of does this in the iPhone vs Google phone wars. Azure and AWS do this in the cloud wars. Fastmail and Proton mail do this in the email wars, etc. Regulation didn't force Fastmail to be awesome; it's just the core of their business model and differentiation.
7952
I wonder if you could require that some data be made accessible to the user. Like transaction logs and actions relating to those transactions. So that you have direct incontrovertible evidennce of the internal state. That could then be used by third party support, or even courts.
undefined
shultays
no accountability
kevingadd
How much money do you want to waste on lawyers while your case drags on for years?
Dave3of5
Sue google? Are you mad? What do you think you'll accomplish with that ?
undefined
chr15m
Stop using their products and giving them money.
coffeefirst
It’s probably too big for one strong personality to get it back to the Don’t Be Evil days.
Regulation, litigation, or split it into pieces.
Croftengea
When was the last time any big company was split? AT&T 50 years ago, any newer examples? Nowadays big tech is too influential to allow politicians to make such decisions.
dzhiurgis
IMO we really need regulation on how support is done by big tech. It should be really hard for them to close accounts like this.
llanowarelves
Agreed. This issue seems to be popping up constantly on the front page. Not just Google but many of these companies.
I should not be able to get customer service from a random McDonalds franchise over a chicken sandwich easier than a company that we spend thousands, tens, or even hundreds of thousands of dollars with.
They do not get to use "We're too big with too many customers" as a defense. They can figure it out.
amirhirsch
It won’t happen. Google is probably going to be like the next IBM: a former tech giant that became irrelevant.
wizofaus
I wouldn't be so sure that will happen to Alphabet before it does to Amazon or MS. Nothing lasts forever, though IBM aren't actually showing any signs of dying outright yet, despite the fact their own attempt to get into the cloud platform market hasn't exactly been inspiring (even if they were only ever intending to cater for niche markets).
thinkingemote
Ironically, empathy is the reason why they are doing this. Empathy for the support worker. They literally call it Human Ops. The idea is that support work is horrible for their intelligent engineers and so they invest in AI to replace human the bad jobs that impact humans.
Of course human ops is originally about reducing actual traumatic jobs like reviewing actual evil images, but its basically now all brain work which techies hate.
londons_explore
I'm kinda surprised they don't voluntarily allow courts to make these tricky decisions for them.
Perhaps even offer to pay court costs.
They can say:
> We suspended your account for violation of XYZ rule. You can appeal internally, and if that is denied, you can ask a court to rule on if you broke this rule. If the court finds you did not violate the rule, we will reinstate your account. We will provide any data from your account to the court to make its decision, including proprietary data that we cannot reveal directly to you, for example data about other linked accounts.
ravenstine
I doubt that can happen organically. The Google is ground zero for ego in Silicon Valley. When ego is inflated, empathy must deflate to make room.
xtracto
They should be freaking broken:
1. Ad business should be one thing 2. Other products should be different companies: Gmail, GCS, Domains, etc etc. They should not be "cushioned" by Ad business revenue.
That way they will play an even playing field with similar companies. And I think Amazon/AWS should have the same split.
If that happens, these companies will have to differentiate to their customers with non-crappy customer support.
tomxor
> They were "investigating" the issue and assured the project will not be suspended.
In my experience, and from everything I have read of others, Google's customer support is mostly a facade - Google will have given them a few buttons to press but they have absolutely no autonomy or authority, they are disconnected from Google and when things go wrong they cannot help you no matter how much they promise.
Get the top HN stories in your inbox every day.
TLDR; never use google cloud systems for production.
Google cloud suspended all our projects due to the billing issue in their system they had.
Despite reassurances "your account will not be suspended" while communicating with billing support, all the projects were suspended at 1am on Saturday.
All the account payments were made and the billing cards are valid. There are no outstanding bills.
Never use GCP for production.
---- Edit: full story by request, long read: ----
Previous month billing didn't went through. Not sure if this was due to the billing outage google had (https://status.cloud.google.com/products/oLCqDYkE9NFWQVgctQTL/history) or financial transaction issue, however we went ahead and made a manual payment covering all the outstanding amount + extra. Despite the payment made, about a week+ later we suddenly started receiving threatening emails "Your Projects are at risk of suspension". Edited and updated the billing cards. Opened a billing support request clearly mentioning this is a production environment and all the bills are paid. They were "investigating" the issue and assured the project will not be suspended.