Brian Lovin
/
Hacker News
Daily Digest email

Get the top HN stories in your inbox every day.

retired

Years ago I was making the case that instead of digging ourselves into the Amazon eco-system with S3 storage, EC2 instances, DynamoDB and various other Amazon specific cloud products... we should just host virtual machines and have everything in there using open source products.

People looked at me like they saw water burning but that would have made the dependency on the US a lot easier to sever. Just move the VM's.

kevin_nisbet

I've operated at companies using both models, and have observed similar reactions to suggestions of using the cloud.

To me it's like anything else in engineering, are the costs, risks, and benefits fully understood, and worth the tradeoff in the particular context.

I worked for a startup doing internet of things, the consumer would buy a device and get lifetime service baked in. And that company was a step further, just renting space in a colo was incredibly cost efficient, which supported the sales model and competitive landscape of that product. But it was also very costly to attention, one of the most valuable resources. But it can also get costly in non-intuitive ways, an example that comes to mind is we started to get interviews where a generation of candidates no longer had experience with metal, it was a foreign world to them.

With more experience, I find it's really the costs that get severely underestimated, both for and against the suggestion.

ghaff

Especially in larger organizations, it's easy to lose track of all the distributed soft costs that DIY can bring (and all the bus factors that may be involved). There are lots of people that kinda want to get paid and get benefits and which require some level of management structure.

At some point, you have people (on here and elsewhere) questioning what all these people in an organization do. PART of the answer is that they're doing internal work that could have been outsourced in various ways.

port11

I’ve struggled with convincing colleagues to host on something other than AWS. I’m not sure they understand the costs and aren’t simply doubling-down on the evil they already know.

In fact, I had no idea our static website at a scale-up in 2019 was costing us 90€/month; it came up when we were told to cut costs. Developers don’t always have a say in these things.

Heck, I then went and got a series of certifications in GCP. Even then, I’m not sure I’d understand the full complexity and pricing options of GCP. Smaller clouds and simple VPS solutions really are the overlooked option.

BrandoElFollito

> fully understood

Unfortunately this is neither the case for very large, or very small companies.

Very large companies may have some experience, but it is usually legacy. Or buried under processes and politics.

Small companies will not have the experience to make the analysis

lmf4lol

I am running my startup out of a self build GPU server from our office with a backup to the cloud. I only pay for the IP address as electricity is included in the rent. If the startup fails, Ill have thousand other potential use case for it and in the worst case, it will make for a awesome gaming machine.

The machine is a beast and I can serve a lot of users with it. In fact, and quite funnily, I already serve much more users with it than a lot of my older clients do with their software running on expensive k8s setup because „scale“ :-)

And last, but not least, I had a lot of fun building it. Its just nice to hear that thing humming away in the corner.

throwaway894345

> The machine is a beast and I can serve a lot of users with it. In fact, and quite funnily, I already serve much more users with it than a lot of my older clients do with their software running on expensive k8s setup because „scale“ :-)

Honestly even if you have a single server, running k8s (or maybe Docker Compose for really simple cases) on it is still the simplest way to manage it (assuming you have more than 1 service, anyway). One configuration file format, one CLI tool, zero special paths to memorize, no filesystem permissions to configure, pretty good security out of the box, access to a whole bunch of helm charts and operators (for example, cert-manager, external-dns, prometheus, alert-manager, some logging operator for centralized logging with a decent UI and search, and a postgres operator for backups / replication / failover), etc.

ozgrakkurt

This depends on what you know. Kubernetes is really not that good for me

dzhiurgis

How does your startup setup applies to a bank?

undefined

[deleted]

xfactorial

The whole business model is around “Optimization through custom tools”.

We can go with your idea, sure: a few months in, an Account Manager from the cloud provider shows up and says your bill could be reduced by 50% if you just adopt some changes, using their custom, super optimized tools (“minor changes” will be the mantra).

And now you have your own company looking back to you on how can they get those savings, people who don’t understand what a VM is and cannot differentiate salesforce from an elastic container, as everything is “cloud”, but heard “50% off”.

walrus01

Preventing this from happening requires a clued-in CTO and equivalent senior level leadership who can defend against such 'attack' methods and knows the difference between, for instance, paying a monthly recurring cost to host a Linux/KVM virtual machine and paying for some totally 'cloud' SaAs.

Further, it needs people in decision making roles who understand and value the strategic differences between having an infrastructure concept that is trapped in one provider's proprietary software tooling ecosystem (aws, azure, etc), vs things built on open standards that are portable.

eddythompson80

> Preventing this from happening requires a clued-in CTO and equivalent senior level leadership

Most CTOs (and increasingly M2s and M3s) I've met are what I call "box architects". You know the ones who love drawing boxes, moving one box inside another box, drawing a line between 2 boxes or changing a unidirectional arrow into a bidirectional one, then declaring the hard part is done and now we need any random engineer to implement that or "Is there an AWS service that does that? I just don't see the value in us doing it in house".

A "super optimized tools" is just a box that you swap for another box and the "minor changes" will be just a couple of arrows than need to change or another box to swap for another box. You get them to feel good about doing architect stuff plus the 10x reduction in the bill. They can always replace that box with another box later after all.

palmotea

> Preventing this from happening requires a clued-in CTO and equivalent senior level leadership who can defend against such 'attack' methods and knows the difference between, for instance, paying a monthly recurring cost to host a Linux/KVM virtual machine and paying for some totally 'cloud' SaAs.

And the reality is eventually you'll get a clueless one, and everything will revert to the mean.

And the mean is heavily influenced by marketing propaganda.

RobotToaster

Do people actually take claims like that from glorified salesmen seriously?

If a car salesman told me I could save 50% of my fuel bill from driving their special car a certain way I'd laugh at them.

throwaw12

You are missing the timeline factor here.

2016 - lets use EC2, its just VM, we can move off

2018 - I see you are hosting your own PostgreSQL in EC2, you can use our managed solution

2020 - you are already using 18 our services (note, at this point you might still be using non-vendor products, like VMs, managed DB, and so on), why not use our IAM instead of rolling out your own auth.

2024 - you are now deeply locked, lets add more lock-in, why don't you use this tool to optimize your costs (welcome DynamoDB)

At this point, no one would ever question next tool from salesman. Because engineers see that company doesnt have strategy to move to another cloud, why should they reject this new tool?

also consider the people who are involved, a lot of times after 2 years you have totally new people in your team, they won't have context and constraints you had in the past when deciding to buy "just VM", they see it as "we already use AWS"

green7ea

I had many conversations with a former boss about the Azure sales team. They would come in, say they can do it cheaper, simpler and better — he was immediately convinced.

I would do a calculation based on their public price plan and come up with a 5-10x price compared to the bare metal OVH solution that perfectly fit our use case. I would then ask the sales team where I made a mistake in my calculation and hear nothing back.

A few months later, they would come back with the same pitch and the whole process would repeat...

piperswe

They're probably not wrong, if they're talking about hypermiling a Prius

thebruce87m

I save 75% on electricity vs diesel

pvtmert

AWS has been (blatantly) using Microsoft method of making their way in. Redis, Elasticsearch, whatnot, all follow the same procedure: 1. Here is a managed service. 2. Here is a fork of the managed service where we manage the server (you don't see) with 15% off in price/credits. Easier backups with clicks etc. 3. We are dropping support of managed-X, move to our fork. 4. Due to the market conditions, our forked service is now 50% more expensive. 5. Ah also, you cannot export/download your backups because they are in proprietary format. 6. Locked-in.

SpicyLemonZest

You'd be wrong to laugh at them, because different cars of the same general size can indeed vary 50% or more in fuel efficiency. It's fair to be skeptical of promises of huge savings, and question why your counterparty would benefit from giving you those savings, but sometimes there's a good reason.

walrus01

> Do people actually take claims like that from glorified salesmen seriously?

People who know the tech, no

Non-technical middle management types, yes. It produces revenue when done aggressively enough, google "solarwinds sales people" for many anecdotal examples of extreme persistence. Not that I agree with it.

BadBadJellyBean

I prefer not using managed services but I kind of understand the appeal. Instead of paying several engineers, that you have to vet first, to configure and maintain the services adjacent to your product you can just pay AWS or Azure or someone else to maintain the service. Then you can concentrate your whole manpower on your product. In case the service goes down you can blame someone else and maybe even recover some money. On the other hand it of course makes you dependent on the provider.

jamesfinlayson

Yep, been in a job like this. Use AWS because the team is three people and they don't want to waste time on patching, database administration, networking etc. I agree you pay more but in that team we were just able to get on with building the products.

lelanthran

> Instead of paying several engineers, that you have to vet first, to configure and maintain the services adjacent to your product you can just pay AWS or Azure or someone else to maintain the service.

Your engineers who all have to possess AWS or similar certs before you hire them, work for free?

A move off VPS to managed services doesn't reduce your headcount or labour costs.

BadBadJellyBean

You are correct. Someone has to manage and plan the infra. But that is the same for on prem or other non cloud. What you don't necessary need is several database admins, several network admins, several kubernetes admins, etc. I don't necessarily agree, but that is the calculation. Azure hires the 24/7 admins for the service and you pay a bit more to get a share of them. I have heard this argument in person.

I think there is a very narrow space where you need the resources that this provides and it's not yet more cost effective to have your own team of admins. At a certain headcount a the number admins don't matter that much anymore.

SkiFire13

> Your engineers who all have to possess AWS or similar certs

If you're using managed services that are so complex you need certified people then you're doing it wrong

mulmen

What you’re describing is outsourcing. It’s still possible with on-prem or cloud VMs. You just hire a contractor provide those services.

BadBadJellyBean

You are correct but I don't know about the cost structure. Also you have to somehow verify that they do a good job. You sometimes only see bad work when something goes wrong. Also you have to first find a company that provides the service.

The cloud makes it simple. They offer you managed service X. They hire experts for service x and you pay a part of the cost on top of your infra cost. No searching. No vetting. You just use the service.

I see the why this might be attractive. It isn't to me. But the pencil pushers like it.

actionfromafar

In my experience it doesn’t take long until you use such complex offerings from the cloud vendors, you need those ops engineers anyways. Just with slightly different skillsets.

BadBadJellyBean

I'd say you need people with certain skill sets anyways but at a certain scale you have to get specialized people for some service. Database admins, kubernetes admins, network admins. At a small scale that can be one or two people. But if you want 24/7 with a bigger scale you need multiple people for each role. You have to find them, pay them, schedule their absences.

To some management types it looks like a good deal to not deal with that and just let Amazon/Microsoft/Google/etc. deal with finding people to support the service and just pay a bit extra to the infra cost. Then you can only hire cloud infra admins. I don't think it works that way but that is what I have observed.

fancythat

Calculations from me and others have proven that cloud providers use 5-10x multipliers when selling you things. The less you use them, the better is your bottom line. At the beginning it maybe makes sense to use cloud credits to get you moving, but when credits expire or your organization grows, it is wise to invest in people that can setup things on their own. The biggest lie that cloud providers managed to sell to the world, that you don't need knowledgeable people to run things in cloud.

PaulKeeble

There was a period when development and system adminstrators were really concerned about vendor lock in and would choose on the basis of the ease of moving to a different platform, Java and J2EE was clearly based on this mindset. I have always found it odd people have been willing to adopt AWS with no apparent easy route off given its price.

pjmlp

Still is, nowadays the standard is Jakarta EE 11, alongside Microprofile, which Spring also uses parts of.

bell-cot

> I have always found it odd people have been willing to adopt AWS ...

It's the new "Nobody ever got fired for buying IBM".

tt24

This is great, your suggestion to replace s3 and ddb is to run some VMs?

I don’t blame people for being skeptical

filleduchaos

Projects like Ceph and Minio have existed for years, though?

Beyond that, I just don't understand your point of view at all. Do people unironically think there is some super special dark magic being done in the bowels of Amazon, as opposed to just...code that runs on (virtual and physical) machines? The open source community yielded Linux but it's just sooo impossible for it to yield an object storage service? What a strangely shackled view of the world.

rcbdev

This is great, your suggestion to replace our sovereignly hosted VMs is to run containers in the U.S.A.?

I don't blame people for being skeptical

tt24

When sovergienty comes at the expense of availability the latter will usually win over the former.

ApolloFortyNine

Yea, op just handwaved away all scalability. Guessing their response would be 'launch more vms'.

izacus

VMs scale just fine.

hermanzegerman

Why does the Dutch Central Bank need scalability?

actionfromafar

Scalability is great, when you need it. Most companies don’t need it.

lelanthran

> This is great, your suggestion to replace s3 and ddb is to run some VMs?

Well... yes?

What do you think the AWS S3 and DDB is running on? Fairy dust?

tt24

No it’s using an army of extremely well paid engineers, something I guarantee the parent comment has no access to

stuaxo

S3 has become a standard outside of AWS but everything else can be done with open tools except IAM which is always a pain anyway.

tjwebbnorfolk

> will sign a major contract tomorrow

Ok so nothing has actually happened. These migrations are difficult and expensive, and often fail. It will be interesting to see an update in 5 years on how this went.

pier25

Wait... Lidl has a cloud service now?

kodama-lens

Yeah, kind of. Lidl and Kaufland is owned by the Schwarz Group. They have been busy replicating the AWS orgin story. Their cloud is called StackIT. I've worked with them. Still some room to grow but a solid foundation. I like that competition is back on

martijnvds

Is it fully custom, or are they using a flavor of OpenStack or similar?

MaKey

They have their own API but under the hood it's OpenStack.

storus

Schwarz seems to be obsessed with how Amazon (book seller) created AWS and they are trying to do the same... with 5 people. Also Aleph Alpha + Cohere is a Lidl work as the current CEO of the former led Lidl digital division.

wasmitnetzen

Lidl famously blew 500M on a failed SAP project, so they're understandably a bit into running things themselves.

https://news.ycombinator.com/item?id=17541092

pronik

But they had the guts to back away. Not everyone does.

scandox

So the 7,500 they say they're employing...is not true?

ivan_gammel

Schwarz Gruppe includes Schwarz Digits, which include StackIT. 7500 is the number of employees at Digits, which also includes online marketplaces like Kaufland e-commerce, so definitely not all of them work on the sovereign cloud.

ambicapter

Lidl is a grocery store chain, I'm assuming GP was talking about the amount of people actually working on the cloud.

undefined

[deleted]

walthamstow

The things you can find in the middle aisle!

apparatur

Sure and it's on SALE right now if you have the customer loyalty card!

KronisLV

Decent amount of features but definitely way more expensive than what you can get on Hetzner: https://stackit.com/en/prices/cloud

lifestyleguru

> Lidl has a cloud service now?

Yes, when you pay you have to print a receipt with QR code, and then have to scan it to log out.

dzhiurgis

And don't forget to always carry 50 cent roman coin if you want to log in

tcp_handshaker

Its terrible...just go and try to open an account...it broken from the start

https://accounts.stackit.cloud/ui/login/user

janmarsal

You need to download the lidl+ app. great value

helsinkiandrew

Here’s the service:

https://stackit.com/en

anfogoat

I'm sure the actual management interface is a separate thing but still, this website is unreal. Feels like a repurposed, half baked WordPress theme.

rsynnott

I mean, have you seen the AWS management interface? I struggle to imagine it being _worse_.

junto

Lidl cloud is a resurgence in OpenStack?

I used to have quite a few customer hosted at Rackspace in the early 2010’s and I always thought it sad they dropped the ball when they got bought out by private investment and they fired most of the talent. I loved their API and the docs were really good.

speedgoose

Yes I guess banks don’t mind the high prices of Lidl’s cloud.

It’s very much not a discount cloud provider. They are costly unlike their physical discount grocery stores.

Havoc

Pretty sure it's the oracle model - the advertised prices don't matter because it's all custom negotiated.

dethos

Makes sense. I never worked with this particular provider, but I must say that for many (many) use cases, Europe has very capable providers, and the big US players are not necessarily the best choices.

tomschwiha

The title is heavy clickbait. To say I just bought a Porsche when it was actually a Volkswagen is also wrong. Just because they belong to the same owner doesn't make it the same brand.

rsynnott

This feels overly pedantic. In practice, the Schwarz Group is "Lidl, plus some random small stuff". Compare Alphabet.

croes

It’s actually the other way around. Porsche is a Volkswagen but a Volkswagen isn’t necessarily a Porsche.

VW bought Porsche

AndroTux

Yes, that's what they're saying. LIDL doesn't have a cloud. The Schwarz Group does.

fodkodrasz

Too bad, a LIDL branded cloud would be something really well marketable. Cloudside services (a'la Parkside)... or something along these lines.

manquer

Kinda, VW indeed owns Porche AG 100% today.

However it was more complicated than that. Porche owned 50+% of Volkswagen at the time of Volkswagen buying them. Porche got over extended and leveraged buying Volkswagen . The management family is closely connected since the start and at the time in early 2010s 20% government ownership rule was just getting stuck down by European courts .

ck45

Well, it's more difficult. The original Volkswagen was designed by Ferdinand Porsche.

rsynnott

Fortunately for them, Lidl's middle aisle deal this week happened to be cloud computing.

(Tesco cloud compute will price-match the lidl cloud-compute, but only if you remember to scan your Tesco clubcard at the self-checkout while buying it.)

xtiansimon

Lidl the grocery store chain, ha! Up yours Amazon.

petervandijck

I just love that it’s Lidl, of all brands

k__

European central bank will probably go for Aldi.

pnt12

They're going with Scaleway, a French company. Seems like the owner is a billionaire with a few businesses, but neither La Redoute nor Intermarché!

https://www.scaleway.com/en/news/scaleway-accelerates-its-eu...

retired

Aldi Nord or Aldi Süd?

MiinusMiinus

I'm so happy that companies are ditching the big tech. Not enough fast enough imo.

holoduke

[flagged]

strangegecko

Expand?

German government is certainly slow and overly limited by bureaucracy, but dangerous?

Who are you comparing to?

hermanzegerman

That's funny coming from an American.

Throwing Stones in a Glasshouse

vrganj

The last war Germany started famously ended 80 years ago.

The last war the US started is still ongoing and was started by them a few weeks ago.

holoduke

You really compare 40s Germany with the US? Nah

Daily Digest email

Get the top HN stories in your inbox every day.