Get the top HN stories in your inbox every day.
slt2021
cortesoft
Isn't this exactly WHY this blog post exists? They are open sourcing this software so that they don't have to maintain it all internally anymore.
They had a need that an existing "off-the-shelf open source" project didn't solve, so they created this an are now turning it into an "off-the-shelf open source" project so they can keep using it without having to maintain it entirely themselves.
How are these open source tools supposed to be created in the first place? This is the process, someone has to do it
rjh29
Usually the corporate needs differ too much and they end up keeping their own fork anyway.
Netflix has the resources to maintain this. It's probably more a PR move for their hiring division.
darkwater
Absolutely this. It literally happened to me with a Netflix OSS we were using at work. I found a bug that was biting us, opened a ticket with a PR attached with a possible fix and got an answer after a few months "ah yeah we fixed this in our internal version time ago, thanks, will merge it now".
gregoriol
Indeed, this is not open-source: this is public-source. They don't really open the project to external contribution, they just publish their code and continue the project as their tool. They will not have incentive to add features that are not useful to their business even if it useful to the community (if provided by a PR for example), because all the developers of the project are employed by the same company and this company doesn't have any reason to review and fix code that is not part of their business.
slt2021
So Netflix expects open source community to pick up the maintenance tab ?
I understand how open source proejcts are born, but I struggle to see what is novelty of this project. Just another Java CRUD app with some questionable design choices that are only applicable to netflix:
1. They claim it is distributed system, but it is just a regular Java crud with SQL backend
2. Java-like DSL with parser and classloader (why? Just why?)
Projects like these are the perfect examples of Enterprise Grade FizzBuzz (https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpris...) and this is exactly what I dont like about it
cortesoft
> So Netflix expects open source community to pick up the maintenance tab?
Isn’t this the deal with all open source? They are giving something (the code and access to the project) in return for help maintaining it?
No one is being forced to do anything. It is not like there is some open source contributor somewhere now saying, “oh damn, now I have to maintain this, too?”
If people like it and find value in it, they can help contribute to the project in ways they want. Netflix gets to use those contributions, in return for letting people use their contributions. That is just how open source works.
geodel
You are making great points. This is power of Netflix marketing and branding that they are considered as cutting edge tech company. In reality most of Netflix Java projects are pretty mediocre Enterprise Java stuff. Last year or so they have mandated Spring Boot as their development platform for all their web services.
This is exactly same stack I have to deal daily and management reason is it is lowest common denominator that works well with 3-month contract developer to deliver Nth micro service whose sole job is to call another service.
jjuliano
> So Netflix expects open source community to pick up the maintenance tab ?
I think the notion of open sourcing a project, is you are literally asking at the community for help and that the community will naturally help you with the maintenance.
digger495
People want an alternative to things like Temporal, and don't want to handle DAGs with Kafka Streams.
cbsmith
> So Netflix expects open source community to pick up the maintenance tab ?
In fairness, the very nature of open source is that the community is only going to pick up the maintenance tab if the value they're getting out of it is worth it.
xcdzvyn
[dead]
bluepizza
> People need to realize that code is a liability
This is an extreme point of view, that is tightly connected to the MBA-driven min-maxing of everything under the sun.
I am glad that there are folks who aren't afraid to code new systems and champion new ideas. Even in the corporate sense, mediocre risk averse solutions will only take you so far. The most profitable companies tend to be quite daring in their tech.
Code is not a liability. Code is what makes a company move its gears.
delecti
Code being a liability is not a contradiction with code being what makes a company move its gears. The trucks of a delivery service are a liability (requiring maintenance, deprecation accounting, fuel), but are also the only thing that lets the company deliver. A delivery company should own as few trucks as necessary, and no fewer. Any company should publish/run/maintain as little code as necessary, and no less.
bhawks
Trucks are literally an asset - you can't do depreciation on a liability.
The only way a 'truck' could be a liability is a lease for said truck.
There are plenty of economically rationale reasons why a company may own more trucks they strictly need to manage delivery. For example wanting to handle seasonal bursts, wanting to ensure reliability, preparing for an expansion, being able to lease capacity to other businesses.
Actually you can go replace truck with server and you describe what made AWS make initial sense.
Please stop misusing accounting concepts.
slt2021
For trucking company owning and developing trucks makes sense.
But does it make sense for a trucking(streaming) company to create own plumbing equipment? I’d rather use Plumbers Supply Inc that every other company uses from Plumber Depot or use open-source-plumbers.com, because I am not in a plumbing business
bilalq
But thinking of those trucks primarily as a liability is exactly the kind of mindset that leads to companies minimizing their liabilities instead of maximizing their potential.
pants2
Using open-source is a liability too, with added problems of code licensing conflicts, supply chain attacks, zero-day vulnerabilities, relying on maintainers that don’t work for you, etc.
cbsmith
Not open source is a liability too, with added problems of code licensing conflicts, supply chain attacks, zero-day vulnerabilities, relying on maintainers that don't work for you, etc... ;-)
YawningAngel
Off-the-shelf open source stuff is often the product of big companies open sourcing internal tools though. Airflow, which you name check, is a great example of this. Temporal is another example in the space. Someone has to be dumb enough to build new stuff
slt2021
airflow and Temporal has teams dedicated to maintain and extend their system. And these systems are business critical for astronomer/temporal, respectively.
And they develop them in a way that works for many customers and use cases, not just netflix.
But for netflix this is just another auxillary system, out of many others. Just a nice GUI to schedule cron jobs basically, does it make sense to sink resources into custom cron?
hulahoof
But when Airbnb created airflow, you could have said the same. It’s just later in its lifecycle.
bhawks
> with long history of maintenance and improvement,
That is a huge load bearing statement.
Do you plan on any contributions back to the community yourself?
Build vs. buy is always an important conversation but claiming that the 'buy'-side path has perfectly 0 maintenance and reliability costs reeks of naivety.
slt2021
If I needed container orchestration I would use k8s. I can improve it and even propose patches/bugs or chip into opensource maintainers fund. I wont write my own orchestrator, especialy being in a streaming business.
Thats what I meant, doesnt even necessarily Build-vs-Buy, but rather Use-Open-Source-and-Contribute or Reinvent-the-wheel-for-L6-promo-and-then-opensource ??
Would the world be better with 10 workflow orchestrator systems or one mature?
bhawks
Netflix is building a workflow orchestrator not a container orchestrator. The viable alternative would be Airflow or maybe something like Temporal. K8s alone isn't going to meet the need in this case.
Does the world need another workflow orchestrator? Who knows - some folks at Netflix seem willing to pay a handful of engineers $ to do so. Good luck to them
undefined
makeset
> anything that is not the business critical stuff That's an important qualifier. For skilled teams in performance-critical domains, the inflection point where any outside code becomes a low-quality/low-control liability is not that far.
ripped_britches
100%. Very few times are these systems built as robustly as external folks who earn a profit on building robustness. Best example of course being Stripe. But I see this from everything from visual snapshot testing tools to custom CI workflows. The good thing is you can always rely on competitive market dynamics to price the off the shelf solution down to a reasonable margin above maintenance costs.
why-el
I am confused by this comment:
> open source stuff with long history of maintenance and improvement
improvement and maintenance is continent on usage, and having been used at Netflix, this project is in a better place to have already faced whatever bug you are worried about (and let's be real, 99% of applications wont ever get the luck to exercise code paths sophisticated enough to find bugs Netflix has not found already).
You might be unnecessarily projecting here. You don't have evidence to support that open sourcing this might have been for any other reason than it is simply good for the community to have.
archerx
This is a naive view, other people’s code is even more of a liability. Look at crowdstrike and opensource infiltrations. Using opensource software doesn’t magically grant you security nor stability.
hintymad
I wonder how many iterations we will need before engineers are happy with a workflow solution. Netflix had multiple solutions before Maestro, such as metaflow. Uber built multiple solutions too. Amazon had at least a dozen internal workflow engines. It's quite curious why engineers are so keen on building their own workflow engines.
Update: I just find it really interesting that many individuals in many companies like to build workflow engines. This is a not deriding comment towards anyone or Netflix in particular. To me, such observation is worth some friendly chitchat.
alfalfasprout
The issue is that "workflow orchestration" is a broad problem space. Companies need to address a lot of disparate issues and so any solution ends up being a giant product w/ a lot of associated functionality and heavily opinionated as it grows into a big monolith. This is why almost universally folks are never happy.
In reality there are five main concerns: 1. Resource scheduling-- "I have a job or collection of jobs to run... allocate them to the machines I have" 2. Dependency solving-- If my jobs have dependencies on each other, perform the topological sort so I can dispatch things to my resource scheduler 3. API/DSL for creating jobs and workflows. I want to define a DAG... sometimes static, sometimes on the fly. 4. Cron-like functionality. I want to be able to run things on a schedule or ad-hoc. 5. Domain awareness-- If doing ETL I want my DAGs to be data aware... if doing ML/AI workflows then I want to be able to surface info about what I'm actually doing with them
No one solution does all these things cleanly. So companies end up building or hacking around off the shelf stuff to deal with the downsides of existing solutions. Hence it's a perpetual cycle of everyone being unhappy.
I don't think that you can just spin up a startup to deliver this as a "solution". This needs to be solved with an open source ecosystem of good pluggable modular components.
SOLAR_FIELDS
The issue indeed is that "workflow orchestration" is a broad problem space. I would argue that the solution is not this:
> I don't think that you can just spin up a startup to deliver this as a "solution". This needs to be solved with an open source ecosystem of good pluggable modular components.
But rather more specialized tools that solve specific issues.
What you describe just sounds like a better implemented version of Airflow or the over 100 other systems that are actively trying to be this today (Flyte, Dagster, Prefect, Argo Workflows, Kubeflow, Nifi, Oozie, Conductor, Cadence, Temporal, Step Functions, Logic Apps, your CI system of choice has their own, need I continue, that is not even scratching the surface). Most of those have some sort of "plugin" ecosystem for custom code, in varying degrees of robustness.
For what it is worth, everyone and their mom thinks they can make and wants to be this orchestrator. It's a problem that is just so generic and such a wide net that you end up with annoying-to-use building blocks because everyone wants to architecture astronaut themselves into being the generic workflow orchestration engine. The ultimate system design trap: Something so fundamentally easy to grok and conceptualize that you can PoC one in hours or days, but near infinite possibilities of what you can do with it, resulting in near infinite edge cases.
Instead, I'd rather companies just focus on the problem space that it lends itself to. Instead of Dagster saying "Automate any workflow" and try to capture that space, just make building blocks for data engineering workflows and get really good at that. Instead of Github Actions being a generic "workflow engine" just have it really good at making CI workflow building blocks.
But we can't have it that way. Because then some architecture astronaut will come around and design a generic workflow engine for orchestrating your domain specific workflow engines and say that you no longer need those.
Actually I think I just convinced myself that what you are suggesting actually IS the right way. If companies just said "we will provide an Airflow plugin" instead of building their own damn Airflow this would be easy. But we won't ever have that either. What we really need is some standards around that. Like if CNCF got together and got tired of this and said "This is THE canonical and supported engine for Kube workflows, bring your plugins here if you want us to pump you up". That might work. They've usually had better luck with putting people in lockstep in the Kube ecosystem at least than Apache has historically for more general FOSS stuff. Probably because the problem space there is more limited.
swyx
great insight, appreciate this. would also point out logging/event sourcing for "free" observability
savin-goyal
Metaflow sits on top of Maestro, and neither replaces the other
> ...Users can use Metaflow library to create workflows in Maestro to execute DAGs consisting of arbitrary Python code. from https://netflixtechblog.com/orchestrating-data-ml-workflows-...
The orchestration section in this article (https://netflixtechblog.com/supporting-diverse-ml-systems-at...) goes into detail on how Metaflow interplays with Maestro (and Airflow, Argo Workflows & Step Functions)
dinobones
We rolled our own workflow engine and it almost crashed one of our unrelated projects for having so many bugs and being so inflexible.
I’m starting to think workflow engines are somewhat of a design smell.
It’s enticing to think you can build this reusable thing once and use it for a ton of different workflows, but besides requiring more than one asynchronous step, these workflows have almost nothing in common.
Different data, different APIs, different feedback required from users or other systems to continue.
ryanianian
> workflow engines are somewhat of a design smell
Probably so, but the real design smell seems to be thinking of a workflow engine as a panacea for sustainable business process automation.
You have to really understand the business flow before you automate it. You have to continuously update your understanding of it as it changes. You have to refactor it into sub-flows or bigger/smaller units of work. You have to have tests, tracer-bullets, and well-defined user-stories that the flows represent.
Else your business flow automation accumulates process debt. Just as much as a full-code-based solution accumulates technical debt.
And, just like technical debt, it's much easier (or at least more interesting) to propose a rewrite or framework change than it is to propose an investment in refactoring, testing, and gradual migrations.
pm90
It’s likely because we haven’t yet found a workflow engine/orchestrator thats capable of handling diverse tasks while still being easy to understand and operate.
It’s really easy to build a custom workflow engine and optimize it for specific use cases. I think we haven’t yet seen a convergence simply because this tool hasn’t yet been built.
Consider the recent rise of tools that quickly dominated their fields: Terraform (IaC), Kubernetes (distributed compute). Both systems are hella complex, but they solve hard problems. Generic workflow engines are complex to understand and difficult to operate and offer a middling experience so many folks don’t even bother.
fragmede
slurm? airflow?
pm90
airflow is notoriously hard to operate https://news.ycombinator.com/item?id=31482217
sgloutnikov
Naming things, cache invalidation, and workflow engines? :)
cbsmith
No, it's just the two things: naming things, cache invalidation, and off by one errors.
Nathanba
It inherently asks for a custom implementation because it's almost like workflows are just how you'd have to code and run everything anyway. Conceptually: why wouldn't we want to reconnect to any work we are currently in progress of, just like in a videogame where if we lose connection for a splitsecond, we want to be able to keep going where we left off? So therefore we must save the current step persistently and make sure that we can resume work and never lose it. Workflow engines also do no magic: They still just run code and if it fails in a place that we didn't manually checkpoint (=by making it into a separate task/workflow/function/action/transaction that is persistable) then we still lose data so.. at that point why not just try doing it this way everywhere whether it's running in a "workflow engine" or not. Before "workflow engines" we already had db transactions but those were mostly for our benefit so we don't mess up the db with partial inserts. Although so far what I've seen in open source workflow engines is that they don't let you work with user input easily, it's sad how they all start a new thread and then just block the thread while it waits for the user to send something. This is obviously not how you'd code a crud operation. In my opinion this is a huge drawback of current workflow engines. If this was solved, we should literally do everything as a workflow I think. Every form submission from the user could offer to let the user continue where he left off and we saved all his data so "he can reconnect to his game" (to revive the videogame metaphor I started with)
dekhn
I wrote my own because I wanted to learn about DAG and toposort and had some ideas about what nodes and edges in the workflow meant (IE, does data flow over edges? Or do the edges just represent the sequence in which things run? Is a node a bundle of code, does it run continuously, or run then exit?). I almost ended up with reflow, which is a functional-programming approach based on python, similar to nextflow, but I found that the whole functional approach to be extremely challenging to reason about and debug.
Often times what happens is the workflow engine is tailored to a specific problem and then other teams discover the engine and want to use it for their projects, but often need some additional feature, sometimes which completely up-ends the mental model of the engine itself.
renewiltord
We all have different use-cases. We also have a workflow engine at work but that's because we wanted immediate execution. From submit to execute time can be 100 ms on our system, which makes it also work well for short jobs. Usually, the task coordinator overhead is greater than that on these things.
rubenfiszel
Founder of https://windmill.dev here which share many similarities with Maestro.
> Maestro is a general-purpose, horizontally scalable workflow orchestrator designed to manage large-scale workflows such as data pipelines and machine learning model training pipelines. It oversees the entire lifecycle of a workflow, from start to finish, including retries, queuing, task distribution to compute engines, etc.. Users can package their business logic in various formats such as Docker images, notebooks, bash script, SQL, Python, and more. Unlike traditional workflow orchestrators that only support Directed Acyclic Graphs (DAGs), Maestro supports both acyclic and cyclic workflows and also includes multiple reusable patterns, including foreach loops, subworkflow, and conditional branch, etc.
You could replace Maestro with Windmill here and it would be precisely correct. Their rollup is what we call the openflow state.
Main differences I see:
- Windmill is written in Rust instead of Java.
- Maestro relies on CockroachDB for state and us Postgresql for everything (state but also queue). I can see why they would use CockroachDB, we had to rollout our own sharding algorithms to make Windmill horizontally scale on our very large scale customer instances
- Maestro is Apache 2.0 vs Windmill AGPL which is less friendly
- It's backed by Netflix so infinite money but although we are profitable, we are a much smaller company
- Maestro doesn't have extensive docs about self-hosting on k8s or docker-compose and either there is no UI to build stuff, or the UI is not yet well surfaced in their documentation
But overall, pretty cool stuff to open-source, will keep an eye on it and benchmark it asap
arresin
Anyone considering windmill needs to look at this first:
https://www.windmill.dev/docs/advanced/local_development.
Why do I need to "sync" with windmill? Why is there an IDE built into windmill? Why is this so convoluted? It's like it's starting with the goal of lock-in before even developing a good product or finding market fit.
ensignavenger
Thanks for the great comparison! While Meastro is Apache licensed, if it depends on CockroachDB, Cokroach itslef isn't even Open Source, so that isn't great. I would rather have an AGPL codebase than a non open source dependency. Of course overtime some one could add alternative DB support.
jamra
I really wonder why they didn’t choose something like RocksDB for more speed.
rwky
Been using windmill for a few months and so far it's rock solid keep it up!
skissane
I'm a bit confused about what is going on here: This project appears to use Netflix/conductor [0]. But you go to that repo, you see it has been archived, with a message saying it is replaced by Netflix's internal non-OSS version, and by unmentioned community forks – by which I assume they mean Orkes Conductor [1]. But this isn't using Orkes Conductor, it looks like it is using the discontinued Netflix version `com.netflix.conductor:conductor-core:2.31.5` [2] – and an outdated version of it too.
[0] https://github.com/Netflix/conductor
[1] https://github.com/conductor-oss/conductor
[2] https://github.com/Netflix/maestro/blob/e8bee3f1625d3f31d84d...
halamadrid
Yes Netflix abandoned Conductor long time ago [0]. The other repo is built and managed by Orkes after Netflix abandoned it.
[0] https://techcrunch.com/2023/12/13/orkes-forks-conductor-as-n...
skissane
I haven't touched Conductor for a few years now, but back in 2020 I did some work trying to implement it, even submitted a few PRs – https://github.com/Netflix/conductor/pulls?q=is%3Apr+author%...
My impression of the code base, is I felt like it needed a lot of work to run in a non-Netflix environment. Which is part of why the project I was working on ended up abandoning Conductor – we were going to embed Conductor in our product as a workflow engine, we ended up building our own workflow engine from scratch instead. Another team did end up using it for some internal use cases, but scalability/reliability/etc are less of a concern for internal use cases as opposed to customer-facing ones.
And then Netflix abandons it – and then they open source something else which depends on an old version of it – well, I'm happy they open source anything, but it fits with my earlier impression – throwing stuff over the fence which can be a struggle to adopt in an outside environment. Still, throwing it over the fence is better than not releasing it at all.
IamLoading
if you look at the linkedin history, the main 2 maintainers left around 6 months before they archived it.
nfkuler
I can tell you coming from Netflix that we have not abandoned Conductor. In fact, we have more that 5x the usage of Conductor across the Company in the last 6 months alone. The news link above is a bit misleading.. We archived the project, and its now managed at conductor-oss/conductor project given that there are a few thousand companies using it and there are a few companies not managing this project and its now graduated to a new foundation, just like many projects before at Netflix (Iceberg for eg)
saturn8601
Anyone here use Activebatch? To me it is the best software I wish had an equivalent for non enterprise users. I have tried and tried to use other "competitors" but Activebatch's simplicity of just attaching a simple MS SQL DB, installing the Windows GUI and execution agent is just click, click, click and now you have a robust GUI based automation environment where you don't have to use code...or if you want, go ahead and use code in any language if you want...but you don't have to.
Airflow may be robust but it is hidden behind a complexity fence that prevents most from seeing whatever its true capability may be. The same goes for other "open source" competitors.
Why can't someone just develop a robust DB backed GUI first system?
I have tried online services as well, they pale in comparison. I guess the cost of maintaining extensions is what kills simpler paid offerings?
Its a complete shame that ActiveBatch is walled off behind a stupid enterprise sales model. This has prevented this wonderful piece of software from being picked up by the wider community. Its like a hidden secret. :/
skywhopper
Advice: don’t rely on any tool open-sourced by Netflix. They have a long history of dropping support for things after they’ve announced them. Someone got a checkmark on their promotion packet by getting this blog post and code sharing out the door, but don’t build your business on a solution like this.
meliora245
why would one consider this over something more established such as Temporal, also I see Maestro is written in Java vs Temporal's Go
trustno2
Temporal's go is... something. They used to use Java (I think), then they switched to Go, and the Go is very Java-like.
Or maybe I just don't know Fx.
https://github.com/temporalio/temporal/blob/main/service/mat...
The issue we hit with Temporal - again and again - is that it's very under-documented, and it's something you install at the core of your business, yet it's really hard to understand what is going on, through all the layers and through the very obtuse documentation.
Maestro has... no documentation? OK Temporal wins by default.
swyx
no just the SDK is Java. temporal is 99% Golang, even at Uber https://github.com/uber/cadence
arresin
They're saying the code style is java like and I agree. Whether or not that's a bad thing depends on your opinion of java and java coding styles.
robryan
Netflix also uses temporal: https://temporal.io/in-use/netflix
iamspoilt
That's also my question.
aimazon
isn’t Maestro an alternative to Airflow, not Temporal? Temporal isn’t a workflow orchestrator. There’s some overlap on the internals but they’re different designs for different use cases.
pantsforbirds
This is a really great-looking project. I know I've considered building (a probably worse) version of exactly this on almost every mixed ML + Data Engineering project I've ever worked on.
I'm looking forward to testing it out.
HugoLu88
I'm building something in the space (orchestra) so here's my take:
Folks making stuff open source and building in the open is obviously brilliant, but when it comes to "orchestrators" (as this is, and identifies) there is already so much that has been before (Airflow and so on) it's quite hard to see how this actually adds anything to the space other than another option nobody is ever going to use in a commercial setting.
Shameless plug: https://getorchestra.io
indiv0
Is this meaningfully different from Conductor (which they archived a while back)? Browsing through the code I see quite a few similarities. Plus the use of JSON as the workflow definition language.
opiniateddev
Conductor was moved here: https://github.com/conductor-oss/conductor Maestro uses conductor as its core.
https://github.com/Netflix/maestro/blob/main/maestro-engine/...
https://netflixtechblog.com/orchestrating-data-ml-workflows-...
iamsanteri
So will this serve as a stand-in replacement for something like Airflow?
then4p
I'm also missing comparisons to other existing tools like airflow, dagster, mlflow...
makestuff
Yeah, also curious if this is meant as a replacement for Airflow.
dboreham
Interesting. My team recently built a thing for managing long running, multi-machine, restartable, cascading batch jobs in an unrelated vehicle. Had no idea it was a category.
Get the top HN stories in your inbox every day.
I used to be impressed with these corporate techblogs and their internal proprietary systems, but not so much anymore. Because code is a liability.
I would rather use off-the-shelf open source stuff with long history of maintenance and improvement, rather than reinvent the cron/celery/airflow/whatever, because code is a liability. Somebody needs to maintain it, fix bugs, add new features. Unless I get +1 grade promotion and salary/rsu bump, ofc.
People need to realize that code is a liability, anything that is not the business critical stuff that earns/makes $$$ for the company is a distraction and resource sink.