Skip to content(if available)orjump to list(if available)

Launch HN: Windmill (YC S22) – Turn scripts into internal apps and workflows

Launch HN: Windmill (YC S22) – Turn scripts into internal apps and workflows

83 comments

·August 9, 2022

Ruben here, software engineer, long-time lurker of Hacker News and founder of Windmill. Windmill is a fully open-source self-hostable platform and runtime to build complex workflows, internal apps and integrations using any scripts in Python or Typescript-deno. I am back after having been revealed a bit too soon on HN and miraculously getting into YC (https://news.ycombinator.com/item?id=31272793).

To build internal apps for ops, integrations between services that cannot talk to each other directly, or to run background jobs that run your business logic and analytics, the two main options today are no-code solutions and old-fashioned, roll-your-own scripting. Both have problems, and our goal with Windmill is to find a new sweet spot between the two. No-code solutions are productive if your problem matches the tool exactly - but it not, they are rigid, hard to extend and quickly become tech debt, annihilating their initial time advantage. Indeed, no-code is just code but made by an opinionated someone else and hidden as a blackbox with an UI.

The alternative is to do it the old-fashioned way, writing everything from scratch, both backend and frontend, perhaps deploying it on the latest flavor of serverless, and pray to never have to touch it again because that took way too much time and it has now became a burden that the ops and business team might poke you about regularly.

Furthermore, the landscape of SaaS is specialized tools for everything—alerting, data analytics, administration panels, support management, integration between services—when it feels like a few scripts would have been as good or even better and spared you the need of depending on one yet another tool. This could be even further facilitated if there was a way to import the right bunch of scripts from a fellow community of engineers, tweak it and deploy it like you can do in communities where automation can be shared as simple JSON files, for instance in the node-red or home assistant community.. That’s the idea of Windmill: to bring back the power of scripting in an easy way.

With Windmill, you write normal scripts, or reuse ones made by others, and we make them production-grade and composable. You shouldn’t have to worry about things like http requests or scheduling jobs. We abstract much of that away, making your scripts be both more focused and more composable. You end up doing things the right way but much quicker.

We reduce the complexity of workflows, integrations and internal apps by uniting them all under one banner. At the heart, they mostly have the same needs: workflows with a UI or a schedule. One tool that does it all out-of-the-box offers greater consistency and allows you to grow the complexity of your toolset at your own pace.

I have an academic background in compilers and industry experience in distributed systems. My compiler work made me wary of solving every problem with a domain-specific-language (DSL) or complex frameworks. We can just do more with the well-crafted existing languages like Python or Typescript. Rolling up your own DSL is nice in theory, you can make it very ergonomic and focused on the task at hand, but then you start adding features and either reinvent existing – albeit worse – programming language or decide to stop there. In the very large majority of cases, a well crafted library is vastly superior to any DSL. By being able to use any library of Python and Typescript, we stand on the shoulders of giants.

I have also observed that the best distributed systems are often the most simple as they are more predictable and have invariants that are easier to reason with and scale horizontally. This is why for Windmill, we rely solely on Postgres + our native workers + our http REST api layer. Later on, we plan to build adapters to host the workers on AWS lambda or Cloudflare workers, and the queue on Kafka if your needs are exceptionally high.

At the heart of what we have built is a queue implemented in Postgres and workers implemented in Rust that create a sandbox (using nsjail), fetch dependencies, and execute scripts. Every script can be triggered through its name with an HTTP POST by passing a JSON payload in which every field corresponds 1:1 to an argument of the script’s main parameters. Most primitive types in Python or Typescript have a natural corresponding type in JSON so the conversion is always what you would expect. We then execute the script inside a new sandbox and then store the results in the same Postgres DB at the end of the job execution.

The HTTP payload can be sent from your own frontend or you can use our automatically generated UI. Indeed, we do a simple, yet effective analysis of the parameters of your script, and from it, generate the jsonschema corresponding to your parameters. That schema is what enables us to convert any script into a no-code like module for flows, or a standalone internal app with its auto-generated UI. In the case of Python, we also look at the imports to deduce the Pypi dependencies without you having to declare them.

For flows, we defined an open spec for building them out of those scripts we call OpenFlow: https://docs.windmill.dev/docs/openflow. It is essentially a json format for describing a sequence of steps with for loops and soon branching. The most interesting bit here is that each input of each step can define its input as a javascript expression that refers to and transforms the output of any previous step. We make it fast by leveraging native v8 integration in Rust (thanks to the deno team) for executing those expressions. This makes this apparently linear sequence a flexible DAG in which one can express complex workflows.

Then on top of that we have an UI builder for flows that hides most of the complexity to give an experience that is similar to a low-code platform where every step is treated as a blackbox. The platform itself offers all the features that you would expect: a variable and object store for storing states, plain values and credentials; a cron scheduler, tight permissioning for the sensitive credentials, groups, a webeditor with smart assistant to edit the scripts directly in the platform etc. Finally, we made a hub (https://hub.windmill.dev) to share flows and scripts with everyone. The goal is to grow over time an exhaustive library of pre-made modules and flows to tweak from so that you can focus on what is actually custom to you.

Windmill is open-source and self-hostable. You can think of it as a superset of both Pipedream and Airplane.dev. Compared to Temporal, the scripts themselves are agnostic of the flow in which they are embedded, which has the benefit of making it easier to build a hub of reusable modules. We are the only ones as far as we know to convert script parameters to UI automatically. We see ourselves as complementary to UI builder solutions like Retool or Tooljet as we do not want to focus too much on the auto-generated UI and could be used solely as the backend part of the two aforementioned tools.

We are now a team of 3 senior engineers and the product is progressing faster than ever with a public roadmap: https://github.com/orgs/windmill-labs/projects/2

We make money from commercial licenses, support and team plans on the hosted solution.

You can self-host it or try it https://app.windmill.dev, the free tier is generous (and the paid one is not enforced yet). Our landing page is: https://windmill.dev. We would appreciate your feedback and ideas and look forward to all your comments!

andrewguenther

I beg of you, please reconsider the SSO tax (https://sso.tax/). This is becoming a big pain point for small orgs trying to become compliant that a small change like this can be huge in decision making. If I look at Pipedream, Airplane, Temporal, and Retool none of them will even list pricing if I need SSO, I'm suddenly a 40 person company with "Enterprise" needs. If you want an easy differentiator, including SSO in your Team tier is a simple way to do that.

jedberg

The reason a lot of companies charge extra for SSO is because it is a support nightmare. Most users can't tell the difference between "my SSO provider is broken" and "your service is broken" so they always blame the service, who now has to either tell the customer "talk to your SSO provider", which of course they don't like, or having to diagnose and fix SSO problems.

I agree that it should be a core feature, but I sympathize with companies that need to charge extra for it.

grinich

(Small plug for my startup, hope that's ok :))

WorkOS makes adding SAML to your app incredibly easy and fast. We're powering Vercel, Planetscale, Webflow, Loom, and hundreds of others. Pricing is transparent and scales as you grow. And you don't need to talk to a sales rep. ;)

https://workos.com/single-sign-on

samlambert

Great product.

somenewaccount1

What? That's such absolute nonsense. Compared to managing your users passwords? Bah! No way is supporting saml that hard. It's. Library, and config. Users usually find out pretty quick when thier idp is down.

deepakprab

Plugging in my startup BoxyHQ here. This is the reason why we open sourced our SAML integration - https://github.com/boxyhq/jackson, it should be commodity.

the_common_man

Isn't this true for just about any integration in a product?

jedberg

Yes, but SSO is a very upfront integration that blocks all access to app. Most other integrations only break a piece of the app or are in the background. And lots of companies charge extra for those integrations too.

Aeolun

I find this hard to believe, setting up SSO was a piece of cake with Github. Sure you have some issues during the setup, but once it’s there it never changes.

Then again, Github also chargea you $20/month instead of $4/month if you want SSO in a small org, but it’s possible.

Saaster

If I as a SaaS provider get my SSO SAML integration via a provider like Okta or Auth0, the auth provider pricing itself is also on a "call us" tier, with a per-federation pricing in the low four figures for each individual company connecting to me via SAML.

It's pretty insane, so I'll state it again: To have a company connect to my SaaS via SAML, I as the SaaS provider have to pay my auth provider $X,000 per year for the privilege. Not counting the base enterprise tier pricing for the auth solution itself. So then I have to roll my own solution if I want to provide it for free, and I get the joy of supporting the long tail of broken SAML implementations on both the service and identity provider sides. For free. In a perfect world SSO wouldn't be a shitshow and everyone could have it for free, unfortunately that is not this world.

aisrael

The SAML world is definitely a fun mess. We’re[1] building out SAML support and are beta testing it with a few customers and it is funny how different even the large IDPs are. Add in things like needing to test the integration, making sure attribute and role mappings are correct, and it’s unfortunate but understandable that companies not specializing in auth wouldn’t want to deal with it except for customers that pay a lot.

[1] Disclaimer, I’m a founder of PropelAuth

deepakprab

Plugging in my startup BoxyHQ here. This is the reason why we open sourced our SAML integration - https://github.com/boxyhq/jackson, it should be commodity.

somenewaccount1

Fwiw, Amazon offers an sso platform with saml for free. That's not the answer for everyone though, and may not be for you even. Just putting it out there for those looking for a cheap saml identity provider.

rubenfiszel

I don't disagree with what you are saying. It's just that everything is already free and as friction-less as possible. I made a point to remove any lock-in. My goal is to make Windmill fully-featured and available to the widest range of people. On the other hand, we do need to make some money at some point. We could have an "enterprise edition" with features that are proprietary. This is what most open-source companies do, but it would create misaligned incentives to not bake those into the product and make the product worse. If I am being honest I will probably bake SSO into the team-tier as soon as we have some healthy financials :)

apugoneappu

From my limited knowledge of how SSO works, is this possible?

1. A company Acme wants to have SSO in the tools A, B and C that it uses.

2. Another company Balloon integrates with A, B, C to use the admin API for an admin account to modify or delete users in that account

3. Acme logs in to Balloon and connects its admin account of A, B, and C to these integrations.

4. Now Acme has access to employee's accounts in A, B, C through the Balloon's dashboard to modify or delete users etc.

jon-wood

Yes, it’s typically referred to as SCIM and is supported by most serious SSO services.

rubenfiszel

I was not aware of this and will look it up. I was of the naive opinion of the parent thread.

krashidov

I wish this website had a wall of fame on it as well

mwcampbell

The ability to run sandboxed workers from a PostgreSQL-based queue sounds useful all by itself, particularly for multi-tenant SaaS applications that need to do compute-intensive tasks on untrusted inputs. The difference, of course, is that in this case, the worker implementation is part of the application, not a user-defined script. How tightly are the work queue and sandboxed worker runner coupled to the rest of the product?

rubenfiszel

They are tightly coupled at the moment but easy to extract out. I did not think there would be interest for that but I could carve it out as a separate dependency project for everyone to reuse!

Here is the magic part to implement the queue in sql: https://github.com/windmill-labs/windmill/blob/main/backend/...

ogarten

We are thinking about creating something like this actually. We are still early-stage and haven't launched yet but we are looking for feedback and input to our idea.

Would you mind talking to us about your problem?

sprsimplestuff

I've only been in the industry for about a year and a half, so forgive my lack of knowledge here...I see these no-code/internal tools/run a script easily startups every so often

What are examples of things people are building with these services? My context - I work at a 200ish person b2b saas - I don't know of anyone in my company using these, at least not in an official capacity I can see

hoping to learn and b informed

rubenfiszel

Great question. The answer varies widely depending on the kind of company you are.

1. The biggest pain we solve is to turn scripts that you would run on your own laptop as internal apps that you can share. If your company make no use of script anywhere and does not need any automation then maybe it doesn't apply to you.

2. Avoid live sql queries in production and use templatized sql query made into apps instead (automatically!). Making live sql queries is common in DevOps, support and ops in general. It can be very error-prone, stressful and inconvenient.

3. Integrations between tools you already use but that are unable to talk to each other.

4. Workflows, code that runs very frequently to react to new events, transform data and run your business logic. Most companies are a frontend on a database that is updated in the background. We make it possible to build those workflows from simple scripts so that you can build it faster, more reliable and easier to maintain.

5. The last one only apply if you are a SaaS that want to provide automation as a feature of your product, or a no-code tools yourself. Because we focused very much on the hard-engineering of orchestration and specs for workflows, you might simply want to wrap Windmill to offer it to your own clients.

walleeee

Have been looking for something like this for several years to replace the custom UI/orchestrator for a science gateway. The other options I've seen don't do auto UI generation or don't allow self-hosting or aren't open source. This looks great. Thank you.

rubenfiszel

Thank you for the kind words!

aliasxneo

I'll admit I only gave the docs a cursory review, but it might help to expand on what self-hosted means. Over at r/selfhosted, for example, it typically translates to, "can run on my hardware with no external dependencies." It's hard for me to tell if Windmill meets this definition, but early indications seem to suggest the answer is no.

-- Edit --

I just saw the Github repo. Does this section exist[1] in the docs anywhere? Might be good to include.

[1]: https://github.com/windmill-labs/windmill#how-to-self-host

rubenfiszel

Thank you for giving me the opportunity to clarify this! https://github.com/windmill-labs/windmill#how-to-self-host

You just need to docker-compose up https://github.com/windmill-labs/windmill/blob/main/docker-c... and you are all set :)

nolok

I believe his question was about "does this docker instance then rely on an external service/api to run", which it appears it does, and thus wouldn't quite qualify for fully self hosted.

Many reasons why it's important to know that, the one that matters to me personnaly is because of data storage location requirements.

rubenfiszel

It does not rely on external api or services, you can run this completely isolated (well you would need a postgres instance). The WindmillHub that Windmill fetch data from is completely unidirectional and something purely out of convenience. In theory shouldn't have an impact in your ability to use the product without it, although we should introduce a flag for the product to not even attempt to list the available scripts from the community if that's your wish. I will write an issue on the roadmap about that.

tailspin2019

For what it's worth, I'm confused about the self-hosting option too.

On the home page it has the phrase "Self-hostable AWS Lambda" which when clicked takes me to a page entitled "Benchmark" [0]. As someone who doesn't use AWS, and the linked page seemingly not giving me much clear indication of being able to host this outside of AWS, I'm left confused about options for self-hosting.

Often with these sorts of products, I won't necessarily self-host initially, but will almost always want to ensure up-front that the option is there if I want to in the future.

It's perhaps slightly counterintuitive that being clear about self-hosting being available is a big tick for me onboarding as a SaaS customer!

Meant as constructive feedback as Windmill looks very cool!

[0] https://docs.windmill.dev/docs/benchmark/

Edit: Ok I see I misunderstood... that page is showing how Windmill benchmarks against AWS Lambda. I had assumed that clicking on the prominent blue link that mentioned "self hosting" would tell me about self-hosting :-)

Narretz

This looks cool. Since I haven't used Airplane etc. it looks to me a bit like a mix between n8n and low code tools for creating internal apps.

A bit of feedback for the website:

- I can't see anything linking to a pricing page / feature tiers. There should be a prominent "Pricing" link in the top menu like all other such services do, even if the paid Tier isn't enforced yet. - the docs should have a page for self-hosting / deplyoment. This is another big one I always look for in any tool. - when I go to https://app.windmill.dev I am redirected to the login, but I can also click the browser back button, which leads to the app page, but all broken because no data is available

rubenfiszel

Pricing is at the bottom of the landing page. Point taken, we should add the link directly to the menu bar.

Good catch for the back-button bug, we will fix it. Thanks!

poulpi

Very cool!

I've seen too many companies with spaghetti integration built on Zapier / Make to orchestrate the user onboarding (CRM, database, Finance tool)

At the beginning it's a good idea and at one point everything start to crash and you wonder how to put everything in a got repository!

Windmill seems really cool

rubenfiszel

Thank you. I agree, everything starts very nice and it's all downhill from there with rigid tools.

bearjaws

I could see my company using this, we have some internal tooling that we were considering moving to Retool. I think for any of these platforms to succeed long term they need to be just as easy to back out of (e.g. once it gets too complicated, go back to normal code) as they are to start using.

One thing that I misread and I believe others would too is on the main landing page where it has the list of features is one of them says "UI? Done (checkmark)" and the rest of the items don't have any Done or checkmark. It reads like the rest of the items listed aren't ready at all, which based on the docs they are at least available.

rubenfiszel

checkmark removed, thanks for the feedback. Looking forward to have demanding users with strong use-cases.

About lock-in I completely agree which is why you can just deploy from github and export a tarball of your entire workspace at any point. The scripts can be used outside of Windmill (except if you use our K/V store but you do not have to). The day you want to migrate out, you will just have to redeploy your scripts as lambdas somewhere else.

debarshri

My 2 cents, choose retool. I have no association with retool but as a product/business owner question is do you want to risk with a startup or an established product that reasonably solves your problem. Retool is pretty reasonable pricing wise, very stable, great support and inherently you dont have to deal with startup like bugs.

raviparikh

Retool is a great product, but I don't think your reasoning is entirely sound.

> do you want to risk with a startup or an established product that reasonably solves your problem

I think it depends greatly and I don't think this is a hard-and-fast rule. If you're adopting a product that is easy to replace and doesn't have huge vendor lock-in, and the startup offering solves your problem better than the established vendor, then a startup might be a better choice. Windmill is an open source product, so even if they went out of business, you'd still have access to the code which is a great way to avoid some amount of the risk surface area.

dang

You've posted four comments now in someone else's launch thread. One was arguably maybe ok (though also arguably kind of rude). But you're going overboard now, so please stop.

Whether it's YC-funded or not, each startup deserves to be the focus of its own launch thread. There will be plenty of other opportunities for you to make a case for your own startup.

Cilvic

This looks amazing and I'd love to try it out for this one thing I have in mind.

I'm not familiar with the Deno part of Typescript. Would it somehow be possible to use a C# library as a dependency?

For example https://github.com/OfficeDev/Open-XML-SDK I need for a way to edit (not create) the openXML content of Microsoft Office files

toomuchtodo

Will you eventually support importing workflows from similar closed low code/no code tools?

rubenfiszel

If there is a way to do this in an automatic manner for sure but it's of course hard to reverse engineer proprietary spec.

On the other hand, one model is to have low code/no code tools, or SaaS that want to add automation as a feature of their product, wrap us to run their backends as we have done a lot of effort on the engineering of the orchestrator/backend and we cannot compete on every vertical. If they all run deno/python code with the OpenFlow format, then everything is interchangeable everywhere!

toomuchtodo

Tremendous, wishing you much success!

tailspin2019

Friendly suggestion to all future "Launch HN's"... I think you may get better engagement with the link to your product right at the top of the announcement comment.

I'm sure lots of people are more interested in immediately checking out your cool thing and not necessarily having to read (or skim) a wall of text first :)

As a side note; I've noticed that Launch HN posts are typically very verbose as a rule. Maybe this reflects more on me, but I almost never read them, but still am often interested in the products/services themselves and will hunt down the link to just go and check it out.

It's great to have the backstory and detail there, but to those writing them, it's worth bearing in mind that a large % of people (possibly a majority?) will skim read at best. So link upfront and maybe a TLDR sentence would always be a good opener IMHO.

Edit: Also... congrats on the launch :-)

dmundhra

This is great. I have been trying out pipedream and would love to try this too. Are there steps available to do a self deploy? Also, can the URL be rendered on my own domain?

rubenfiszel

Self-host: https://github.com/windmill-labs/windmill#how-to-self-host

Custom domain: not yet but for sure we will add it as an option for the teams plan