Pkl, a Programming Language for Configuration

You mean you don’t have a distributed merge post microservice that emits post migration events, which are then consumed by a post owner conversion service using your existing event-driven architecture to facilitate seamless data synchronization and user notification processes??? That is not very hacker news of you from the guy who owns hacker news.

Really? Link?

Gently teasing: linking a 2 hour video with "shameless self plug" definitely did _not_ help obviate the surreality.

Same - it doesn't help that I read Futamura as Futurama the first 3 times.

Same. There was a mini subthread years ago that applies.

https://news.ycombinator.com/item?id=13752964

Those aren't "buzzwords" though, it's a very specific way to implement programming languages. It's not really meaningful except for the PL implementation nerds.

Especially the Futamura projections. It's almost magic and very few people have even heard of them.

If Futamurma means what I think it means skimming across the Wikipedia entry, it would mean that simple value-holder-file configurations would be parsed and checked at the speed of a general purpose tokenizer. But without closing the door to what the language can express in more elaborate configuration file "landscapes". Best of both worlds and presumably all without requiring anybody but the toolmakers to understand what the buzzwords really mean.

Fantastic talk! Thanks for sharing.

That all sounds cool, but is any of that especially useful for a configuration language?

> With partial evaluation you have static (known at compile time) arguments to your function, and dynamic (known at runtime) arguments.

That's pretty clever... How is this implemented in actual code though? I can't even begin to imagine how that magic machinery works.

> Truffle is a framework for building languages, that sits on top of Graal.

wtf is Graal? That sounds like a supporting character from Beowulf.

I definitely did a double take to make sure they didn’t write Futurama.

holy graals

Too close to the "reliably solvable by simple heuristic" end of the spectrum to be a good game: if the text is short it's probably a joke, if it is a very long wall of words it's LLM and anything of somewhat reasonable length can only be a legitimate technical explanation, no matter how unlikely.

FWIW Graal is probably one of the most exciting technologies to come out of Oracle in a long time.

Agreed. Anything by Oracle is an automatic hard nope.

You'll really like the Quantum Resistant key exchange algorithm - Kyber. A related project is Cryptographic Suite for Algebraic Lattices or CRYSTALS.

Sadly they renamed Kyber to MLKEM.

[dead]

Not scalable though, like JSON with nested objects.

If your configurations are simple, you don’t need this. If they are not, you might.

Because you have not run into the problems this has addressed in your career, does not mean that you know better than Apple how to solve them. In fact, it means like you are uniquely unqualified to solve them. Acting like you are in a severely condescending dismissal of the engineers who worked on this makes for boring conversation.

Misappropriating a trendy word to use as a descriptor for anything you don’t like. This is how someone speaks immediately before taking a look in the mirror and realising that they sound like their parents. You are not the be-all end-all and your drive-by analysis of a technology is going to be biased by its suitability for what you do day-to-day. No amount of experience justifies your attitude.

It is no different when you are trying to solve a complex problem and complex layers upon layers obfuscate and confound you when you are trying to figure out why some component ends up with configuration values that are nowhere to be found in your input files.

I think they are being sarcastic about windows registry.

Yeah, it’s really great that compiling C is a straightforward endeavor that doesn’t require any tools, and definitely doesn’t depend on executables that generate or parse configuration files. I’m confident that no C developers have ever spent time on tool setup just to program.

You are proposing to use general propose language vs domain-specific ones. I think, this discussion has been settled quite some time ago. For example, awk itself is domain-specific, built for processing text (or filed-separated) files. On the other had, m4 is as old as awk. m4 was heavily used even before autoconf (sendmail configuration is probably the best-known and possibly the ugliest application of it), for configuration, so one can argue, the need for macro expansion and rendering of configuration (as well as othe types) files was recognised even back then.

m4 just shows its age, and we’ve learned what is good (being Turing-complete) and what’s less useful (multiple output streams).

m4 could’ve evolved into m5, m6, etc., but nobody did it, instead people developed tools like jinja. One could argue, this is wrong and should’ve been evolutionary, but the point still is, domain-specific language to render configuration existed because there was use for them. Perl indeed could be such, I guess it did not happen for totally unrelated reasons. If Perl worked, Python would not gain popularity.

Pkl is not conceptually a new invention, it’s a new attempt to solve a real problem.

I stopped using sed because it behaved differently on my coworker's machine than it did on mine.

I'm surprised there's no widely-adopted library for general-purpose languages that handles user-friendly object input and outputs static config (perhaps in JSON) for machines to read. Instead, there are several DSLs that act as dynamic config, and even some that generate dynamic config, or something in the middle like k8s YAMLs.

Have I just not thought hard enough about this problem yet, and a DSL is really needed for it? Or should I go write that lib?

> TOML also is a good example of "creating problems instead of solutions": They deliberately (!) broke compatibility to the INI format due to "I can't stand unquoted strings". Yeah, emotional feelings about CONFIG FILE FORMATS.

INI is hard to parse because without quotes the parser would not know whether it’s a “True” as a string or True as Boolean value. Formal parser that can be included into a program as a library and would reliably generate internal config representation (e.g., read config file into an object in application memory that will be used to modify the behaviour) is a good thing and TOML helps with that while INI does not.

> They deliberately (!) broke compatibility to the INI format

There is no "ini format" though? INI is a weak set of conventions that different programs parse in a completely different way, it's the opposite of standardization.

> ONE actual standard that everyone uses would be fine

Sounds like a social problem. How do you get _everyone_ to agree on anything?

It’s pretty clear that you are putting “things that I [think I] understand above things that are better. Have you ever written an ini parser? If you have, I doubt it was good, because none of them are, because ini is a flawed format.

Right, so its still just about 9 MB:

    guile=$(readlink -f $(which guile))
    sizes=$({ echo $guile; ldd $guile|grep /|sed 's+^[^/]*\(/[^ ]*\).*+\1+'; }|xargs -n 1 readlink -f|xargs du -ab|cut -f 1)
    sumsize=0
    for size in $sizes; do
        sumsize=$((sumsize+size));
    done
    echo $sumsize

While I learned to accept YAML it messes up editor usage.

It is so sensitive that basic text editing like copy and paste, tab, in/decreasing indent never quite do what I expect in IntelliJ.

I paste parts of yaml into another yaml and it ends up somewhere unpredictable.

Curly braces are great at ensuring correctness though, as goto fail; has shown.

How many people will run a malicious config at least once without the flags? At some point it becomes a numbers game.

Agreed. If we're getting the point where you're saying "it's like writing your configuration in [insert Domain Specific Language Here]" then I'd prefer to simply use that language. I understand the point Pkl is making in that "configuration won't work across DSLs so here's one language for all of them" but I don't know that that's enough motivation for people to adopt it.

However, I haven't built anything cool like this so what do I know. I'm just procrastinating on the my personal project and browsing hacker news.

A big problem we've hit with allowing users to write Typescript (or any other general-purpose programming language) for our product is that it's too powerful.

Observationally, it seems that all that power eventually gets used, and then you end up with config that has complex interfaces, or becomes non-portable because it's doing arbitrary file reads, or is non-deterministic because of an ill-advised call to random or the system clock. The config then becomes something not maintainable by the rest of the team - just the few who know it.

Config languages seem to need to strike an interesting balance between being complex enough to allow for reasonable DRY code (which helps maintainability at the expense of readability), but not so complex that they're not generally-maintainable.

My impression of typescript is that if you do:

a = 1;

and elsewhere:

a = 2;

The ultimate value of 'a' will depend on the order in which those statements are executed, right?

A configuration language should surface that as an error and tell you where all of the conflicting references are.

> Ah just give me typescript, I do not need to learn a new thing for configuration languages

People who don't know Typescript will still have to learn a new thing in that case, so there isn't really any reason to pick TS over any other existing language

> syntax highlighting in every editor

There is a tree-sitter parser for pkl, so it'll work anywhere where tree-sitter works. And for VSCode and JetBrains stuff, they have official extensions.

Nix is a bunch of things, one of which is a configuration language.

I agree that there's a huge documentation and user awareness gap, but NixOS is obviously possible to configure this way so it's definitely possible.

Might be room for a tool that exposes just the configuration management side of Nix in a more approachable way... on the other hand it would be a bit silly to use nix for conf files and not also for the underlying packages.

How come? I don't know anything in say, pycharm, that can't be done with a LSP server and plugins. But that's probably because I don't know enough so I'm curious about what limitations an LSP would involve!

Yes, I was using it when I joined Apple in mid 2020

Clang leaps to mind. Most of LLVM was developed under Apple's auspices, although the project predates Apple hiring Chris Latner. Swift is also open source, although in practice it's almost always used in a macOS context. Which I think is a shame, it's a great language for writing the sorts of servers for which people usually reach for Go.

That's not a complete list of programs Apple has open-sourced, although clearly open-source software isn't what they're best known for. But clang alone is a worthy contribution which deserves to be recognized.

I think the answer to your question is yes: LSPs can’t do everything that a plugin can do.

https://blog.jetbrains.com/platform/2023/07/lsp-for-plugin-d...

From https://pkl-lang.org/blog/introducing-pkl.html#editor-suppor...:

"We are also releasing two other plugins: our VS Code plugin, and our neovim plugin. Today, these plugins only provide basic editing features like syntax highlighting and code folding."

> For you a "date" type might mean ISO 8601, but for me it might mean RFC 3339

A generic date type doesn’t come with any specific string format. ISO 8601 and RFC 3339 are both ways of representing a date as a string. Which has little to do with Date as a type.

There’s also perfectly good solutions to those problems. Use a type alias to create a date type backed by a string, with formatting constraints (such as a regex). Then people can define dates using any string representation they want!

Incidentally this is exactly what Pkl lets you do. You can use Pkl schema to define in detail how you want to validate your data using various primitives like regex. And then separately create a config template that uses those custom data types. As a dev you can choose how tightly bound your config template is to a data validation schema, define all the validation in line, or import an external module to provide you with richly validated data types.

Even more reason to standardize the format used in configurations and to validate it early, rather than at runtime.

Seems like someone should create a new datetime standard, documenting the safe intersection of ISO 8601 and RFC 3339. I did find this useful comparison: https://ijmacd.github.io/rfc3339-iso8601/

That’s not a reason for giving up on all config validation before runtime. Just because we can’t solve a problem in every possible situation doesn’t mean shouldn’t solve the problem for any situation.

duplicate port number can be checked in the type system if you’re using racket, at least

Ah! Well this is the hole that Pkl does a very good job of filling!

Being able to use Pkl code-gen to create language bindings means you can take any arbitrary Pkl schema and turn it into native structures in your language, typed based on the Pkl schema. Then you can let Pkl do all the heavy lifting of parsing and validating a specific config, and turning it into native objects in your language.

So no need for double declaring types. Declare them once in Pkl, generate the equivalent types in your language, then just start using those types like any other primitive. The Pkl bindings will handle the loading glue for you.

The whole _point_ of Pkl is that it is both rich enough and integrated enough though?

Yaml predates toml by ten years or so _and_ has an extensible schema to define types.

Sadly, nobody ever cared about that.

Hmm isn’t pickle designed for exactly this use case? External config module deps you pull for overall config validation.

E.g. so you always write valid k8s manifests.

And then you can extend them with your own additional validation rules for what you think your app needs? I’ve just skimmed the docs but it seems it allows you to be as loose or as precise as possible, plus packaging and publishing those rules for others to use.

Seems kinda awesome.

Fixing the app implementation details so that the configuration stays "clean", or at least forcibly documenting them so that the configuration can be written correctly, is vastly better than allowing the app to make undocumented surprises with its private validation.

I am convinced the Pkl config will grow in complexity until it has a yaml or json config for the configuration program.

That’s a consequence of configuration languages not have proper type systems, and robust was to manipulate data.

Helm isn’t a configuration language, it’s a dressed up string templating system with ability to do kubectl apply.

> you can't guarantee your config is valid anyway if your config language can't see what class you're going to feed it to.

Obviously, but that’s hardly an insurmountable problem. We’ve had code gen and language introspection for decades.

Helm is not a proper configuration language. It just does string replace. Unbelievable that people actually use it.

A config language that does have the type information can give proper errors. Try for example terraform.

Doing so would necessitate the addition of another type: character/grapheme cluster.

Representing arrays as maps would impose no additional requirements outside of validation which is already considered as part of the proposal in question.

What does "morally" mean here? I've not seen that term used in compsci except when about ethics.

Yeah I actually haven't found a great answer yet. Here's everything I've tried and why it sucks:

* JSON. No comments. Deal-breaker

* JSONC. No unique file extension so its difficult to distinguish from JSON. Poor library support due to library authors drinking the "comments are bad" koolaid.

* JSON5. This would be an excellent option IMO except that library and IDE support is not great.

* JSON6. This just complicates JSON5 for minimal benefits. Pointless.

* Cue. As described.

* Jsonnet. Pretty good option tbh! However I couldn't get the Rust library to work. It's a low level parser, seems like you can't just plug it into Serde, which is what 99% of people really want. Also I ran into the "uncanny valley" effect where you can do some things but not all. So it tricks you into writing some programmatic config (e.g. with string manipulation) but then you find you can't do that string manipulation.

* Dhall. Weird syntax (backslash to declare functions. I've also heard it is slow. Didn't try this too much.

* YAML. Obviously YAML is the worst option. However I did realise you can use it as basically JSON5 except with a different comment character, which is not too bad.

* Starlark. Actually I haven't tried this yet but it looks promising.

So yeah I have no idea at the moment.

I wonder if it would be worth defining a "YAML JSON5" format, that's basically YAML-compatible JSON5.

I really like CUE, but for most use cases I have I would want to embed it in an application, and Go is the only language with support.

For it to gain more adoption it really needs a rewrite in a low-level language (C/Rust), so it can be exposed in various languages through an extension/FFI.

I've been somewhat surprised that CUE bills itself as "tooling friendly" and doesn't yet have a language server- the number one bit of tooling most devs use for a particular language.

I'm assuming it's becaus CUE is still unstable?

Anyway, if others are interested in CUE's LSP work, I think https://github.com/cue-lang/cue/issues/142 is the issue to subscribe to

There are three (maybe more?) ways things can be Turing-incomplete:

1. You are limited to N evaluation/reduction steps.

2. The language doesn't include primitives like recursion or loops.

3. You can have recursion or loops, but the language makes you somehow prove that your program will terminate.

I think (1) would be fine, but I don't know any configuration languages that use this approach.

(2) is restrictive/annoying whenever you want to implement any logic in the config language library. Eg. a tool uses a homegrown data format BAML and you need to convert JSON to BAML in the config. Now either you have to write and manually call a preprocessor, or you need to use a patched version of the <config language evaluator> that will have JSON->BAML as a built-in, or you must implement JSON->BAML without loops or recursion. For a more realistic example, imagine that a certain config string has to be HTML-escaped and the config language doesn't provide a built-in for that purpose.

(3) -- you don't want it. There are languages (like Agda) that let you prove things like "this program terminates", but writing those proofs can be harder than writing the program itself.

Many data transformations that you take for granted in other languages are either impossible or require amazing feats of contortion of the language to make happen.

Step two of installing Copybara is to install Bazel [0], so that doesn't exactly contradict my claim that if you're not already using Bazel you probably won't use Starlark.

[0] https://github.com/google/copybara