Literate programming: Knuth is doing it wrong (2014)

I think it's clear to everyone that Knuth's style of literate programming didn't achieve any sort of mainstream adoption.

However, this article doesn't mention a newer form of literate programming that has gone mainstream. Notebooks! Whether it's jupyter, colab, whatever the Julia one is called, etc. People use literate programming all the time for data science and machine learning.

You wrote "There's a fundamental problem with generating a beautifully typeset document for a codebase: it's dead. It can't render inside just about any actual programming environment (editor or IDE) on this planet, and so we can't make changes to it while we work on the codebase."

Sorry, that's not correct. When writing a literate program you should also include a chunk containing a Makefile. Extract the Makefile and let it construct the program.

I did this with the whole Clojure programming language. The whole source code for all of Clojure, as well as the test cases, are in the PDF. The sequence (from scratch is):

Extract the Makefile from the Latex document. Run the Makefile. This:

  1) extracts the code and tests

  2) compiles the code

  3) runs the tests

  4) recreates the PDF from the Latex

If you have the PDF open next to your editor it is likely that the PDF viewer will reload the changes immediately.

This ensures that (a) your program compiles, (b) your program passes tests, and (c) the PDF builds correctly.

I've been working on literate programs for years. See https://en.wikipedia.org/wiki/Axiom_(computer_algebra_system.... I've even given a talk on the subject See https://www.youtube.com/watch?v=Av0PQDVTP4A).

I find literate programming to be the MOST important change in programming since I started in 1970.

Once a program has been developed and the developers have moved on to other tasks it needs to be maintained. The fundamental problem is that if you modify code you didn't write, you don't see the big picture and you don't understand the reasons why the code is written the way it is. Thus changing code without a larger context is almost always going to introduce new bugs.

The only way to correctly change code is to deeply understand the implications of the change. This requires a deep understanding of the code and an awareness of the big picture. Yet the "why" of code is rarely ever written down in standard programming practice. The goal is only to elaborate the "how" so the machine can perform the task. The programmer communicates with the machine.

Literate programming, as used in Axiom, is an attempt to communicate with other users, developers, and researchers in addition to the machine. The goal is to have the program read like a story so that others can understand the rational, the theory, the choices, the implications, and the implementation context as well as the "how".

This code is intended to live forever but it is highly probable that you will not. Write to communicate with the next person to pick up the torch. When you explore code, write down what you learn. When you change code, explain why you made your choices. When you write new code explain what others need to know to maintain it.

    Yet to me, literate programming is certainly the most important thing that came out of the TeX project. Not only has it enabled me to write and maintain programs faster and more reliably than ever before, and been one of my greatest sources of joy since the 1980s -- it has actually been indispensable at times. Some of my major programs, such as the MMIX meta-simulator, could not have been written with any other methodology that I've ever heard of. The complexity was simply too daunting for my limited brain to handle; without literate programming, the whole enterprise would have flopped miserably.

    If people discover nice ways to use the newfangled multithreaded machines, I would expect the discovery to come from people who routinely use literate programming. Literate programming is what you need to rise above the ordinary level of achievement. But I don't believe in forcing ideas on anybody.

It took 30 years, but literate programming paradigm is now very commonly referenced when working with notebooks, even though it's not what Knuth was going for in the first place and the original intent was different.

Notebooks are a direct evolution of REPLs and at the time of writing (Knuth published his paper on literate programming in 1984), REPLs have been well known (they had been around since 1970s). Yet there is not a single mention of them in the original paper, nor any prediction how literate-programming-capable notebooks could look like. At the time, these were two different concepts each serving its own need (documentation vs exploratory programming).

I have been using ObservableHQ as my main environment for over a year. I have found that I have started adopting a "documentation driven development" methodology organically over time. The opening paragraph of the notebook, I review and edit almost every time I open the notebook. The opening paragraph provides a qualitative north star and tone setting for the whole project contained within. You can have a dynamic table to contents structuring the whole notebook. When documentation is included with source code, it is easy to skep between micro projects, as all information required to run the project and perform maintenance can be put where it makes most sense. So as a solo developer, I am able to scale more broadly than I used to be able to.

To me, literate programming is massive productivity left on the table. I love it.

I love it so much I developed https://webcode.run to extend observable to the backend with a fresh retake of a serverless compute environment. I have a working OAUTH server "hosted" here: https://observablehq.com/@endpointservices/auth if you are concerned this does not scale to interesting infra. You can eject source code as ES6 modules and publish to NPM which I have done for generally useful tools like the first REDIS client for web https://observablehq.com/@tomlarkworthy/redis

Literate programming is certainly a great idea. The problem starts when it is implemented badly in a project, and instead of code that is easier to read, the devs are faced with files stuffed with "comment-noise".

Should we try to write code that is primarily meant to be read? YES! Oh god yes! Code is read 1000x more than it is written. But that doesn't mean "stuff the code with comments".

Primarily, it means write code that is clear, concise, conveys the meaning, is logically structured, is not "optimized" into obscurity before even the first performance measurement has been done, and doesn't implement something just to fulfill some dogmatic paradigm.

Once that's in the bank, we can start using comments where they are needed: Explain Functions that are part of the interface, explain functions that have complex logic, explain imports that aren't part of the stdlib or well known 3rd party libraries. I don't need a comment telling me what `import requests´ is needed for. I certainly do need a comment telling me what the hell `from apputil.tools import tvectorization` does.

Yeah, the word "read" seems to have seduced us into an obviously false analogy. "Reading" code is not like "reading" a novel. Perhaps this could seem more plausible if your work was primarily along the lines of Knuth's, rather than combining the same bag of shopworn, well-known techniques into academically rather uninteresting production code.

Another article on this topic I recommend is "Code Is Not Literature," from Peter Siebel: https://gigamonkeys.com/code-reading/

I've read a fair number of Knuth's literate programs, and I've read this post a few times over the years, and my opinion is mixed.

The first point in the post is that many (non-Knuth) LP systems provide typesetting, but allowing arbitrary code re-ordering (as in Knuth's WEB/CWEB) is actually the more important feature. This is a good point.

But next it faults Knuth for putting #include statements at the top of the program, instead of using his system's reordering feature to have an "⟨includes⟩" section that is later filled in when each #include is needed (the way he does for "⟨Type definitions⟩" and "⟨Global variables⟩" in those examples). This slightly misunderstands the purpose of literate programming, I think.

As far as Knuth is concerned, the main idea in literate programming is an attitude, an orientation: treating programming as writing, for a human reader. (This particularly makes sense for him as he is primarily, in a very deep way, a writer.) Now, writing is always (best) done with a specific reader in mind: you assume the reader has a certain background/prerequisites: some things that don't need explaining, and some things that do. Depending on the reader you're targeting (e.g., yourself a few years from now), and how polished you're trying to make your presentation, you may well choose to take it for granted and not bother explaining that a C program will have some obvious #includes at the top. Take it for granted, as understood between friends.

From this point of view, as long as you're writing for a human reader, it is literate programming whether you're at the extreme of explaining every part of every line of code, to a reader who does not even know the language, or at the other extreme of "here's my program: ⟨the entire code of the program⟩": it's only a matter of degree, how much you want to explain. [This is actually one of the misconceptions about LP: it doesn't necessarily require you to write lots of comments, or to explain everything verbosely; the main thing is to treat the activity as writing, and explain to whatever degree it makes sense given your intended reader. In Knuth's own TeX program http://mirrors.ctan.org/info/knuth-pdf/tex/tex.pdf , the one he came up with literate programming for, there are sections where he explains things very thoroughly but see also section §315 where, after explaining, he says "This is easier to program than to explain", and §69 where he says "Readers who like puzzles might enjoy trying to figure out how this tricky code works; therefore no explanation will be given" :-)]

A few other criticisms in the post are understandable given unfamiliarity with some further context about Knuth's work and these programs:

• Many of the examples being criticized are from programs that Knuth wrote for himself, as he mentions at the top of https://cs.stanford.edu/~knuth/programs.html — obviously the standard of polish and degree of explaining will be different when something is written up for public presentation rather than for oneself. (As I said, these examples also show that to write and enjoy LP you don't need to overdo it, feel free to explain only as much as you want.)

• An example is criticized for the fact that "GraphBase data structures" are not described in-line in the program, but again, as the webpage says, this is a program that uses "the conventions and library of The Stanford GraphBase" — Knuth has published an entire book about these data structures, so it isn't necessary (or realistic) to re-describe them inline. In fact, many of Knuth's programs can be criticized for being monolithic rather than modular, but the Stanford GraphBase is a rare case of Knuth actually factoring out a library for use in many programs.

• There's a criticism for "a steep jump across several levels of abstraction", but, contrary to common opinion, Knuth has repeatedly mentioned this as a good thing:

> "I have felt for a long time that a talent for programming consists largely of the ability to switch readily from microscopic to macroscopic views of things, i.e., to change levels of abstraction fluently." — 1974, Structured Programming with Go To Statements (http://www.kohala.com/start/papers.others/knuth.dec74.html, https://pic.plover.com/knuth-GOTO.pdf#page=32 )

> "I believe the thing that marks a computer scientist most is an ability to jump across levels of abstraction: We see the small and the large and several things in between as a continuum, and we don't even notice that we're jumping levels. — 2013 (https://programmingphilosophy.org/posts/jumping-across-level...)

> "after decades of observation I’ve come to believe that one particular talent stands out among the world-class programmers I’ve known— namely, an ability to move effortlessly between different levels of abstraction […] A programmer must deal with high-level concepts related to a problem area, with low-level concepts related to basic steps of computation, and with numerous levels in between. We represent reality by creating structures that are composed of progressively simpler and simpler parts. We don’t only need to understand how those parts fit together; we also need to be able somehow to envision the whole show — to see everything in the large while seeing it simultaneously in the small and in the middle. Without blinking an eye, we need to understand why a major goal can be accomplished if we begin by increasing the contents of a lowly computer register by 1. The best way to enhance our level-jumping skills is to exercise them frequently." — Foreword to The MMIX Supplement, published 2015.

• The second major criticism in the post is that the woven program (the readable output of the LP system: a typeset PDF file or book or whatever) is "dead". This is reasonable given the majority of readers, including myself in the past. But in fact, the intention is the very opposite of "we can't make changes to it while we work on the codebase. Everybody reads a pdf about a program at most once, when they first encounter it." For Knuth, the printed/woven version is the program, the one he thinks of as canonical and the one he uses whenever debugging or making changes to the program (with pencil on paper, I think). It serves him well: here he is in 2021:

> While I was preparing this round of updates, I was overjoyed to see how well the philosophy of literate programming has facilitated everything. This multifaceted program [TeX] was written 40 years ago, yet I could still get back into TeX’s darkest corners without trouble, just by rereading [B] and using its index and mini-indexes! — https://tug.org/tugboat/tb42-1/tb130knuth-tuneup21.pdf

And unlike the common criticism (not present in this post, to be fair) of the author's order not being what a reader may want, in fact a book is not meant to be read linearly. ("Books are not scrolls. […] Books are random access" — Manuel Blum, https://www.cs.cmu.edu/~mblum/research/pdf/grad.html) Knuth also recommends reading in whatever order makes sense; he thinks that his indexes are enough to enable this, though readers like us may disagree.

His idea of reading (a book, or a program) does include "poke at it and run what-if experiments", just that he does it in his head. See Peter Seibel's post at https://gigamonkeys.com/code-reading/ where he describes how Knuth was one of the only two interviewees in his Coders at Work who do a lot of reading other people's code, and he also includes some detail of how Knuth read some program. (See also this great post by Dylan Lederle-Ensign on Knuth's reading practice: http://www.dlederle.com/2018/11/25/reading-programs.html In fact, one of these programs Knuth "read" this way while spending a long time in hospital with a broken arm, he says somewhere IIRC.) So though I agree with the ideal of "viewing live, modifiable, interactive code", the intention of Knuth-style woven output is far from "passive reading" as the post says.

Nevertheless, given most readers today (probably this was true in 1980s too, to be fair), the post is probably right that given a typeset version people will read it passively, so overall this criticism is probably worth making, at least as a warning.

There are lots of other criticisms that can be made about how Knuth does literate programming (here's a discussion I found just now: https://wiki.c2.com/?LongFunctionsByDonaldKnuth), and I've made quite a few myself, but this comment is already too long (and too late). For the "ADVENT[URE]" program in particular, see this discussion from about a year ago: https://news.ycombinator.com/item?id=25597842

The examples give me the creeps. How can one obsess over readability of code (in form of comments) but not think about writing readable code?!

For example:

   cell\* run(string s) {
     stringstream in(s);
     return run(in);
   }

What is "in"?!

What the hell does "run" do?!

All this code commenting (which is bound to deviate from the implementation!) but no thought into naming and structuring code in an easy to understand fashion.

Replace "run()" with what it actually does even if it means naming the function "replaceAllNamesInTheInputStringByDoingAFancyComputation()" or whatever.

It's all better then "run()".

Replace the variables with easy to understand names!

I'd be mad as hell if my colleagues wrote that code.

I’m all for hyperlinks but I also think there’s something crucially valuable about presenting some kind of ordered, intelligible, linear narrative structure. The idea of organizing a program as a sequence of chapters is to me the core of literate programming. I do it by simply ordering my files with numbers. You can start reading at file 1, or you can skip the first chapters and jump to a more substantial chapter where the key concepts are already established… Feel free to start reading at the very end; this is also a good strategy for prose, see Adler’s “How To Read a Book.” Actually I think Adler’s framework would be very interesting to apply to programs. Literature isn’t just novels, it’s also monographs and anthologies and textbooks.

It is interesting to read this after reading about Pluto, the notebook system for Julia.

Pluto is exciting because - it provides nice text formatting easily via MarkDown and simple production of graphics - it allows you to build a data flow out of small understandable steps

and, unusually, - it understands the dependency graph between cells of computation. That means can rearrange them as you see fit but they will still execute in the right order. Moreover, it makes it so everybody sees the results of executing things in the right order. This is worlds different from most notebook systems

- you can embed HTML controls directly in your notebook so that you get reactive computation

This meets many of the goals Kartik was talking about in his blog.