Get the top HN stories in your inbox every day.
keithwinstein
duskwuff
> - First step (for a UTF-8-input terminal) is interpreting the input bytestream as UTF-8
> - Second step is "parsing" the scalar values by running them through the DEC parser/state machine.
Unfortunately, you may need to intermingle some logic between these two steps.
While VT100 style control sequences are usually introduced with an ESC, they can also be represented as a C1 control sequence, e.g. 0x84 instead of ESC + D, 0x9b instead of ESC + [. These sequences are raw bytes, not Unicode codepoints, and their encoding collides unpleasantly with UTF-8 continuation characters.
Further documentation: https://vt100.net/docs/vt220-rm/chapter4.html
Since there's no standard which specifies how UTF-8 should interact with the terminal parser, you're a little bit on your own here. But probably the simplest fix is to introduce a special case into the UTF-8 decoder which allows stray continuation characters to be passed through to the DEC parser, rather than transforming them to replacement characters immediately.
keithwinstein
:-) The UTF-8/Unix FAQ and existing terminal emulators don't agree with you here. As you say, there's no spec for this, but here's what Kuhn's FAQ says (https://www.cl.cam.ac.uk/~mgk25/unicode.html#term):
"UTF-8 still allows you to use C1 control characters such as CSI, even though UTF-8 also uses bytes in the range 0x80-0x9F. It is important to understand that a terminal emulator in UTF-8 mode must apply the UTF-8 decoder to the incoming byte stream before interpreting any control characters. C1 characters are UTF-8 decoded just like any other character above U+007F."
The existing ANSI terminal emulators that support UTF-8 input and C1 controls seem to agree on this (VTE, GNU screen, Mosh). xterm, urxvt, tmux, PuTTY, and st don't seem to support C1 controls in UTF-8 mode. So I don't think poking holes in the UTF-8 decoder is necessary, especially since allowing C1 in UTF-8 mode is rare anyway.
dgl
Note very few terminals implement UTF-8 and C1 controls and in particular xterm (which is kind of the defacto standard) doesn't because of the issues you outline. My opinion is they should just die as a legacy thing. No programs depend on them.
duskwuff
That's a perfectly reasonable answer too. There's a couple of other VT100 features that are safe to omit if you aren't going for full "historical accuracy" -- VT52 mode, for instance, has been obsolete for 30-40 years now.
As an aside: I wonder how useful it'd be to assemble a report documenting all known terminal control sequences and other behaviors, what terminals they're available in, and how frequently they're used in modern software. There are some big gaps between the DEC documentation, ECMA035/043/048, and actual implementations of terminal emulators.
saurik
https://www.xfree86.org/current/ctlseqs.html <- seems to claim they are supported? Oh! Maybe I misunderstand, and you mean those two features simultaneously?
azinman2
The comments here don’t seem to reflect what I think is the most interesting point here: quick loops of satisfaction. So much of programming often takes forever to get any real utility or see progress. That can really be depressing, especially for a side project. That’s what I love about cooking or sewing; you quickly see the process come together. I wish programming was like that more often.
vidarh
Yeah, I'm using my own terminal, and my own editor. The terminal also relies on a font-engine I have heavily modified (I converted the original from C to Ruby). So I "control" the whole pipeline from the editor to the actual pixels, and on one hand it has all kinds of quirks I wouldn't wish on someone else, on the other hand they all have bits and pieces that are custom-written to fit exactly what I want, and which features gets implemented are decided almost entirely based on which little change feels like it'll immediately improve my life right now (and I'm not joking - I spend enough time in front of my terminal that fixing small aspects of the terminal or my editor does feel like it is making an actual improvement in my happiness).
wnolens
What do you do? That all sounds like so so much effort, I'm very curious.
I'm in a terminal most of the day but never have I ever thought of once writing my own. Installing ohmyzsh was a huge step - I'm good for life now.
8n4vidtmkvmk
I'm constantly disappointed by my terminal. It always feels sluggish, especially with something like ohmyzsh. Boggles my mind why the whole pipeline is synchronous too.
vidarh
Devops and development.
It is effort, but it's also relaxing. I'll start by saying I don't think this is for everyone and it is far easier if you like minimalist environments. E.g. to me Visual Studio Code for example is an utter nightmare of visual overload. I want my editor to at most show a minimal one-line or less status area and maybe, optionally line numbers. Nothing more. I actively don't want it to pop up auto-completion options or documentation. And so on.
A lot of my requirements makes things simpler rather than harder, and that's not necessarily the case for others.
I started with the editor when I realised my Emacs config was several thousand lines and I though "I can write a whole editor in less". As it happens, a basic editor can be extremely small even in a language like C (e.g. see Antirez' "kilo"), but important: It can be extra small when you write it only for you. I can overlook bugs that doesn't bother me, and only implement functions I care about, and make assumptions about the environment others can't.
E.g. my editor presumes you're running under bspwm. It can run without it, but requires adapting helper scripts, because e.g. "split horizontal" and "split vertical" are implemented by opening a container and starting a new instance of the editor that will fill that area on screen and attach to the same view (the buffers are all held in a server, similar to if you use Emacs in client server mode).
It also contains no code for picking files, because it spawns an external one-liner which uses rofi to select files (same for themes etc.).
After a while, I started thinking about minor nuisances as a result of my terminal. E.g. I like a high degree of translucency, but that makes text hard to read unless there's an outline, so I started hacking outlines into st and kitty, but found other things that annoyed me, and before you knew it I had a "I bet I could replace this" moment.
I don't really care about packing up these for others as-is because so much is specific to how I want things, but I am slowly working my way through and pulling out generic components to release, and then I'll clean up the pase so it's at least there to look at.
EDIT: Just to add that e.g. for the terminal, the entire codebase, including packaging up all dependences other than the standard Ruby library, is currently ~6.5k lines. That includes ca. 1.5k lines for the Toml parser for the config files, for example, so I think it can be trimmed down significantly. For comparison `st` is around 8k lines, excluding dependencies like Xlib, freetype etc, and Xterm is around 88k lines. That's not to crow about how tiny mine is, because a lot of it I get "for free" thanks to Ruby, and a lot of it is down to just not implementing features I don't care about. But it is an illustration of how you can start really small and make something usable when you're free to keep a really tight focus. "Just" the terminal codebase itself is about 1.5k lines. But it will fail to correctly render a bunch of apps that I don't care about because there are still escapes I haven't bothered to implement. That's fine for me, but not for a terminal for general use.
m463
> I wish programming was like that more often.
Sometimes I think a shitty visible working prototype is a better goal than a more elaborate deep prototype that takes more time to see results.
for example, I know you can design in a log system, but you can start with printf and do the logging later. It's more important to tickle your motivation brain cells with something that works.
Or, you can start with a working project, strip stuff out, and get something visible. Then add your code.
There's also another (related?) trick - when something is working stop in the middle of a feature that is almost done. The next day you tickle those motivation braincells when you can dive right in and finish it instead of diving into the motivation-killing "planning the next step" stage.
Instantnoodl
I wrote a small terminal emulator a while ago to have a portable terminal for my terminal based game. It's very specific but I had great fun with it.
clemailacct1
That terminal and even the associated game look incredible!
sotix
This is awesome! Excited to see what you do with the project. Definitely keep us updated.
Instantnoodl
Thanks for the kind words :) It's currently in a bit of a hiatus. While the games "engine" is in a workable state it lacks content. Unfortunately I like building systems far more than actually creating game content. Might need to find a few people that are interested in helping with that at some point
sotix
Well lucky for you I like building content but lack experience in building engines (something I’d love to learn). Happy to contribute when I have some time. Email is in my profile if you feel like reaching out.
undefined
er4hn
Mitchell Hashimoto, Hashicorps longest serving IC, has been working on his own terminal emulator as a side project: https://mitchellh.com/ghostty . It's been interesting to read through his logs and see how it develops along with the gnarly bugs he gets to work through.
mtlynch
Agreed! I started reading not understanding anything about terminal emulators, and it's been interesting following his progress.
>Mitchell Hashimoto, Hashicorps longest serving IC
Small correction: I don't think this is right. He only became an IC two years ago.[0]
[0] https://www.hashicorp.com/blog/mitchell-s-new-role-at-hashic...
er4hn
That's just my bad attempt at a joke. "Longest serving code writer" may be more accurate, but IC gets lumped in as that so frequently and he publicly took that title back on. This is getting less funny the more I try to explain it..
billconan
I wanted to write a terminal emulator, the biggest hurdle is understanding the escape sequences. all documents seem to be unreadable, including those mentioned in the post, and those often referenced in projects, like https://vt100.net/emu/dec_ansi_parser
the second difficulty is handling reflow. a real terminal can't resize its screen, but an emulator can. how to implement that correctly with cursor movements?
the third difficulty is handling font fallback and rendering emojis and other combinatory glyphs correctly.
jws
I've written ANSI terminal handling before. The original VT100 paper manual that came with the terminal is a nice start, because the complexity hadn't really happened yet and people used to write useful documentation. With that as a starting point for understanding it isn't hard to extend into handling the full spec.
That diagram you linked is actually quite nice, but visually intimidating. If you think of it as a handful of regular expressions exploded into a state diagram that helps. For instance, the entire left half of the diagram is just the CSI code acceptor, see "CSI (Control Sequence Introducer) sequences" on the "ANSI escape code" wikipedia page.
You can write a regular expression to match a CSI and carry on. This is 2023 and you aren't using an 8080 with 3k of RAM. (probably) The only tiny trick is that you have to handle the "incomplete trailing regex" and wait for more data to arrive and try again.
As for handling reflow. I wouldn't call that an "implementation" problem. I'd call it a "specification" problem. I'd approach it by seeing what Apple's Terminal program does, write that down, and call that my specification.
vidarh
Ignore all of this, and start simple.
You can get something going with just the most rudimentary escape handling just by spitting what programs write to your terminal to debug output in the terminal you run your new terminal from, and add a proper parser a bit later.
You can totally ignore reflow. It'll look ugly. It doesn't matter. When running full screen applications you need to handle width/height reporting, that's all.
Font fallback and nice font handling is a detail to worry about well down the line. There are libraries that can do a lot of the lifting for you depending on language/platform. Just pick a font with reasonable coverage and worry about the rest later.
duskwuff
> the second difficulty is handling reflow. a real terminal can't resize its screen, but an emulator can. how to implement that correctly with cursor movements?
The answer to this one, incidentally, is ¯\_(ツ)_/¯.
Every terminal emulator engine I've examined handles reflow a little differently. There's no official spec, and everyone has their own idea about how to make it work "best". That being said, I've wrestled with these questions myself, and here's what I've come up with:
* Each line should have a flag on it which tracks whether it should be reflowed with the next line. By default, this flag should not be set.
* The reflow flag should only be set when the cursor leaves that line because a character was printed with the cursor in the last column ("wrapnext" state), and no special modes are set which change the wrapping columns.
* Performing any other action which changes the contents of a line should clear the reflow flag on that line.
The same logic can also be used to join wrapped lines for copy/paste. It won't work 100% of the time, but it will work for most situations.
alpaca128
I always used Wikipedia's article on ANSI escape sequences. A few details could be explained a bit better but overall I found it useful. The diagram you linked is probably a more complete and compact overview of all possible combinations, but I don't find it very intuitive either.
sureglymop
I think it lacks some important escape sequences. E.g. how do programs like vim and tmux switch to another buffer and then restore the buffer? I vaguely know about it but never saw an actually complete documentation.
vidarh
The diagram is complete. It shows the collection of the raw sequences, which includes a bunch of parameters that you then need to process separately to determine what to actually do.
To switch to/from the alternate screen mode is \e[?47h and \e[?47l. "\e[?" is DEC private mode which are DEC private mode codes. The number specifies a range of settings to switch on or off. The "h" and "l" determines if you're setting or clearing the setting respectively.
The parsing of those are handled by the escape, csi entry, and csi param boxes in the diagram.
ksherlock
They need to keep the screen and scrollback history in memory, then redraw it as needed. As mentioned next door, xterm has support for alternate screen mode which can save and restore the current contents. But tmux works fine with more than 2 buffers and tmux works fine if the terminal underneath it doesn't support alternate screen mode (like a physical vt100)
undefined
blueflow
Ignore all that ANSI stuff, implement escape sequences like you think its easy to implement and then write a terminfo file for that so applications know how to use it.
yencabulator
And hope your users never SSH to other machines, because shuffling terminfo files around is simply painful.
blueflow
What else are you gonna do? Impersonate some other terminal type and then only support half of its features? Looking at the many xterm "clones".
wkat4242
> the second difficulty is handling reflow. a real terminal can't resize its screen, but an emulator can. how to implement that correctly with cursor movements?
That's not entirely true. Most serial terminals did have multiple text size modes eg 80x25 but also 128x50 or something. I still have a real VT520 here that can do that. Not a Dec labeled one sadly but one of the later boundless models.
undefined
SoftTalker
Some real terminals had a few different row/column modes, e.g. 80x24 or 132x40 (IIRC). I don't recall if text was reflowed when switching.
wyan
Usually the screen was cleared when switching modes, so no text reflow.
jws
The early DEC terminals did not reflow on switching.
c-smile
Just in case, Sciter has built-in element <terminal> that can be used for various purposes.
Escape codes are supported, see: https://sciter.com/wp-content/uploads/2022/10/terminal.png
Docs/API: https://docs.sciter.com/docs/behaviors/behavior-terminal
jmmv
I can confirm that writing a terminal is fun, for the reasons mentioned in the article: it’s easy to get “self-hosting”, but then the possibilities are endless :)
In my case, this was about creating the terminal for EndBASIC (https://www.endbasic.dev/). I wanted to mix text and graphics in the same console, so I had to ditch Xterm.js and create my own thing. It was really exciting to see graphics rendering mix with text “just fine” when I was able to render the first line.
ishuah_
Shameless plug: I wrote an article on building a terminal emulator in 100 lines of code - https://ishuah.com/2021/03/10/build-a-terminal-emulator-in-1...
kragen
this is cool! golang definitely seems to make this simpler than c, and less error-prone too
you sure did have to import a lot of libraries, tho
kragen
mine emulates an adm-3a, which is probably about the simplest thing already in termcap/terminfo that can support vi and curses
https://gitlab.com/kragen/bubbleos/-/blob/master/yeso/admu_s...
it's 96 lines of c, which ought to remove any intimidation factor from the task; if i can do it so can you
https://gitlab.com/kragen/bubbleos/-/blob/master/yeso/admu.h https://gitlab.com/kragen/bubbleos/-/blob/master/yeso/admu.c
then all the crap to interface with unix braindamage and draw glyphs on the screen, without any actual terminal emulation, is another 232 lines of c
https://gitlab.com/kragen/bubbleos/-/blob/master/yeso/admu-s...
yeso makes it pretty easy to get pixels on the screen (opposite extreme from 'OpenGL code is very heavy on boiler plate and repetition'), but its text handling capabilities are limited at best
norir
> It’s also possible to write a terminal in a terminal, like tmux, but I’d save this for my second attempt. It’s very helpful to have a place to dump logging info that’s not also the screen we’re writing to.
I don't fully understand what the author is saying here or precisely what they mean by writing a terminal in a terminal. From my perspective though, it is easier to write a hosted terminal that runs inside of an existing terminal. Writing the full thing from scratch is a much harder problem. A terminal has many subproblems that are best attacked separately in my opinion.
At its heart, a terminal reads formatted text from standard input and writes formatted text to standard output. It is essentially a REPL. So the first step is to write a (R)ead function. Then you pass the result of read to the (E)valuate function which will process the input and finally pass it in to the (P)rint function. If you start with a hosted terminal, the read and print functions can be modeled with posix read and write so you can devote most of your time to the evaluate function.
Once you have a good evaluate function and the terminal works as you like in the hosted environment, then it makes sense to go back and write new implementations of read and write that target a new host environment. This is when it makes sense to switch to QT or opengl: when you have already implemented the core logic of the terminal and want better io performance. But it also might make sense to target html/js for maximum portability. You can either reuse or rewrite the backend that was used in the bootstrap terminal depending on how it was written. Even if you are changing languages, the rewrite should be much easier than the initial implementation since you already know what functionality is necessary and how to do it.
If you start with QT or opengl, you might never even get to a useful terminal because you get so bogged down in the incidental details.
What I am describing is essentially quite similar to bootstrapping a new programming language. The initial implementation should be done in the most convenient language/environment possible for the author. People commonly make the mistake of implementing a bootstrap compiler in a low level language, which is almost always a premature optimization and forces you to take on accidental complexity (such as memory management) that is secondary to your primary goals. Remember Fred Brook's advice to plan to throw the first implementation away. It is so much easier to do something that you have already done before than something new.
vidarh
The entire backend rendering to raw X11 calls for my personal terminals is ~160 lines of Ruby, and that includes support for oddities like double width/double height, and optimizations you can drop at first like scroll up/down (as opposed to taking the slow approach of redrawing, which is enough for a first approximation). You need very little to do the bare minimum graphical output.
brainbag
Is your Ruby-based terminal source online? I'd love to see it.
vidarh
No current version, but I'm preparing it. But actually, to see a really ridiculously minimalist start, this was my starting point, which used a tiny C extension to do the X rendering (though it optimistically included a dummy class intended to be the start for the Ruby X backend). It's awfully limited, and awfully broken, but it shows how little it takes to be able to start writing:
https://github.com/vidarh/rubyterm
It's totally useless for anything other that testing or expanding on, but it was the starting point for the terminal I now run every day, and I'll be updating that repo as I clean up my current version at some point.
The current version uses this for a pure Ruby (no Xlib) X11 client implementation:
https://github.com/vidarh/ruby-x11
And this pure-Ruby TrueType font renderer (I did the Ruby conversion; the C code it's based on was not mine, and is a beautiful example of compact C - look up libschrift) as I got tired of using the bitmap fonts and didn't want to add a FreeType dependency (the renderer is ~500 lines of code):
norir
Exactly. So start there and build up.
qweqwe14
Check out st[1] for a minimal terminal implementation. They also have user-submitted patches that you can apply to add desired functionality.
wkat4242
Why would I want to? Terminal emulators are one of the categories of software with the most options available already.
And understanding terminal codes which are historically a bunch of kludges on kludges is a guarantee for bugs. I don't think there's any terminal emulator that does it perfectly and the quirks of the most prolific ones like xterm have simply become the ad-hoc standard.
kragen
all that is true, but writing your own terminal emulator is the first step towards fixing it
Get the top HN stories in your inbox every day.
FWIW, I wouldn't try to parse escape sequences directly from the input bytestream -- it's easy to end up with annoying edge cases. :-/ In my experience you'll thank yourself if you can separate the logic into something like:
- First step (for a UTF-8-input terminal) is interpreting the input bytestream as UTF-8 and "lexing" into a stream of Unicode Scalar Values (https://www.unicode.org/versions/Unicode15.1.0/ch03.pdf#P.12... ; https://github.com/mobile-shell/mosh/blob/master/src/termina...).
- Second step is "parsing" the scalar values by running them through the DEC parser/state machine. This is independent of the escape sequences (https://vt100.net/emu/dec_ansi_parser ; https://github.com/mobile-shell/mosh/blob/master/src/termina...).
- And then the third step is for the terminal to execute the dispatch/execute/etc. actions coming from the parser, which is where the escape sequences and control chars get implemented (https://www.vt100.net/docs/vt220-rm/ ; https://invisible-island.net/xterm/ctlseqs/ctlseqs.html ; https://github.com/mobile-shell/mosh/blob/master/src/termina...).
Without this separation, it's easier to end up with bugs where, e.g., a UTF-8 sequence or an ANSI escape sequence is treated differently if it's split between read() calls (https://bugs.chromium.org/p/chromium/issues/detail?id=212702), or invalid input isn't correctly recovered-from, etc.